GenAI4ELab/papercli-papers huggingface dataset:
Parameterized Quasi-Physical Simulators for Dexterous Manipulations Transfer
Authors: Xueyi Liu; Kangbo Lyu; jieqiong zhang; Tao Du; Li Yi
Venue: ECCV 2024 (main)

Abstract: "We explore the dexterous manipulation transfer problem by designing simulators. The task wishes to transfer human manipulations to dexterous robot hand simulations and is inherently difficult due to its intricate, highly-constrained, and discontinuous dynamics and the need to control a dexterous hand with a DoF to accurately replicate human manipulations. Previous approaches that optimize in high-fidelity black-box simulators or a modified one with relaxed constraints only demonstrate limited capabilities or are restricted by insufficient simulation fidelity. We introduce parameterized quasi-physical simulators and a physics curriculum to overcome these limitations. The key ideas are 1) balancing between fidelity and optimizability of the simulation via a curriculum of parameterized simulators, and 2) solving the problem in each of the simulators from the curriculum, with properties ranging from high task optimizability to high fidelity. We successfully enable a dexterous hand to track complex and diverse manipulations in high-fidelity simulated environments, boosting the success rate by 11%+ from the best-performed baseline. The project website is available at QuasiSim."
3D Hand Pose Estimation in Everyday Egocentric Images
Authors: Aditya Prakash; Ruisen Tu; Matthew Chang; Saurabh Gupta
Venue: ECCV 2024 (main)

Abstract: "3D hand pose estimation in everyday egocentric images is challenging for several reasons: poor visual signal (occlusion from the object of interaction, low resolution & motion blur), large perspective distortion (hands are close to the camera), and lack of 3D annotations outside of controlled settings. While existing methods often use hand crops as input to focus on fine-grained visual information to deal with poor visual signal, the challenges arising from perspective distortion and lack of 3D annotations in the wild have not been systematically studied. We focus on this gap and explore the impact of different practices, crops as input, incorporating camera information, auxiliary supervision, scaling up datasets. We provide several insights that are applicable to both convolutional and transformer models, leading to better performance. Based on our findings, we also present , a system for 3D hand pose estimation in everyday egocentric images. Zero-shot evaluation on 4 diverse datasets (H2O, , , ) demonstrate the effectiveness of our approach across 2D and 3D metrics, where we beat past methods by 7.4% – 66%. In system level comparisons, achieves the best 3D hand pose on egocentric split, outperforms FrankMocap across all metrics and HaMeR on 3 out of 6 metrics while being 10× smaller and trained on 5× less data."
Mitigating Perspective Distortion-induced Shape Ambiguity in Image Crops
Authors: Aditya Prakash; Arjun Gupta; Saurabh Gupta
Venue: ECCV 2024 (main)

Abstract: "Objects undergo varying amounts of perspective distortion as they move across a camera’s field of view. Models for predicting 3D from a single image often work with crops around the object of interest and ignore the location of the object in the camera’s field of view. We note that ignoring this location information further exaggerates the inherent ambiguity in making 3D inferences from 2D images and can prevent models from even fitting to the training data. To mitigate this ambiguity, we propose Intrinsics-Aware Positional Encoding (), which incorporates information about the location of crops in the image and camera intrinsics. Experiments on three popular 3D-from-a-single-image benchmarks: depth prediction on NYU, 3D object detection on KITTI & nuScenes, and predicting 3D shapes of articulated objects on ARCTIC, show the benefits of ."
Towards Neuro-Symbolic Video Understanding
Authors: Minkyu Choi; Harsh Goel; Mohammad Omama; Yunhao Yang; Sahil Shah; Sandeep Chinchali
Venue: ECCV 2024 (main)

Abstract: "The unprecedented surge in video data production in recent years necessitates efficient tools to extract meaningful frames from videos for downstream tasks. Long-term temporal reasoning is a key desideratum for frame retrieval systems. While state-of-the-art foundation models, like VideoLLaMA and ViCLIP, are proficient in short-term semantic understanding, they surprisingly fail at long-term reasoning across frames. A key reason for this failure is that they intertwine per-frame perception and temporal reasoning into a single deep network. Hence, decoupling but co-designing the semantic understanding and temporal reasoning is essential for efficient scene identification. We propose a system that leverages vision-language models for semantic understanding of individual frames and effectively reasons about the long-term evolution of events using state machines and temporal logic (TL) formulae that inherently capture memory. Our TL-based reasoning improves the F1 score of complex event identification by 9 − 15%, compared to benchmarks that use GPT-4 for reasoning, on state-of-the-art self-driving datasets such as Waymo and NuScenes. The source code is available at https://github.com/UTAustin-SwarmLab/Neuro-Symbolic-Video-Search-Temporal-Logic."
Optimization-based Uncertainty Attribution Via Learning Informative Perturbations
Authors: Hanjing Wang; Bashirul Azam Biswas; Qiang Ji
Venue: ECCV 2024 (main)

Abstract: "Uncertainty attribution (UA) aims to identify key contributors to predictive uncertainty in deep learning models. To improve the faithfulness of existing UA methods, we formulate UA as an optimization problem to learn a binary mask on the input. The learned mask identifies regions that significantly contribute to output uncertainty and allows uncertainty reduction through learning informative perturbations on the masked input. Our method enhances UA interpretability and maintains high efficiency by integrating three key improvements: Segment Anything Model (SAM)-guided mask parameterization for efficient and interpretable mask learning; learnable perturbations that adaptively target and refine problematic regions specific to each input without manually tuning the perturbation parameters; and a novel application of Gumbel-sigmoid reparameterization for efficiently learning Bernoulli-distributed binary masks under continuous optimization. Our experiments on problematic region detection and faithfulness tests demonstrate our method’s superiority over state-of-the-art UA methods."
Context-Aware Action Recognition: Introducing a Comprehensive Dataset for Behavior Contrast
Authors: Tatsuya Sasaki; Yoshiki Ito; Satoshi Kondo
Venue: ECCV 2024 (main)

Abstract: "While datasets on everyday actions, sports, and cooking are abundant, there’s a significant scarcity in datasets focused on industrial domain activities, especially for distinguishing between proper and improper actions. This shortage poses a unique challenge, necessitating highly precise, context-sensitive feature extraction due to the subtle class distinctions, which are more nuanced than in general action recognition. To address this gap, we introduce a dataset featuring contrasting pairs of proper and improper actions, aimed at exploring these specific challenges, assessing the limitations of current methods, and establishing a new standard. Our dataset not only encompasses traditional industrial tasks, such as working at heights, but also extends to everyday situations like basketball, underscoring the task’s broad relevance. By evaluating leading techniques on this dataset, we aim to unearth valuable insights, pushing the boundaries of action understanding in both industrial and everyday contexts."
Semi-supervised Segmentation of Histopathology Images with Noise-Aware Topological Consistency
Authors: Meilong Xu; Xiaoling Hu; Saumya Gupta; Shahira Abousamra; Chao Chen
Venue: ECCV 2024 (main)

Abstract: "In digital pathology, segmenting densely distributed objects like glands and nuclei is crucial for downstream analysis. Since detailed pixel-wise annotations are very time-consuming, we need semi-supervised segmentation methods that can learn from unlabeled images. Existing semi-supervised methods are often prone to topological errors, e.g., missing or incorrectly merged/separated glands or nuclei. To address this issue, we propose TopoSemiSeg, the first semi-supervised method that learns the topological representation from unlabeled histopathology images. The major challenge is for unlabeled images; we only have predictions carrying noisy topology. To this end, we introduce a noise-aware topological consistency loss to align the representations of a teacher and a student model. By decomposing the topology of the prediction into signal topology and noisy topology, we ensure that the models learn the true topological signals and become robust to noise. Extensive experiments on public histopathology image datasets show the superiority of our method, especially on topology-aware evaluation metrics. Code is available at https://github.com/Melon-Xu/TopoSemiSeg."
Adaptive Compressed Sensing with Diffusion-Based Posterior Sampling
Authors: Noam Elata; Tomer Michaeli; Michael Elad
Venue: ECCV 2024 (main)

Abstract: "Compressed Sensing (CS) facilitates rapid image acquisition by selecting a small subset of measurements sufficient for high-fidelity reconstruction. Adaptive CS seeks to further enhance this process by dynamically choosing future measurements based on information gleaned from data that is already acquired. However, many existing frameworks are often tailored to specific tasks and require intricate training procedures. We propose AdaSense, a novel Adaptive CS approach that leverages zero-shot posterior sampling with pre-trained diffusion models. By sequentially sampling from the posterior distribution, we can quantify the uncertainty of each possible future linear measurement throughout the acquisition process. AdaSense eliminates the need for additional training and boasts seamless adaptation to diverse domains with minimal tuning requirements. Our experiments demonstrate the effectiveness of AdaSense in reconstructing facial images from a small number of measurements. Furthermore, we apply AdaSense for active acquisition of medical images in the domains of magnetic resonance imaging (MRI) and computed tomography (CT), highlighting its potential for tangible real-world acceleration.1 1 Our code is available at https://github.com/noamelata/AdaSense."
Instant Uncertainty Calibration of NeRFs Using a Meta-Calibrator
Authors: Niki Amini-Naieni; Tomas Jakab; Andrea Vedaldi; Ronald Clark
Venue: ECCV 2024 (main)

Abstract: "Neural Radiance Fields (NeRFs) have markedly improved novel view synthesis, but accurate uncertainty quantification in their image predictions remains an open problem. The prevailing methods for estimating uncertainty, including the state-of-the-art Density-aware NeRF Ensembles (DANE) [?], quantify uncertainty without calibration. This frequently leads to over- or under-confidence in image predictions, which can undermine their real-world applications. In this paper, we propose a method which, for the first time, achieves calibrated uncertainties for NeRFs. To accomplish this, we overcome a significant challenge in adapting existing calibration techniques to NeRFs: a need to hold out ground truth images from the target scene, reducing the number of images left to train the NeRF. This issue is particularly problematic in sparse-view settings, where we can operate with as few as three images. To address this, we introduce the concept of a meta-calibrator that performs uncertainty calibration for NeRFs with a single forward pass without the need for holding out any images from the target scene. Our meta-calibrator is a neural network that takes as input the NeRF images and uncalibrated uncertainty maps and outputs a scene-specific calibration curve that corrects the NeRF’s uncalibrated uncertainties. We show that the meta-calibrator can generalize on unseen scenes and achieves well-calibrated and state-of-the-art uncertainty for NeRFs, significantly beating DANE and other approaches. This opens opportunities to improve applications that rely on accurate NeRF uncertainty estimates such as next-best view planning and potentially more trustworthy image reconstruction for medical diagnosis."
MetaAT: Active Testing for Label-Efficient Evaluation of Dense Recognition Tasks
Authors: Sanbao Su; Xin Li; Thang Doan; Sima Behpour; Wenbin He; Liang Gou; Fei Miao; Liu Ren
Venue: ECCV 2024 (main)

Abstract: "In this study, we investigate the task of active testing for label-efficient evaluation, which aims to estimate a model’s performance on an unlabeled test dataset with a limited annotation budget. Previous approaches relied on deep ensemble models to identify highly informative instances for labeling, but fell short in dense recognition tasks like segmentation and object detection due to their high computational costs. In this work, we present MetaAT, a simple yet effective approach that adapts a Vision Transformer as a Meta Model for active testing. Specifically, we introduce a region loss estimation head to identify challenging regions for more accurate and informative instance acquisition. Importantly, the design of MetaAT allows it to handle annotation granularity at the region level, significantly reducing annotation costs in dense recognition tasks. As a result, our approach demonstrates consistent and substantial performance improvements over five popular benchmarks compared with state-of-the-art methods. Notably, on the CityScapes dataset, MetaAT achieves a 1.36% error rate in performance estimation using only 0.07% of annotations, marking a 10× improvement over existing state-of-the-art methods. To the best of our knowledge, MetaAT represents the first framework for active testing of dense recognition tasks."
Salience-Based Adaptive Masking: Revisiting Token Dynamics for Enhanced Pre-training
Authors: Hyesong Choi; Hyejin Park; Kwang Moo Yi; Sungmin Cha; Dongbo Min
Venue: ECCV 2024 (main)

Abstract: "In this paper, we introduce Saliency-Based Adaptive Masking (SBAM), a novel and cost-effective approach that significantly enhances the pre-training performance of Masked Image Modeling (MIM) approaches by prioritizing token salience. Our method provides robustness against variations in masking ratios, effectively mitigating the performance instability issues common in existing methods. This relaxes the sensitivity of MIM-based pre-training to masking ratios, which in turn allows us to propose an adaptive strategy for ‘tailored’ masking ratios for each data sample, which no existing method can provide. Toward this goal, we propose an Adaptive Masking Ratio (AMR) strategy that dynamically adjusts the proportion of masking for the unique content of each image based on token salience. We show that our method significantly improves over the state-of-the-art in mask-based pre-training on the ImageNet-1K dataset. Code and model parameters are available at https://github.com/doihye/ SBAM."
Data Augmentation via Latent Diffusion for Saliency Prediction
Authors: Bahar Aydemir; Deblina Bhattacharjee; Tong Zhang; Mathieu Salzmann; Sabine Süsstrunk
Venue: ECCV 2024 (main)

Abstract: "Saliency prediction models are constrained by the limited diversity and quantity of labeled data. Standard data augmentation techniques such as rotating and cropping alter scene composition, affecting saliency. We propose a novel data augmentation method for deep saliency prediction that edits natural images while preserving the complexity and variability of real-world scenes. Since saliency depends on high-level and low-level features, our approach involves learning both by incorporating photometric and semantic attributes such as color, contrast, brightness, and class. To that end, we introduce a saliency-guided cross-attention mechanism that enables targeted edits on the photometric properties, thereby enhancing saliency within specific image regions. Experimental results show that our data augmentation method consistently improves the performance of various saliency models. Moreover, leveraging the augmentation features for saliency prediction yields superior performance on publicly available saliency benchmarks. Our predictions align closely with human visual attention patterns in the edited images, as validated by a user study. Our code is publicly available on GitHub1 ."
Explorative Inbetweening of Time and Space
Authors: Haiwen Feng; Zheng Ding; Zhihao Xia; Simon Niklaus; Victoria Fernandez Abrevaya; Michael J. Black; Xuaner Zhang
Venue: ECCV 2024 (main)

Abstract: "We introduce bounded generation as a generalized task to control video generation to synthesize arbitrary camera and subject motion based only on a given start and end frame. Our objective is to fully leverage the inherent generalization capability of an image-to-video model without additional training or fine-tuning of the original model. This is achieved through the proposed new sampling strategy, which we call , that fuses the temporally forward and backward denoising paths conditioned on the start and end frame, respectively. The fused path results in a video that smoothly connects the two frames, generating inbetweening of faithful subject motion, novel views of static scenes, and seamless video looping when the two bounding frames are identical. We curate a diverse evaluation dataset of image pairs and compare against the closest existing methods. We find that outperforms related work on all subtasks, exhibiting the ability to generate complex motions and 3D-consistent views guided by bounded frames. See project page at https://time-reversal. github.io."
A Diffusion Model for Simulation Ready Coronary Anatomy with Morpho-skeletal Control
Authors: Karim Kadry; Shreya Gupta; Jonas Sogbadji; Michiel Schaap; Kersten Petersen; Takuya Mizukami; Carlos Collet; Farhad R. Nezami; Elazer R Edelman
Venue: ECCV 2024 (main)

Abstract: "Virtual interventions enable the physics-based simulation of device deployment within coronary arteries. This framework allows for counterfactual reasoning by deploying the same device in different arterial anatomies. However, current methods to create such counterfactual arteries face a trade-off between controllability and realism. In this study, we investigate how Latent Diffusion Models (LDMs) can custom synthesize coronary anatomy for virtual intervention studies based on mid-level anatomic constraints such as topological validity, local morphological shape, and global skeletal structure. We also extend diffusion model guidance strategies to the context of morpho-skeletal conditioning and propose a novel guidance method for continuous attributes that adaptively updates the negative guiding condition throughout sampling. Our framework enables the generation and editing of coronary anatomy in a controllable manner, allowing device designers to derive mechanistic insights regarding anatomic variation and simulated device deployment. Our code is available at https://github.com/kkadry/Morphoskel-Diffusion."
Learning to Make Keypoints Sub-Pixel Accurate
Authors: Shinjeong Kim; Marc Pollefeys; Daniel Barath
Venue: ECCV 2024 (main)

Abstract: "This work addresses the challenge of sub-pixel accuracy in detecting 2D local features, a cornerstone problem in computer vision. Despite the advancements brought by neural network-based methods like SuperPoint and ALIKED, these modern approaches lag behind classical ones such as SIFT in keypoint localization accuracy due to their lack of sub-pixel precision. We propose a novel network that enhances any detector with sub-pixel precision by learning an offset vector for detected features, thereby eliminating the need for designing specialized sub-pixel accurate detectors. This optimization directly minimizes test-time evaluation metrics like relative pose error. Through extensive testing with both nearest neighbors matching and the recent LightGlue matcher across various real-world datasets, our method consistently outperforms existing methods in accuracy. Moreover, it adds only around 7 ms to the time of a particular detector. The code is available at https://github.com/KimSinjeong/keypt2subpx."
Imaging with Confidence: Uncertainty Quantification for High-dimensional Undersampled MR Images
Authors: Frederik Hoppe; Claudio Mayrink Verdun; Hannah Sophie Laus; Sebastian Endt; Marion Irene Menzel; Felix Krahmer; Holger Rauhut
Venue: ECCV 2024 (main)

Abstract: "Establishing certified uncertainty quantification (UQ) in imaging processing applications continues to pose a significant challenge. In particular, such a goal is crucial for accurate and reliable medical imaging if one aims for precise diagnostics and appropriate intervention. In the case of magnetic resonance imaging, one of the essential tools of modern medicine, enormous advancements in fast image acquisition were possible after the introduction of compressive sensing and, more recently, deep learning methods. Still, as of now, there is no UQ method that is both fully rigorous and scalable. This work takes a step towards closing this gap by proposing a total variation minimization-based method for pixel-wise sharp confidence intervals for undersampled MRI. We demonstrate that our method empirically achieves the predicted confidence levels. We expect that our approach will also have implications for other imaging modalities as well as deep learning applications in computer vision. Our code is available on GitHub https://github.com/HannahLaus/Project_UQ_TV.git."
Generalizable Human Gaussians for Sparse View Synthesis
Authors: YoungJoong Kwon; Baole Fang; Yixing Lu; Haoye Dong; Cheng Zhang; Francisco Vicente Carrasco; Albert Mosella-Montoro; Jianjin Xu; Shingo J Takagi; Daeil Kim; Aayush Prakash; Fernando de la Torre
Venue: ECCV 2024 (main)

Abstract: "Recent progress in neural rendering has brought forth pioneering methods, such as NeRF and Gaussian Splatting, which revolutionize view rendering across various domains like AR/VR, gaming, and content creation. While these methods excel at interpolating within the training data, the challenge of generalizing to new scenes and objects from very sparse views persists. Specifically, modeling 3D humans from sparse views presents formidable hurdles due to the inherent complexity of human geometry, resulting in inaccurate reconstructions of geometry and textures. To tackle this challenge, this paper leverages recent advancements in Gaussian Splatting and introduces a new method to learn generalizable human Gaussians that allows photorealistic and accurate view-rendering of a new human subject from a limited set of sparse views in a feed-forward manner. A pivotal innovation of our approach involves reformulating the learning of 3D Gaussian parameters into a regression process defined on the 2D UV space of a human template, which allows leveraging the strong geometry prior and the advantages of 2D convolutions. In addition, a multi-scaffold is proposed to effectively represent the offset details. Our method outperforms recent methods on both within-dataset generalization as well as cross-dataset generalization settings."