GenAI4ELab/papercli-papers huggingface dataset:
OAT: Object-Level Attention Transformer for Gaze Scanpath Prediction
Authors: Yini Fang; Jingling Yu; Haozheng Zhang; Ralf van der Lans; Bertram E Shi
Venue: ECCV 2024 (main)

Abstract: "Visual search is important in our daily life. The efficient allocation of visual attention is critical to effectively complete visual search tasks. Prior research has predominantly modelled the spatial allocation of visual attention in images at the pixel level, e.g. using a saliency map. However, emerging evidence shows that visual attention is guided by objects rather than pixel intensities. This paper introduces the Object-level Attention Transformer (OAT), which predicts human scanpaths as they search for a target object within a cluttered scene of distractors. OAT uses an encoder-decoder architecture. The encoder captures information about the position and appearance of the objects within an image and about the target. The decoder predicts the gaze scanpath as a sequence of object fixations, by integrating output features from both the encoder and decoder. We also propose a new positional encoding that better reflects spatial relationships between objects. We evaluated OAT on the Amazon book cover dataset and a new dataset for visual search that we collected. OAT’s predicted gaze scanpaths align more closely with human gaze patterns, compared to predictions by algorithms based on spatial attention on both established metrics and a novel behavioural-based metric. Our results demonstrate the generalization ability of OAT, as it accurately predicts human scanpaths for unseen layouts and target objects. The code is available at: https://github.com/HKUST-NISL/ oat_eccv24. 1 The first two authors contributed equally in the paper."
SelfSwapper: Self-Supervised Face Swapping via Shape Agnostic Masked AutoEncoder
Authors: Jaeseong Lee; Junha Hyung; Sohyun Jeong; Jaegul Choo
Venue: ECCV 2024 (main)

Abstract: "Face swapping has gained significant attention for its varied applications. Most previous face swapping approaches have relied on the seesaw game training scheme, also known as the target-oriented approach. However, this often leads to instability in model training and results in undesired samples with blended identities due to the target identity leakage problem. Source-oriented methods achieve more stable training with self-reconstruction objective but often fail to accurately reflect target image’s skin color and illumination. This paper introduces the Shape Agnostic Masked AutoEncoder (SAMAE) training scheme, a novel self-supervised approach that combines the strengths of both target-oriented and source-oriented approaches. Our training scheme addresses the limitations of traditional training methods by circumventing the conventional seesaw game and introducing clear ground truth through its self-reconstruction training regime. Our model effectively mitigates identity leakage and reflects target albedo and illumination through learned disentangled identity and non-identity features. Additionally, we closely tackle the shape misalignment and volume discrepancy problems with new techniques, including perforation confusion and random mesh scaling. SAMAE establishes a new state-of-the-art, surpassing other baseline methods, preserving both identity and non-identity attributes without sacrificing on either aspect."
EgoPoseFormer: A Simple Baseline for Stereo Egocentric 3D Human Pose Estimation
Authors: Chenhongyi Yang; Anastasia Tkach; Shreyas Hampali; Linguang Zhang; Elliot J Crowley; Cem Keskin
Venue: ECCV 2024 (main)

Abstract: "We present , a simple yet effective transformer-based model for stereo egocentric human pose estimation. The main challenge in egocentric pose estimation is overcoming joint invisibility, which is caused by self-occlusion or a limited field of view (FOV) of head-mounted cameras. Our approach overcomes this challenge by incorporating a two-stage pose estimation paradigm: in the first stage, our model leverages the global information to estimate each joint’s coarse location, then in the second stage, it employs a DETR style transformer to refine the coarse locations by exploiting fine-grained stereo visual features. In addition, we present a operation to enable our transformer to effectively process multi-view features, which enables it to accurately localize each joint in the 3D world. We evaluate our method on the stereo UnrealEgo dataset and show it significantly outperforms previous approaches while being computationally efficient: it improves MPJPE by 27.4mm (45% improvement) with only 7.9% model parameters and 13.1% FLOPs compared to the state-of-the-art. Surprisingly, with proper training settings, we find that even our first-stage pose proposal network can achieve superior performance compared to previous arts. We also show that our method can be seamlessly extended to monocular settings, which achieves state-of-the-art performance on the SceneEgo dataset, improving MPJPE by 25.5mm (21% improvement) compared to the best existing method with only 60.7% model parameters and 36.4% FLOPs. Code is available at https://github.com/ChenhongyiYang/egoposeformer."
An Information Theoretical View for Out-Of-Distribution Detection
Authors: Hu Jinjing; Wenrui Liu; Hong Chang; Bingpeng MA; Shiguang Shan; Xilin Chen
Venue: ECCV 2024 (main)

Abstract: "Detecting out-of-distribution (OOD) inputs are pivotal for real-world applications. However, due to the inaccessibility of OODs during training phase, applying supervised binary classification with in-distribution (ID) and OOD labels is not feasible. Therefore, previous works typically employ the proxy ID classification task to learn feature representation for OOD detection task. In this study, we delve into the relationship between the two tasks through the lens of Information Theory. Our analysis reveals that optimizing the classification objective could inevitably cause the over-confidence and undesired compression of OOD detection-relevant information. To address these two problems, we propose OOD Entropy Regularization (OER) to regularize the information captured in classification-oriented representation learning for detecting OOD samples. Both theoretical analyses and experimental results underscore the consistent improvement of OER on OOD detection."
DMiT: Deformable Mipmapped Tri-Plane Representation for Dynamic Scenes
Authors: Jing-Wen Yang; Jia-Mu Sun; Yong-Liang Yang; Jie Yang; Ying Shan; Yan-Pei Cao; Lin Gao
Venue: ECCV 2024 (main)

Abstract: "Neural Radiance Fields (NeRF) have achieved remarkable progress on dynamic scenes with deformable objects. Nonetheless, most previous works required multi-view inputs or long training time (several hours), making it hard to apply them for real-world scenarios. Recent works dedicated to addressing blurry artifacts may fail to predict stable and accurate deformation while keeping high-frequency details when rendering at various resolutions. To this end, we introduce a novel framework DMiT (Deformable Mipmapped Tri-Plane) that adopts the mipmaps to render dynamic scenes at various resolutions from novel views. With the help of hierarchical mipmapped tri-planes, we incorporate an MLP to effectively predict a mapping between the observation space and the canonical space, enabling not only high-fidelity dynamic scene rendering but also high-performance training and inference. Moreover, a training scheme for joint geometry and deformation refinement is designed for canonical regularization to reconstruct high-quality geometries. Extensive experiments on both synthetic and real dynamic scenes demonstrate the efficacy and efficiency of our method."
Gated Temporal Diffusion for Stochastic Long-term Dense Anticipation
Authors: Olga Zatsarynna; Emad Bahrami; Yazan Abu Farha; Gianpiero Francesca; Jürgen Gall
Venue: ECCV 2024 (main)

Abstract: "Long-term action anticipation has become an important task for many applications such as autonomous driving and human-robot interaction. Unlike short-term anticipation, predicting more actions into the future imposes a real challenge with the increasing uncertainty in longer horizons. While there has been a significant progress in predicting more actions into the future, most of the proposed methods address the task in a deterministic setup and ignore the underlying uncertainty. In this paper, we propose a novel Gated Temporal Diffusion (GTD) network that models the uncertainty of both the observation and the future predictions. As generator, we introduce a Gated Anticipation Network (GTAN) to model both observed and unobserved frames of a video in a mutual representation. On the one hand, using a mutual representation for past and future allows us to jointly model ambiguities in the observation and future, while on the other hand GTAN can by design treat the observed and unobserved parts differently and steer the information flow between them. Our model achieves state-of-the-art results on the Breakfast, Assembly101 and 50Salads datasets in both stochastic and deterministic settings."
Gradient-Aware for Class-Imbalanced Semi-supervised Medical Image Segmentation
Authors: Wenbo Qi; Jiafei Wu; S. C. Chan
Venue: ECCV 2024 (main)

Abstract: "Class imbalance poses a significant challenge in semi-supervised medical image segmentation (SSLMIS). Existing techniques face problems such as poor performance on tail classes, instability, and slow convergence speed. We propose a novel Gradient-Aware (GA) method, structured on a clear paradigm: identify extrinsic data-bias → analyze intrinsic gradient-bias → propose solutions, to address this issue. Through theoretical analysis, we identify the intrinsic gradient bias instigated by extrinsic data bias in class-imbalanced SSLMIS. To combat this, we propose a GA loss, featuring GADice loss, which leverages a probability-aware gradient for absent classes, and GACE, designed to alleviate gradient bias through class equilibrium and dynamic weight equilibrium. Our proposed method is plug-and-play, simple yet very effective and robust, exhibiting a fast convergence speed. Comprehensive experiments on three public datasets (CT&MRI, 2D&3D) demonstrate our method’s superior performance, significantly outperforming other SOTA of SSLMIS and class-imbalanced designs (+ 17.90% with CPS on 20% labeled Synapse). Code is available at https://github.com/cicailalala/GALoss."
HowToCaption: Prompting LLMs to Transform Video Annotations at Scale
Authors: Nina Shvetsova; Anna Kukleva; Xudong Hong; Christian Rupprecht; Bernt Schiele; Hilde Kuehne
Venue: ECCV 2024 (main)

Abstract: "Instructional videos are a common source for learning text-video or even multimodal representations by leveraging subtitles extracted with automatic speech recognition systems (ASR) from the audio signal in the videos. However, in contrast to human-annotated captions, both speech and subtitles naturally differ from the visual content of the videos and thus provide only noisy supervision. As a result, large-scale annotation-free web video training data remains sub-optimal for training text-video models. In this work, we propose to leverage the capabilities of large language models (LLMs) to obtain high-quality video descriptions aligned with videos at scale. Specifically, we prompt an LLM to create plausible video captions based on ASR subtitles of instructional videos. To this end, we introduce a prompting method that is able to take into account a longer text of subtitles, allowing us to capture the contextual information beyond one single sentence. We further prompt the LLM to generate timestamps for each produced caption based on the timestamps of the subtitles and finally align the generated captions to the video temporally. In this way, we obtain human-style video captions at scale without human supervision. We apply our method to the subtitles of the HowTo100M dataset, creating a new large-scale dataset, HowToCaption. Our evaluation shows that the resulting captions not only significantly improve the performance over many different benchmark datasets for zero-shot text-video retrieval and video captioning, but also lead to a disentangling of textual narration from the audio, boosting the performance in text-video-audio tasks.1 1 All data and code is available at https://github.com/ninatu/howtocaption."
LabelDistill: Label-guided Cross-modal Knowledge Distillation for Camera-based 3D Object Detection
Authors: Sanmin Kim; Youngseok Kim; Sihwan Hwang; Hyeonjun Jeong; Dongsuk Kum
Venue: ECCV 2024 (main)

Abstract: "Recent advancements in camera-based 3D object detection have introduced cross-modal knowledge distillation to bridge the performance gap with LiDAR 3D detectors, leveraging the precise geometric information in LiDAR point clouds. However, existing cross-modal knowledge distillation methods tend to overlook the inherent imperfections of LiDAR, such as the ambiguity of measurements on distant or occluded objects, which should not be transferred to the image detector. To mitigate these imperfections in LiDAR teacher, we propose a novel method that leverages aleatoric uncertainty-free features from ground truth labels. In contrast to conventional label guidance approaches, we approximate the inverse function of the teacher’s head to effectively embed label inputs into feature space. This approach provides additional accurate guidance alongside LiDAR teacher, thereby boosting the performance of the image detector. Additionally, we introduce feature partitioning, which effectively transfers knowledge from the teacher modality while preserving the distinctive features of the student, thereby maximizing the potential of both modalities. Experimental results demonstrate that our approach improves mAP and NDS by 5.1 points and 4.9 points compared to the baseline model, proving the effectiveness of our approach. The code is available at https://github.com/ sanmin0312/LabelDistill"
Beyond the Data Imbalance: Employing the Heterogeneous Datasets for Vehicle Maneuver Prediction
Authors: Hyeongseok Jeon; Sanmin Kim; Abi Rahman Syamil; Junsoo Kim; Dongsuk Kum
Venue: ECCV 2024 (main)

Abstract: "Predicting the maneuvers of surrounding vehicles is imperative for the safe navigation of autonomous vehicles. However, naturalistic driving datasets tend to be highly imbalanced, with a bias towards the ”going straight” maneuver. Consequently, learning and accurately predicting turning maneuvers pose significant challenges. In this study, we propose a novel two-stage maneuver learning method that can overcome such strong biases by leveraging two heterogeneous datasets in a complementary manner. In the first training phase, we utilize an intersection-centric dataset characterized by balanced distribution of maneuver classes to learn the representations of each maneuver. Subsequently, in the second training phase, we incorporate an ego-centric driving dataset to account for various geometrical road shapes, by transferring the knowledge of geometric diversity to the maneuver prediction model. To facilitate this, we constructed an in-house intersection-centric trajectory dataset with a well-balanced maneuver distribution. By harnessing the power of heterogeneous datasets, our framework significantly improves maneuver prediction performance, particularly for minority maneuver classes such as turning maneuvers. The dataset is available at https: //github.com/KAIST-VDCLab/VDC-Trajectory-Dataset."
On Pretraining Data Diversity for Self-Supervised Learning
Authors: Hasan Abed Al Kader Hammoud; Tuhin Das; Fabio Pizzati; Philip Torr; Adel Bibi; Bernard Ghanem
Venue: ECCV 2024 (main)

Abstract: "We explore the impact of training with more diverse datasets, characterized by the number of unique samples, on the performance of self-supervised learning (SSL) under a fixed computational budget. Our findings demonstrate that increasing pretraining data diversity enhances SSL performance, albeit only when the distribution distance to the downstream data is minimal. Notably, even with an exceptionally large pretraining data diversity achieved through methods like web crawling or diffusion-generated data, among other ways, the distribution shift remains a challenge. Our experiments are comprehensive with seven SSL methods using large-scale datasets such as ImageNet and YFCC100M amounting to over 200 GPU days. The code and trained models will be available at https://github.com/hammoudhasan/DiversitySSL."
Look Around and Learn: Self-Training Object Detection by Exploration
Authors: Gianluca Scarpellini; Stefano Rosa; Pietro Morerio; Lorenzo Natale; Alessio Del Bue
Venue: ECCV 2024 (main)

Abstract: "When an object detector is deployed in a novel setting it often experiences a drop in performance. This paper studies how an embodied agent can automatically fine-tune a pre-existing object detector while exploring and acquiring images in a new environment without relying on human intervention, i.e., a fully self-supervised approach. In our setting, an agent initially learns to explore the environment using a pre-trained off-the-shelf detector to locate objects and associate pseudo-labels. By assuming that pseudo-labels for the same object must be consistent across different views, we learn the exploration policy “Look Around” to mine hard samples, and we devise a novel mechanism called “Disagreement Reconciliation” for producing refined pseudo-labels from the consensus among observations. We implement a unified benchmark of the current state-of-the-art and compare our approach with pre-existing exploration policies and perception mechanisms. Our method is shown to outperform existing approaches, improving the object detector by 6.2% in a simulated scenario, a 3.59% advancement over other state-of-the-art methods, and by 9.97% in the real robotic test without relying on ground-truth. Code for the proposed approach and baselines are available at https:// iit-pavis.github.io/Look_Around_And_Learn/."
Bayesian Self-Training for Semi-Supervised 3D Segmentation
Authors: Ozan Unal; Christos Sakaridis; Luc Van Gool
Venue: ECCV 2024 (main)

Abstract: "3D segmentation is a core problem in computer vision and, similarly to many other dense prediction tasks, it requires large amounts of annotated data for adequate training. However, densely labeling 3D point clouds to employ fully-supervised training remains too labor intensive and expensive. Semi-supervised training provides a more practical alternative, where only a small set of labeled data is given, accompanied by a larger unlabeled set. This area thus studies the effective use of unlabeled data to reduce the performance gap that arises due to the lack of annotations. In this work, inspired by Bayesian deep learning, we first propose a Bayesian self-training framework for semi-supervised 3D semantic segmentation. Employing stochastic inference, we generate an initial set of pseudo-labels and then filter these based on estimated point-wise uncertainty. By constructing a heuristic n-partite matching algorithm, we extend the method to semi-supervised 3D instance segmentation, and finally, with the same building blocks, to dense 3D visual grounding. We demonstrate state-of-the-art results for our semi-supervised method on SemanticKITTI and ScribbleKITTI for 3D semantic segmentation and on ScanNet and S3DIS for 3D instance segmentation. We further achieve substantial improvements in dense 3D visual grounding over supervised-only baselines on ScanRefer. Our project page is available at ouenal.github.io/bst/."
Motion and Structure from Event-based Normal Flow
Authors: Zhongyang Ren; Bangyan Liao; Delei Kong; Jinghang Li; Peidong Liu; Laurent Kneip; Guillermo Gallego; Yi Zhou
Venue: ECCV 2024 (main)

Abstract: "Recovering the camera motion and scene geometry from visual data is a fundamental problem in computer vision. Its success in conventional (frame-based) vision is attributed to the maturity of feature extraction, data association and multi-view geometry. The emergence of asynchronous (event-based) cameras calls for new approaches that use raw event data as input to solve this fundamental problem. State-of-the-art solutions typically infer data association implicitly by iteratively reversing the event data generation process. However, the nonlinear nature of these methods limits their applicability in real-time tasks, and the constant-motion assumption leads to unstable results under agile motion. To this end, we reformulate the problem in a way that aligns better with the differential working principle of event cameras. We show that event-based normal flow can be used, via the proposed geometric error term, as an alternative to the full (optical) flow in solving a family of geometric problems that involve instantaneous first-order kinematics and scene geometry. Furthermore, we develop a fast linear solver and a continuous-time nonlinear solver on top of the proposed geometric error term. Experiments on both synthetic and real data show the superiority of our linear solver in terms of accuracy and efficiency, and its practicality as an initializer for previous nonlinear solvers. Besides, our continuous-time non-linear solver exhibits exceptional capabilities in accommodating sudden variations in motion since it does not rely on the constant-motion assumption. Our project page can be found at https://nail-hnu.github.io/EvLinearSolver/."
ParCo: Part-Coordinating Text-to-Motion Synthesis
Authors: Qiran Zou; Shangyuan Yuan; Shian Du; Yu Wang; Chang Liu; Yi Xu; Jie Chen; Xiangyang Ji
Venue: ECCV 2024 (main)

Abstract: "We study a challenging task: text-to-motion synthesis, aiming to generate motions that align with textual descriptions and exhibit coordinated movements. Currently, the part-based methods introduce part partition into the motion synthesis process to achieve finer-grained generation. However, these methods encounter challenges such as the lack of coordination between different part motions and difficulties for networks to understand part concepts. Moreover, introducing finer-grained part concepts poses computational complexity challenges. In this paper, we propose Part-Coordinating Text-to-Motion Synthesis (ParCo), endowed with enhanced capabilities for understanding part motions and communication among different part motion generators, ensuring a coordinated and fined-grained motion synthesis. Specifically, we discretize whole-body motion into multiple part motions to establish the prior concept of different parts. Afterward, we employ multiple lightweight generators designed to synthesize different part motions and coordinate them through our part coordination module. Our approach demonstrates superior performance on common benchmarks with economic computations, including HumanML3D and KIT-ML, providing substantial evidence of its effectiveness. Code is available at: https://github.com/qrzou/ParCo."