GenAI4ELab/papercli-papers huggingface dataset:
Generative End-to-End Autonomous Driving
Authors: Wenzhao Zheng; Ruiqi Song; Xianda Guo; Chenming Zhang; Long Chen
Venue: ECCV 2024 (main)

Abstract: "Directly producing planning results from raw sensors has been a long-desired solution for autonomous driving and has attracted increasing attention recently. Most existing end-to-end autonomous driving methods factorize this problem into perception, motion prediction, and planning. However, we argue that the conventional progressive pipeline still cannot comprehensively model the entire traffic evolution process, e.g., the future interaction between the ego car and other traffic participants and the structural trajectory prior. In this paper, we explore a new paradigm for end-to-end autonomous driving, where the key is to predict how the ego car and the surroundings evolve given past scenes. We propose GenAD, a generative framework that casts autonomous driving into a generative modeling problem. We propose an instance-centric scene tokenizer that first transforms the surrounding scenes into map-aware instance tokens. We then employ a variational autoencoder to learn the future trajectory distribution in a structural latent space for trajectory prior modeling. We further adopt a temporal model to capture the agent and ego movements in the latent space to generate more effective future trajectories. GenAD finally simultaneously performs motion prediction and planning by sampling distributions in the learned structural latent space conditioned on the instance tokens and using the learned temporal model to generate futures. Extensive experiments on the widely used nuScenes benchmark show that the proposed GenAD achieves state-of-the-art performance on vision-centric end-to-end autonomous driving with high efficiency. Code: https://github.com/wzzheng/GenAD."
Learning to Distinguish Samples for Generalized Category Discovery
Authors: Fengxiang Yang; Nan Pu; Wenjing Li; Zhiming Luo; Shaozi Li; Nicu Sebe; Zhun Zhong
Venue: ECCV 2024 (main)

Abstract: "Generalized Category Discovery (GCD) utilizes labelled data from seen categories to cluster unlabelled samples from both seen and unseen categories. Previous methods have demonstrated that assigning pseudo-labels for representation learning is effective. However, these methods commonly predict pseudo-labels based on pairwise similarities, while the overall relationship among each instance’s k -nearest neighbors (k NNs) is largely overlooked, leading to inaccurate pseudo-labeling. To address this issue, we introduce a Neighbor Graph Convolutional Network (NGCN) that learns to predict pairwise similarities between instances using only labelled data. NGCN explicitly leverages the relationships among each instance’s k NNs and is generalizable to samples of both seen and unseen classes. This helps produce more accurate positive samples by injecting the predicted similarities into subsequent clustering. Furthermore, we design a Cross-View Consistency Strategy (CVCS) to exclude samples with noisy pseudo-labels generated by clustering. This is achieved by comparing clusters from two different clustering algorithms. The filtered unlabelled data with pseudo-labels and the labelled data are then used to optimize the model through cluster- and instance-level contrastive objectives. The collaboration between NGCN and CVCS ensures the learning of a robust model, resulting in significant improvements in both seen and unseen class accuracies. Extensive experiments demonstrate that our method achieves state-of-the-art performance on both generic and fine-grained GCD benchmarks. Code: https://github.com/FlyingRoastDuck/NGCN CVCS.git."
COM Kitchens: An Unedited Overhead-view Procedural Videos Dataset a Vision-Language Benchmark
Authors: Atsushi Hashimoto; Koki Maeda; Tosho Hirasawa; Jun Harashima; Leszek Rybicki; Yusuke Fukasawa; Yoshitaka Ushiku
Venue: ECCV 2024 (main)

Abstract: "Procedural video understanding is gaining attention in the vision and language community. Deep learning-based video analysis requires extensive data. Consequently, existing works often use web videos as training resources, making it challenging to query instructional contents from raw video observations. To address this issue, we propose a new dataset, COM Kitchens. The dataset consists of unedited overhead-view videos captured by smartphones, in which participants performed food preparation based on given recipes. Fixed-viewpoint video datasets often lack environmental diversity due to high camera setup costs. We used modern wide-angle smartphone lenses to cover cooking counters from sink to cooktop in an overhead view, capturing activity without in-person assistance. With this setup, we collected a diverse dataset by distributing smartphones to participants. With this dataset, we propose the novel video-to-text retrieval task Online Recipe Retrieval (OnRR) and new video captioning domain Dense Video Captioning on unedited Overhead-View videos (DVC-OV). Our experiments verified the capabilities and limitations of current web-video-based SOTA methods in handling these tasks. The dataset and code are available at https://doi.org/10.32130/rdata.6.1 and https://github. com/omron-sinicx/com_kitchens, respectively."
PILoRA: Prototype Guided Incremental LoRA for Federated Class-Incremental Learning
Authors: Haiyang Guo; Fei Zhu; Wenzhuo Liu; Xu-Yao Zhang; Cheng-Lin Liu
Venue: ECCV 2024 (main)

Abstract: "Existing federated learning methods have effectively dealt with decentralized learning in scenarios involving data privacy and non-IID data. However, in real-world situations, each client dynamically learns new classes, requiring the global model to classify all seen classes. To effectively mitigate catastrophic forgetting and data heterogeneity under low communication costs, we propose a simple and effective method named PILoRA. On the one hand, we adopt prototype learning to learn better feature representations and leverage the heuristic information between prototypes and class features to design a prototype re-weight module to solve the classifier bias caused by data heterogeneity without retraining the classifier. On the other hand, we view incremental learning as the process of learning distinct task vectors and encoding them within different LoRA parameters. Accordingly, we propose Incremental LoRA to mitigate catastrophic forgetting. Experimental results on standard datasets indicate that our method outperforms the state-of-the-art approaches significantly. More importantly, our method exhibits strong robustness and superiority in different settings and degrees of data heterogeneity. The code is available at https://github.com/ Ghy0501/PILoRA."
Diff-Reg: Diffusion Model in Doubly Stochastic Matrix Space for Registration Problem
Authors: Qianliang Wu; Haobo Jiang; Lei Luo; Jun Li; Yaqing Ding; Jin Xie; Jian Yang
Venue: ECCV 2024 (main)

Abstract: "Establishing reliable correspondences is essential for 3D and 2D-3D registration tasks. Existing methods commonly leverage geometric or semantic point features to generate potential correspondences. However, these features may face challenges such as large deformation, scale inconsistency, and ambiguous matching problems (e.g., symmetry). Additionally, many previous methods, which rely on single-pass prediction, may struggle with local minima in complex scenarios. To mitigate these challenges, we introduce a diffusion matching model for robust correspondence construction. Our approach treats correspondence estimation as a denoising diffusion process within the doubly stochastic matrix space, which gradually denoises (refines) a doubly stochastic matching matrix to the ground-truth one for high-quality correspondence estimation. It involves a forward diffusion process that gradually introduces Gaussian noise into the ground truth matching matrix and a reverse denoising process that iteratively refines the noisy one. In particular, we deploy a lightweight denoising strategy during the inference phase. Specifically, once points/image features are extracted and fixed, we utilize them to conduct multiple-pass denoising predictions in the reverse sampling process. Evaluation of our method on both 3D and 2D-3D registration tasks confirms its effectiveness. The code is available at https://github.com/wuqianliang/Diff-Reg."
WBP: Training-time Backdoor Attacks through Hardware-based Weight Bit Poisoning
Authors: Kunbei Cai; Zhenkai Zhang; Qian Lou; Fan Yao
Venue: ECCV 2024 (main)

Abstract: "Training from pre-trained models (PTM) is a popular approach for fast machine learning (ML) service deployment. Recent studies on hardware security have revealed that ML systems could be compromised through flipping bits in model parameters (e.g., weights) with memory faults. In this paper, we introduce (i.e., weight bit poisoning), a novel task-agnostic backdoor attack that manifests during the victim’s training time (i.e., fine-tuning from a public and clean PTM) by inducing hardware-based weight bit flips. utilizes a novel distance-aware algorithm that identifies bit flips to maximize the distance between the distribution of poisoned output representations (ORs) and clean ORs based on the public PTM. This unique set of bit flips can be applied to backdoor any victim model during the fine-tuning of the same public PTM, regardless of the downstream tasks. We evaluate on state-of-the-art CNNs and Vision Transformer models with representative downstream tasks. The results show that can compromise a wide range of PTMs and downstream tasks with an average 99.3% attack success rate by flipping as few as 11 model weight bits. can be effective in various training configurations with respect to learning rate, optimizer, and fine-tuning duration. We investigate limitations of existing backdoor protection techniques against and discuss potential future mitigation. 1 1 Our code can be accessed at: https://github.com/casrl/WBP"
"Towards Dual Transparent Liquid Level Estimation in Biomedical Lab: Dataset, Methods and Practice"
Authors: Xiayu Wang; Ke Ma; Ruiyun Zhong; Xinggang Wang; Yi Fang; Yang Xiao; Tian Xia
Venue: ECCV 2024 (main)

Abstract: "“Dual Transparent Liquid” refers to a liquid and its container, both being transparent. Accurately estimating the levels of such a liquid from arbitrary viewpoints is fundamental and crucial, especially in AI-guided autonomous biomedical laboratories for tasks like liquid dispensing, aspiration, and mixing. However, current methods for estimating liquid level focus on scenarios of a single instance captured from a fixed view. We propose a new dual transparent liquid level estimation paradigm, including a dataset, methods, and practices. The dual transparent liquid dataset, named DTLD, comprises 27,458 images with four object instances captured from multiple views across three biomedical lab scenes. Based on DTLD, we propose an end-to-end learning method for detecting the liquid contact line, followed by an approach to estimate the liquid level. To enhance contact line detection, a color rectification module is proposed to stabilize the color distribution at the local region of the air-liquid interface. Additionally, our method surpasses the current best approach, reducing the mean absolute percentage error by a percentage of 43.4. The dataset and code are available at https://github.com/dualtransparency/TCLD."
Encapsulating Knowledge in One Prompt
Authors: Qi Li; Runpeng Yu; Xinchao Wang
Venue: ECCV 2024 (main)

Abstract: "In this paper, we propose a new knowledge transfer paradigm called Knowledge in One Prompt (KiOP). This paradigm encapsulates knowledge from various models into a solitary prompt without altering the original models or requiring access to the training data, which enables us to achieve efficient and convenient knowledge transfer in more realistic scenarios. From a practicality standpoint, this paradigm not only for the first time proves the effectiveness of Visual Prompt in data inaccessible contexts, but also solves the problems of low model reusability and high storage resource consumption faced by traditional Data-Free Knowledge Transfer, which means that we can realize the parallel knowledge transfer of multiple models without modifying any source model. Extensive experiments across various datasets and models demonstrate the efficacy of the proposed KiOP knowledge transfer paradigm. Without access to real training data and with rigorous storage capacity constraints, it is also capable of yielding considerable outcomes when dealing with cross-model backbone setups and handling parallel knowledge transfer processing requests with multiple (more than 2) models. Code is available at https://github.com/LiQiiiii/Encapsulating-Knowledge-In-One-Prompt."
Cross-Input Certified Training for Universal Perturbations
Authors: Changming Xu; Gagandeep Singh
Venue: ECCV 2024 (main)

Abstract: "Existing work in trustworthy machine learning primarily focuses on single-input adversarial perturbations. In many real-world attack scenarios, input-agnostic adversarial attacks, e.g. universal adversarial perturbations (UAPs), are much more feasible. Current certified training methods train models robust to single-input perturbations but achieve suboptimal clean and UAP accuracy, thereby limiting their applicability in practical applications. We propose a novel method, , for certified training of networks robust against UAP attackers. We show in an extensive evaluation across different datasets, architectures, and perturbation magnitudes that our method outperforms traditional certified training methods on standard accuracy (up to 10.3%) and achieves SOTA performance on the more practical certified UAP accuracy metric."
Visual Relationship Transformation
Authors: Xiaoyu Xu; Jiayan Qiu; Baosheng Yu; Zhou Wang
Venue: ECCV 2024 (main)

Abstract: "What will be the relationships between objects in a novel view? We strive to answer this question by investigating a new visual cognition task, termed visual relationship transformation or VRT. Unlike prior visual relationship detection task that works on visible view images, VRT aims to predict the relationships in unseen novel views from a single observed source view. Towards solving VRT, we propose an end-to-end deep approach that, given an observed view image and inter-view transformations, learns to predict the relationships in novel views. Specifically, we introduce an equivariant graph neural network to predict the relationships between objects in novel views, which is achieved by enforcing the transformation equivariance of the learned relationship representations. Simultaneously, a relationship presentness mask is learned for pruning the invisible ones, thus enabling the visible relationship prediction in novel views. To this end, VRT provides supplementary cues for accomplishing novel-view-related tasks, such as visual grounding (VG), novel view synthesis (NVS), and pedestrian intention estimation (PIE). In the experiments, adopting VRT as a plug-in module results in considerable performance improvements in VG, NVS, and PIE across all datasets."
"Not Just Change the Labels, Learn the Features: Watermarking Deep Neural Networks with Multi-View Data"
Authors: Yuxuan Li; Sarthak Kumar Maharana; Yunhui Guo
Venue: ECCV 2024 (main)

Abstract: "With the increasing prevalence of Machine Learning as a Service (MLaaS) platforms, there is a growing focus on deep neural network (DNN) watermarking techniques. These methods are used to facilitate the verification of ownership for a target DNN model to protect intellectual property. One of the most widely employed watermarking techniques involves embedding a trigger set into the source model. Unfortunately, existing methodologies based on trigger sets are still susceptible to functionality-stealing attacks, potentially enabling adversaries to steal the functionality of the source model without a reliable means of verifying ownership. In this paper, we first introduce a novel perspective on trigger set-based watermarking methods from a feature learning perspective. Specifically, we demonstrate that by selecting data exhibiting multiple features, also referred to as multi-view data, it becomes feasible to effectively defend functionality stealing attacks. Based on this perspective, we introduce a novel watermarking technique based on Multi-view dATa, called MAT, for efficiently embedding watermarks within DNNs. This approach involves constructing a trigger set with multi-view data and incorporating a simple feature-based regularization method for training the source model. We validate our method across various benchmarks and demonstrate its efficacy in defending against model extraction attacks, surpassing relevant baselines by a significant margin. The code is available at https://github.com/liyuxuan-github/MAT."
Delving into Adversarial Robustness on Document Tampering Localization
Authors: Huiru Shao; Zhuang Qian; Kaizhu Huang; Wei Wang; Xiaowei Huang; Qiufeng Wang
Venue: ECCV 2024 (main)

Abstract: "Recent advances in document forgery techniques produce malicious yet nearly visually untraceable alterations, imposing a big challenge for document tampering localization (DTL). Despite significant recent progress, there has been surprisingly limited exploration of adversarial robustness in DTL. This paper presents the first effort to uncover the vulnerability of most existing DTL models to adversarial attacks, highlighting the need for greater attention within the DTL community. In pursuit of robust DTL, we demonstrate that adversarial training can promote the model’s robustness and effectively protect against adversarial attacks. As a notable advancement, we further introduce a latent manifold adversarial training approach that enhances adversarial robustness in DTL by incorporating perturbations on the latent manifold of adversarial examples, rather than exclusively relying on label-guided information. Extensive experiments on DTL benchmark datasets show the necessity of adversarial training and our proposed manifold-based method significantly improves the adversarial robustness on both white-box and black-box attacks. Codes will be available at https://github.com/SHR-77/DTL-ARob.git."
Adaptive Selection of Sampling-Reconstruction in Fourier Compressed Sensing
Authors: Seongmin Hong; Jaehyeok Bae; Jongho Lee; Se Young Chun
Venue: ECCV 2024 (main)

Abstract: "Compressed sensing (CS) has emerged to overcome the inefficiency of Nyquist sampling. However, traditional optimization-based reconstruction is slow and may not yield a high-quality image in practice. Deep learning-based reconstruction has been a promising alternative to optimization-based reconstruction, outperforming it in accuracy and computation speed. Finding an efficient sampling method with deep learning-based reconstruction, especially for Fourier CS remains a challenge. Existing joint optimization of sampling-reconstruction works (H1 ) optimize the sampling mask but yield suboptimal results because it is not adaptive to each data point. Adaptive sampling (H2 ) has also disadvantages of difficult optimization and Pareto sub-optimality. Here, we propose a novel adaptive selection of sampling-reconstruction (H1.5 ) framework that selects the best sampling mask and reconstruction network for each input data. We provide theorems that our method has a lower infimum of the true risk compared to H1 and effectively solves the Pareto sub-optimality problem in sampling-reconstruction by using separate reconstruction networks for different sampling masks. To select the best sampling mask, we propose to quantify the high-frequency Bayesian uncertainty of the input, using a super-resolution space generation model. Our method outperforms joint optimization of sampling-reconstruction (H1 ) and adaptive sampling (H2 ) by achieving significant improvements on several Fourier CS problems."
Confidence-Based Iterative Generation for Real-World Image Super-Resolution
Authors: Jialun Peng; Xin Luo; Jingjing Fu; Dong Liu
Venue: ECCV 2024 (main)

Abstract: "Real-world image super-resolution deals with complex and unknown degradations, making it challenging to produce plausible results in a single step. In this work, we propose a transformer model with an iterative generation process that iteratively refines the results based on predicted confidences. It allows the model to focus on regions with low confidences and generate more confident and accurate results. Specifically, our model learns to predict the visual tokens of the high-resolution image and their corresponding confidence scores, conditioned on the low-resolution image. By keeping only the most confident tokens at each iteration and re-predicting the other tokens in the next iteration, our model generates all high-resolution tokens within a few steps. To ensure consistency with the low-resolution input image, we further propose a conditional controlling module that utilizes the low-resolution image to control the decoding process from high-resolution tokens to image pixels. Experiments demonstrate that our model achieves state-of-the-art performance on real-world datasets while requiring fewer iteration steps compared to recent diffusion models."
Learning Scalable Model Soup on a Single GPU: An Efficient Subspace Training Strategy
Authors: Tao Li; Weisen Jiang; Fanghui Liu; Xiaolin Huang; James Kwok
Venue: ECCV 2024 (main)

Abstract: "Pre-training followed by fine-tuning is widely adopted among practitioners. The performance can be improved by “model soups” [?] via exploring various hyperparameter configurations. The Learned-Soup, a variant of model soups, significantly improves the performance but suffers from substantial memory and time costs due to the requirements of (i) having to load all fine-tuned models simultaneously, and (ii) a large computational graph encompassing all fine-tuned models. In this paper, we propose Memory Efficient Hyperplane Learned Soup (MEHL-Soup) to tackle this issue by formulating the learned soup as a hyperplane optimization problem and introducing block coordinate gradient descent to learn the mixing coefficients. At each iteration, MEHL-Soup only needs to load a few fine-tuned models and build a computational graph with one combined model. We further extend MEHL-Soup to MEHL-Soup+ in a layer-wise manner. Experimental results on various ViT models and data sets show that MEHL-Soup(+) outperforms Learned-Soup(+) in terms of test accuracy, and also reduces memory usage by more than 13×. Moreover, MEHL-Soup(+) can be run on a single GPU and achieves 9× speed up in soup construction compared with the Learned-Soup. The code is released at https://github. com/nblt/MEHL-Soup."
Correspondences of the Third Kind: Camera Pose Estimation from Object Reflection
Authors: Kohei Yamashita; Vincent Lepetit; Ko Nishino
Venue: ECCV 2024 (main)

Abstract: "Computer vision has long relied on two kinds of correspondences: pixel correspondences in images and 3D correspondences on object surfaces. Is there another kind, and if there is, what can they do for us? In this paper, we introduce correspondences of the third kind we call reflection correspondences and show that they can help estimate camera pose by just looking at objects without relying on the background. Reflection correspondences are point correspondences in the reflected world, , the scene reflected by the object surface. The object geometry and reflectance alter the scene geometrically and radiometrically, respectively, causing incorrect pixel correspondences. Geometry recovered from each image is also hampered by distortions, namely generalized bas-relief ambiguity, leading to erroneous 3D correspondences. We show that reflection correspondences can resolve the ambiguities arising from these distortions. We introduce a neural correspondence estimator and a RANSAC algorithm that fully leverages all three kinds of correspondences for robust and accurate joint camera pose and object shape estimation just from the object appearance. The method expands the horizon of numerous downstream tasks, including camera pose estimation for appearance modeling (, NeRF) and motion estimation of reflective objects (, cars on the road), to name a few, as it relieves the requirement of overlapping background."