GenAI4ELab/papercli-papers huggingface dataset:
SelEx: Self-Expertise in Fine-Grained Generalized Category Discovery
Authors: Sarah Rastegar; Mohammadreza Salehi; Yuki M Asano; Hazel Doughty; Cees Snoek
Venue: ECCV 2024 (main)

Abstract: "In this paper, we address Generalized Category Discovery, aiming to simultaneously uncover novel categories and accurately classify known ones. Traditional methods, which lean heavily on self-supervision and contrastive learning, often fall short when distinguishing between fine-grained categories. To address this, we introduce a novel concept called ‘self-expertise’, which enhances the model’s ability to recognize subtle differences and uncover unknown categories. Our approach combines unsupervised and supervised self-expertise strategies to refine the model’s discernment and generalization. Initially, hierarchical pseudo-labeling is used to provide ‘soft supervision’, improving the effectiveness of self-expertise. Our supervised technique differs from traditional methods by utilizing more abstract positive and negative samples, aiding in the formation of clusters that can generalize to novel categories. Meanwhile, our unsupervised strategy encourages the model to sharpen its category distinctions by considering within-category examples as ‘hard’ negatives. Supported by theoretical insights, our empirical results showcase that our method outperforms existing state-of-the-art techniques in Generalized Category Discovery across several fine-grained datasets. Our code is available at: https://github.com/SarahRastegar/SelEx."
Self-Cooperation Knowledge Distillation for Novel Class Discovery
Authors: Yuzheng Wang; Zhaoyu Chen; Dingkang Yang; Yunquan Sun; Lizhe Qi
Venue: ECCV 2024 (main)

Abstract: "Novel Class Discovery (NCD) aims to discover unknown and novel classes in an unlabeled set by leveraging knowledge already learned about known classes. Existing works focus on instance-level or class-level knowledge representation and build a shared representation space to achieve performance improvements. However, a long-neglected issue is the potential imbalanced number of samples from known and novel classes, pushing the model towards dominant classes. Therefore, these methods suffer from a challenging trade-off between reviewing known classes and discovering novel classes. Based on this observation, we propose a Self-Cooperation Knowledge Distillation (SCKD) method to utilize each training sample (whether known or novel, labeled or unlabeled) for both review and discovery. Specifically, the model’s feature representations of known and novel classes are used to construct two disjoint representation spaces. Through spatial mutual information, we design a self-cooperation learning method to encourage model learning from the two feature representation spaces. Extensive experiments on six datasets demonstrate that our method can achieve significant performance improvements, achieving state-of-the-art performance. 1 1† Corresponding Author"
EventBind: Learning a Unified Representation to Bind Them All for Event-based Open-world Understanding
Authors: jiazhou zhou; Xu Zheng; Yuanhuiyi Lyu; Lin Wang
Venue: ECCV 2024 (main)

Abstract: "In this paper, we propose EventBind, a novel and effective framework that unleashes the potential of vision-language models (VLMs) for event-based recognition to compensate for the lack of large-scale event-based datasets. In particular, due to the distinct modality gap with the image-text data and the lack of large-scale datasets, learning a common representation space for images, texts, and events is non-trivial. Intuitively, we need to address two key challenges: 1) how to generalize CLIP’s visual encoder to event data while fully leveraging events’ unique properties, , sparsity and high temporal resolution; 2) how to effectively align the multi-modal embeddings, , image, text, and events. Accordingly, we first introduce a novel event encoder that subtly models the temporal information from events and meanwhile generates event prompts for modality bridging. We then design a text encoder that generates content prompts and utilizes hybrid text prompts to enhance EventBind’s generalization ability across diverse datasets. With the proposed event encoder, text encoder, and image encoder, a novel Hierarchical Triple Contrastive Alignment HTCA module is introduced to jointly optimize the correlation and enable efficient knowledge transfer among the three modalities. We evaluate various settings, including fine-tuning and few-shot on three benchmarks and our EventBind achieves new state-of-art accuracy compared with the previous methods, such as on N-Caltech101 (+5.34% and +1.70%) and N-Imagenet (+5.65% and +1.99%) with fine-tuning and 20-shot settings respectively. Moreover, our EventBind can be flexibly extended to the event retrieval task using text or image queries, showing plausible performance."
GLAD: Towards Better Reconstruction with Global and Local Adaptive Diffusion Models for Unsupervised Anomaly Detection
Authors: Hang Yao; Ming Liu; Zhicun Yin; Zifei Yan; Xiaopeng Hong; Wangmeng Zuo
Venue: ECCV 2024 (main)

Abstract: "Diffusion models have shown superior performance on unsupervised anomaly detection tasks. Since trained with normal data only, diffusion models tend to reconstruct normal counterparts of test images with certain noises added. However, these methods treat all potential anomalies equally, which may cause two main problems. From the global perspective, the difficulty of reconstructing images with different anomalies is uneven. For example, adding back a missing element is harder than dealing with a scratch, thus requiring a larger number of denoising steps. Therefore, instead of utilizing the same setting for all samples, we propose to predict a particular denoising step for each sample by evaluating the difference between image contents and the priors extracted from diffusion models. From the local perspective, reconstructing abnormal regions differs from normal areas even in the same image. Theoretically, the diffusion model predicts a noise for each step, typically following a standard Gaussian distribution. However, due to the difference between the anomaly and its potential normal counterpart, the predicted noise in abnormal regions will inevitably deviate from the standard Gaussian distribution. To this end, we propose introducing synthetic abnormal samples in training to encourage the diffusion models to break through the limitation of standard Gaussian distribution, and a spatial-adaptive feature fusion scheme is utilized during inference. With the above modifications, we propose a global and local adaptive diffusion model (abbreviated to GLAD) for unsupervised anomaly detection, which introduces appealing flexibility and achieves anomaly-free reconstruction while retaining as much normal information as possible. Extensive experiments are conducted on three commonly used anomaly detection datasets (MVTec-AD, MPDD, and VisA) and a printed circuit board dataset (PCB-Bank) we integrated, showing the effectiveness of the proposed method. The source code and pre-trained models are publicly available at https://github.com/hyao1/GLAD."
MedRAT: Unpaired Medical Report Generation via Auxiliary Tasks
Authors: Elad Hirsch; Gefen Dawidowicz; Ayellet Tal
Venue: ECCV 2024 (main)

Abstract: "Medical report generation from X-ray images is a challenging task, particularly in an unpaired setting where paired image-report data is unavailable for training. To address this challenge, we propose a novel model that leverages the available information in two distinct datasets, one comprising reports and the other consisting of images. The core idea of our model revolves around the notion that combining auto-encoding report generation with multi-modal (report-image) alignment can offer a solution. However, the challenge persists regarding how to achieve this alignment when pair correspondence is absent. Our proposed solution involves the use of auxiliary tasks, particularly contrastive learning and classification, to position related images and reports in close proximity to each other. This approach differs from previous methods that rely on pre-processing steps, such as using external information stored in a knowledge graph. Our model, named MedRAT, surpasses previous state-of-the-art methods, demonstrating the feasibility of generating comprehensive medical reports without the need for paired data or external tools. Our code is publicly available 1 . 1 https://github.com/eladhi/MedRAT"
Are Synthetic Data Useful for Egocentric Hand-Object Interaction Detection?
Authors: Rosario Leonardi; Antonino Furnari; Francesco Ragusa; Giovanni Maria Farinella
Venue: ECCV 2024 (main)

Abstract: "In this study, we investigate the effectiveness of synthetic data in enhancing egocentric hand-object interaction detection. Via extensive experiments and comparative analyses on three egocentric datasets, VISOR, EgoHOS, and ENIGMA-51, our findings reveal how to exploit synthetic data for the HOI detection task when real labeled data are scarce or unavailable. Specifically, by leveraging only 10% of real labeled data, we achieve improvements in Overall AP compared to baselines trained exclusively on real data of: +5.67% on EPIC-KITCHENS VISOR, +8.24% on EgoHOS, and +11.69% on ENIGMA-51. Our analysis is supported by a novel data generation pipeline and the newly introduced HOI-Synth benchmark which augments existing datasets with synthetic images of hand-object interactions automatically labeled with hand-object contact states, bounding boxes, and pixel-wise segmentation masks. Data, code, and data generation tools to support future research are released at: https://fpv-iplab. github.io/HOI-Synth/."
"PoseEmbroider: Towards a 3D, Visual, Semantic-aware Human Pose Representation"
Authors: Ginger Delmas; Philippe Weinzaepfel; Francesc Moreno-Noguer; Gregory Rogez
Venue: ECCV 2024 (main)

Abstract: "Aligning multiple modalities in a latent space, such as images and texts, has shown to produce powerful semantic visual representations, fueling tasks like image captioning, text-to-image generation, or image grounding. In the context of human-centric vision, albeit CLIP-like representations encode most standard human poses relatively well (such as standing or sitting), they lack sufficient acuteness to discern detailed or uncommon ones. Actually, while 3D human poses have been often associated with images (to perform pose estimation or pose-conditioned image generation), or more recently with text (for text-to-pose generation), they have seldom been paired with both. In this work, we combine 3D poses, person’s pictures and textual pose descriptions to produce an enhanced 3D-, visual- and semantic-aware human pose representation. We introduce a new transformer-based model, trained in a retrieval fashion, which can take as input any combination of the aforementioned modalities. When composing modalities, it outperforms a standard multi-modal alignment retrieval model, making it possible to sort out partial information (image with the lower body occluded). We showcase the potential of such an embroidered pose representation for (1) SMPL regression from image with optional text cue; and (2) on the task of fine-grained instruction generation, which consists in generating a text that describes how to move from one 3D pose to another (as a fitness coach). Unlike prior works, our model can take any kind of input (image and/or pose) without retraining."
A Comparative Study of Image Restoration Networks for General Backbone Network Design
Authors: Xiangyu Chen; Zheyuan Li; Yuandong Pu; Yihao Liu; Jiantao Zhou; Yu Qiao; Chao Dong
Venue: ECCV 2024 (main)

Abstract: "Despite the significant progress made by deep models in various image restoration tasks, existing image restoration networks still face challenges in terms of task generality. An intuitive manifestation is that networks which excel in certain tasks often fail to deliver satisfactory results in others. To illustrate this point, we select five representative networks and conduct a comparative study on five classic image restoration tasks. First, we provide a detailed explanation of the characteristics of different image restoration tasks and backbone networks. Following this, we present the benchmark results and analyze the reasons behind the performance disparity of different models across various tasks. Drawing from this comparative study, we propose that a general image restoration backbone network needs to meet the functional requirements of diverse tasks. Based on this principle, we design a new general image restoration backbone network, X-Restormer. Extensive experiments demonstrate that X-Restormer possesses good task generality and achieves state-of-the-art performance across a variety of tasks."
Learned Image Enhancement via Color Naming
Authors: David Serrano-Lozano; Luis Herranz; Michael S Brown; Javier Vazquez-Corral
Venue: ECCV 2024 (main)

Abstract: "A popular method for enhancing images involves learning the style of a professional photo editor using pairs of training images comprised of the original input with the editor-enhanced version. When manipulating images, many editing tools offer a feature that allows the user to manipulate a limited selection of familiar colors. Editing by color name allows easy adjustment of elements like the ”blue” of the sky or the ”green” of trees. Inspired by this approach to color manipulation, we propose NamedCurves, a learning-based image enhancement technique that separates the image into a small set of named colors. Our method learns to globally adjust the image for each specific named color via tone curves and then combines the images using an attention-based fusion mechanism to mimic spatial editing. We demonstrate the effectiveness of our method against several competing methods on the well-known Adobe 5K dataset and the PPR10K dataset, showing notable improvements."
Synthesizing Time-varying BRDFs via Latent Space
Authors: Takuto Narumoto; Hiroaki Santo; Fumio Okura
Venue: ECCV 2024 (main)

Abstract: "This paper introduces a method for synthesizing time-varying bidirectional reflectance distribution functions (BRDFs) by applying learned temporal changes to static BRDFs. Achieving realistic and natural changes in material appearance over time is crucial in computer graphics and virtual reality. Existing methods employ a parametric BRDF model, and the temporal changes in BRDFs are modeled by polynomial functions that represent the transitions of the BRDF parameters. However, the limited representational capabilities of both the parametric BRDF model and the polynomial temporal model restrict the fidelity of the appearance reproduction. In this paper, to overcome this limitation, we introduce a neural embedding for BRDFs and propose a neural temporal model that represents the temporal changes of BRDFs in the latent space, which allows flexible representations of BRDFs and temporal changes. The experiments using synthetic and real-world datasets demonstrate that the flexibility of the proposed approach achieves a faithful synthesis of temporal changes in material appearance."
HoloADMM: High-Quality Holographic Complex Field Recovery
Authors: Mazen Mel; Paul Springer; Pietro Zanuttigh; Haitao Zhou; Alexander Gatto
Venue: ECCV 2024 (main)

Abstract: "Holography enables intriguing microscopic imaging modalities, particularly through Quantitative Phase Imaging (QPI), which utilizes the phase of coherent light as a way to reveal the contrast in transparent and thin microscopic specimens. Despite the limitation of image sensors, which detect only light intensity, phase information can still be recorded within a two-dimensional interference pattern between two distinct light waves. Numerical reconstruction is later needed to retrieve the amplitude and phase from such holographic measurements. To this end, we introduce HoloADMM, a novel interpretable, learning-based approach for in-line holographic image reconstruction. HoloADMM enhances imaging capability with spatial image super-resolution, offering a versatile framework that accommodates multiple illumination wavelengths and supports extensive refocusing ranges with up to 10 µm precision. Our results indicate a substantial improvement in reconstruction quality over existing methods and demonstrate HoloADMM’s effective adaptation to real holographic data captured by our Digital in-line Holographic Microscope (DIHM). This work not only advances holographic imaging techniques but also broadens the potential for non-invasive microscopic analysis applications."
Fundamental Matrix Estimation Using Relative Depths
Authors: Yaqing Ding; Václav Vávra; Snehal Bhayani; Qianliang Wu; Jian Yang; Zuzana Kukelova
Venue: ECCV 2024 (main)

Abstract: "We propose a novel approach to estimate the fundamental matrix from point correspondences and their relative depths. Relative depths can be approximated from the scales of local features, which are commonly available or can be obtained from non-metric monocular depth estimates provided by popular deep learning-based methods. This makes the considered problem very relevant. To derive efficient solutions, we explore new geometric constraints on the fundamental matrix with known relative depths and present new algebraic constraints between the fundamental matrix and the translation vector. Using point correspondences and their relative depths, we derive novel efficient minimal solvers for two fully uncalibrated cameras, two cameras with different unknown focal lengths, and two cameras with equal unknown focal lengths, respectively. We propose different variants of these solvers based on the source of the relative depth information. We present detailed analyses and comparisons with state-of-the-art solvers, including results with 86, 306 image pairs from three large-scale datasets."
Gaussian Splatting on the Move: Blur and Rolling Shutter Compensation for Natural Camera Motion
Authors: Otto Seiskari; Jerry Ylilammi; Valtteri Kaatrasalo; Pekka Rantalankila; Matias Turkulainen; Juho Kannala; Esa Rahtu; Arno Solin
Venue: ECCV 2024 (main)

Abstract: "High-quality scene reconstruction and novel view synthesis based on Gaussian Splatting (3DGS) typically require steady, high-quality photographs, often impractical to capture with handheld cameras. We present a method that adapts to camera motion and allows high-quality scene reconstruction with handheld video data suffering from motion blur and rolling shutter distortion. Our approach is based on detailed modelling of the physical image formation process and utilizes velocities estimated using visual-inertial odometry (VIO). Camera poses are considered non-static during the exposure time of a single image frame and camera poses are further optimized in the reconstruction process. We formulate a differentiable rendering pipeline that leverages screen space approximation to efficiently incorporate rolling-shutter and motion blur effects into the 3DGS framework. Our results with both synthetic and real data demonstrate superior performance in mitigating camera motion over existing methods, thereby advancing 3DGS in naturalistic settings."
MTaDCS: Moving Trace and Feature Density-based Confidence Sample Selection under Label Noise
Authors: Qingzheng Huang; Xilin He; Xiaole Xian; Qinliang Lin; Weicheng Xie; Siyang Song; Linlin Shen; Zitong Yu
Venue: ECCV 2024 (main)

Abstract: "Learning from noisy labels is a challenging task, as noisy labels can compromise decision boundaries and result in suboptimal generalization performance. Most previous approaches for dealing noisy labels are based on sample selection, which utilized the small loss criterion to reduce the adverse effects of noisy labels. Nevertheless, they encounter a critical limitation in being unable to effectively separate challenging samples from those that were merely mislabeled. Meanwhile, there is a lack of researches on the trace changes of samples during training. To this end, we propose a novel moving trace and feature density-based confidence sample selection strategy (called MTaDCS). Different from existing small loss-based approaches, the local feature density of samples in the latent space is explored to construct a confidence set by selectively choosing confident samples in a progressive manner in terms of moving trace. Therefore, our MTaDCS can gradually isolate noisy labels through the setting of confidence set and achieve the goal of learning discriminative features from hard samples. Extensive experiments conducted on datasets with simulated and real-world noises validate that the proposed MTaDCS outperforms the state-of-the-art methods in terms of various metrics. The code is available at https://github.com/QZ-CODER/-ECCV-24-MTaD"
Towards Open-World Object-based Anomaly Detection via Self-Supervised Outlier Synthesis
Authors: Brian Kostadinov Shalon Isaac-Medina; Yona Falinie Abdul Gaus; Neelanjan Bhowmik; Toby P Breckon
Venue: ECCV 2024 (main)

Abstract: "Object detection is a pivotal task in computer vision that has received significant attention in previous years. Nonetheless, the capability of a detector to localise objects out of the training distribution remains unexplored. Whilst recent approaches in object-level out-of-distribution (OoD) detection heavily rely on class labels, such approaches contradict truly open-world scenarios where the class distribution is often unknown. In this context, anomaly detection focuses on detecting unseen instances rather than classifying detections as OoD. This work aims to bridge this gap by leveraging an open-world object detector and an OoD detector via virtual outlier synthesis. This is achieved by using the detector backbone features to first learn object pseudo-classes via self-supervision. These pseudo-classes serve as the basis for class-conditional virtual outlier sampling of anomalous features that are classified by an OoD head. Our approach empowers our overall object detector architecture to learn anomaly-aware feature representations without relying on class labels, hence enabling truly open-world object anomaly detection. Empirical validation of our approach demonstrates its effectiveness across diverse datasets encompassing various imaging modalities (visible, infrared, and X-ray). Moreover, our method establishes state-of-the-art performance on object-level anomaly detection, achieving an average recall score improvement of over 5.4% for natural images and 23.5% for a security X-ray dataset compared to the current approaches. In addition, our method detects anomalies in datasets where current approaches fail. Code available at https: //github.com/KostadinovShalon/oln-ssos."