GenAI4ELab/papercli-papers huggingface dataset:
CLIP-Fusion: A Spatio-Temporal Quality Metric for Frame Interpolation
Authors: Goksel Mert Çökmez; Yang Zhang; Christopher Schroers; Tunç Ozan Aydin
Venue: WACV 2025 (main)

Abstract: Video frame interpolation is an ill-posed problem and a wide variety of methods have been proposed ranging from more traditional computer vision strategies to the most recent developments with neural network models. While there are many methods to interpolate video frames quality assessment regarding the resulting artifacts from these methods remains dependent on off-the-shelf methods. Although these methods can make accurate quality predictions for many visual artifacts such as compression blurring and banding their performance is mediocre for video frame interpolation artifacts due to the unique spatio-temporal qualities of such artifacts. To address this we aim to leverage semantic feature extraction capabilities of the pre-trained visual backbone of CLIP. Specifically we adapt its multi-scale approach to our feature extraction network and combine it with the spatio-temporal attention mechanism of the Video Swin Transformer. This allows our model to detect interpolation-related artifacts across frames and predict the relevant differential mean opinion score. Our model outperforms existing state-of-the-art quality metrics for assessing the quality of interpolated frames in both full-reference and no-reference settings.
Dual-Representation Interaction Driven Image Quality Assessment with Restoration Assistance
Authors: Jingtong Yue; Xin Lin; Zijiu Yang; Chao Ren
Venue: WACV 2025 (main)

Abstract: No-Reference Image Quality Assessment for distorted images has always been a challenging problem due to image content variance and distortion diversity. Previous IQA models mostly encode explicit single-quality features of synthetic images to obtain quality-aware representations for quality score prediction. However performance decreases when facing real-world distortion and restored images from restoration models. The reason is that they do not consider the degradation factors of the low-quality images adequately. To address this issue we first introduce the DRI method to obtain degradation vectors and quality vectors of images which separately model the degradation and quality information of low-quality images. After that we add the restoration network to provide the MOS score predictor with degradation information. Then we design the Representation-based Semantic Loss (RS Loss) to assist in enhancing effective interaction between representations. Extensive experimental results demonstrate that the proposed method performs favorably against existing state-of-the-art models on both synthetic and real-world datasets.
Gaussian Deja-vu: Creating Controllable 3D Gaussian Head-Avatars with Enhanced Generalization and Personalization Abilities
Authors: Peizhi Yan; Rabab Ward; Qiang Tang; Shan Du
Venue: WACV 2025 (main)

Abstract: Recent advancements in 3D Gaussian Splatting (3DGS) have unlocked significant potential for modeling 3D head avatars providing greater flexibility than mesh-based methods and more efficient rendering compared to NeRF-based approaches. Despite these advancements the creation of controllable 3DGS-based head avatars remains time-intensive often requiring tens of minutes to hours. To expedite this process we here introduce the "Gaussian Deja-vu" framework which first obtains a generalized model of the head avatar and then personalizes the result. The generalized model is trained on large 2D (synthetic and real) image datasets. This model provides a well-initialized 3D Gaussian head that is further refined using a monocular video to achieve the personalized head avatar. For personalizing we propose learnable expression-aware rectification blendmaps to correct the initial 3D Gaussians ensuring rapid convergence without the reliance on neural networks. Experiments demonstrate that the proposed method meets its objectives. It outperforms state-of-the-art 3D Gaussian head avatars in terms of photorealistic quality as well as reduces training time consumption to at least a quarter of the existing methods producing the avatar in minutes. Project homepage: https://peizhiyan.github.io/docs/dejavu
No Annotations for Object Detection in Art through Stable Diffusion
Authors: Patrick Ramos; Nicolas Gonthier; Selina Khan; Yuta Nakashima; Noa Garcia
Venue: WACV 2025 (main)

Abstract: Object detection in art is a valuable tool for the digital humanities as it allows for faster identification of objects in artistic and historical images compared to humans. However annotating such images poses significant challenges due to the need for specialized domain expertise. We present NADA (no annotations for detection in art) a pipeline that leverages diffusion models' art-related knowledge for object detection in paintings without the need for full bounding box supervision. Our method which supports both weakly-supervised and zero-shot scenarios and does not require any fine-tuning of its pretrained components consists of a class proposer based on large vision-language models and a class-conditioned detector based on Stable Diffusion. NADA is evaluated on two artwork datasets ArtDL 2.0 and IconArt outperforming prior work in weakly-supervised detection while being the first work for zero-shot object detection in art. Code is available at https://github.com/patrick-john-ramos/nada
Unifying Low-Resolution and High-Resolution Alignment by Event Cameras for Space-Time Video Super-Resolution
Authors: Hoonhee Cho; Jae-Young Kang; Taewoo Kim; Yuhwan Jeong; Kuk-Jin Yoon
Venue: WACV 2025 (main)

Abstract: Event cameras deliver asynchronous pixel intensity changes which result in sparse event data that offers the advantages of high temporal resolution. These high temporal characteristics make researchers naturally incorporate event cameras into video frame interpolation (VFI) and video super-resolution (VSR). In this paper we make the first attempt to solve the space-time video super-resolution (STVSR) task effectively addressing both VFI and VSR simultaneously by leveraging temporally dense events. STVSR aims to generate intermediate high-resolution (HR) videos between consecutive low-resolution (LR) frames. To fully exploit the high temporal frequency of events for STVSR we focus on temporal alignment in two stages at low-resolution and after up-sampling in high-resolution. In temporal alignment at low-resolution to upsample spatial dimensions effectively we leverage high temporal features to preserve spatial context. On the other hand for temporal alignment at the high-resolution stage we employ a deformable sampling process from events to achieve accurate alignment with forward and backward directions. In addition we provide the SuperREST dataset which features high-frequency details and complex motion in an RGB-Event setup. Experimental results on several datasets demonstrate that our method achieves a significant performance gain on STVSR tasks with low computational cost. Our codes and datasets are available at https://github.com/Chohoonhee/ESTNet.
GeoPos: A Minimal Positional Encoding for Enhanced Fine-Grained Details in Image Synthesis using Convolutional Neural Networks
Authors: Mehran Hosseini; Peyman Hosseini
Venue: WACV 2025 (main)

Abstract: The enduring inability of image generative models to recreate intricate geometric features such as those present in human hands and fingers has been an ongoing problem in image generation for nearly a decade. While strides have been made by increasing model sizes and diversifying training datasets this issue remains prevalent across all models from denoising diffusion models to Generative Adversarial Networks (GAN) pointing to a fundamental shortcoming in the underlying architectures. In this paper we demonstrate how this problem can be mitigated by augmenting convolution layers geometric capabilities through providing them with a single input channel incorporating the relative n-dimensional Cartesian coordinate system. We show this drastically improves quality of hand and face images generated by GANs and Variational AutoEncoders (VAE).
Diffusion-Based Visual Anagram as Multi-Task Learning
Authors: Zhiyuan Xu; Yinhe Chen; Huan-ang Gao; Weiyan Zhao; Guiyu Zhang; Hao Zhao
Venue: WACV 2025 (main)

Abstract: Visual anagrams are images that change appearance upon transformation like flipping or rotation. With the advent of diffusion models generating such optical illusions can be achieved by averaging noise across multiple views during the reverse denoising process. However we observe two critical failure modes in this approach: (i) concept segregation where concepts in different views are independently generated which can not be considered a true anagram and (ii) concept domination where certain concepts overpower others. In this work we cast the visual anagram generation problem in a multi-task learning setting where different viewpoint prompts are analogous to different tasks and derive denoising trajectories that align well across tasks simultaneously. At the core of our designed framework are two newly introduced techniques where (i) an anti-segregation optimization strategy that promotes overlap in cross-attention maps between different concepts and (ii) a noise vector balancing method that adaptively adjusts the influence of different tasks. Additionally we observe that directly averaging noise predictions yields suboptimal performance because statistical properties may not be preserved prompting us to derive a noise variance rectification method. Extensive qualitative and quantitative experiments demonstrate our method's superior ability to generate visual anagrams spanning diverse concepts.
MambaRecon: MRI Reconstruction with Structured State Space Models
Authors: Yilmaz Korkmaz; Vishal M. Patel
Venue: WACV 2025 (main)

Abstract: Magnetic Resonance Imaging (MRI) is one of the most important medical imaging modalities as it provides superior resolution of soft tissues albeit with a notable limitation in scanning speed. The advent of deep learning has catalyzed the development of cutting-edge methods for the expedited reconstruction of MRI scans utilizing convolutional neural networks and more recently vision transformers. Recently proposed structured state space models (e.g. Mamba) have gained some traction due to their efficiency and low computational requirements compared to transformer models. We propose an innovative MRI reconstruction framework that employs structured state space models at its core aimed at amplifying both long-range contextual sensitivity and reconstruction efficacy. Comprehensive experiments on public brain MRI datasets show that our model sets new benchmarks beating state-of-the-art reconstruction baselines. Code is available at https://github.com/yilmazkorkmaz1/MambaRecon.
LoSA: Long-Short-Range Adapter for Scaling End-to-End Temporal Action Localization
Authors: Akshita Gupta; Gaurav Mittal; Ahmed Magooda; Ye Yu; Graham Taylor; Mei Chen
Venue: WACV 2025 (main)

Abstract: Temporal Action Localization (TAL) involves localizing and classifying action snippets in an untrimmed video. The emergence of large video foundation models has led RGB-only video backbones to outperform previous methods needing both RGB and optical flow modalities. Leveraging these large models is often limited to training only the TAL head due to the prohibitively large GPU memory required to adapt the video backbone for TAL. To overcome this limitation we introduce LoSA the first memory-and-parameter-efficient backbone adapter designed specifically for TAL to handle untrimmed videos. LoSA specializes for TAL by introducing Long-Short-range Adapters that adapt the intermediate layers of the video backbone over different temporal ranges. These adapters run parallel to the video backbone to significantly reduce memory footprint. LoSA also includes Long-Short-range Gated Fusion that strategically combines the output of these adapters from the video backbone layers to enhance the video features provided to the TAL head. Experiments show that LoSA significantly outperforms all existing methods on standard TAL benchmarks THUMOS-14 and ActivityNet-v1.3 by scaling end-to-end backbone adaptation to billion-parameter-plus models like VideoMAEv2 (ViT-g) and leveraging them beyond head-only transfer learning.
Investigating Imaging Annotation and Self-Supervision for the Classification of Continuously Developing Cells in Histological Whole Slide Images
Authors: Sebastian Thiele; Jacqueline Kockwelp; Joachim Wistuba; Sabine Kliesch; Jörg Gromoll; Benjamin Risse
Venue: WACV 2025 (main)

Abstract: The analysis of individual cells is increasingly automated through deep learning techniques. This is particularly relevant for high-resolution whole slide images (WSIs) which can contain thousands of cells making manual evaluation impractical. This increase in automation however requires higher levels of standardisation (with respect to the scanning hardware settings and staining) and is further aggravated by the dynamics of the underlying cellular processes rendering unique cell classifications difficult. To address these difficulties we investigated the entire processing pipeline (from imaging over annotation to model training) and study its underlying trade-offs. In particular we created a new dataset comprising of more than 6300 labelled and 500000 unlabelled cells scanned using two different scan settings resulting in fully registered image pairs with varying level of detail and quality. Using these alternative dataset versions we analysed the impact of inter- and intra-variability between three different annotators and addressed the challenge of limited labelled data by comparing the impact of different self-supervised pretraining strategies. Overall our analyses provide new insights into the dependencies between imaging annotation self-supervision and deep learning-based classification especially in the context of continuously developing cells and demonstrate the beneficial impact of these considerations on the overall classification accuracy. Code is available at https://zivgitlab.uni-muenster.de/cvmls/icdc and the data will be shared upon qualified request due to data privacy laws.
Spatio-Temporal Context Prompting for Zero-Shot Action Detection
Authors: Wei-Jhe Huang; Min-Hung Chen; Shang-Hong Lai
Venue: WACV 2025 (main)

Abstract: Spatio-temporal action detection encompasses the tasks of localizing and classifying individual actions within a video. Recent works aim to enhance this process by incorporating interaction modeling which captures the relationship between people and their surrounding context. However these approaches have primarily focused on fully-supervised learning and the current limitation lies in the lack of generalization capability to recognize unseen action categories. In this paper we aim to adapt the pretrained image-language models to detect unseen actions. To this end we propose a method which can effectively leverage the rich knowledge of visual-language models to perform Person-Context Interaction. Meanwhile our Context Prompting module will utilize contextual information to prompt labels thereby enhancing the generation of more representative text features. Moreover to address the challenge of recognizing distinct actions by multiple people at the same timestamp we design the Interest Token Spotting mechanism which employs pretrained visual knowledge to find each person's interest context tokens and then these tokens will be used for prompting to generate text features tailored to each individual. To evaluate the ability to detect unseen actions we propose a comprehensive benchmark on J-HMDB UCF101-24 and AVA datasets. The experiments show that our method achieves superior results compared to previous approaches and can be further extended to multi-action videos bringing it closer to real-world applications. The code and data can be found in ST-CLIP.
COSNet: A Novel Semantic Segmentation Network using Enhanced Boundaries in Cluttered Scenes
Authors: Muhammad Ali; Mamoona Javaid; Mubashir Noman; Mustansar Fiaz; Salman Khan
Venue: WACV 2025 (main)

Abstract: Automated waste recycling aims to efficiently separate the recyclable objects from the waste by employing vision-based systems. However the presence of varying shaped objects having different material types makes it a challenging problem especially in cluttered environments. Existing segmentation methods perform reasonably on many semantic segmentation datasets by employing multi-contextual representations however their performance is degraded when utilized for waste object segmentation in cluttered scenarios. In addition plastic objects further increase the complexity of the problem due to their translucent nature. To address these limitations we introduce an efficacious segmentation network named COSNet that uses boundary cues along with multi-contextual information to accurately segment the objects in cluttered scenes. COSNet introduces novel components including feature sharpening block (FSB) and boundary enhancement module (BEM) for enhancing the features and highlighting the boundary information of irregular waste objects in cluttered environment. Extensive experiments on three challenging datasets including ZeroWaste-f SpectralWaste and ADE20K demonstrate the effectiveness of the proposed method. Our COSNet achieves a significant gain of 1.8% on ZeroWaste-f and 2.1% on SpectralWaste datasets respectively in terms of mIoU metric.
Learning Instance-Specific Parameters of Black-Box Models using Differentiable Surrogates
Authors: Arnisha Khondaker; Nilanjan Ray
Venue: WACV 2025 (main)

Abstract: Tuning parameters of a non-differentiable or black-box compute is challenging. Existing methods rely mostly on random sampling or grid sampling from the parameter space. Further with all the current methods it is not possible to supply any input specific parameters to the black-box. To the best of our knowledge for the first time we are able to learn input-specific parameters for a black box in this work. As a test application we choose a popular image denoising method BM3D as our black-box compute. Then we use a differentiable surrogate model (a neural network) to approximate the black-box behaviour. Next another neural network is used in an end-to-end fashion to learn input instance-specific parameters for the black-box. Motivated by prior advances in surrogate-based optimization we applied our method to the Smartphone Image Denoising Dataset (SIDD) and the Color Berkeley Segmentation Dataset (CBSD68) for image denoising. The results are compelling demonstrating a significant increase in PSNR and a notable improvement in SSIM nearing 0.93. Experimental results underscore the effectiveness of our approach in achieving substantial improvements in both model performance and optimization efficiency. For code and implementation details please refer to our GitHub repository: https://github.com/arnisha-k/instance-specific-param.
Swap Path Network for Robust Person Search Pre-Training
Authors: Lucas Jaffe; Avideh Zakhor
Venue: WACV 2025 (main)

Abstract: In person search we detect and rank matches to a query person image within a set of gallery scenes. Most person search models make use of a feature extraction backbone followed by separate heads for detection and re-identification. While pre-training methods for vision backbones are well-established pre-training additional modules for the person search task has not been previously examined. In this work we present the first framework for end-to-end person search pre-training. Our framework splits person search into object-centric and query-centric methodologies and we show that the query-centric framing is robust to label noise and trainable using only weakly-labeled person bounding boxes. Further we provide a novel model dubbed Swap Path Net (SPNet) which implements both query-centric and object-centric training objectives and can swap between the two while using the same weights. Using SPNet we show that query-centric pre-training followed by object-centric fine-tuning achieves state-of-the-art results on the standard PRW and CUHK-SYSU person search benchmarks with 96.4% mAP on CUHK-SYSU and 61.2% mAP on PRW. In addition we show that our method is more effective efficient and robust for person search pre-training than recent backbone-only pre-training alternatives.
Dropout Connects Transformers and CNNs: Transfer General Knowledge for Knowledge Distillation
Authors: Bokyeung Lee; Jonghwan Hong; Hyunuk Shin; Bonwha Ku; Hanseok Ko
Venue: WACV 2025 (main)

Abstract: Thanks to their long-range dependencies transformers obtain state-of-the-art performance in diverse research fields such as computer vision and audio processing. In practical scenarios convolutional neural networks (CNNs) are used more than Transformers due to their low complexity. So Transformer-to-CNN knowledge distillation (KD) research where the Transformer is the teacher and the CNN is the student is in demand and receiving attention. In Transformer-to-CNN KD training the capacity gap problem arising from structural differences between the teacher and student networks is the main factor of performance degradation of the student network unlike homogenous architecture KD. However previous KD studies transfer all of a teacher's knowledge to the student without considering structural differences. They cannot overcome problems caused by structural differences and show poor performance in Transformer-to-CNN KD. In this paper we identify general and specific knowledge in feature maps of the teacher and student. General and specific knowledge are the generalized and non-generalized feature representation. We propose a novel KD framework DropKD which extracts general knowledge from the teacher and student while removing specific knowledge and then allows general knowledge of the student network to learn general knowledge of the teacher. Our DropKD empowers the student network to achieve generalization by effectively managing general and specific knowledge. Through extensive experiments on challenging image classification datasets we demonstrate that the proposed method is superior to existing methods.
Feature Design for Bridging SAM and CLIP toward Referring Image Segmentation
Authors: Koichiro Ito
Venue: WACV 2025 (main)

Abstract: Referring Image Segmentation (RIS) is a task aimed at segmenting objects expressed in natural language within an image. This task requires an understanding of the relationship between vision and language along with precise segmentation capabilities. In the field of computer vision CLIP and Segment anything model (SAM) have gained significant attention for their classification and the segmentation capabilities. Given that both models possess essential skills for RIS combining them seems to be an effective strategy. In this paper we propose a model that integrates CLIP and SAM to enhance RIS. Since SAM lacks classification capabilities we developed a module that supplies the SAM mask decoder with features that specify the target object. We introduce a new module which is trained on additional instance segmentation tasks. The features utilized and derived from this module serve as inputs for the SAM decoder. With these inputs SAM is expected to effectively segment areas corresponding to the given natural language expressions. We conducted experiments using the traditional RefCOCO/+/g as well as the recently introduced gRefCOCO and Ref-zom datasets demonstrating the advantages of our approach. Code will be available on https://github.com/hitachi-rd-cv/dfam.