GenAI4ELab/papercli-papers huggingface dataset:
Simultaneous Acquisition of High Quality RGB Image and Polarization Information Using a Sparse Polarization Sensor
Authors: Teppei Kurita; Yuhi Kondo; Legong Sun; Yusuke Moriuchi
Venue: WACV 2023 (main)

Abstract: This paper proposes a novel polarization sensor structure and network architecture to obtain a high-quality RGB image and polarization information. Conventional polarization sensors can simultaneously acquire RGB images and polarization information, but the polarizers on the sensor degrade the quality of the RGB images. There is a trade-off between the quality of the RGB image and polarization information as fewer polarization pixels reduce the degradation of the RGB image but decrease the resolution of polarization information. Therefore, we propose an approach that resolves the trade-off by sparsely arranging polarization pixels on the sensor and compensating for low-resolution polarization information with higher resolution using the RGB image as a guide. Our proposed network architecture consists of an RGB image refinement network and a polarization information compensation network. We confirmed the superiority of our proposed network in compensating the differential component of polarization intensity by comparing its performance with state-of-the-art methods for similar tasks: depth completion. Furthermore, we confirmed that our approach could simultaneously acquire higher quality RGB images and polarization information than conventional polarization sensors, resolving the trade-off between the quality of RGB images and polarization information. The baseline code and newly generated real and synthetic large-scale polarization image datasets are available for further research and development.
Skew-Robust Human-Object Interactions in Videos
Authors: Apoorva Agarwal; Rishabh Dabral; Arjun Jain; Ganesh Ramakrishnan
Venue: WACV 2023 (main)

Abstract: Humans are, arguably, one of the most important regions of interest in a visual analysis pipeline. Detecting how the human interacts with the surrounding environment, thus, becomes an important problem and has several potential use-cases. While this has been adequately addressed in the literature in the image setting, there exist very few methods addressing the case for in-the-wild videos. The problem is further exacerbated by the high degree of label skew. To this end, we propose SeRVo-HOI, a robust end-to-end framework for recognizing human-object interactions from a video, particularly in high label-skew settings. The network contextualizes multiple image representations and is trained to explicitly handle dataset skew. We propose and analyse methods to address the long-tail distribution of the labels and show improvements on the tail-labels. SeRVo-HOI outperforms the state-of-the-art by a significant margin 21.1% vs 17.6% mAP on the large-scale, in-the-wild VidHOI dataset while particularly demonstrating solid improvements in the tail-classes 19.9% vs 17.3% mAP.
Video Joint Denoising and Demosaicing With Recurrent CNNs
Authors: Valéry Dewil; Adrien Courtois; Mariano Rodríguez; Thibaud Ehret; Nicola Brandonisio; Denis Bujoreanu; Gabriele Facciolo; Pablo Arias
Venue: WACV 2023 (main)

Abstract: Denoising and demosaicing are two critical components of the image/video processing pipeline. While historically these two tasks have mainly been considered separately, current neural network approaches allow to obtain state-of-the-art results by treating them jointly. However, most existing research focuses in single image or burst joint denoising and demosaicing (JDD). Although related to burst JDD, video JDD deserves its own treatment. In this work we present an empirical exploration of different design aspects of video joint denoising and demosaicing using neural networks. We compare recurrent and non-recurrent approaches and explore aspects such as type of propagated information in recurrent networks, motion compensation, video stabilization, and network architecture. We found that recurrent networks with motion compensation achieve best results. Our work should serve as a strong baseline for future research in video JDD.
Gallery Filter Network for Person Search
Authors: Lucas Jaffe; Avideh Zakhor
Venue: WACV 2023 (main)

Abstract: In person search, we aim to localize a query person from one scene in other gallery scenes. The cost of this search operation is dependent on the number of gallery scenes, making it beneficial to reduce the pool of likely scenes. We describe and demonstrate the Gallery Filter Network (GFN), a novel module which can efficiently discard gallery scenes from the search process, and benefit scoring for persons detected in remaining scenes. We show that the GFN is robust under a range of different conditions by testing on different retrieval sets, including cross-camera, occluded, and low-resolution scenarios. In addition, we develop the base SeqNeXt person search model, which improves and simplifies the original SeqNet model. We show that the SeqNeXt+GFN combination yields significant performance gains over other state-of-the-art methods on the standard PRW and CUHK-SYSU person search datasets. To aid experimentation for this and other models, we provide standardized tooling for the data processing and evaluation pipeline typically used for person search research.
A Continual Deepfake Detection Benchmark: Dataset, Methods, and Essentials
Authors: Chuqiao Li; Zhiwu Huang; Danda Pani Paudel; Yabin Wang; Mohamad Shahbazi; Xiaopeng Hong; Luc Van Gool
Venue: WACV 2023 (main)

Abstract: There have been emerging a number of benchmarks and techniques for the detection of deepfakes. However, very few works study the detection of incrementally appearing deepfakes in the real-world scenarios. To simulate the wild scenes, this paper suggests a continual deepfake detection benchmark (CDDB) over a new collection of deepfakes from both known and unknown generative models. The suggested CDDB designs multiple evaluations on the detection over easy, hard, and long sequence of deepfake tasks, with a set of appropriate measures. In addition, we exploit multiple approaches to adapt multiclass incremental learning methods, commonly used in the continual visual recognition, to the continual deepfake detection problem. We evaluate existing methods, including their adapted ones, on the proposed CDDB. Within the proposed benchmark, we explore some commonly known essentials of standard continual learning. Our study provides new insights on these essentials in the context of continual deepfake detection. The suggested CDDB is clearly more challenging than the existing benchmarks, which thus offers a suitable evaluation avenue to the future research. Both data and code are available at https://github.com/Coral79/CDDB.
Semantic Segmentation of Degraded Images Using Layer-Wise Feature Adjustor
Authors: Kazuki Endo; Masayuki Tanaka; Masatoshi Okutomi
Venue: WACV 2023 (main)

Abstract: Semantic segmentation of degraded images is important for practical applications such as autonomous driving and surveillance systems. The degradation level, which represents the strength of degradation, is usually unknown in practice. Therefore, the semantic segmentation algorithm needs to take account of various levels of degradation. In this paper, we propose a convolutional neural network of semantic segmentation which can cope with various levels of degradation. The proposed network is based on the knowledge distillation from a source network trained with only clean images. More concretely, the proposed network is trained to acquire multi-layer features keeping consistency with the source network, while adjusting for various levels of degradation. The effectiveness of the proposed method is confirmed for different types of degradations: JPEG distortion, Gaussian blur and salt&pepper noise. The experimental comparisons validate that the proposed network outperforms existing networks for semantic segmentation of degraded images with various degradation levels.
Addressing Feature Suppression in Unsupervised Visual Representations
Authors: Tianhong Li; Lijie Fan; Yuan Yuan; Hao He; Yonglong Tian; Rogerio Feris; Piotr Indyk; Dina Katabi
Venue: WACV 2023 (main)

Abstract: Contrastive learning is one of the fastest growing research areas in machine learning due to its ability to learn useful representations without labeled data. However, contrastive learning is susceptible to feature suppression - i.e., it may discard important information relevant to the task of interest, and learn irrelevant features. Past work has addressed this limitation via handcrafted data augmentations that eliminate irrelevant information. This approach however does not work across all datasets and tasks. Further, data augmentations fail in addressing feature suppression in multi-attribute classification when one attribute can suppress features relevant to other attributes. In this paper, we analyze the objective function of contrastive learning and formally prove that it is vulnerable to feature suppression. We then present Predictive Contrastive Learning (PrCL), a framework for learning unsupervised representations that are robust to feature suppression. The key idea is to force the learned representation to predict the input, and hence prevent it from discarding important information. Extensive experiments verify that PrCL is robust to feature suppression and outperforms state-of-the-art contrastive learning methods on a variety of datasets and tasks.
Guiding Users to Where To Give Color Hints for Efficient Interactive Sketch Colorization via Unsupervised Region Prioritization
Authors: Youngin Cho; Junsoo Lee; Soyoung Yang; Juntae Kim; Yeojeong Park; Haneol Lee; Mohammad Azam Khan; Daesik Kim; Jaegul Choo
Venue: WACV 2023 (main)

Abstract: Existing deep interactive colorization models have focused on ways to utilize various types of interactions, such as point-wise color hints, scribbles, or natural-language texts, as methods to reflect a user's intent at runtime. However, another approach, which actively informs the user of the most effective regions to give hints for sketch image colorization, has been under-explored. This paper proposes a novel model-guided deep interactive colorization framework that reduces the required amount of user interactions, by prioritizing the regions in a colorization model. Our method, called GuidingPainter, prioritizes these regions where the model most needs a color hint, rather than just relying on the user's manual decision on where to give a color hint. In our extensive experiments, we show that our approach outperforms existing interactive colorization methods in terms of the conventional metrics, such as PSNR and FID, and reduces required amount of interactions.
Learning Incoherent Light Emission Steering From Metasurfaces Using Generative Models
Authors: Prasad P. Iyer; Saaketh Desai; Sadhvikas Addamane; Remi Dingreville; Igal Brener
Venue: WACV 2023 (main)

Abstract: Spatiotemporal control over incoherent light sources is critically important for applications such as displays, remote sensing, clean energy, and illumination. Incoherent light emission made up of randomized wavefronts is incompatible with known beam steering techniques that rely on coherent electromagnetic wave interference. The emerging field of tunable dielectric metasurfaces consisting of sub- wavelength arrays of optical nanoresonators has recently enabled active re-direction of incoherent light (photoluminescence, PL) emission. This was achieved by illuminating (pumping) the metasurface with a pump laser reflecting off a programmable spatial light modulator (SLM) with sawtooth grating patterns as input. Achieving efficient beam steering requires the generation of optimal pump patterns programmed into the SLM to maximize the PL emitted towards a given direction. Given the innumerable possibilities and the lack of a theoretical physical framework to guide the exploration of pump patterns, we use an active learning algorithm running a closed loop optical experiment with a generative model to explore and optimize novel pump patterns. We achieve up to an order of magnitude enhancement in the steering efficiency by using pump patterns that are generated by a variational auto-encoder, with minimal number of experiments. The results presented in this paper highlight the unique ability of generative models and active learning to dramatically improve steering efficiency by finding novel optical pump patterns that are beyond human intuition. Our combination of advanced machine learning techniques driving closed loop nanophotonic experiments might pave the way to derive the underlying physics of emergent light-matter phenomena.
Multimodal Vision Transformers With Forced Attention for Behavior Analysis
Authors: Tanay Agrawal; Michal Balazia; Philipp Müller; François Brémond
Venue: WACV 2023 (main)

Abstract: Human behavior understanding requires looking at minute details in the large context of a scene containing multiple input modalities. It is necessary as it allows the design of more human-like machines. While transformer approaches have shown great improvements, they face multiple challenges such as lack of data or background noise. To tackle these, we introduce the Forced Attention (FAt) Transformer which utilize forced attention with a modified backbone for input encoding and a use of additional inputs. In addition to improving the performance on different tasks and inputs, the modification requires less time and memory resources. We provide a model for a generalised feature extraction for tasks concerning social signals and behavior analysis. Our focus is on understanding behavior in videos where people are interacting with each other or talking into the camera which simulates the first person point of view in social interaction. FAt Transformers are applied to two downstream tasks: personality recognition and body language recognition. We achieve state-of-the-art results for Udiva v0.5, First Impressions v2 and MPII Group Interaction datasets. We further provide an extensive ablation study of the proposed architecture.
Anomaly Detection in 3D Point Clouds Using Deep Geometric Descriptors
Authors: Paul Bergmann; David Sattlegger
Venue: WACV 2023 (main)

Abstract: We present a new method for the unsupervised detection of geometric anomalies in high-resolution 3D point clouds. In particular, we propose an adaptation of the established student-teacher anomaly detection framework to three dimensions. A student network is trained to match the output of a pretrained teacher network on anomaly-free point clouds. When applied to test data, regression errors between the teacher and the student allow reliable localization of anomalous structures. To construct an expressive teacher network that extracts dense local geometric descriptors, we introduce a novel self-supervised pretraining strategy. The teacher is trained by reconstructing local receptive fields and does not require annotations. Extensive experiments on the comprehensive MVTec 3D Anomaly Detection dataset highlight the effectiveness of our approach, which outperforms the existing methods by a large margin. Ablation studies show that our approach meets the requirements of practical applications regarding performance, runtime, and memory consumption.
Self-Supervised Clustering Based on Manifold Learning and Graph Convolutional Networks
Authors: Leonardo Tadeu Lopes; Daniel Carlos Guimarães Pedronette
Venue: WACV 2023 (main)

Abstract: In spite of the huge advances in supervised learning, the common requirement for extensive labeled datasets represents a severe bottleneck. In this scenario, other learning paradigms capable of addressing the challenge associated with the scarcity of labeled data represent a relevant alternative solution. This paper presents a novel clustering method called Self-Supervised Graph Convolutional Clustering (SGCC), which aims to exploit the strengths of different learning paradigms, combining unsupervised, semi-supervised, and self-supervised perspectives. An unsupervised manifold learning algorithm based on hypergraphs and ranking information is used to provide more effective and global similarity information. The hypergraph structures allow identifying representative items for each cluster, which are used to derive a set of small but high confident clusters. Such clusters are taken as soft-labels for training a Graph Convolutional Network (GCN) in a semi-supervised classification task. Once trained in a self-supervised setting, the GCN is used to predict the cluster of remaining items. The proposed SGCC method was evaluated both in image and citation networks datasets and compared with classic and recent clustering methods, obtaining high-effective results in all scenarios.
Orthogonal Transforms for Learning Invariant Representations in Equivariant Neural Networks
Authors: Jaspreet Singh; Chandan Singh; Ankur Rana
Venue: WACV 2023 (main)

Abstract: The convolutional layers of the standard convolutional neural networks (CNNs) are equivariant to translation. Recently, a new class of CNNs is introduced which is equivariant to other affine geometric transformations such as rotation and reflection by replacing the standard convolutional layer with the group convolutional layer or using the steerable filters in the convloutional layer. We propose to embed the 2D positional encoding which is invariant to rotation, reflection and translation using orthogonal polar harmonic transforms (PHTs) before flattening the feature maps for fully-connected or classification layer in the equivariant CNN architecture. We select the PHTs among several invariant transforms, as they are very efficient in performance and speed. The proposed 2D positional encoding scheme between the convolutional and fully-connected layers of the equivariant networks is shown to provide significant improvement in performance on the rotated MNIST, CIFAR-10 and CIFAR-100 datasets.
ViewCLR: Learning Self-Supervised Video Representation for Unseen Viewpoints
Authors: Srijan Das; Michael S. Ryoo
Venue: WACV 2023 (main)

Abstract: Learning self-supervised video representation predominantly focuses on discriminating instances generated from simple data augmentation schemes. However, the learned representation often fails to generalize over unseen camera viewpoints. To this end, we propose ViewCLR, that learns self-supervised video representation invariant to camera viewpoint changes. We introduce a viewpoint-generator that can be considered as a learnable augmentation for any self-supervised pre-text tasks, to generate latent viewpoint representation of a video. ViewCLR maximizes the similarities between the representation of the latent viewpoint and that of the original viewpoint, enabling the learned video encoder to generalize over unseen camera viewpoints. Experiments on cross-view benchmark datasets including NTU RGB+D dataset show that ViewCLR stands as a state-of-the-art viewpoint invariant self-supervised method.
MRI Imputation Based on Fused Index- and Intensity-Registration
Authors: Jiyoon Shin; Jungwoo Lee
Venue: WACV 2023 (main)

Abstract: 3D MRI imaging is based on a number of imaging sequences such as T1, T2, T1ce, and Flair, and each of them is performed by a group of two-dimensional scans. In practical MRI, some scans are often missing while many medical applications require a full set of scans. An MRI imputation method is presented, which synthesizes such missing scans. Key components in this method are the index registration and the intensity registration. The index registration models anatomical differences between two different scans in the same imaging sequence, and the intensity registration reflects the image contrast differences between two different scans of the same index. Two registration fields are learned to be invariant, and accordingly, allow two estimates of a missing scan, one within corresponding imaging sequence and another along scan index; the two estimates are combined to yield the final synthesized scan. Experimental results highlight that the proposed method improves prevalent limitations existing in previous synthesis methods, blending both structural and contrast aspects and capturing subtle parts of the brain. Quantitative results also show the superiority in various data sets, transitions, and measures.
Masked Image Modeling Advances 3D Medical Image Analysis
Authors: Zekai Chen; Devansh Agarwal; Kshitij Aggarwal; Wiem Safta; Mariann Micsinai Balan; Kevin Brown
Venue: WACV 2023 (main)

Abstract: Recently, masked image modeling (MIM) has gained considerable attention due to its capacity to learn from vast amounts of unlabeled data and has been demonstrated to be effective on a wide variety of vision tasks involving natural images. Meanwhile, the potential of self-supervised learning in modeling 3D medical images is anticipated to be immense due to the high quantities of unlabeled images, and the expense and difficulty of quality labels. However, MIM's applicability to medical images remains uncertain. In this paper, we demonstrate that masked image modeling approaches can also advance 3D medical images analysis in addition to natural images. We study how masked image modeling strategies leverage performance from the viewpoints of 3D medical image segmentation as a representative downstream task: i) when compared to naive contrastive learning, masked image modeling approaches accelerate the convergence of supervised training even faster (1.40x) and ultimately produce a higher dice score; ii) predicting raw voxel values with a high masking ratio and a relatively smaller patch size is non-trivial self-supervised pretext-task for medical images modeling; iii) a lightweight decoder or projection head design for reconstruction is powerful for masked image modeling on 3D medical images which speeds up training and reduce cost; iv) finally, we also investigate the effectiveness of MIM methods under different practical scenarios where different image resolutions and labeled data ratios are applied. Anonymized codes are available at https://anonymous.4open.science/r/MIM-Med3D.
PreViTS: Contrastive Pretraining With Video Tracking Supervision
Authors: Brian Chen; Ramprasaath R. Selvaraju; Shih-Fu Chang; Juan Carlos Niebles; Nikhil Naik
Venue: WACV 2023 (main)

Abstract: Videos are a rich source for self-supervised learning (SSL) of visual representations due to the presence of natural temporal transformations of objects. However, current methods typically randomly sample video clips for learning, which results in an imperfect supervisory signal. In this work, we propose PreViTS, an SSL framework that utilizes an unsupervised tracking signal for selecting clips containing the same object, which helps better utilize temporal transformations of objects. PreViTS further uses the tracking signal to spatially constrain the frame regions to learn from and trains the model to locate meaningful objects by providing supervision on Grad-CAM attention maps. To evaluate our approach, we train a momentum contrastive (MoCo) encoder on VGG-Sound and Kinetics-400 datasets with PreViTS. Training with PreViTS outperforms representations learnt by contrastive strategy alone on video downstream tasks, obtaining state-of-the-art performance on action classification. PreViTS helps learn feature representations that are more robust to changes in background and context, as seen by experiments on datasets with background changes. Learning from large-scale videos with PreViTS could lead to more accurate and robust visual feature representations.