GenAI4ELab/papercli-papers huggingface dataset:
Foreground Guidance and Multi-Layer Feature Fusion for Unsupervised Object Discovery With Transformers
Authors: Zhiwei Lin; Zengyu Yang; Yongtao Wang
Venue: WACV 2023 (main)

Abstract: Unsupervised object discovery (UOD) has recently shown encouraging progress with the adoption of pre-trained Transformer features. However, current methods based on Transformers mainly focus on designing the localization head (e.g., seed selection-expansion and normalized cut) and overlook the importance of improving Transformer features. In this work, we handle UOD task from the perspective of feature enhancement and propose FOReground guidance and MUlti-LAyer feature fusion for unsupervised object discovery, dubbed FORMULA. Firstly, we present a foreground guidance strategy with an off-the-shelf UOD detector to highlight the foreground regions on the feature maps and then refine object locations in an iterative fashion. Moreover, to solve the scale variation issues in object detection, we design a multi-layer feature fusion module that aggregates features responding to objects at different scales. The experiments on VOC07, VOC12, and COCO_20k show that the proposed FORMULA achieves new state-of-the-art results on unsupervised object discovery. The code will be released at https://github.com/VDIGPKU/FORMULA.
Gait Recognition Using 3-D Human Body Shape Inference
Authors: Haidong Zhu; Zhaoheng Zheng; Ram Nevatia
Venue: WACV 2023 (main)

Abstract: Gait recognition, which identifies individuals based on their walking patterns, is an important biometric technique since it can be observed from a distance and does not require the subject's cooperation. Recognizing a person's gait is difficult because of the appearance variants in human silhouette sequences produced by varying viewing angles, carrying objects, and clothing. Recent research has produced a number of ways for coping with these variants. In this paper, we present the usage of inferring 3-D body shapes distilled from limited images, which are, in principle, invariant to the specified variants. Inference of 3-D shape is a difficult task, especially when only silhouettes are provided in a dataset. We provide a method for learning 3-D body inference from silhouettes by transferring knowledge from 3-D shape prior from RGB photos. We use our method on multiple existing state-of-the-art gait baselines and obtain consistent improvements for gait identification on two public datasets, CASIA-B and OUMVLP, on several variants and settings, including a new setting of novel views not seen during training.
Delving Into Masked Autoencoders for Multi-Label Thorax Disease Classification
Authors: Junfei Xiao; Yutong Bai; Alan Yuille; Zongwei Zhou
Venue: WACV 2023 (main)

Abstract: Vision Transformer (ViT) has become one of the most popular neural architectures due to its simplicity, scalability, and compelling performance in multiple vision tasks. However, since the scales of medical datasets are relatively small, ViT has shown inferior performance on medical datasets even after pre-trained on ImageNet. In this paper, we unleash the potential of ViT by pre-training on 266,340 unlabeled chest X-rays. Specifically, we explore Masked Autoencoders (MAE) whose task is to reconstruct missing pixels from a small proportion of each image and figure out a strong recipe for pre-training MAE and fine-tuning on chest X-ray datasets, revealing that medical reconstruction needs a much smaller proportion of an image than natural images (10% vs. 25%) and a more moderate RandomResizedCrop cropping range than natural images (0.5 1.0 vs. 0.2 1.0). With our recipe, ViT-S shows competitive results with the state-of-the-art CNN model (DenseNet-121) on three public chest X-ray datasets and 2.5x faster pre-training on the NIH ChestX-ray14 dataset and CheXpert. To the best of our knowledge, we are the first to make vanilla ViT achieve state-of-the-art performance on chest X-ray datasets. We hope that this study can direct future research on the application of Transformers to a larger variety of medical imaging tasks. Code will be made available.
Dissecting Deep Metric Learning Losses for Image-Text Retrieval
Authors: Hong Xuan; Xi (Stephen) Chen
Venue: WACV 2023 (main)

Abstract: Visual-Semantic Embedding (VSE) is a prevalent approach in image-text retrieval by learning a joint embedding space between the image and language modalities where semantic similarities would be preserved. The triplet loss with hard-negative mining has become the de-facto objective for most VSE methods. Inspired by recent progress in deep metric learning (DML) in the image domain which gives rise to new loss functions that outperform triplet loss, in this paper we revisit the problem of finding better objectives for VSE in image-text matching. Despite some attempts in designing losses based on gradient movement, most DML losses are defined empirically in the embedding space. Instead of directly applying these loss functions which may lead to sub-optimal gradient updates in model parameters, in this paper we present a novel Gradient-based Objective AnaLysis framework, or GOAL, to systematically analyze the combinations and reweighting of the gradients in existing DML functions. With the help of this analysis framework, we further propose a new family of objectives in the gradient space exploring different gradient combinations. In the event that the gradients are not integrable to a valid loss function, we implement our proposed objectives such that they would directly operate in the gradient space instead of on the losses in the embedding space. Comprehensive experiments have demonstrated that our novel objectives have consistently improved performance over baselines across different visual/text features and model frameworks. We also showed the generalizability of the GOAL framework by extending it to other models using triplet family losses including vision-language model with heavy cross-modal interactions and have achieved state-of-the-art results on the image-text retrieval tasks on COCO and Flick30K.
BoxMask: Revisiting Bounding Box Supervision for Video Object Detection
Authors: Khurram Azeem Hashmi; Alain Pagani; Didier Stricker; Muhammad Zeshan Afzal
Venue: WACV 2023 (main)

Abstract: We present a new, simple yet effective approach to uplift video object detection. We observe that prior works operate on instance-level feature aggregation that imminently neglects the refined pixel-level representation, resulting in confusion among objects sharing similar appearance or motion characteristics. To address this limitation, we pro- pose BoxMask, which effectively learns discriminative representations by incorporating class-aware pixel-level information. We simply consider bounding box-level annotations as a coarse mask for each object to supervise our method. The proposed module can be effortlessly integrated into any region-based detector to boost detection. Extensive experiments on ImageNet VID and EPIC KITCHENS datasets demonstrate consistent and significant improvement when we plug our BoxMask module into numerous recent state-of-the-art methods. The code will be available at https://github.com/khurramHashmi/BoxMask.
A Simple and Powerful Global Optimization for Unsupervised Video Object Segmentation
Authors: Georgy Ponimatkin; Nermin Samet; Yang Xiao; Yuming Du; Renaud Marlet; Vincent Lepetit
Venue: WACV 2023 (main)

Abstract: We propose a simple, yet powerful approach for unsupervised object segmentation in videos. We introduce an objective function whose minimum represents the mask of the main salient object over the input sequence. It only relies on independent image features and optical flows, which can be obtained using off-the-shelf self-supervised methods. It scales with the length of the sequence with no need for superpixels or sparsification, and it generalizes to different datasets without any specific training. This objective function can actually be derived from a form of spectral clustering applied to the entire video. Our method achieves on-par performance with the state of the art on standard benchmarks (DAVIS2016, SegTrack-v2, FBMS59), while being conceptually and practically much simpler.
Randomness Is the Root of All Evil: More Reliable Evaluation of Deep Active Learning
Authors: Yilin Ji; Daniel Kaestner; Oliver Wirth; Christian Wressnegger
Venue: WACV 2023 (main)

Abstract: Using deep neural networks for active learning (AL) poses significant challenges for the stability and the reproducibility of experimental results. Inconsistent settings continue to be the root causes for contradictory conclusions and in worst cases, for incorrect appraisal of methods. Our community is in search of a unified framework for exhaustive and fair evaluation of deep active learning. In this paper, we provide just such a framework, one which is built upon systematically fixing, containing and interpreting sources of randomness. We isolate different influence factors, such as neural-network initialization or hardware specifics, to assess their impact on the learning performance. We then use our framework to analyze the effects of basic AL settings, such as the query-batch size and the use of subset selection, and different datasets on AL performance. Our findings enable us to derive specific recommendations for the reliable evaluation of deep active learning, thus helping advance the community toward a more normative evaluation of results.
Visually Explaining 3D-CNN Predictions for Video Classification With an Adaptive Occlusion Sensitivity Analysis
Authors: Tomoki Uchiyama; Naoya Sogi; Koichiro Niinuma; Kazuhiro Fukui
Venue: WACV 2023 (main)

Abstract: This paper proposes a method for visually explaining the decision-making process of 3D convolutional neural networks (CNN) with a temporal extension of occlusion sensitivity analysis. The key idea here is to occlude a specific volume of data by a 3D mask in an input 3D temporal-spatial data space and then measure the change degree in the output score. The occluded volume data that produces a larger change degree is regarded as a more critical element for classification. However, while the occlusion sensitivity analysis is commonly used to analyze single image classification, it is not so straightforward to apply this idea to video classification as a simple fixed cuboid cannot deal with the motions. To this end, we adapt the shape of a 3D occlusion mask to complicated motions of target objects. Our flexible mask adaptation is performed by considering the temporal continuity and spatial co-occurrence of the optical flows extracted from the input video data. We further propose to approximate our method by using the first-order partial derivative of the score with respect to an input image to reduce its computational cost. We demonstrate the effectiveness of our method through various and extensive comparisons with the conventional methods in terms of the deletion/insertion metric and the pointing metric on the UCF-101. The code is available at: https://github.com/uchiyama33/AOSA.
Self-Supervised Learning With Local Contrastive Loss for Detection and Semantic Segmentation
Authors: Ashraful Islam; Benjamin Lundell; Harpreet Sawhney; Sudipta N. Sinha; Peter Morales; Richard J. Radke
Venue: WACV 2023 (main)

Abstract: We present a self-supervised learning (SSL) method suitable for semi-global tasks such as object detection and semantic segmentation. We enforce local consistency between self-learned features, representing corresponding image locations of transformed versions of the same image, by minimizing a pixel-level local contrastive (LC) loss during training. LC-loss can be added to existing self-supervised learning methods with minimal overhead. We evaluate our SSL approach on two downstream tasks -- object detection and semantic segmentation, using COCO, PASCAL VOC, and CityScapes datasets. Our method outperforms the existing state-of-the-art SSL approaches by 1.9% on COCO object detection, 1.4% on PASCAL VOC detection, and 0.6% on CityScapes segmentation.
Concept Correlation and Its Effects on Concept-Based Models
Authors: Lena Heidemann; Maureen Monnet; Karsten Roscher
Venue: WACV 2023 (main)

Abstract: Concept-based learning approaches for image classification, such as Concept Bottleneck Models, aim to enable interpretation and increase robustness by directly learning high-level concepts which are used for predicting the main class. They achieve competitive test accuracies compared to standard end-to-end models. However, with multiple concepts per image and binary concept annotations (without concept localization), it is not evident if the output of the concept model is truly based on the predicted concepts or other features in the image. Additionally, high correlations between concepts would allow a model to predict a concept with high test accuracy by simply using a correlated concept as a proxy. In this paper, we analyze these correlations between concepts in the CUB and GTSRB datasets and propose methods beyond test accuracy for evaluating their effects on the performance of a concept-based model trained on this data. To this end, we also perform a more detailed analysis on the effects of concept correlation using synthetically generated datasets of 3D shapes. We see that high concept correlation increases the risk of a model's inability to distinguish these concepts. Yet simple techniques, like loss weighting, show promising initial results for mitigating this issue.
A Neural Video Codec With Spatial Rate-Distortion Control
Authors: Noor Fathima; Jens Petersen; Guillaume Sautière; Auke Wiggers; Reza Pourreza
Venue: WACV 2023 (main)

Abstract: Neural video compression algorithms are nearly competitive with hand-crafted codecs in terms of rate-distortion performance and subjective quality. However, many neural codecs are inflexible black boxes, and give users little to no control over the reconstruction quality and bitrate. In this work, we present a flexible neural video codec that combines ideas from variable-bitrate codecs and region-of-interest-based coding. By conditioning our model on a global rate-distortion tradeoff parameter and a region-of-interest (ROI) mask, we obtain fine control over the per-frame bitrate and the reconstruction quality in the ROI. The resulting codec enables practical use cases such as coding under bitrate constraints with fixed ROI quality, while taking a negligible hit in overall rate-distortion performance. We find that our codec is best utilized when the sequence contains complex motion, such as scenes with camera panning or sports videos, where we substantially outperform non-ROI codecs in the region of interest with BD-rate savings exceeding 60% in some cases.
Planar Object Tracking via Weighted Optical Flow
Authors: Jonáš Šerých; Jiří Matas
Venue: WACV 2023 (main)

Abstract: We propose WOFT - a novel method for planar object tracking that estimates a full 8 degrees-of-freedom pose, i.e., the homography w.r.t. a reference view. The method uses a novel module that leverages dense optical flow and assigns a weight to each optical flow correspondence, estimating a homography by weighted least squares in a fully differentiable manner. The trained module assigns zero weights to incorrect correspondences (outliers) in most cases, making the method robust and eliminating the need of the typically used non-differentiable robust estimators like RANSAC. The proposed weighted optical flow tracker (WOFT) achieves state-of-the-art performance on two benchmarks, POT-210 and POIC, tracking consistently well across a wide range of scenarios.
Dance Style Transfer With Cross-Modal Transformer
Authors: Wenjie Yin; Hang Yin; Kim Baraka; Danica Kragic; Mårten Björkman
Venue: WACV 2023 (main)

Abstract: We present CycleDance, a dance style transfer system to transform an existing motion clip in one dance style to a motion clip in another dance style while attempting to preserve motion context of the dance. Our method extends an existing CycleGAN architecture for modeling audio sequences and integrates multimodal transformer encoders to account for music context. We adopt sequence length-based curriculum learning to stabilize training. Our approach captures rich and long-term intra-relations between motion frames, which is a common challenge in motion transfer and synthesis work. We further introduce new metrics for gauging transfer strength and content preservation in the context of dance movements. We perform an extensive ablation study as well as a human study including 30 participants with 5 or more years of dance experience. The results demonstrate that CycleDance generates realistic movements with the target style, significantly outperforming the baseline CycleGAN on naturalness, transfer strength, and content preservation.
Interpreting Disparate Privacy-Utility Tradeoff in Adversarial Learning via Attribute Correlation
Authors: Likun Zhang; Yahong Chen; Ang Li; Binghui Wang; Yiran Chen; Fenghua Li; Jin Cao; Ben Niu
Venue: WACV 2023 (main)

Abstract: Adversarial learning is commonly used to extract latent data representations which are expressive to predict the target attribute but indistinguishable in the privacy attribute. However, whether they can achieve an expected privacy-utility tradeoff is of great uncertainty. In this paper, we posit it is the complex interaction between different attributes in the training set that causes disparate tradeoff results. We first formulate the measurement of utility, privacy and their tradeoff in adversarial learning. Then we propose the metrics of Statistical Reliability (SR) and Feature Reliability (FR) to quantify the relationship between attributes. Specifically, SR reflects the co-occurrence sampling bias of the joint distribution between two attributes. Beyond the explicit dependence, FR exploits the intrinsic interaction one attribute exerts on the other via exploring the representation disentanglement. We validate the metrics in an adversarial learning scheme on CelebA and LFW dataset with a suite of target-privacy attribute pairs. Experiments demonstrate the strong correlations between the metrics and utility, privacy and their tradeoff. We further conclude how to use SR and FR as a guide to the selection of the privacy-utility tradeoff parameter.
Control-NeRF: Editable Feature Volumes for Scene Rendering and Manipulation
Authors: Verica Lazova; Vladimir Guzov; Kyle Olszewski; Sergey Tulyakov; Gerard Pons-Moll
Venue: WACV 2023 (main)

Abstract: We present Control-NeRF, a method for performing flexible, 3D-aware image content manipulation while enabling high-quality novel view synthesis, from a set of posed input images. NeRF-based approaches are effective for novel view synthesis, however such models memorize the radiance for every point in a scene within a neural network. Since these models are scene-specific and lack a 3D scene representation, classical editing such as shape manipulation, or combining scenes is not possible. While there are some recent hybrid approaches that combine NeRF with external scene representations such as sparse voxels, planes, hash tables, etc., they focus mostly on efficiency and don't explore the scene editing and manipulation capabilities of hybrid approaches. With the aim of exploring controllable scene representations for novel view synthesis, our model couples learnt scene-specific 3D feature volumes with a general NeRF rendering network. We can generalize to novel scenes by optimizing only the scene-specific 3D feature volume, while keeping the parameters of the rendering network fixed. Since the feature volumes are independent of the rendering model, we can manipulate and combine scenes by editing their corresponding feature volumes. The edited volume can then be plugged into the rendering model to synthesise high-quality novel views. We demonstrate scene manipulations including: scene mixing; applying rigid and non-rigid transformations; inserting, moving and deleting objects in a scene; while producing photo-realistic novel-view synthesis results.
Towards Equivariant Optical Flow Estimation With Deep Learning
Authors: Stefano Savian; Pietro Morerio; Alessio Del Bue; Andrea A. Janes; Tammam Tillo
Venue: WACV 2023 (main)

Abstract: Methods for Optical Flow (OF) estimation based on Deep Learning have considerably improved traditional approaches in challenging and realistic conditions. However, data-driven approaches can inherently be biased, leading to unexpected under-performance in real application scenarios. In this paper, we first observe that the OF estimation accuracy varies with motion direction, and name this phenomenon 'OF sign imbalance'. The sign imbalance cannot be assessed by means of the endpoint-error (EPE), the typical training and evaluation metric for Deep Optical Flow estimators. This paper tackles this issue by proposing a new metric to assess the sign imbalance, which is compared to the endpoint-error. We provide an extensive evaluation of the sign imbalance for the state-of-the-art optical flow estimators. Based on the evaluation, we propose two strategies to mitigate the phenomenon, i) by constraining the model estimations during inference, and, ii) by constraining the loss function during training. Testing and training code is available at: www.github.com/stsavian/equivariant_of_estimation.
Hyperblock Floating Point: Generalised Quantization Scheme for Gradient and Inference Computation
Authors: Marcelo Gennari do Nascimento; Victor Adrian Prisacariu; Roger Fawcett; Martin Langhammer
Venue: WACV 2023 (main)

Abstract: Prior quantization methods focus on producing networks for fast and lightweight inference. However, the cost of unquantised training is overlooked, despite requiring significantly more time and energy than inference. We present a method for quantizing convolutional neural networks for efficient training. Quantizing gradients is challenging because it requires higher granularity and their values span a wider range than the weight and feature maps. We propose an extension of the Channel-wise Block Floating Point format that allows for quick gradient computation, using a minimal amount of quantization time. This is achieved through sharing an exponent across both depth and batch dimensions in order to quantize tensors once and reuse them during backpropagation. We test our method using standard models such as AlexNet, VGG, and ResNet, on the CIFAR10, SVHN and ImageNet datasets. We show no loss of accuracy when quantizing AlexNet weights, activations and gradients to only 4 bits training ImageNet.
FeTrIL: Feature Translation for Exemplar-Free Class-Incremental Learning
Authors: Grégoire Petit; Adrian Popescu; Hugo Schindler; David Picard; Bertrand Delezoide
Venue: WACV 2023 (main)

Abstract: Exemplar-free class-incremental learning is very challenging due to the negative effect of catastrophic forgetting. A balance between stability and plasticity of the incremental process is needed in order to obtain good accuracy for past as well as new classes. Existing exemplar-free class-incremental methods focus either on successive fine tuning of the model, thus favoring plasticity, or on using a feature extractor fixed after the initial incremental state, thus favoring stability. We introduce a method which combines a fixed feature extractor and a pseudo-features generator to improve the stability-plasticity balance. The generator uses a simple yet effective geometric translation of new class features to create representations of past classes, made of pseudo-features. The translation of features only requires the storage of the centroid representations of past classes to produce their pseudo-features. Actual features of new classes and pseudo-features of past classes are fed into a linear classifier which is trained incrementally to discriminate between all classes. The incremental process is much faster with the proposed method compared to mainstream ones which update the entire deep model. Experiments are performed with three challenging datasets, and different incremental settings. A comparison with ten existing methods shows that our method outperforms the others in most cases. FeTrIL code is available at https://github.com/GregoirePetit/FeTrIL