GenAI4ELab/papercli-papers huggingface dataset:
Homeomorphism Alignment for Unsupervised Domain Adaptation
Authors: Lihua Zhou; Mao Ye; Xiatian Zhu; Siying Xiao; Xu-Qian Fan; Ferrante Neri
Venue: ICCV 2023 (main)

Abstract: Existing unsupervised domain adaptation (UDA) methods rely on aligning the features from the source and target domains explicitly or implicitly in a common space (i.e., the domain invariant space). Explicit distribution matching ignores the discriminability of learned features, while the implicit counterpart such as self-supervised learning suffers from pseudo-label noises. With distribution alignment, it is challenging to acquire a common space which maintains fully the discriminative structure of both domains. In this work, we propose a novel HomeomorphisM Alignment (HMA) approach characterized by aligning the source and target data in two separate spaces. Specifically, an invertible neural network based homeomorphism is constructed. Distribution matching is then used as a sewing up tool for connecting this homeomorphism mapping between the source and target feature spaces. Theoretically, we show that this mapping can preserve the data topological structure (e.g., the cluster/group structure). This property allows for more discriminative model adaptation by leveraging both the original and transformed features of source data in a supervised manner, and those of target domain in an unsupervised manner (e.g., prediction consistency). Extensive experiments demonstrate that our method can achieve the state-of-the-art results. Code is released at https://github.com/buerzlh/HMA.
MBPTrack: Improving 3D Point Cloud Tracking with Memory Networks and Box Priors
Authors: Tian-Xing Xu; Yuan-Chen Guo; Yu-Kun Lai; Song-Hai Zhang
Venue: ICCV 2023 (main)

Abstract: 3D single object tracking has been a crucial problem for decades with numerous applications such as autonomous driving. Despite its wide-ranging use, this task remains challenging due to the significant appearance variation caused by occlusion and size differences among tracked targets. To address these issues, we present MBPTrack, which adopts a Memory mechanism to utilize past information and formulates localization in a coarse-to-fine scheme using Box Priors given in the first frame. Specifically, past frames with targetness masks serve as an external memory, and a transformer-based module propagates tracked target cues from the memory to the current frame. To precisely localize objects of all sizes, MBPTrack first predicts the target center via Hough voting. By leveraging box priors given in the first frame, we adaptively sample reference points around the target center that roughly cover the target of different sizes. Then, we obtain dense feature maps by aggregating point features into the reference points, where localization can be performed more effectively. Extensive experiments demonstrate that MBPTrack achieves state-of-the-art performance on KITTI, nuScenes and Waymo Open Dataset, while running at 50 FPS on a single RTX3090 GPU.
Novel-View Synthesis and Pose Estimation for Hand-Object Interaction from Sparse Views
Authors: Wentian Qu; Zhaopeng Cui; Yinda Zhang; Chenyu Meng; Cuixia Ma; Xiaoming Deng; Hongan Wang
Venue: ICCV 2023 (main)

Abstract: Hand-object interaction understanding and the barely addressed novel view synthesis are highly desired in the immersive communication, whereas it is challenging due to the high deformation of hand and heavy occlusions between hand and object. In this paper, we propose a neural rendering and pose estimation system for hand-object interaction from sparse views, which can also enable 3D hand-object interaction editing. We share the inspiration from recent scene understanding work that shows a scene specific model built beforehand can significantly improve and unblock vision tasks especially when inputs are sparse, and extend it to the dynamic hand-object interaction scenario and propose to solve the problem in two stages. We first learn the shape and appearance prior knowledge of hands and objects separately with the neural representation at the offline stage. During the online stage, we design a rendering-based joint model fitting framework to understand the dynamic hand-object interaction with the pre-built hand and object models as well as interaction priors, which thereby overcomes penetration and separation issues between hand and object and also enables novel view synthesis. In order to get stable contact during the hand-object interaction process in a sequence, we propose a stable contact loss to make the contact region to be consistent. Experiments demonstrate that our method outperforms the state-of-the-art methods. Code and dataset are available in project webpage https://iscas3dv.github.io/HO-NeRF.
EmoSet: A Large-scale Visual Emotion Dataset with Rich Attributes
Authors: Jingyuan Yang; Qirui Huang; Tingting Ding; Dani Lischinski; Danny Cohen-Or; Hui Huang
Venue: ICCV 2023 (main)

Abstract: Visual Emotion Analysis (VEA) aims at predicting people's emotional responses to visual stimuli. This is a promising, yet challenging, task in affective computing, which has drawn increasing attention in recent years. Most of the existing work in this area focuses on feature design, while little attention has been paid to dataset construction. In this work, we introduce EmoSet, the first large-scale visual emotion dataset annotated with rich attributes, which is superior to existing datasets in four aspects: scale, annotation richness, diversity, and data balance. EmoSet comprises 3.3 million images in total, with 118,102 of these images carefully labeled by human annotators, making it five times larger than the largest existing dataset. EmoSet includes images from social networks, as well as artistic images, and it is well balanced between different emotion categories. Motivated by psychological studies, in addition to emotion category, each image is also annotated with a set of describable emotion attributes: brightness, colorfulness, scene type, object class, facial expression, and human action, which can help understand visual emotions in a precise and interpretable way. The relevance of these emotion attributes is validated by analyzing the correlations between them and visual emotion, as well as by designing an attribute module to help visual emotion recognition. We believe EmoSet will bring some key insights and encourage further research in visual emotion analysis and understanding. Project page: https://vcc.tech/EmoSet.
Distilling from Similar Tasks for Transfer Learning on a Budget
Authors: Kenneth Borup; Cheng Perng Phoo; Bharath Hariharan
Venue: ICCV 2023 (main)

Abstract: We address the challenge of getting efficient yet accurate recognition systems with limited labels. While recognition models improve with model size and amount of data, many specialized applications of computer vision have severe resource constraints both during training and inference. Transfer learning is an effective solution for training with few labels, however often at the expense of a computationally costly fine-tuning of large base models. We propose to mitigate this unpleasant trade-off between compute and accuracy via semi-supervised cross-domain distillation from a set of diverse source models. Initially, we show how to use task similarity metrics to select a single suitable source model to distill from, and that a good selection process is imperative for good downstream performance of a target model. We dub this approach DistillNearest. Though effective, DistillNearest assumes a single source model matches the target task, which is not always the case. To alleviate this, we propose a weighted multi-source distillation method to distill multiple source models trained on different domains weighted by their relevance for the target task into a single efficient model (named DistillWeighted). Our methods need no access to source data and merely need features and pseudo-labels of the source models. When the goal is accurate recognition under computational constraints, both DistillNearest and DistillWeighted approaches outperform both transfer learning from strong ImageNet initializations as well as state-of-the-art semi-supervised techniques such as FixMatch. Averaged over 8 diverse target tasks our multi-source method outperforms the baselines by 5.6%-points and 4.5%-points, respectively.
Self-Supervised Burst Super-Resolution
Authors: Goutam Bhat; Michaël Gharbi; Jiawen Chen; Luc Van Gool; Zhihao Xia
Venue: ICCV 2023 (main)

Abstract: We introduce a self-supervised training strategy for burst super-resolution that only uses noisy low-resolution bursts during training. Our approach eliminates the need to carefully tune synthetic data simulation pipelines, which often do not match real-world image statistics. Compared to weakly-paired training strategies, which require noisy smartphone burst photos of static scenes, paired with a clean reference obtained from a tripod-mounted DSLR camera, our approach is more scalable, and avoids the color mismatch between the smartphone and DSLR. To achieve this, we propose a new self-supervised objective that uses a forward imaging model to recover a high-resolution image from aliased high frequencies in the burst. Our approach does not require any manual tuning of the forward model's parameters; we learn them from data. Furthermore, we show our training strategy is robust to dynamic scene motion in the burst, which enables training burst super-resolution models using in-the-wild data. Extensive experiments on real and synthetic data show that, despite only using noisy bursts during training, models trained with our self-supervised strategy match, and sometimes surpass, the quality of fully-supervised baselines trained with synthetic data or weakly-paired ground-truth. Finally, we show our training strategy is general using four different burst super-resolution architectures.
Class-relation Knowledge Distillation for Novel Class Discovery
Authors: Peiyan Gu; Chuyu Zhang; Ruijie Xu; Xuming He
Venue: ICCV 2023 (main)

Abstract: We tackle the problem of novel class discovery, which aims to learn novel classes without supervision based on labeled data from known classes. A key challenge lies in transferring the knowledge in the known-class data to the learning of novel classes. Previous methods mainly focus on building a shared representation space for knowledge transfer and often ignore modeling class relations. To address this, we introduce a class relation representation for the novel classes based on the predicted class distribution of a model trained on known classes. Empirically, we find that such class relation becomes less informative during typical discovery training. To prevent such information loss, we propose a novel knowledge distillation framework, which utilizes our class-relation representation to regularize the learning of novel classes. In addition, to enable a flexible knowledge distillation scheme for each data point in novel classes, we develop a learnable weighting function for the regularization, which adaptively promotes knowledge transfer based on the semantic similarity between the novel and known classes. To validate the effectiveness and generalization of our method, we conduct extensive experiments on multiple benchmarks, including CIFAR100, Stanford Cars, CUB, and FGVC-Aircraft datasets. Our results demonstrate that the proposed method outperforms the previous state-of-the-art methods by a significant margin on almost all benchmarks.
PARTNER: Level up the Polar Representation for LiDAR 3D Object Detection
Authors: Ming Nie; Yujing Xue; Chunwei Wang; Chaoqiang Ye; Hang Xu; Xinge Zhu; Qingqiu Huang; Michael Bi Mi; Xinchao Wang; Li Zhang
Venue: ICCV 2023 (main)

Abstract: Recently, polar-based representation has shown promising properties in perceptual tasks. In addition to Cartesian-based approaches, which separate point clouds unevenly, representing point clouds as polar grids has been recognized as an alternative due to (1) its advantage in robust performance under different resolutions and (2) its superiority in streaming-based approaches. However, state-of-the-art polar-based detection methods inevitably suffer from the feature distortion problem because of the non-uniform division of polar representation, resulting in a non-negligible performance gap compared to Cartesian-based approaches. To tackle this issue, we present PARTNER, a novel 3D object detector in the polar coordinate. PARTNER alleviates the dilemma of feature distortion with global representation re-alignment and facilitates the regression by introducing instance-level geometric information into the detection head. Extensive experiments show overwhelming advantages in streaming-based detection and different resolutions. Furthermore, our method outperforms the previous polar-based works with remarkable margins of 3.68% and 9.15% on Waymo and ONCE validation set, thus achieving competitive results over the state-of-the-art methods.
Data-Free Class-Incremental Hand Gesture Recognition
Authors: Shubhra Aich; Jesus Ruiz-Santaquiteria; Zhenyu Lu; Prachi Garg; K J Joseph; Alvaro Fernandez Garcia; Vineeth N Balasubramanian; Kenrick Kin; Chengde Wan; Necati Cihan Camgoz; Shugao Ma; Fernando De la Torre
Venue: ICCV 2023 (main)

Abstract: This paper investigates data-free class-incremental learning (DFCIL) 
 for hand gesture recognition from 3D skeleton sequences. 
 In this class-incremental learning (CIL) setting, while incrementally 
 registering the new classes, we do not have access to the training 
 samples (i.e. data-free) of the already known classes due to privacy. 
 Existing DFCIL methods primarily focus on various forms of 
 knowledge distillation for model inversion to mitigate 
 catastrophic forgetting. Unlike SOTA methods, 
 we delve deeper into the choice of the best samples for inversion. 
 Inspired by the well-grounded theory of max-margin classification, 
 we find that the best samples tend to lie close to the approximate 
 decision boundary within a reasonable margin. To this end, 
 we propose BOAT-MI -- a simple and effective boundary-aware prototypical 
 sampling mechanism for model inversion for DFCIL. 
 Our sampling scheme outperforms SOTA methods significantly 
 on two 3D skeleton gesture datasets, the publicly available 
 SHREC 2017, and EgoGesture3D -- which we extract from a publicly 
 available RGBD dataset. Both our codebase and the EgoGesture3D 
 skeleton dataset are publicly available: https://github.com/humansensinglab/dfcil-hgr
Corrupting Neuron Explanations of Deep Visual Features
Authors: Divyansh Srivastava; Tuomas Oikarinen; Tsui-Wei Weng
Venue: ICCV 2023 (main)

Abstract: The inability of DNNs to explain their black-box behavior has led to a recent surge of explainability methods. However, there are growing concerns that these explainability methods are not robust and trustworthy. In this work, we perform the first robustness analysis of Neuron Explanation Methods under a unified pipeline and show that these explanations can be significantly corrupted by random noises and well-designed perturbations added to their probing data. We find that even adding small random noise with a standard deviation of 0.02 can already change the assigned concepts of up to 28% neurons in the deeper layers. Furthermore, we devise a novel corruption algorithm and show that our algorithm can manipulate the explanation of more than 80% neurons by poisoning less than 10% of probing data. This raises the concern of trusting Neuron Explanation Methods in real-life safety and fairness critical applications.
PNI : Industrial Anomaly Detection using Position and Neighborhood Information
Authors: Jaehyeok Bae; Jae-Han Lee; Seyun Kim
Venue: ICCV 2023 (main)

Abstract: Because anomalous samples cannot be used for training,
 many anomaly detection and localization methods use
 pre-trained networks and non-parametric modeling to estimate
 encoded feature distribution. However, these methods
 neglect the impact of position and neighborhood information
 on the distribution of normal features. To overcome
 this, we propose a new algorithm, PNI, which estimates
 the normal distribution using conditional probability given
 neighborhood features, modeled with a multi-layer perceptron
 network. Moreover, position information is utilized by
 creating a histogram of representative features at each position.
 Instead of simply resizing the anomaly map, the proposed
 method employs an additional refine network trained
 on synthetic anomaly images to better interpolate and account
 for the shape and edge of the input image. We conducted
 experiments on the MVTec AD benchmark dataset
 and achieved state-of-the-art performance, with 99.56%
 and 98.98% AUROC scores in anomaly detection and localization,
 respectively. Code is available at https://github.com/wogur110/PNI_Anomaly_Detection.
PC-Adapter: Topology-Aware Adapter for Efficient Domain Adaption on Point Clouds with Rectified Pseudo-label
Authors: Joonhyung Park; Hyunjin Seo; Eunho Yang
Venue: ICCV 2023 (main)

Abstract: Understanding point clouds captured from the real-world is challenging due to shifts in data distribution caused by varying object scales, sensor angles, and self-occlusion. Prior works have addressed this issue by combining recent learning principles such as self-supervised learning, self-training and adversarial training, which leads to to significant computational overhead. Toward succinct yet powerful domain adaptation for point clouds, we revisit the unique challenges of point cloud data under domain shift scenarios and discover the importance of the global geometry of source data and trends of target pseudo-labels biased to the source label distribution. Motivated by our observations, we propose an adapter-guided domain adaptation method, PC-Adapter, that preserves the global shape information of the source domain using an attention-based adapter, while learning the local characteristics of the target domain via another adapter equipped with graph convolution. Additionally, we propose a novel pseudo-labeling strategy resilient to the classifier bias by adjusting confidence scores using their class-wise confidence distributions to consider relative confidences. Our method demonstrates superiority over baselines on various domain shift settings in benchmark datasets - PointDA, GraspNetPC, and PointSegDA.
Cyclic Test-Time Adaptation on Monocular Video for 3D Human Mesh Reconstruction
Authors: Hyeongjin Nam; Daniel Sungho Jung; Yeonguk Oh; Kyoung Mu Lee
Venue: ICCV 2023 (main)

Abstract: Despite recent advances in 3D human mesh reconstruction, domain gap between training and test data is still a major challenge. Several prior works tackle the domain gap problem via test-time adaptation that fine-tunes a network relying on 2D evidence (e.g., 2D human keypoints) from test images. However, the high reliance on 2D evidence during adaptation causes two major issues. First, 2D evidence induces depth ambiguity, preventing the learning of accurate 3D human geometry. Second, 2D evidence is noisy or partially non-existent during test time, and such imperfect 2D evidence leads to erroneous adaptation. To overcome the above issues, we introduce CycleAdapt, which cyclically adapts two networks: a human mesh reconstruction network (HMRNet) and a human motion denoising network (MDNet), given a test video. In our framework, to alleviate high reliance on 2D evidence, we fully supervise HMRNet with generated 3D supervision targets by MDNet. Our cyclic adaptation scheme progressively elaborates the 3D supervision targets, which compensate for imperfect 2D evidence. As a result, our CycleAdapt achieves state-of-the-art performance compared to previous test-time adaptation methods. The codes are available in here: https://github.com/hygenie1228/CycleAdapt_RELEASE.
2D3D-MATR: 2D-3D Matching Transformer for Detection-Free Registration Between Images and Point Clouds
Authors: Minhao Li; Zheng Qin; Zhirui Gao; Renjiao Yi; Chenyang Zhu; Yulan Guo; Kai Xu
Venue: ICCV 2023 (main)

Abstract: The commonly adopted detect-then-match approach to registration finds difficulties in the cross-modality cases due to the incompatible keypoint detection and inconsistent feature description. We propose, 2D3D-MATR, a detection-free method for accurate and robust registration between images and point clouds. Our method adopts a coarse-to-fine pipeline where it first computes coarse correspondences between downsampled patches of the input image and the point cloud and then extends them to form dense correspondences between pixels and points within the patch region. The coarse-level patch matching is based on transformer which jointly learns global contextual constraints with self-attention and cross-modality correlations with cross-attention. To resolve the scale ambiguity in patch matching, we construct a multi-scale pyramid for each image patch and learn to find for each point patch the best matching image patch at a proper resolution level. Extensive experiments on two public benchmarks demonstrate that 2D3D-MATR outperforms the previous state-of-the-art P2-Net by around 20 percentage points on inlier ratio and over 10 points on registration recall. Our code and models will be publicly released.
Mixed Neural Voxels for Fast Multi-view Video Synthesis
Authors: Feng Wang; Sinan Tan; Xinghang Li; Zeyue Tian; Yafei Song; Huaping Liu
Venue: ICCV 2023 (main)

Abstract: Synthesizing high-fidelity videos from real-world multiview input is challenging due to the complexities of real-world environments and high-dynamic movements. Previous works based on neural radiance fields have demonstrated high-quality reconstructions of dynamic scenes. However, training such models on real-world scenes is time-consuming, usually taking days or weeks. In this paper, we present a novel method named MixVoxels to efficiently represent dynamic scenes which leads to fast training and rendering speed. The proposed MixVoxels represents the 4D dynamic scenes as a mixture of static and dynamic voxels and processes them with different networks. In this way, the computation of the required modalities for static voxels can be processed by a lightweight model, which essentially reduces
 the amount of computation as many daily dynamic scenes are dominated by static backgrounds. To distinguish the two kinds of voxels, we propose a novel variation field to estimate the temporal variance of each voxel. For the dynamic representations, we design an inner-product time query method to efficiently query multiple time steps, which is essential to recover the high-dynamic movements. As
 a result, with 15 minutes of training for dynamic scenes with inputs of 300-frame videos, MixVoxels achieves better PSNR than previous methods. For rendering, MixVoxels can render a novel view video with 1K resolution at 37 fps. Codes and trained models are available at https://github.com/fengres/mixvoxels.