GenAI4ELab/papercli-papers huggingface dataset:
DiverGen: Improving Instance Segmentation by Learning Wider Data Distribution with More Diverse Generative Data
Authors: Chengxiang Fan; Muzhi Zhu; Hao Chen; Yang Liu; Weijia Wu; Huaqi Zhang; Chunhua Shen
Venue: CVPR 2024 (main)

Abstract: Instance segmentation is data-hungry and as model capacity increases data scale becomes crucial for improving the accuracy. Most instance segmentation datasets today require costly manual annotation limiting their data scale. Models trained on such data are prone to overfitting on the training set especially for those rare categories. While recent works have delved into exploiting generative models to create synthetic datasets for data augmentation these approaches do not efficiently harness the full potential of generative models. To address these issues we introduce a more efficient strategy to construct generative datasets for data augmentation termed DiverGen. Firstly we provide an explanation of the role of generative data from the perspective of distribution discrepancy. We investigate the impact of different data on the distribution learned by the model. We argue that generative data can expand the data distribution that the model can learn thus mitigating overfitting. Additionally we find that the diversity of generative data is crucial for improving model performance and enhance it through various strategies including category diversity prompt diversity and generative model diversity. With these strategies we can scale the data to millions while maintaining the trend of model performance improvement. On the LVIS dataset DiverGen significantly outperforms the strong model X-Paste achieving +1.1 box AP and +1.1 mask AP across all categories and +1.9 box AP and +2.5 mask AP for rare categories. Our codes are available at https://github.com/aim-uofa/DiverGen.
Neural Refinement for Absolute Pose Regression with Feature Synthesis
Authors: Shuai Chen; Yash Bhalgat; Xinghui Li; Jia-Wang Bian; Kejie Li; Zirui Wang; Victor Adrian Prisacariu
Venue: CVPR 2024 (main)

Abstract: Absolute Pose Regression (APR) methods use deep neural networks to directly regress camera poses from RGB images. However the predominant APR architectures only rely on 2D operations during inference resulting in limited accuracy of pose estimation due to the lack of 3D geometry constraints or priors. In this work we propose a test-time refinement pipeline that leverages implicit geometric constraints using a robust feature field to enhance the ability of APR methods to use 3D information during inference. We also introduce a novel Neural Feature Synthesizer (NeFeS) model which encodes 3D geometric features during training and directly renders dense novel view features at test time to refine APR methods. To enhance the robustness of our model we introduce a feature fusion module and a progressive training strategy. Our proposed method achieves state-of-the-art single-image APR accuracy on indoor and outdoor datasets. Code will be released at https://github.com/ActiveVisionLab/NeFeS.
Learning Disentangled Identifiers for Action-Customized Text-to-Image Generation
Authors: Siteng Huang; Biao Gong; Yutong Feng; Xi Chen; Yuqian Fu; Yu Liu; Donglin Wang
Venue: CVPR 2024 (main)

Abstract: This study focuses on a novel task in text-to-image (T2I) generation namely action customization. The objective of this task is to learn the co-existing action from limited data and generalize it to unseen humans or even animals. Experimental results show that existing subject-driven customization methods fail to learn the representative characteristics of actions and struggle in decoupling actions from context features including appearance. To overcome the preference for low-level features and the entanglement of high-level features we propose an inversion-based method Action-Disentangled Identifier (ADI) to learn action-specific identifiers from the exemplar images. ADI first expands the semantic conditioning space by introducing layer-wise identifier tokens thereby increasing the representational richness while distributing the inversion across different features. Then to block the inversion of action-agnostic features ADI extracts the gradient invariance from the constructed sample triples and masks the updates of irrelevant channels. To comprehensively evaluate the task we present an ActionBench that includes a variety of actions each accompanied by meticulously selected samples. Both quantitative and qualitative results show that our ADI outperforms existing baselines in action-customized T2I generation. Our project page is at https://adi-t2i.github.io/ADI.
Automatic Controllable Colorization via Imagination
Authors: Xiaoyan Cong; Yue Wu; Qifeng Chen; Chenyang Lei
Venue: CVPR 2024 (main)

Abstract: We propose a framework for automatic colorization that allows for iterative editing and modifications. The core of our framework lies in an imagination module: by understanding the content within a grayscale image we utilize a pre-trained image generation model to generate multiple images that contain the same content. These images serve as references for coloring mimicking the process of human experts. As the synthesized images can be imperfect or different from the original grayscale image we propose a Reference Refinement Module to select the optimal reference composition. Unlike most previous end-to-end automatic colorization algorithms our framework allows for iterative and localized modifications of the colorization results because we explicitly model the coloring samples. Extensive experiments demonstrate the superiority of our framework over existing automatic colorization algorithms in editability and flexibility. Project page: https://xy-cong.github.io/imagine-colorization/.
Point Transformer V3: Simpler Faster Stronger
Authors: Xiaoyang Wu; Li Jiang; Peng-Shuai Wang; Zhijian Liu; Xihui Liu; Yu Qiao; Wanli Ouyang; Tong He; Hengshuang Zhao
Venue: CVPR 2024 (main)

Abstract: This paper is not motivated to seek innovation within the attention mechanism. Instead it focuses on overcoming the existing trade-offs between accuracy and efficiency within the context of point cloud processing leveraging the power of scale. Drawing inspiration from recent advances in 3D large-scale representation learning we recognize that model performance is more influenced by scale than by intricate design. Therefore we present Point Transformer V3 (PTv3) which prioritizes simplicity and efficiency over the accuracy of certain mechanisms that are minor to the overall performance after scaling such as replacing the precise neighbor search by KNN with an efficient serialized neighbor mapping of point clouds organized with specific patterns. This principle enables significant scaling expanding the receptive field from 16 to 1024 points while remaining efficient (a 3x increase in processing speed and a 10x improvement in memory efficiency compared with its predecessor PTv2). PTv3 attains state-of-the-art results on over 20 downstream tasks that span both indoor and outdoor scenarios. Further enhanced with multi-dataset joint training PTv3 pushes these results to a higher level.
DiffCast: A Unified Framework via Residual Diffusion for Precipitation Nowcasting
Authors: Demin Yu; Xutao Li; Yunming Ye; Baoquan Zhang; Chuyao Luo; Kuai Dai; Rui Wang; Xunlai Chen
Venue: CVPR 2024 (main)

Abstract: Precipitation nowcasting is an important spatio-temporal prediction task to predict the radar echoes sequences based on current observations which can serve both meteorological science and smart city applications. Due to the chaotic evolution nature of the precipitation systems it is a very challenging problem. Previous studies address the problem either from the perspectives of deterministic modeling or probabilistic modeling. However their predictions suffer from the blurry high-value echoes fading away and position inaccurate issues. The root reason of these issues is that the chaotic evolutionary precipitation systems are not appropriately modeled. Inspired by the nature of the systems we propose to decompose and model them from the perspective of global deterministic motion and local stochastic variations with residual mechanism. A unified and flexible framework that can equip any type of spatio-temporal models is proposed based on residual diffusion which effectively tackles the shortcomings of previous methods. Extensive experimental results on four publicly available radar datasets demonstrate the effectiveness and superiority of the proposed framework compared to state-of-the-art techniques. Our code is publicly available at https://github.com/DeminYu98/DiffCast.
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives
Authors: Kristen Grauman; Andrew Westbury; Lorenzo Torresani; Kris Kitani; Jitendra Malik; Triantafyllos Afouras; Kumar Ashutosh; Vijay Baiyya; Siddhant Bansal; Bikram Boote; Eugene Byrne; Zach Chavis; Joya Chen; Feng Cheng; Fu-Jen Chu; Sean Crane; Avijit Dasgupta; Jing Dong; Maria Escobar; Cristhian Forigua; Abrham Gebreselasie; Sanjay Haresh; Jing Huang; Md Mohaiminul Islam; Suyog Jain; Rawal Khirodkar; Devansh Kukreja; Kevin J Liang; Jia-Wei Liu; Sagnik Majumder; Yongsen Mao; Miguel Martin; Effrosyni Mavroudi; Tushar Nagarajan; Francesco Ragusa; Santhosh Kumar Ramakrishnan; Luigi Seminara; Arjun Somayazulu; Yale Song; Shan Su; Zihui Xue; Edward Zhang; Jinxu Zhang; Angela Castillo; Changan Chen; Xinzhu Fu; Ryosuke Furuta; Cristina Gonzalez; Prince Gupta; Jiabo Hu; Yifei Huang; Yiming Huang; Weslie Khoo; Anush Kumar; Robert Kuo; Sach Lakhavani; Miao Liu; Mi Luo; Zhengyi Luo; Brighid Meredith; Austin Miller; Oluwatumininu Oguntola; Xiaqing Pan; Penny Peng; Shraman Pramanick; Merey Ramazanova; Fiona Ryan; Wei Shan; Kiran Somasundaram; Chenan Song; Audrey Southerland; Masatoshi Tateno; Huiyu Wang; Yuchen Wang; Takuma Yagi; Mingfei Yan; Xitong Yang; Zecheng Yu; Shengxin Cindy Zha; Chen Zhao; Ziwei Zhao; Zhifan Zhu; Jeff Zhuo; Pablo Arbelaez; Gedas Bertasius; Dima Damen; Jakob Engel; Giovanni Maria Farinella; Antonino Furnari; Bernard Ghanem; Judy Hoffman; C.V. Jawahar; Richard Newcombe; Hyun Soo Park; James M. Rehg; Yoichi Sato; Manolis Savva; Jianbo Shi; Mike Zheng Shou; Michael Wray
Venue: CVPR 2024 (main)

Abstract: We present Ego-Exo4D a diverse large-scale multimodal multiview video dataset and benchmark challenge. Ego-Exo4D centers around simultaneously-captured egocentric and exocentric video of skilled human activities (e.g. sports music dance bike repair). 740 participants from 13 cities worldwide performed these activities in 123 different natural scene contexts yielding long-form captures from 1 to 42 minutes each and 1286 hours of video combined. The multimodal nature of the dataset is unprecedented: the video is accompanied by multichannel audio eye gaze 3D point clouds camera poses IMU and multiple paired language descriptions---including a novel "expert commentary" done by coaches and teachers and tailored to the skilled-activity domain. To push the frontier of first-person video understanding of skilled human activity we also present a suite of benchmark tasks and their annotations including fine-grained activity understanding proficiency estimation cross-view translation and 3D hand/body pose. All resources are open sourced to fuel new research in the community.
Point Cloud Pre-training with Diffusion Models
Authors: Xiao Zheng; Xiaoshui Huang; Guofeng Mei; Yuenan Hou; Zhaoyang Lyu; Bo Dai; Wanli Ouyang; Yongshun Gong
Venue: CVPR 2024 (main)

Abstract: Pre-training a model and then fine-tuning it on downstream tasks has demonstrated significant success in the 2D image and NLP domains. However due to the unordered and non-uniform density characteristics of point clouds it is non-trivial to explore the prior knowledge of point clouds and pre-train a point cloud backbone. In this paper we propose a novel pre-training method called Point cloud Diffusion pre-training PointDif. We consider the point cloud pre-training task as a conditional point-to-point generation problem and introduce a conditional point generator. This generator aggregates the features extracted by the backbone and employs them as the condition to guide the point-to-point recovery from the noisy point cloud thereby assisting the backbone in capturing both local and global geometric priors as well as the global point density distribution of the object. We also present a recurrent uniform sampling optimization strategy which enables the model to uniformly recover from various noise levels and learn from balanced supervision. Our PointDif achieves substantial improvement across various real-world datasets for diverse downstream tasks such as classification segmentation and detection. Specifically PointDif attains 70.0% mIoU on S3DIS Area 5 for the segmentation task and achieves an average improvement of 2.4% on ScanObjectNN for the classification task compared to TAP. Furthermore our pre-training framework can be flexibly applied to diverse point cloud backbones and bring considerable gains. Code is available at https://github.com/zhengxiaozx/PointDif
Mask4Align: Aligned Entity Prompting with Color Masks for Multi-Entity Localization Problems
Authors: Haoquan Zhang; Ronggang Huang; Yi Xie; Huaidong Zhang
Venue: CVPR 2024 (main)

Abstract: In Visual Question Answering (VQA) recognizing and localizing entities pose significant challenges. Pretrained vision-and-language models have addressed this problem by providing a text description as the answer. However in visual scenes with multiple entities textual descriptions struggle to distinguish the entities from the same category effectively. Consequently the VQA dataset is limited by the limitations of text description and cannot adequately cover scenarios involving multiple entities. To address this challenge we introduce a Mask for Align (Mask4Align) method which can determine the entity's position in the given image that best matches the user-input question. This method incorporates colored masks into the image enabling the VQA model to handle discrimination and localization challenges associated with multiple entities. To process an arbitrary number of similar entities Mask4Align is designed hierarchically to discern subtle differences achieving precise localization. Since Mask4Align directly utilizes pre-trained models it does not introduce additional training overhead. Extensive experiments conducted on both the gaze target prediction task dataset and our proposed multi-entity localization dataset showcase the superiority of Mask4Align.
RCL: Reliable Continual Learning for Unified Failure Detection
Authors: Fei Zhu; Zhen Cheng; Xu-Yao Zhang; Cheng-Lin Liu; Zhaoxiang Zhang
Venue: CVPR 2024 (main)

Abstract: Deep neural networks are known to be overconfident for what they don't know in the wild which is undesirable for decision-making in high-stakes applications. Despite quantities of existing works most of them focus on detecting out-of-distribution (OOD) samples from unseen classes while ignoring large parts of relevant failure sources like misclassified samples from known classes. In particular recent studies reveal that prevalent OOD detection methods are actually harmful for misclassification detection (MisD) indicating that there seems to be a tradeoff between those two tasks. In this paper we study the critical yet under-explored problem of unified failure detection which aims to detect both misclassified and OOD examples. Concretely we identify the failure of simply integrating learning objectives of misclassification and OOD detection and show the potential of sequence learning. Inspired by this we propose a reliable continual learning paradigm whose spirit is to equip the model with MisD ability first and then improve the OOD detection ability without degrading the already adequate MisD performance. Extensive experiments demonstrate that our method achieves strong unified failure detection performance. The code is available at https://github.com/Impression2805/RCL.
Referring Image Editing: Object-level Image Editing via Referring Expressions
Authors: Chang Liu; Xiangtai Li; Henghui Ding
Venue: CVPR 2024 (main)

Abstract: Significant advancements have been made in image editing with the recent advance of the Diffusion model. However most of the current methods primarily focus on global or subject-level modifications and often face limitations when it comes to editing specific objects when there are other objects coexisting in the scene given solely textual prompts. In response to this challenge we introduce an object-level generative task called Referring Image Editing (RIE) which enables the identification and editing of specific source objects in an image using text prompts. To tackle this task effectively we propose a tailored framework called ReferDiffusion. It aims to disentangle input prompts into multiple embeddings and employs a mixed-supervised multi-stage training strategy. To facilitate further research in this domain we introduce the RefCOCO-Edit dataset comprising images editing prompts source object segmentation masks and reference edited images for training and evaluation. Our extensive experiments demonstrate the effectiveness of our approach in identifying and editing target objects while conventional general image editing and region-based image editing methods have difficulties in this challenging task.
CAMixerSR: Only Details Need More "Attention"
Authors: Yan Wang; Yi Liu; Shijie Zhao; Junlin Li; Li Zhang
Venue: CVPR 2024 (main)

Abstract: To satisfy the rapidly increasing demands on the large image (2K-8K) super-resolution (SR) prevailing methods follow two independent tracks: 1) accelerate existing networks by content-aware routing and 2) design better super-resolution networks via token mixer refining. Despite directness they encounter unavoidable defects (e.g. inflexible route or non-discriminative processing) limiting further improvements of quality-complexity trade-off. To erase the drawbacks we integrate these schemes by proposing a content-aware mixer (CAMixer) which assigns convolution for simple contexts and additional deformable window-attention for sparse textures. Specifically the CAMixer uses a learnable predictor to generate multiple bootstraps including offsets for windows warping a mask for classifying windows and convolutional attentions for endowing convolution with the dynamic property which modulates attention to include more useful textures self-adaptively and improves the representation capability of convolution. We further introduce a global classification loss to improve the accuracy of predictors. By simply stacking CAMixers we obtain CAMixerSR which achieves superior performance on large-image SR lightweight SR and omnidirectional-image SR.
Towards Backward-Compatible Continual Learning of Image Compression
Authors: Zhihao Duan; Ming Lu; Justin Yang; Jiangpeng He; Zhan Ma; Fengqing Zhu
Venue: CVPR 2024 (main)

Abstract: This paper explores the possibility of extending the capability of pre-trained neural image compressors (e.g. adapting to new data or target bitrates) without breaking backward compatibility the ability to decode bitstreams encoded by the original model. We refer to this problem as continual learning of image compression. Our initial findings show that baseline solutions such as end-to-end fine-tuning do not preserve the desired backward compatibility. To tackle this we propose a knowledge replay training strategy that effectively addresses this issue. We also design a new model architecture that enables more effective continual learning than existing baselines. Experiments are conducted for two scenarios: data-incremental learning and rate-incremental learning. The main conclusion of this paper is that neural image compressors can be fine-tuned to achieve better performance (compared to their pre-trained version) on new data and rates without compromising backward compatibility. The code is publicly available online.
Latent Modulated Function for Computational Optimal Continuous Image Representation
Authors: Zongyao He; Zhi Jin
Venue: CVPR 2024 (main)

Abstract: The recent work Local Implicit Image Function (LIIF) and subsequent Implicit Neural Representation (INR) based works have achieved remarkable success in Arbitrary-Scale Super-Resolution (ASSR) by using MLP to decode Low-Resolution (LR) features. However these continuous image representations typically implement decoding in High-Resolution (HR) High-Dimensional (HD) space leading to a quadratic increase in computational cost and seriously hindering the practical applications of ASSR. To tackle this problem we propose a novel Latent Modulated Function (LMF) which decouples the HR-HD decoding process into shared latent decoding in LR-HD space and independent rendering in HR Low-Dimensional (LD) space thereby realizing the first computational optimal paradigm of continuous image representation. Specifically LMF utilizes an HD MLP in latent space to generate latent modulations of each LR feature vector. This enables a modulated LD MLP in render space to quickly adapt to any input feature vector and perform rendering at arbitrary resolution. Furthermore we leverage the positive correlation between modulation intensity and input image complexity to design a Controllable Multi-Scale Rendering (CMSR) algorithm offering the flexibility to adjust the decoding efficiency based on the rendering precision. Extensive experiments demonstrate that converting existing INR-based ASSR methods to LMF can reduce the computational cost by up to 99.9% accelerate inference by up to 57x and save up to 76% of parameters while maintaining competitive performance. The code is available at https://github.com/HeZongyao/LMF.
Unsupervised Video Domain Adaptation with Masked Pre-Training and Collaborative Self-Training
Authors: Arun Reddy; William Paul; Corban Rivera; Ketul Shah; Celso M. de Melo; Rama Chellappa
Venue: CVPR 2024 (main)

Abstract: In this work we tackle the problem of unsupervised domain adaptation (UDA) for video action recognition. Our approach which we call UNITE uses an image teacher model to adapt a video student model to the target domain. UNITE first employs self-supervised pre-training to promote discriminative feature learning on target domain videos using a teacher-guided masked distillation objective. We then perform self-training on masked target data using the video student model and image teacher model together to generate improved pseudolabels for unlabeled target videos. Our self-training process successfully leverages the strengths of both models to achieve strong transfer performance across domains. We evaluate our approach on multiple video domain adaptation benchmarks and observe significant improvements upon previously reported results.
UniDepth: Universal Monocular Metric Depth Estimation
Authors: Luigi Piccinelli; Yung-Hsu Yang; Christos Sakaridis; Mattia Segu; Siyuan Li; Luc Van Gool; Fisher Yu
Venue: CVPR 2024 (main)

Abstract: Accurate monocular metric depth estimation (MMDE) is crucial to solving downstream tasks in 3D perception and modeling. However the remarkable accuracy of recent MMDE methods is confined to their training domains. These methods fail to generalize to unseen domains even in the presence of moderate domain gaps which hinders their practical applicability. We propose a new model UniDepth capable of reconstructing metric 3D scenes from solely single images across domains. Departing from the existing MMDE methods UniDepth directly predicts metric 3D points from the input image at inference time without any additional information striving for a universal and flexible MMDE solution. In particular UniDepth implements a self-promptable camera module predicting dense camera representation to condition depth features. Our model exploits a pseudo-spherical output representation which disentangles camera and depth representations. In addition we propose a geometric invariance loss that promotes the invariance of camera-prompted depth features. Thorough evaluations on ten datasets in a zero-shot regime consistently demonstrate the superior performance of UniDepth even when compared with methods directly trained on the testing domains. Code and models are available at: github.com/lpiccinelli-eth/unidepth