GenAI4ELab/papercli-papers huggingface dataset:
Inside-Out: Measuring Generalization in Vision Transformers Through Inner Workings
Authors: Yunxiang Peng; Mengmeng Ma; Ziyu Yao; Xi Peng
Venue: CVPR 2026 (main)

Abstract: Reliable generalization metrics are fundamental to the evaluation of machine learning models. Especially in high-stakes applications where labeled target data are scarce, evaluation of models' generalization performance under distribution shift is a pressing need. We focus on two practical scenarios: (1) Before deployment, how to select the best model for unlabeled target data? (2) After deployment, how to monitor model performance under distribution shift? The central need in both cases is a reliable and label-free proxy metric. Yet existing proxy metrics, such as model confidence or accuracy-on-the-line, are often unreliable as they only assess model output while ignoring the internal mechanisms that produce them. We address this limitation by introducing a new perspective: using the inner workings of a model, i.e., circuits, as a predictive metric of generalization performance. Leveraging circuit discovery, we extract the causal interactions between internal representations as a circuit, from which we derive two metrics tailored to the two practical scenarios. (1) Before deployment, we introduce Dependency Depth Bias, which measures different models' generalization capability on target data. (2) After deployment, we propose Circuit Shift Score, which predicts a model's generalization under different distribution shifts. Across various tasks, both metrics demonstrate significantly improved correlation with generalization performance, outperforming existing proxies by an average of 13.4% and 34.1%, respectively. Our code is available at https://github.com/deep-real/GenCircuit.
AniMimic: Imitating 3D Animation from Video Priors
Authors: Tianyi Xie; Yunuo Chen; Yaowei Guo; Yin Yang; Bolei Zhou; Demetri Terzopoulos; Ying Jiang; Chenfanfu Jiang
Venue: CVPR 2026 (main)

Abstract: Creating realistic 3D animation remains a time-consuming and expertise-dependent process, requiring manual rigging, keyframing, and fine-tuning of complex motions. Meanwhile, video diffusion models have recently demonstrated remarkable 2D motion imagination, generating dynamic and visually coherent motion from text or image prompts. However, their outputs lack explicit 3D structure and cannot be directly used for animation or simulation. We present AniMimic, a framework that animates static 3D meshes using motion priors learned from video diffusion models. Starting from an input mesh, AniMimic synthesizes a monocular animation video, automatically constructs a skeleton with skinning weights, and refines its joint parameters through differentiable rendering and video-based supervision. To further enhance realism, we integrate a differentiable simulation module that refines mesh deformation through physically grounded soft-tissue dynamics. Our method bridges the creativity of video diffusion and the structural control of 3D rigged animation, producing physically plausible, temporally coherent, and artist-editable motion sequences that integrate seamlessly into standard animation pipelines.
Building Robust Vision Encoders for Cross-Dataset Evaluation in Immunofluorescent Microscopy
Authors: Umar Marikkar; Syed Sameed Husain; Muhammad Awais; Sara Atito
Venue: CVPR 2026 (main)

Abstract: Immunofluorescence (IF) images reveal detailed information about structures and functions at the subcellular level. However, unlike RGB images, IF datasets pose challenges for deep learning models due to their inconsistencies in channel count and configuration, stemming from varying staining protocols across laboratories and studies. Although existing approaches build channel-adaptive models for training, they do not perform evaluations across IF datasets with unseen channel configurations. To address this, we first introduce a biologically informed view of cellular image channels by grouping them into either context or concept, where we treat the context channels as a reference for the concept channels in the image. We leverage this view to propose Channel Conditioned Cell Representations (C3R), a framework that learns representations that transfers well to both in-distribution (ID) and out-of-distribution (OOD) datasets which contain same and different channel configurations, respectively. C3R is a two-fold framework comprising a channel-adaptive encoder architecture and a masked knowledge distillation training strategy, both built around the context-concept principle. We find that C3R outperforms existing benchmarks on both ID and OOD tasks, while yielding state-of-the-art results on frozen encoder evaluation on the CHAMMI benchmark. Our method opens a new pathway for cross-dataset generalization between IF datasets, with no need for retraining on unseen channel configurations.
Gated KalmaNet: A Fading Memory Layer through Test-time Ridge Regression
Authors: Liangzu Peng; Aditya Chattopadhyay; Luca Zancato; Elvis Nunez; Wei Xia; Stefano Soatto
Venue: CVPR 2026 (main)

Abstract: As efficient alternatives to softmax Attention, linear state space models (SSMs) achieve constant memory and linear compute, but maintain only a lossy, fading summary of the past, often leading to inferior performance in recall oriented settings. We propose Gated KalmaNet (GKA), a layer that reduces this gap by accounting for the full past when predicting the next token, while maintaining SSM-style efficiency. GKA is inspired by the Kalman Filter and solves ridge regression problems online at test time, with constant memory and linear time in the sequence length. An insight is that standard Kalman filter equations are numerically unstable in low-precision environments (e.g., bfloat16) and difficult to parallelize on modern GPUs. We address both challenges via two innovations: (1) an adaptive regularization strategy with input-dependent gating that controls the condition number of the problem, ensuring numerical stability and balancing memory retention; (2) the use of Chebyshev Iteration instead of conventional iterative solvers, which we show to be more stable in low-precision settings. To improve scalability, we implement Chebyshev Iteration in a hardware-aware, chunk-wise manner, along with custom kernels for backpropagating through our adaptive regularization and gating mechanisms. On short-context tasks, GKA shows strong language understanding capabilities and outperforms existing SSMs (e.g., Mamba2, and Gated DeltaNet). On long-context tasks, GKA excels at real-world RAG and LongQA tasks up to 128k tokens with more than 10% relative improvement over baselines. Finally, we show GKA outperforms Mamba when extended for ImageNet classification.
The Missing Point in Vision Transformers for Universal Image Segmentation
Authors: Sajjad Shahabodini; Mobina Mansoori; Farnoush Bayatmakou; Jamshid Abouei; Konstantinos Plataniotis; Arash Mohammadi
Venue: CVPR 2026 (main)

Abstract: Image segmentation remains a challenging task in computer vision, demanding robust mask generation and precise classification. Recent mask-based approaches yield high-quality masks by capturing global context. However, accurately classifying these masks, especially in the presence of ambiguous boundaries and imbalanced class distributions, remains an open challenge. In this work, we introduce ViT-P, a novel two-stage segmentation framework that decouples mask generation from classification. The first stage employs a proposal generator to produce class-agnostic mask proposals, while the second stage utilizes a point-based classification model built on the Vision Transformer (ViT) to refine predictions. ViT-P serves as a pre-training-free adapter, allowing the integration of various pre-trained vision transformers without modifying their architecture, ensuring adaptability to dense prediction tasks. Furthermore, we demonstrate that coarse and bounding box annotations can effectively enhance classification without requiring additional training on fine annotation datasets, reducing annotation costs while maintaining strong performance. Extensive experiments across COCO, ADE20K, and Cityscapes datasets validate the effectiveness of ViT-P, achieving state-of-the-art results with 54.0 PQ on ADE20K panoptic segmentation, 87.4 mIoU on Cityscapes semantic segmentation, and 63.6 mIoU on ADE20K semantic segmentation.
AeroAgent: A Vision-Physics-Decision Framework for Aerodynamic Vehicle Design
Authors: Ye Liu; Shouyi Liu; Huiyu Yang; Jianghang Gu; Wenhao Fan; Zhongxin Yang; Ding Wang; Simeng Chen; Zirun Jiang; Yuanwei Bin; Shiyi Chen; Yuntian Chen
Venue: CVPR 2026 (main)

Abstract: Modern generative models can propose striking 3D vehicle shapes from text and images, but turning these sketches into aerodynamically efficient, regulation compliant designs still requires weeks of high-fidelity computational fluid dynamics (CFD) and manual iteration. As a result, fast 3D generation without trustworthy physics in the loop does little to reduce end-to-end design time. We study how an AI agent can close this loop under a strict CFD budget. We introduce AeroAgent, a vision-physics-decision framework built around a single 3D, editable surface representation for vehicle shapes. A vision module turns text and 2D references into diverse, standardized 3D candidates and supports image-level edits. A physics module, AeroFormer is a geometry-guided Transformer surrogate trained on a large-scale vehicle aerodynamics dataset of roughly 50k CFD simulations; three task-specific heads predict drag Cd, surface pressure, and velocity fields on shared 3D grids. A decision module encodes regulatory size limits and aesthetic constraints as feasibility tests, uses prototype priors and surrogate sensitivities to guide free-form deformation edits, and runs a budget-aware propose evaluate refine loop in which only the final top K shapes are confirmed by high-fidelity CFD. In extensive experiments across five common vehicle classes, running only five propose evaluate refine iterations per vehicle reduces drag by an average of 2-12% and cuts high-fidelity CFD calls by 50-80% compared to baseline workflows, while preserving or improving styling quality.
Reviving ConvNeXt for Efficient Convolutional Diffusion Models
Authors: Taesung Kwon; Lorenzo Bianchi; Lennart Wittke; Felix Watine; Fabio Carrara; Jong Chul Ye; Romann Weber; Vinicius Azevedo
Venue: CVPR 2026 (main)

Abstract: Recent diffusion models increasingly favor Transformer backbones, motivated by the remarkable scalability of fully attentional architectures. Yet the locality bias, parameter efficiency, and hardware friendliness--the attributes that established ConvNets as the efficient vision backbone--have seen limited exploration in modern generative modeling. Here we introduce the fully convolutional diffusion model (FCDM), a model having a backbone similar to ConvNeXt, but designed for conditional diffusion modeling. We find that using only 50% of the FLOPs of DiT-XL/2, FCDM-XL achieves competitive performance with 7x and 7.5x fewer training steps at 256x256 and 512x512 resolutions, respectively. Remarkably, FCDM-XL can be trained on a 4-GPU system, highlighting the exceptional training efficiency of our architecture. Our results demonstrate that modern convolutional designs provide a competitive and highly efficient alternative for scaling diffusion models, reviving ConvNeXt as a simple yet powerful building block for efficient generative modeling.
StreamAvatar: Streaming Diffusion Models for Real-Time Interactive Human Avatars
Authors: Zhiyao Sun; Ziqiao Peng; Yifeng Ma; Yi Chen; Zhengguang Zhou; Zixiang Zhou; Guozhen Zhang; Youliang Zhang; Yuan Zhou; Qinglin Lu; Yong-Jin Liu
Venue: CVPR 2026 (main)

Abstract: Real-time, streaming interactive avatars represent a critical yet challenging goal in digital human research. Although diffusion-based human avatar generation methods achieve remarkable success, their non-causal architecture and high computational costs make them unsuitable for streaming. Moreover, existing interactive approaches are typically restricted to the head-and-shoulder region, limiting their ability to produce gestures and body motions. To address these challenges, we propose a two-stage autoregressive adaptation and acceleration framework that applies autoregressive distillation and adversarial refinement to adapt a high-fidelity human video diffusion model for real-time, interactive streaming. To ensure long-term stability and consistency, we introduce three key components: a Reference Sink, a Reference-Anchored Positional Re-encoding (RAPR) strategy, and a Consistency-Aware Discriminator. Building on this framework, we develop a one-shot, interactive, human avatar model capable of generating both natural talking and listening behaviors with coherent gestures. Extensive experiments demonstrate that our method achieves state-of-the-art performance, surpassing existing approaches in generation quality, real-time efficiency, and interaction naturalness.
DetAny4D: Detect Anything 4D Temporally in a Streaming RGB Video
Authors: Jiawei Hou; Shenghao Zhang; Can Wang; Zheng Gu; Yonggen Ling; Taiping Zeng; Xiangyang Xue; Jingbo Zhang
Venue: CVPR 2026 (main)

Abstract: Reliable 4D object detection, which refers to 3D object detection in streaming video, is crucial for perceiving and understanding the real world. Existing open-set 4D object detection methods typically make predictions on a frame-by-frame basis without modeling temporal consistency, or rely on complex multi-stage pipelines that are prone to error propagation across cascaded stages. Progress in this area has been hindered by the lack of large-scale datasets that capture continuous reliable 3D bounding box (b-box) annotations. To overcome these challenges, we first introduce DA4D, a large-scale 4D detection dataset containing over 280k sequences with high-quality b-box annotations collected under diverse conditions. Building on DA4D, we propose DetAny4D, an open-set end-to-end framework that predicts 3D b-boxes directly from sequential inputs. DetAny4D fuses multi-modal features from pre-trained foundational models and designs a geometry-aware spatiotemporal decoder to effectively capture both spatial and temporal dynamics. Furthermore, it adopts a multi-task learning architecture coupled with a dedicated training strategy to maintain global consistency across sequences of varying lengths. Extensive experiments show that DetAny4D achieves competitive detection accuracy and significantly improves temporal stability, effectively addressing long-standing issues of jitter and inconsistency in 4D object detection. Data and code will be released upon acceptance.
Rethinking Token Reduction for Large Vision-Language Models
Authors: Yi Wang; Haofei Zhang; Qihan Huang; Anda Cao; Gongfan Fang; Wei Wang; Xuan Jin; Jie Song; Mingli Song; Xinchao Wang
Venue: CVPR 2026 (main)

Abstract: Large Vision-Language Models (LVLMs) excel in visual understanding and reasoning, but the excessive visual tokens lead to high inference costs. Although recent token reduction methods mitigate this issue, they mainly target single-turn Visual Question Answering (VQA), leaving the more practical multi-turn VQA (MT-VQA) scenario largely unexplored. MT-VQA introduces additional challenges, as subsequent questions are unknown beforehand and may refer to arbitrary image regions, making existing reduction strategies ineffective. Specifically, current approaches fall into two categories: prompt-dependent methods, which bias toward the initial text prompt and discard information useful for subsequent turns; prompt-agnostic ones, which, though technically applicable to multi-turn settings, rely on heuristic reduction metrics such as attention scores, leading to suboptimal performance. In this paper, we propose a learning-based prompt-agnostic method, termed MetaCompress, overcoming the limitations of heuristic designs. We begin by formulating token reduction as a learnable compression mapping, unifying existing formats such as pruning and merging into a single learning objective. Upon this formulation, we introduce a data-efficient training paradigm capable of learning optimal compression mappings with limited computational costs. Extensive experiments on MT-VQA benchmarks and across multiple LVLM architectures demonstrate that MetaCompress achieves superior efficiency-accuracy trade-offs while maintaining strong generalization across dialogue turns. Our code is available at https://github.com/MArSha1147/MetaCompress.
CaptionQA: Is Your Caption as Useful as the Image Itself?
Authors: Shijia Yang; Yunong Liu; Bohan Zhai; Ximeng Sun; Zicheng Liu; Emad Barsoum; Manling Li; Chenfeng Xu
Venue: CVPR 2026 (main)

Abstract: Image captions serve as efficient surrogates for visual content in multimodal systems such as retrieval, recommendation, multi-step agentic inference pipelines. Yet current evaluation practices miss a fundamental question: Can captions stand-in for images in real downstream tasks? We propose a utility-based benchmark, CaptionQA, to evaluate model-generated captions, where caption quality is measured by how well it supports downstream tasks. CaptionQA is an extensible domain-dependent benchmark covering 4 domains, Natural, Document, E-commerce, and Embodied AI, each with fine-grained taxonomies (25 top-level and 69 subcategories) that identify useful information for domain-specific tasks. CaptionQA builds 33,027 densely annotated multiple-choice questions (50.3 per image on average) that explicitly require visual information to answer, providing a comprehensive probe of caption utility. In our evaluation protocol, an LLM answers these questions using captions alone, directly measuring whether captions preserve image-level utility and are utilizable by a downstream LLM. Evaluating state-of-the-art MLLMs reveals substantial gaps between the image and its caption utility. Notably, models nearly identical on traditional image-QA benchmarks lower by up to 32% in caption utility. We release CaptionQA along with an open-source pipeline for extension to new domains.
Dynamics-Aware Preference Optimization for Vision-Language Models
Authors: Jusheng Zhang; Kaitong Cai; Jing Yang; Jian Wang; Keze Wang
Venue: CVPR 2026 (main)

Abstract: Preference-based finetuning of vision-language models (VLMs) is notoriously unstable, as trivially wrong negatives inject uninformative gradients that distort optimization and degrade calibration. This work revisits this issue through the lens of learning dynamics and identifies a core pathology, the squeezing effect, where easy negatives retain large, misaligned gradients despite having negligible loss.To address this, we propose Cooling-Weighted Direct Preference Optimization (CW-DPO), a two-stage framework that first smooths and then stabilizes the alignment process. Stage 1 employs a constrained SFT phase with low-weight "gentle negatives" to regularize overconfident distributions and flatten the loss landscape. Stage 2 introduces a competence-aware cooling weight that adaptively scales negative gradients according to the model's average per-token log-probability, suppressing uninformative updates while emphasizing hard, on-policy contrasts. This dynamics-aware weighting effectively mitigates the squeezing effect and enables smoother convergence.Extensive experiments on mainstream benchmarks--including COCO, Flickr30k, NoCaps, MMMU, and MMBench1.1--show that CW-DPO achieves state-of-the-art performance, for example +3.4 CIDEr over PPO and +2.4% absolute accuracy on MMMU, while improving calibration and halving convergence steps. This demonstrates that smoothing before cooling constitutes a simple yet general principle for robust VLM preference optimization.
Hierarchical Long Video Understanding with Audiovisual Entity Cohesion and Agentic Search
Authors: Xinlei Yin; Xiulian Peng; Xiao Li; Zhiwei Xiong; Yan Lu
Venue: CVPR 2026 (main)

Abstract: Long video understanding presents significant challenges for vision-language models due to extremely long context windows. Existing solutions relying on naive chunking strategies with retrieval-augmented generation, typically suffer from information fragmentation and a loss of global coherence. We present HAVEN, a unified framework for long-video understanding that enables coherent and comprehensive reasoning by integrating audiovisual entity cohesion and hierarchical video indexing with agentic search. First, we preserve semantic consistency by integrating entity-level representations across visual and auditory streams, while organizing content into a structured hierarchy spanning global summary, scene, segment, and entity levels. Then we employ an agentic search mechanism to enable dynamic retrieval and reasoning across these layers, facilitating coherent narrative reconstruction and fine-grained entity tracking. Extensive experiments demonstrate that our method achieves good temporal coherence, entity consistency, and retrieval efficiency, establishing a new state-of-the-art with an overall accuracy of 84.1% on LVBench. Notably, it achieves outstanding performance in the challenging reasoning category, reaching 80.1%. These results highlight the effectiveness of structured, multimodal reasoning for comprehensive and context-consistent understanding of long-form videos.
FlowDC: Flow-Based Decoupling-Decay for Complex Image Editing
Authors: Yilei Jiang; Zhen Wang; Yanghao Wang; Jun Yu; Yueting Zhuang; Jun Xiao; Long Chen
Venue: CVPR 2026 (main)

Abstract: With the surge of pre-trained text-to-image flow matching models, text-based image editing performance has gained remarkable improvement, especially for **simple editing** that only contains a single editing target. However, to satisfy the exploding editing requirements, the **complex editing** that contains multiple editing targets is posed as a more challenging task. However, current complex editing solutions: single-round and multi-round editing are limited by long text following and cumulative inconsistency, respectively. Thus, they struggle to strike a balance between semantic alignment and source consistency.In this paper, we propose **FlowDC**, which decouples the complex editing into multiple sub-editing effects and superposes them in parallel during the editing process. Meanwhile, we observed that the velocity quantity that is orthogonal to the editing displacement harms the source structure preserving. Thus, we decompose the velocity and decay the orthogonal part for better source consistency.To evaluate the effectiveness of complex editing settings, we construct a complex editing benchmark: Complex-PIE-Bench. On two benchmarks, FlowDC shows superior results compared with existing methods. We also detail the ablations of our module designs.