Orionnnn/hub-stats huggingface dataset:
A Comprehensive Survey on Long Context Language Modeling
Authors: name: Jiaheng Liu, name: Dawei Zhu, name: Zhiqi Bai, name: Yancheng He, name: Huanxuan Liao, name: Haoran Que, name: Zekun Wang, name: Chenchen Zhang, name: Ge Zhang, name: Jiebin Zhang, name: Yuanxing Zhang, name: Zhuo Chen, name: Hangyu Guo, name: Shilong Li, name: Ziqiang Liu, name: Yong Shan, name: Yifan Song, name: Jiayi Tian, name: Wenhao Wu, name: Zhejian Zhou, name: Ruijie Zhu, name: Junlan Feng, name: Yang Gao, name: Shizhu He, name: Zhoujun Li, name: Tianyu Liu, name: Fanyu Meng, name: Wenbo Su, name: Yingshui Tan, name: Zili Wang, name: Jian Yang, name: Wei Ye, name: Bo Zheng, name: Wangchunshu Zhou, name: Wenhao Huang, name: Sujian Li, name: Zhaoxiang Zhang
Published: 2025-03-20 17:06:28

Efficient processing of long contexts has been a persistent pursuit in
Natural Language Processing. With the growing number of long documents,
dialogues, and other textual data, it is important to develop Long Context
Language Models (LCLMs) that can process and analyze extensive inputs in an
effective and efficient way. In this paper, we present a comprehensive survey
on recent advances in long-context modeling for large language models. Our
survey is structured around three key aspects: how to obtain effective and
efficient LCLMs, how to train and deploy LCLMs efficiently, and how to evaluate
and analyze LCLMs comprehensively. For the first aspect, we discuss data
strategies, architectural designs, and workflow approaches oriented with long
context processing. For the second aspect, we provide a detailed examination of
the infrastructure required for LCLM training and inference. For the third
aspect, we present evaluation paradigms for long-context comprehension and
long-form generation, as well as behavioral analysis and mechanism
interpretability of LCLMs. Beyond these three key aspects, we thoroughly
explore the diverse application scenarios where existing LCLMs have been
deployed and outline promising future development directions. This survey
provides an up-to-date review of the literature on long-context LLMs, which we
wish to serve as a valuable resource for both researchers and engineers. An
associated GitHub repository collecting the latest papers and repos is
available at:
https://github.com/LCLM-Horizon/A-Comprehensive-Survey-For-Long-Context-Language-Modeling{\color[RGB]{175,36,67}{LCLM-Horizon}}.
Can Large Vision Language Models Read Maps Like a Human?
Authors: name: Shuo Xing, name: Zezhou Sun, name: Shuangyu Xie, name: Kaiyuan Chen, name: Yanjia Huang, name: Yuping Wang, name: Jiachen Li, name: Dezhen Song, name: Zhengzhong Tu
Published: 2025-03-18 18:05:38

In this paper, we introduce MapBench-the first dataset specifically designed
for human-readable, pixel-based map-based outdoor navigation, curated from
complex path finding scenarios. MapBench comprises over 1600 pixel space map
path finding problems from 100 diverse maps. In MapBench, LVLMs generate
language-based navigation instructions given a map image and a query with
beginning and end landmarks. For each map, MapBench provides Map Space Scene
Graph (MSSG) as an indexing data structure to convert between natural language
and evaluate LVLM-generated results. We demonstrate that MapBench significantly
challenges state-of-the-art LVLMs both zero-shot prompting and a
Chain-of-Thought (CoT) augmented reasoning framework that decomposes map
navigation into sequential cognitive processes. Our evaluation of both
open-source and closed-source LVLMs underscores the substantial difficulty
posed by MapBench, revealing critical limitations in their spatial reasoning
and structured decision-making capabilities. We release all the code and
dataset in https://github.com/taco-group/MapBench.
GAEA: A Geolocation Aware Conversational Model
Authors: name: Ron Campos, name: Ashmal Vayani, name: Parth Parag Kulkarni, name: Rohit Gupta, name: Aritra Dutta, name: Mubarak Shah
Published: 2025-03-20 17:59:47

Image geolocalization, in which, traditionally, an AI model predicts the
precise GPS coordinates of an image is a challenging task with many downstream
applications. However, the user cannot utilize the model to further their
knowledge other than the GPS coordinate; the model lacks an understanding of
the location and the conversational ability to communicate with the user. In
recent days, with tremendous progress of large multimodal models (LMMs)
proprietary and open-source researchers have attempted to geolocalize images
via LMMs. However, the issues remain unaddressed; beyond general tasks, for
more specialized downstream tasks, one of which is geolocalization, LMMs
struggle. In this work, we propose to solve this problem by introducing a
conversational model GAEA that can provide information regarding the location
of an image, as required by a user. No large-scale dataset enabling the
training of such a model exists. Thus we propose a comprehensive dataset GAEA
with 800K images and around 1.6M question answer pairs constructed by
leveraging OpenStreetMap (OSM) attributes and geographical context clues. For
quantitative evaluation, we propose a diverse benchmark comprising 4K
image-text pairs to evaluate conversational capabilities equipped with diverse
question types. We consider 11 state-of-the-art open-source and proprietary
LMMs and demonstrate that GAEA significantly outperforms the best open-source
model, LLaVA-OneVision by 25.69% and the best proprietary model, GPT-4o by
8.28%. Our dataset, model and codes are available
FFaceNeRF: Few-shot Face Editing in Neural Radiance Fields
Authors: name: Kwan Yun, name: Chaelin Kim, name: Hangyeul Shin, name: Junyong Noh
Published: 2025-03-21 12:24:58

Recent 3D face editing methods using masks have produced high-quality edited
images by leveraging Neural Radiance Fields (NeRF). Despite their impressive
performance, existing methods often provide limited user control due to the use
of pre-trained segmentation masks. To utilize masks with a desired layout, an
extensive training dataset is required, which is challenging to gather. We
present FFaceNeRF, a NeRF-based face editing technique that can overcome the
challenge of limited user control due to the use of fixed mask layouts. Our
method employs a geometry adapter with feature injection, allowing for
effective manipulation of geometry attributes. Additionally, we adopt latent
mixing for tri-plane augmentation, which enables training with a few samples.
This facilitates rapid model adaptation to desired mask layouts, crucial for
applications in fields like personalized medical imaging or creative face
editing. Our comparative evaluations demonstrate that FFaceNeRF surpasses
existing mask based face editing methods in terms of flexibility, control, and
generated image quality, paving the way for future advancements in customized
and high-fidelity 3D face editing. The code is available on the
{https://kwanyun.github.io/FFaceNeRF_page/{project-page}}.
Generalized Few-shot 3D Point Cloud Segmentation with Vision-Language
  Model
Authors: name: Zhaochong An, name: Guolei Sun, name: Yun Liu, name: Runjia Li, name: Junlin Han, name: Ender Konukoglu, name: Serge Belongie
Published: 2025-03-20 16:10:33

Generalized few-shot 3D point cloud segmentation (GFS-PCS) adapts models to
new classes with few support samples while retaining base class segmentation.
Existing GFS-PCS methods enhance prototypes via interacting with support or
query features but remain limited by sparse knowledge from few-shot samples.
Meanwhile, 3D vision-language models (3D VLMs), generalizing across open-world
novel classes, contain rich but noisy novel class knowledge. In this work, we
introduce a GFS-PCS framework that synergizes dense but noisy pseudo-labels
from 3D VLMs with precise yet sparse few-shot samples to maximize the strengths
of both, named GFS-VL. Specifically, we present a prototype-guided pseudo-label
selection to filter low-quality regions, followed by an adaptive infilling
strategy that combines knowledge from pseudo-label contexts and few-shot
samples to adaptively label the filtered, unlabeled areas. Additionally, we
design a novel-base mix strategy to embed few-shot samples into training
scenes, preserving essential context for improved novel class learning.
Moreover, recognizing the limited diversity in current GFS-PCS benchmarks, we
introduce two challenging benchmarks with diverse novel classes for
comprehensive generalization evaluation. Experiments validate the effectiveness
of our framework across models and datasets. Our approach and benchmarks
provide a solid foundation for advancing GFS-PCS in the real world. The code is
at https://github.com/ZhaochongAn/GFS-VL
TaoAvatar: Real-Time Lifelike Full-Body Talking Avatars for Augmented
  Reality via 3D Gaussian Splatting
Authors: name: Jianchuan Chen, name: Jingchuan Hu, name: Gaige Wang, name: Zhonghua Jiang, name: Tiansong Zhou, name: Zhiwen Chen, name: Chengfei Lv
Published: 2025-03-21 10:40:37

Realistic 3D full-body talking avatars hold great potential in AR, with
applications ranging from e-commerce live streaming to holographic
communication. Despite advances in 3D Gaussian Splatting (3DGS) for lifelike
avatar creation, existing methods struggle with fine-grained control of facial
expressions and body movements in full-body talking tasks. Additionally, they
often lack sufficient details and cannot run in real-time on mobile devices. We
present TaoAvatar, a high-fidelity, lightweight, 3DGS-based full-body talking
avatar driven by various signals. Our approach starts by creating a
personalized clothed human parametric template that binds Gaussians to
represent appearances. We then pre-train a StyleUnet-based network to handle
complex pose-dependent non-rigid deformation, which can capture high-frequency
appearance details but is too resource-intensive for mobile devices. To
overcome this, we "bake" the non-rigid deformations into a lightweight
MLP-based network using a distillation technique and develop blend shapes to
compensate for details. Extensive experiments show that TaoAvatar achieves
state-of-the-art rendering quality while running in real-time across various
devices, maintaining 90 FPS on high-definition stereo devices such as the Apple
Vision Pro.
Single Image Iterative Subject-driven Generation and Editing
Authors: name: Yair Shpitzer, name: Gal Chechik, name: Idan Schwartz
Published: 2025-03-20 10:45:04

Personalizing image generation and editing is particularly challenging when
we only have a few images of the subject, or even a single image. A common
approach to personalization is concept learning, which can integrate the
subject into existing models relatively quickly, but produces images whose
quality tends to deteriorate quickly when the number of subject images is
small. Quality can be improved by pre-training an encoder, but training
restricts generation to the training distribution, and is time consuming. It is
still an open hard challenge to personalize image generation and editing from a
single image without training. Here, we present SISO, a novel, training-free
approach based on optimizing a similarity score with an input subject image.
More specifically, SISO iteratively generates images and optimizes the model
based on loss of similarity with the given subject image until a satisfactory
level of similarity is achieved, allowing plug-and-play optimization to any
image generator. We evaluated SISO in two tasks, image editing and image
generation, using a diverse data set of personal subjects, and demonstrate
significant improvements over existing methods in image quality, subject
fidelity, and background preservation.
PVChat: Personalized Video Chat with One-Shot Learning
Authors: name: Yufei Shi, name: Weilong Yan, name: Gang Xu, name: Yumeng Li, name: Yuchen Li, name: Zhenxi Li, name: Fei Richard Yu, name: Ming Li, name: Si Yong Yeo
Published: 2025-03-21 11:50:06

Video large language models (ViLLMs) excel in general video understanding,
e.g., recognizing activities like talking and eating, but struggle with
identity-aware comprehension, such as "Wilson is receiving chemotherapy" or
"Tom is discussing with Sarah", limiting their applicability in smart
healthcare and smart home environments. To address this limitation, we propose
a one-shot learning framework PVChat, the first personalized ViLLM that enables
subject-aware question answering (QA) from a single video for each subject. Our
approach optimizes a Mixture-of-Heads (MoH) enhanced ViLLM on a synthetically
augmented video-QA dataset, leveraging a progressive image-to-video learning
strategy. Specifically, we introduce an automated augmentation pipeline that
synthesizes identity-preserving positive samples and retrieves hard negatives
from existing video corpora, generating a diverse training dataset with four QA
types: existence, appearance, action, and location inquiries. To enhance
subject-specific learning, we propose a ReLU Routing MoH attention mechanism,
alongside two novel objectives: (1) Smooth Proximity Regularization for
progressive learning through exponential distance scaling and (2) Head
Activation Enhancement for balanced attention routing. Finally, we adopt a
two-stage training strategy, transitioning from image pre-training to video
fine-tuning, enabling a gradual learning process from static attributes to
dynamic representations. We evaluate PVChat on diverse datasets covering
medical scenarios, TV series, anime, and real-world footage, demonstrating its
superiority in personalized feature understanding after learning from a single
video, compared to state-of-the-art ViLLMs.
ETVA: Evaluation of Text-to-Video Alignment via Fine-grained Question
  Generation and Answering
Authors: name: Kaisi Guan, name: Zhengfeng Lai, name: Yuchong Sun, name: Peng Zhang, name: Wei Liu, name: Kieran Liu, name: Meng Cao, name: Ruihua Song
Published: 2025-03-21 05:52:50

Precisely evaluating semantic alignment between text prompts and generated
videos remains a challenge in Text-to-Video (T2V) Generation. Existing
text-to-video alignment metrics like CLIPScore only generate coarse-grained
scores without fine-grained alignment details, failing to align with human
preference. To address this limitation, we propose ETVA, a novel Evaluation
method of Text-to-Video Alignment via fine-grained question generation and
answering. First, a multi-agent system parses prompts into semantic scene
graphs to generate atomic questions. Then we design a knowledge-augmented
multi-stage reasoning framework for question answering, where an auxiliary LLM
first retrieves relevant common-sense knowledge (e.g., physical laws), and then
video LLM answers the generated questions through a multi-stage reasoning
mechanism. Extensive experiments demonstrate that ETVA achieves a Spearman's
correlation coefficient of 58.47, showing a much higher correlation with human
judgment than existing metrics which attain only 31.0. We also construct a
comprehensive benchmark specifically designed for text-to-video alignment
evaluation, featuring 2k diverse prompts and 12k atomic questions spanning 10
categories. Through a systematic evaluation of 15 existing text-to-video
models, we identify their key capabilities and limitations, paving the way for
next-generation T2V generation.
When Less is Enough: Adaptive Token Reduction for Efficient Image
  Representation
Authors: name: Eduard Allakhverdov, name: Elizaveta Goncharova, name: Andrey Kuznetsov
Published: 2025-03-20 19:17:08

Vision encoders typically generate a large number of visual tokens, providing
information-rich representations but significantly increasing computational
demands. This raises the question of whether all generated tokens are equally
valuable or if some of them can be discarded to reduce computational costs
without compromising quality. In this paper, we introduce a new method for
determining feature utility based on the idea that less valuable features can
be reconstructed from more valuable ones. We implement this concept by
integrating an autoencoder with a Gumbel-Softmax selection mechanism, that
allows identifying and retaining only the most informative visual tokens. To
validate our approach, we compared the performance of the LLaVA-NeXT model,
using features selected by our method with randomly selected features. We found
that on OCR-based tasks, more than 50% of the visual context can be removed
with minimal performance loss, whereas randomly discarding the same proportion
of features significantly affects the model capabilities. Furthermore, in
general-domain tasks, even randomly retaining only 30% of tokens achieves
performance comparable to using the full set of visual tokens. Our results
highlight a promising direction towards adaptive and efficient multimodal
pruning that facilitates scalable and low-overhead inference without
compromising performance.
From Head to Tail: Towards Balanced Representation in Large
  Vision-Language Models through Adaptive Data Calibration
Authors: name: Mingyang Song, name: Xiaoye Qu, name: Jiawei Zhou, name: Yu Cheng
Published: 2025-03-17 05:01:09

Large Vision-Language Models (LVLMs) have achieved significant progress in
combining visual comprehension with language generation. Despite this success,
the training data of LVLMs still suffers from Long-Tail (LT) problems, where
the data distribution is highly imbalanced. Previous works have mainly focused
on traditional VLM architectures, i.e., CLIP or ViT, and specific tasks such as
recognition and classification. Nevertheless, the exploration of LVLM (e.g.
LLaVA) and more general tasks (e.g. Visual Question Answering and Visual
Reasoning) remains under-explored. In this paper, we first conduct an in-depth
analysis of the LT issues in LVLMs and identify two core causes: the
overrepresentation of head concepts and the underrepresentation of tail
concepts. Based on the above observation, we propose an Adaptive
Data Refinement Framework (ADR), which
consists of two stages: Data Rebalancing (DR)
and Data Synthesis (DS). In the DR stage, we
adaptively rebalance the redundant data based on entity distributions, while in
the DS stage, we leverage Denoising Diffusion Probabilistic Models (DDPMs) and
scarce images to supplement underrepresented portions. Through comprehensive
evaluations across eleven benchmarks, our proposed ADR effectively mitigates
the long-tail problem in the training data, improving the average performance
of LLaVA 1.5 relatively by 4.36%, without increasing the training data volume.
Implicit Bias-Like Patterns in Reasoning Models
Authors: name: Messi H. J. Lee, name: Calvin K. Lai
Published: 2025-03-14 16:40:02

Implicit bias refers to automatic or spontaneous mental processes that shape
perceptions, judgments, and behaviors. Previous research examining `implicit
bias' in large language models (LLMs) has often approached the phenomenon
differently than how it is studied in humans by focusing primarily on model
outputs rather than on model processing. To examine model processing, we
present a method called the Reasoning Model Implicit Association Test (RM-IAT)
for studying implicit bias-like patterns in reasoning models: LLMs that employ
step-by-step reasoning to solve complex tasks. Using this method, we find that
reasoning models require more tokens when processing association-incompatible
information compared to association-compatible information. These findings
suggest AI systems harbor patterns in processing information that are analogous
to human implicit bias. We consider the implications of these implicit
bias-like patterns for their deployment in real-world applications.
MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical
  Problems
Authors: name: Felix Chen, name: Hangjie Yuan, name: Yunqiu Xu, name: Tao Feng, name: Jun Cen, name: Pengwei Liu, name: Zeying Huang, name: Yi Yang
Published: 2025-03-19 11:46:19

Despite impressive performance across diverse tasks, Multimodal Large
Language Models (MLLMs) have yet to fully demonstrate their potential in visual
mathematical problem-solving, particularly in accurately perceiving and
interpreting diagrams. Inspired by typical processes of humans, we hypothesize
that the perception capabilities to extract meaningful information from
diagrams is crucial, as it directly impacts subsequent inference processes. To
validate this hypothesis, we developed FlowVerse, a comprehensive benchmark
that categorizes all information used during problem-solving into four
components, which are then combined into six problem versions for evaluation.
Our preliminary results on FlowVerse reveal that existing MLLMs exhibit
substantial limitations when extracting essential information and reasoned
property from diagrams and performing complex reasoning based on these visual
inputs. In response, we introduce MathFlow, a modular problem-solving pipeline
that decouples perception and inference into distinct stages, thereby
optimizing each independently. Given the perceptual limitations observed in
current MLLMs, we trained MathFlow-P-7B as a dedicated perception model.
Experimental results indicate that MathFlow-P-7B yields substantial performance
gains when integrated with various closed-source and open-source inference
models. This demonstrates the effectiveness of the MathFlow pipeline and its
compatibility to diverse inference frameworks. The FlowVerse benchmark and code
are available at https://github.com/MathFlow-zju/MathFlow.
RoboFactory: Exploring Embodied Agent Collaboration with Compositional
  Constraints
Authors: name: Yiran Qin, name: Li Kang, name: Xiufeng Song, name: Zhenfei Yin, name: Xiaohong Liu, name: Xihui Liu, name: Ruimao Zhang, name: Lei Bai
Published: 2025-03-20 17:58:38

Designing effective embodied multi-agent systems is critical for solving
complex real-world tasks across domains. Due to the complexity of multi-agent
embodied systems, existing methods fail to automatically generate safe and
efficient training data for such systems. To this end, we propose the concept
of compositional constraints for embodied multi-agent systems, addressing the
challenges arising from collaboration among embodied agents. We design various
interfaces tailored to different types of constraints, enabling seamless
interaction with the physical world. Leveraging compositional constraints and
specifically designed interfaces, we develop an automated data collection
framework for embodied multi-agent systems and introduce the first benchmark
for embodied multi-agent manipulation, RoboFactory. Based on RoboFactory
benchmark, we adapt and evaluate the method of imitation learning and analyzed
its performance in different difficulty agent tasks. Furthermore, we explore
the architectures and training strategies for multi-agent imitation learning,
aiming to build safe and efficient embodied multi-agent systems.
Enabling Versatile Controls for Video Diffusion Models
Authors: name: Xu Zhang, name: Hao Zhou, name: Haoming Qin, name: Xiaobin Lu, name: Jiaxing Yan, name: Guanzhong Wang, name: Zeyu Chen, name: Yi Liu
Published: 2025-03-21 09:48:00

Despite substantial progress in text-to-video generation, achieving precise
and flexible control over fine-grained spatiotemporal attributes remains a
significant unresolved challenge in video generation research. To address these
limitations, we introduce VCtrl (also termed PP-VCtrl), a novel framework
designed to enable fine-grained control over pre-trained video diffusion models
in a unified manner. VCtrl integrates diverse user-specified control
signals-such as Canny edges, segmentation masks, and human keypoints-into
pretrained video diffusion models via a generalizable conditional module
capable of uniformly encoding multiple types of auxiliary signals without
modifying the underlying generator. Additionally, we design a unified control
signal encoding pipeline and a sparse residual connection mechanism to
efficiently incorporate control representations. Comprehensive experiments and
human evaluations demonstrate that VCtrl effectively enhances controllability
and generation quality. The source code and pre-trained models are publicly
available and implemented using the PaddlePaddle framework at
http://github.com/PaddlePaddle/PaddleMIX/tree/develop/ppdiffusers/examples/ppvctrl.