Orionnnn/hub-stats huggingface dataset:
LiveCC: Learning Video LLM with Streaming Speech Transcription at Scale
Authors: name: Joya Chen, name: Ziyun Zeng, name: Yiqi Lin, name: Wei Li, name: Zejun Ma, name: Mike Zheng Shou
Published: 2025-04-22 16:52:09

Recent video large language models (Video LLMs) often depend on costly human
annotations or proprietary model APIs (e.g., GPT-4o) to produce training data,
which limits their training at scale. In this paper, we explore large-scale
training for Video LLM with cheap automatic speech recognition (ASR)
transcripts. Specifically, we propose a novel streaming training approach that
densely interleaves the ASR words and video frames according to their
timestamps. Compared to previous studies in vision-language representation with
ASR, our method naturally fits the streaming characteristics of ASR, thus
enabling the model to learn temporally-aligned, fine-grained vision-language
modeling. To support the training algorithm, we introduce a data production
pipeline to process YouTube videos and their closed captions (CC, same as ASR),
resulting in Live-CC-5M dataset for pre-training and Live-WhisperX-526K dataset
for high-quality supervised fine-tuning (SFT). Remarkably, even without SFT,
the ASR-only pre-trained LiveCC-7B-Base model demonstrates competitive general
video QA performance and exhibits a new capability in real-time video
commentary. To evaluate this, we carefully design a new LiveSports-3K
benchmark, using LLM-as-a-judge to measure the free-form commentary.
Experiments show our final LiveCC-7B-Instruct model can surpass advanced 72B
models (Qwen2.5-VL-72B-Instruct, LLaVA-Video-72B) in commentary quality even
working in a real-time mode. Meanwhile, it achieves state-of-the-art results at
the 7B/8B scale on popular video QA benchmarks such as VideoMME and OVOBench,
demonstrating the broad generalizability of our approach. All resources of this
paper have been released at https://showlab.github.io/livecc.
RealisDance-DiT: Simple yet Strong Baseline towards Controllable
  Character Animation in the Wild
Authors: name: Jingkai Zhou, name: Yifan Wu, name: Shikai Li, name: Min Wei, name: Chao Fan, name: Weihua Chen, name: Wei Jiang, name: Fan Wang
Published: 2025-04-21 09:09:21

Controllable character animation remains a challenging problem, particularly
in handling rare poses, stylized characters, character-object interactions,
complex illumination, and dynamic scenes. To tackle these issues, prior work
has largely focused on injecting pose and appearance guidance via elaborate
bypass networks, but often struggles to generalize to open-world scenarios. In
this paper, we propose a new perspective that, as long as the foundation model
is powerful enough, straightforward model modifications with flexible
fine-tuning strategies can largely address the above challenges, taking a step
towards controllable character animation in the wild. Specifically, we
introduce RealisDance-DiT, built upon the Wan-2.1 video foundation model. Our
sufficient analysis reveals that the widely adopted Reference Net design is
suboptimal for large-scale DiT models. Instead, we demonstrate that minimal
modifications to the foundation model architecture yield a surprisingly strong
baseline. We further propose the low-noise warmup and "large batches and small
iterations" strategies to accelerate model convergence during fine-tuning while
maximally preserving the priors of the foundation model. In addition, we
introduce a new test dataset that captures diverse real-world challenges,
complementing existing benchmarks such as TikTok dataset and UBC fashion video
dataset, to comprehensively evaluate the proposed method. Extensive experiments
show that RealisDance-DiT outperforms existing methods by a large margin.
Efficient Pretraining Length Scaling
Authors: name: Bohong Wu, name: Shen Yan, name: Sijun Zhang, name: Jianqiao Lu, name: Yutao Zeng, name: Ya Wang, name: Xun Zhou
Published: 2025-04-21 09:41:26

Recent advances in large language models have demonstrated the effectiveness
of length scaling during post-training, yet its potential in pre-training
remains underexplored. We present the Parallel Hidden Decoding Transformer
(PHD-Transformer), a novel framework that enables efficient length
scaling during pre-training while maintaining inference efficiency.
PHD-Transformer achieves this through an innovative KV cache
management strategy that distinguishes between original tokens and hidden
decoding tokens. By retaining only the KV cache of original tokens for
long-range dependencies while immediately discarding hidden decoding tokens
after use, our approach maintains the same KV cache size as the vanilla
transformer while enabling effective length scaling. To further enhance
performance, we introduce two optimized variants: PHD-SWA employs
sliding window attention to preserve local dependencies, while
PHD-CSWA implements chunk-wise sliding window attention to eliminate
linear growth in pre-filling time. Extensive experiments demonstrate consistent
improvements across multiple benchmarks.
Learning Adaptive Parallel Reasoning with Language Models
Authors: name: Jiayi Pan, name: Xiuyu Li, name: Long Lian, name: Charlie Snell, name: Yifei Zhou, name: Adam Yala, name: Trevor Darrell, name: Kurt Keutzer, name: Alane Suhr
Published: 2025-04-21 22:29:02

Scaling inference-time computation has substantially improved the reasoning
capabilities of language models. However, existing methods have significant
limitations: serialized chain-of-thought approaches generate overly long
outputs, leading to increased latency and exhausted context windows, while
parallel methods such as self-consistency suffer from insufficient
coordination, resulting in redundant computations and limited performance
gains. To address these shortcomings, we propose Adaptive Parallel Reasoning
(APR), a novel reasoning framework that enables language models to orchestrate
both serialized and parallel computations end-to-end. APR generalizes existing
reasoning methods by enabling adaptive multi-threaded inference using spawn()
and join() operations. A key innovation is our end-to-end reinforcement
learning strategy, optimizing both parent and child inference threads to
enhance task success rate without requiring predefined reasoning structures.
Experiments on the Countdown reasoning task demonstrate significant benefits of
APR: (1) higher performance within the same context window (83.4% vs. 60.0% at
4k context); (2) superior scalability with increased computation (80.1% vs.
66.6% at 20k total tokens); (3) improved accuracy at equivalent latency (75.2%
vs. 57.3% at approximately 5,000ms). APR represents a step towards enabling
language models to autonomously optimize their reasoning processes through
adaptive allocation of computation.
Progent: Programmable Privilege Control for LLM Agents
Authors: name: Tianneng Shi, name: Jingxuan He, name: Zhun Wang, name: Linyu Wu, name: Hongwei Li, name: Wenbo Guo, name: Dawn Song
Published: 2025-04-16 01:58:40

LLM agents are an emerging form of AI systems where large language models
(LLMs) serve as the central component, utilizing a diverse set of tools to
complete user-assigned tasks. Despite their great potential, LLM agents pose
significant security risks. When interacting with the external world, they may
encounter malicious commands from attackers, leading to the execution of
dangerous actions. A promising way to address this is by enforcing the
principle of least privilege: allowing only essential actions for task
completion while blocking unnecessary ones. However, achieving this is
challenging, as it requires covering diverse agent scenarios while preserving
both security and utility.
  We introduce Progent, the first privilege control mechanism for LLM agents.
At its core is a domain-specific language for flexibly expressing privilege
control policies applied during agent execution. These policies provide
fine-grained constraints over tool calls, deciding when tool calls are
permissible and specifying fallbacks if they are not. This enables agent
developers and users to craft suitable policies for their specific use cases
and enforce them deterministically to guarantee security. Thanks to its modular
design, integrating Progent does not alter agent internals and requires only
minimal changes to agent implementation, enhancing its practicality and
potential for widespread adoption. To automate policy writing, we leverage LLMs
to generate policies based on user queries, which are then updated dynamically
for improved security and utility. Our extensive evaluation shows that it
enables strong security while preserving high utility across three distinct
scenarios or benchmarks: AgentDojo, ASB, and AgentPoison. Furthermore, we
perform an in-depth analysis, showcasing the effectiveness of its core
components and the resilience of its automated policy generation against
adaptive attacks.
MR. Video: "MapReduce" is the Principle for Long Video Understanding
Authors: name: Ziqi Pang, name: Yu-Xiong Wang
Published: 2025-04-22 17:59:41

We propose MR. Video, an agentic long video understanding framework that
demonstrates the simple yet effective MapReduce principle for processing long
videos: (1) Map: independently and densely perceiving short video clips, and
(2) Reduce: jointly aggregating information from all clips. Compared with
sequence-to-sequence vision-language models (VLMs), MR. Video performs detailed
short video perception without being limited by context length. Compared with
existing video agents that typically rely on sequential key segment selection,
the Map operation enables simpler and more scalable sequence parallel
perception of short video segments. Its Reduce step allows for more
comprehensive context aggregation and reasoning, surpassing explicit key
segment retrieval. This MapReduce principle is applicable to both VLMs and
video agents, and we use LLM agents to validate its effectiveness.
  In practice, MR. Video employs two MapReduce stages: (A) Captioning:
generating captions for short video clips (map), then standardizing repeated
characters and objects into shared names (reduce); (B) Analysis: for each user
question, analyzing relevant information from individual short videos (map),
and integrating them into a final answer (reduce). MR. Video achieves over 10%
accuracy improvement on the challenging LVBench compared to state-of-the-art
VLMs and video agents.
  Code is available at: https://github.com/ziqipang/MR-Video
Describe Anything: Detailed Localized Image and Video Captioning
Authors: name: Long Lian, name: Yifan Ding, name: Yunhao Ge, name: Sifei Liu, name: Hanzi Mao, name: Boyi Li, name: Marco Pavone, name: Ming-Yu Liu, name: Trevor Darrell, name: Adam Yala, name: Yin Cui
Published: 2025-04-22 17:51:41

Generating detailed and accurate descriptions for specific regions in images
and videos remains a fundamental challenge for vision-language models. We
introduce the Describe Anything Model (DAM), a model designed for detailed
localized captioning (DLC). DAM preserves both local details and global context
through two key innovations: a focal prompt, which ensures high-resolution
encoding of targeted regions, and a localized vision backbone, which integrates
precise localization with its broader context. To tackle the scarcity of
high-quality DLC data, we propose a Semi-supervised learning (SSL)-based Data
Pipeline (DLC-SDP). DLC-SDP starts with existing segmentation datasets and
expands to unlabeled web images using SSL. We introduce DLC-Bench, a benchmark
designed to evaluate DLC without relying on reference captions. DAM sets new
state-of-the-art on 7 benchmarks spanning keyword-level, phrase-level, and
detailed multi-sentence localized image and video captioning.
Vidi: Large Multimodal Models for Video Understanding and Editing
Authors: name: Vidi Team, name: Celong Liu, name: Chia-Wen Kuo, name: Dawei Du, name: Fan Chen, name: Guang Chen, name: Jiamin Yuan, name: Lingxi Zhang, name: Lu Guo, name: Lusha Li, name: Longyin Wen, name: Qingyu Chen, name: Rachel Deng, name: Sijie Zhu, name: Stuart Siew, name: Tong Jin, name: Wei Lu, name: Wen Zhong, name: Xiaohui Shen, name: Xin Gu, name: Xing Mei, name: Xueqiong Qu
Published: 2025-04-22 08:04:45

Humans naturally share information with those they are connected to, and
video has become one of the dominant mediums for communication and expression
on the Internet. To support the creation of high-quality large-scale video
content, a modern pipeline requires a comprehensive understanding of both the
raw input materials (e.g., the unedited footage captured by cameras) and the
editing components (e.g., visual effects). In video editing scenarios, models
must process multiple modalities (e.g., vision, audio, text) with strong
background knowledge and handle flexible input lengths (e.g., hour-long raw
videos), which poses significant challenges for traditional models. In this
report, we introduce Vidi, a family of Large Multimodal Models (LMMs) for a
wide range of video understand editing scenarios. The first release focuses on
temporal retrieval, i.e., identifying the time ranges within the input videos
corresponding to a given text query, which plays a critical role in intelligent
editing. The model is capable of processing hour-long videos with strong
temporal understanding capability, e.g., retrieve time ranges for certain
queries. To support a comprehensive evaluation in real-world scenarios, we also
present the VUE-TR benchmark, which introduces five key advancements. 1) Video
duration: significantly longer than existing temporal retrival datasets, 2)
Audio support: includes audio-based queries, 3) Query format: diverse query
lengths/formats, 4) Annotation quality: ground-truth time ranges are manually
annotated. 5) Evaluation metric: a refined IoU metric to support evaluation
over multiple time ranges. Remarkably, Vidi significantly outperforms leading
proprietary models, e.g., GPT-4o and Gemini, on the temporal retrieval task,
indicating its superiority in video editing scenarios.
From Reflection to Perfection: Scaling Inference-Time Optimization for
  Text-to-Image Diffusion Models via Reflection Tuning
Authors: name: Le Zhuo, name: Liangbing Zhao, name: Sayak Paul, name: Yue Liao, name: Renrui Zhang, name: Yi Xin, name: Peng Gao, name: Mohamed Elhoseiny, name: Hongsheng Li
Published: 2025-04-22 17:58:07

Recent text-to-image diffusion models achieve impressive visual quality
through extensive scaling of training data and model parameters, yet they often
struggle with complex scenes and fine-grained details. Inspired by the
self-reflection capabilities emergent in large language models, we propose
ReflectionFlow, an inference-time framework enabling diffusion models to
iteratively reflect upon and refine their outputs. ReflectionFlow introduces
three complementary inference-time scaling axes: (1) noise-level scaling to
optimize latent initialization; (2) prompt-level scaling for precise semantic
guidance; and most notably, (3) reflection-level scaling, which explicitly
provides actionable reflections to iteratively assess and correct previous
generations. To facilitate reflection-level scaling, we construct GenRef, a
large-scale dataset comprising 1 million triplets, each containing a
reflection, a flawed image, and an enhanced image. Leveraging this dataset, we
efficiently perform reflection tuning on state-of-the-art diffusion
transformer, FLUX.1-dev, by jointly modeling multimodal inputs within a unified
framework. Experimental results show that ReflectionFlow significantly
outperforms naive noise-level scaling methods, offering a scalable and
compute-efficient solution toward higher-quality image synthesis on challenging
tasks.
The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks
Authors: name: Minghao Wu, name: Weixuan Wang, name: Sinuo Liu, name: Huifeng Yin, name: Xintong Wang, name: Yu Zhao, name: Chenyang Lyu, name: Longyue Wang, name: Weihua Luo, name: Kaifu Zhang
Published: 2025-04-22 01:47:37

As large language models (LLMs) continue to advance in linguistic
capabilities, robust multilingual evaluation has become essential for promoting
equitable technological progress. This position paper examines over 2,000
multilingual (non-English) benchmarks from 148 countries, published between
2021 and 2024, to evaluate past, present, and future practices in multilingual
benchmarking. Our findings reveal that, despite significant investments
amounting to tens of millions of dollars, English remains significantly
overrepresented in these benchmarks. Additionally, most benchmarks rely on
original language content rather than translations, with the majority sourced
from high-resource countries such as China, India, Germany, the UK, and the
USA. Furthermore, a comparison of benchmark performance with human judgments
highlights notable disparities. STEM-related tasks exhibit strong correlations
with human evaluations (0.70 to 0.85), while traditional NLP tasks like
question answering (e.g., XQuAD) show much weaker correlations (0.11 to 0.30).
Moreover, translating English benchmarks into other languages proves
insufficient, as localized benchmarks demonstrate significantly higher
alignment with local human judgments (0.68) than their translated counterparts
(0.47). This underscores the importance of creating culturally and
linguistically tailored benchmarks rather than relying solely on translations.
Through this comprehensive analysis, we highlight six key limitations in
current multilingual evaluation practices, propose the guiding principles
accordingly for effective multilingual benchmarking, and outline five critical
research directions to drive progress in the field. Finally, we call for a
global collaborative effort to develop human-aligned benchmarks that prioritize
real-world applications.
Roll the dice & look before you leap: Going beyond the creative limits
  of next-token prediction
Authors: name: Vaishnavh Nagarajan, name: Chen Henry Wu, name: Charles Ding, name: Aditi Raghunathan
Published: 2025-04-21 17:47:46

We design a suite of minimal algorithmic tasks that are a loose abstraction
of open-ended real-world tasks. This allows us to cleanly and controllably
quantify the creative limits of the present-day language model. Much like
real-world tasks that require a creative, far-sighted leap of thought, our
tasks require an implicit, open-ended stochastic planning step that either (a)
discovers new connections in an abstract knowledge graph (like in wordplay,
drawing analogies, or research) or (b) constructs new patterns (like in
designing math problems or new proteins). In these tasks, we empirically and
conceptually argue how next-token learning is myopic and memorizes excessively;
comparatively, multi-token approaches, namely teacherless training and
diffusion models, excel in producing diverse and original output. Secondly, in
our tasks, we find that to elicit randomness from the Transformer without
hurting coherence, it is better to inject noise right at the input layer (via a
method we dub hash-conditioning) rather than defer to temperature sampling from
the output layer. Thus, our work offers a principled, minimal test-bed for
analyzing open-ended creative skills, and offers new arguments for going beyond
next-token learning and softmax-based sampling. We make part of the code
available under https://github.com/chenwu98/algorithmic-creativity
LoftUp: Learning a Coordinate-Based Feature Upsampler for Vision
  Foundation Models
Authors: name: Haiwen Huang, name: Anpei Chen, name: Volodymyr Havrylov, name: Andreas Geiger, name: Dan Zhang
Published: 2025-04-18 18:46:08

Vision foundation models (VFMs) such as DINOv2 and CLIP have achieved
impressive results on various downstream tasks, but their limited feature
resolution hampers performance in applications requiring pixel-level
understanding. Feature upsampling offers a promising direction to address this
challenge. In this work, we identify two critical factors for enhancing feature
upsampling: the upsampler architecture and the training objective. For the
upsampler architecture, we introduce a coordinate-based cross-attention
transformer that integrates the high-resolution images with coordinates and
low-resolution VFM features to generate sharp, high-quality features. For the
training objective, we propose constructing high-resolution pseudo-groundtruth
features by leveraging class-agnostic masks and self-distillation. Our approach
effectively captures fine-grained details and adapts flexibly to various input
and feature resolutions. Through experiments, we demonstrate that our approach
significantly outperforms existing feature upsampling techniques across various
downstream tasks. Our code is released at https://github.com/andrehuang/loftup.
PROMPTEVALS: A Dataset of Assertions and Guardrails for Custom
  Production Large Language Model Pipelines
Authors: name: Reya Vir, name: Shreya Shankar, name: Harrison Chase, name: Will Fu-Hinthorn, name: Aditya Parameswaran
Published: 2025-04-20 21:04:23

Large language models (LLMs) are increasingly deployed in specialized
production data processing pipelines across diverse domains -- such as finance,
marketing, and e-commerce. However, when running them in production across many
inputs, they often fail to follow instructions or meet developer expectations.
To improve reliability in these applications, creating assertions or guardrails
for LLM outputs to run alongside the pipelines is essential. Yet, determining
the right set of assertions that capture developer requirements for a task is
challenging. In this paper, we introduce PROMPTEVALS, a dataset of 2087 LLM
pipeline prompts with 12623 corresponding assertion criteria, sourced from
developers using our open-source LLM pipeline tools. This dataset is 5x larger
than previous collections. Using a hold-out test split of PROMPTEVALS as a
benchmark, we evaluated closed- and open-source models in generating relevant
assertions. Notably, our fine-tuned Mistral and Llama 3 models outperform
GPT-4o by 20.93% on average, offering both reduced latency and improved
performance. We believe our dataset can spur further research in LLM
reliability, alignment, and prompt engineering.
THOUGHTTERMINATOR: Benchmarking, Calibrating, and Mitigating
  Overthinking in Reasoning Models
Authors: name: Xiao Pu, name: Michael Saxon, name: Wenyue Hua, name: William Yang Wang
Published: 2025-04-17 22:16:30

Reasoning models have demonstrated impressive performance on difficult tasks
that traditional language models struggle at. However, many are plagued with
the problem of overthinking--generating large amounts of unnecessary tokens
which don't improve accuracy on a question. We introduce approximate measures
of problem-level difficulty and demonstrate that a clear relationship between
problem difficulty and optimal token spend exists, and evaluate how well
calibrated a variety of reasoning models are in terms of efficiently allocating
the optimal token count. We find that in general, reasoning models are poorly
calibrated, particularly on easy problems. To evaluate calibration on easy
questions we introduce DUMB500, a dataset of extremely easy math, reasoning,
code, and task problems, and jointly evaluate reasoning model on these simple
examples and extremely difficult examples from existing frontier benchmarks on
the same task domain. Finally, we introduce THOUGHTTERMINATOR, a training-free
black box decoding technique that significantly improves reasoning model
calibration.