Orionnnn/hub-stats huggingface dataset:
PerceptionLM: Open-Access Data and Models for Detailed Visual
  Understanding
Authors: name: Jang Hyun Cho, name: Andrea Madotto, name: Effrosyni Mavroudi, name: Triantafyllos Afouras, name: Tushar Nagarajan, name: Muhammad Maaz, name: Yale Song, name: Tengyu Ma, name: Shuming Hu, name: Suyog Jain, name: Miguel Martin, name: Huiyu Wang, name: Hanoona Rasheed, name: Peize Sun, name: Po-Yao Huang, name: Daniel Bolya, name: Nikhila Ravi, name: Shashank Jain, name: Tammy Stark, name: Shane Moon, name: Babak Damavandi, name: Vivian Lee, name: Andrew Westbury, name: Salman Khan, name: Philipp Krähenbühl, name: Piotr Dollár, name: Lorenzo Torresani, name: Kristen Grauman, name: Christoph Feichtenhofer
Published: 2025-04-17 17:59:56

Vision-language models are integral to computer vision research, yet many
high-performing models remain closed-source, obscuring their data, design and
training recipe. The research community has responded by using distillation
from black-box models to label training data, achieving strong benchmark
results, at the cost of measurable scientific progress. However, without
knowing the details of the teacher model and its data sources, scientific
progress remains difficult to measure. In this paper, we study building a
Perception Language Model (PLM) in a fully open and reproducible framework for
transparent research in image and video understanding. We analyze standard
training pipelines without distillation from proprietary models and explore
large-scale synthetic data to identify critical data gaps, particularly in
detailed video understanding. To bridge these gaps, we release 2.8M
human-labeled instances of fine-grained video question-answer pairs and
spatio-temporally grounded video captions. Additionally, we introduce
PLM-VideoBench, a suite for evaluating challenging video understanding tasks
focusing on the ability to reason about "what", "where", "when", and "how" of a
video. We make our work fully reproducible by providing data, training recipes,
code & models.
70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU
  Inference via Dynamic-Length Float
Authors: name: Tianyi Zhang, name: Yang Sui, name: Shaochen Zhong, name: Vipin Chaudhary, name: Xia Hu, name: Anshumali Shrivastava
Published: 2025-04-15 22:38:38

Large Language Models (LLMs) have grown rapidly in size, creating significant
challenges for efficient deployment on resource-constrained hardware. In this
paper, we introduce Dynamic-Length Float (DFloat11), a lossless compression
framework that reduces LLM size by 30% while preserving outputs that are
bit-for-bit identical to the original model. DFloat11 is motivated by the low
entropy in the BFloat16 weight representation of LLMs, which reveals
significant inefficiency in existing storage format. By applying entropy
coding, DFloat11 assigns dynamic-length encodings to weights based on
frequency, achieving near information-optimal compression without any loss of
precision. To facilitate efficient inference with dynamic-length encodings, we
develop a custom GPU kernel for fast online decompression. Our design
incorporates the following: (i) decomposition of memory-intensive lookup tables
(LUTs) into compact LUTs that fit in GPU SRAM, (ii) a two-phase kernel for
coordinating thread read/write positions using lightweight auxiliary variables,
and (iii) transformer-block-level decompression to minimize latency.
Experiments on recent models, including Llama-3.1, Qwen-2.5, and Gemma-3,
validates our hypothesis that DFloat11 achieves around 30% model size reduction
while preserving bit-for-bit exact outputs. Compared to a potential alternative
of offloading parts of an uncompressed model to the CPU to meet memory
constraints, DFloat11 achieves 1.9-38.8x higher throughput in token generation.
With a fixed GPU memory budget, DFloat11 enables 5.3-13.17x longer context
lengths than uncompressed models. Notably, our method enables lossless
inference of Llama-3.1-405B, an 810GB model, on a single node equipped with
8x80GB GPUs. Our code and models are available at
https://github.com/LeanModels/DFloat11.
MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic
  Data Generation
Authors: name: Haris Riaz, name: Sourav Bhabesh, name: Vinayak Arannil, name: Miguel Ballesteros, name: Graham Horwood
Published: 2025-04-17 01:25:15

Recent smaller language models such Phi-3.5 and Phi-4 rely on synthetic data
generated using larger Language models. Questions remain about leveraging
synthetic data for other use cases, such as adapting LLMs to specific domains.
A key limitation of synthetic data is low diversity, which negatively impacts
its downstream applicability for improving other models. To address this, we
propose MetaSynth, a method for generating synthetic data that enhances
diversity through meta-prompting, where a language model orchestrates multiple
"expert" LLM agents to collaboratively generate data. Using only 25 million
tokens of synthetic data generated with MetaSynth, we successfully adapt a
well-trained LLM (Mistral-7B-v0.3) to two specialized domains-Finance and
Biomedicine-without compromising the capabilities of the resulting model in
general tasks. In addition, we evaluate the diversity of our synthetic data
using seven automated metrics, and find that it approaches the diversity of LLM
pre-training corpora.
  Continually pre-training Mistral-7B-v0.3 with MetaSynth notably outperforms
the base LLM, showing improvements of up to 4.08% in Finance and 13.75% in
Biomedicine. The same model shows degraded performance when trained on data
generated using a template prompt, even when the template includes prior
generations and varying In-Context exemplars of real data. Our findings suggest
that a few million tokens of diverse synthetic data without mixing any real
data, is sufficient for effective domain adaptation when using MetaSynth.
Perception Encoder: The best visual embeddings are not at the output of
  the network
Authors: name: Daniel Bolya, name: Po-Yao Huang, name: Peize Sun, name: Jang Hyun Cho, name: Andrea Madotto, name: Chen Wei, name: Tengyu Ma, name: Jiale Zhi, name: Jathushan Rajasegaran, name: Hanoona Rasheed, name: Junke Wang, name: Marco Monteiro, name: Hu Xu, name: Shiyu Dong, name: Nikhila Ravi, name: Daniel Li, name: Piotr Dollár, name: Christoph Feichtenhofer
Published: 2025-04-17 17:59:57

We introduce Perception Encoder (PE), a state-of-the-art encoder for image
and video understanding trained via simple vision-language learning.
Traditionally, vision encoders have relied on a variety of pretraining
objectives, each tailored to specific downstream tasks such as classification,
captioning, or localization. Surprisingly, after scaling our carefully tuned
image pretraining recipe and refining with our robust video data engine, we
find that contrastive vision-language training alone can produce strong,
general embeddings for all of these downstream tasks. There is only one caveat:
these embeddings are hidden within the intermediate layers of the network. To
draw them out, we introduce two alignment methods, language alignment for
multimodal language modeling, and spatial alignment for dense prediction.
Together with the core contrastive checkpoint, our PE family of models achieves
state-of-the-art performance on a wide variety of tasks, including zero-shot
image and video classification and retrieval; document, image, and video Q&A;
and spatial tasks such as detection, depth estimation, and tracking. To foster
further research, we are releasing our models, code, and a novel dataset of
synthetically and human-annotated videos.
CCMNet: Leveraging Calibrated Color Correction Matrices for Cross-Camera
  Color Constancy
Authors: name: Dongyoung Kim, name: Mahmoud Afifi, name: Dongyun Kim, name: Michael S. Brown, name: Seon Joo Kim
Published: 2025-04-10 17:59:31

Computational color constancy, or white balancing, is a key module in a
camera's image signal processor (ISP) that corrects color casts from scene
lighting. Because this operation occurs in the camera-specific raw color space,
white balance algorithms must adapt to different cameras. This paper introduces
a learning-based method for cross-camera color constancy that generalizes to
new cameras without retraining. Our method leverages pre-calibrated color
correction matrices (CCMs) available on ISPs that map the camera's raw color
space to a standard space (e.g., CIE XYZ). Our method uses these CCMs to
transform predefined illumination colors (i.e., along the Planckian locus) into
the test camera's raw space. The mapped illuminants are encoded into a compact
camera fingerprint embedding (CFE) that enables the network to adapt to unseen
cameras. To prevent overfitting due to limited cameras and CCMs during
training, we introduce a data augmentation technique that interpolates between
cameras and their CCMs. Experimental results across multiple datasets and
backbones show that our method achieves state-of-the-art cross-camera color
constancy while remaining lightweight and relying only on data readily
available in camera ISPs.
Sleep-time Compute: Beyond Inference Scaling at Test-time
Authors: name: Kevin Lin, name: Charlie Snell, name: Yu Wang, name: Charles Packer, name: Sarah Wooders, name: Ion Stoica, name: Joseph E. Gonzalez
Published: 2025-04-17 17:59:25

Scaling test-time compute has emerged as a key ingredient for enabling large
language models (LLMs) to solve difficult problems, but comes with high latency
and inference cost. We introduce sleep-time compute, which allows models to
"think" offline about contexts before queries are presented: by anticipating
what queries users might ask and pre-computing useful quantities, we can
significantly reduce the compute requirements at test-time. To demonstrate the
efficacy of our method, we create modified versions of two reasoning tasks -
Stateful GSM-Symbolic and Stateful AIME. We find that sleep-time compute can
reduce the amount of test-time compute needed to achieve the same accuracy by ~
5x on Stateful GSM-Symbolic and Stateful AIME and that by scaling sleep-time
compute we can further increase accuracy by up to 13% on Stateful GSM-Symbolic
and 18% on Stateful AIME. Furthermore, we introduce Multi-Query GSM-Symbolic,
which extends GSM-Symbolic by including multiple related queries per context.
By amortizing sleep-time compute across related queries about the same context
using Multi-Query GSM-Symbolic, we can decrease the average cost per query by
2.5x. We then conduct additional analysis to understand when sleep-time compute
is most effective, finding the predictability of the user query to be well
correlated with the efficacy of sleep-time compute. Finally, we conduct a
case-study of applying sleep-time compute to a realistic agentic SWE task.
InstantCharacter: Personalize Any Characters with a Scalable Diffusion
  Transformer Framework
Authors: name: Jiale Tao, name: Yanbing Zhang, name: Qixun Wang, name: Yiji Cheng, name: Haofan Wang, name: Xu Bai, name: Zhengguang Zhou, name: Ruihuang Li, name: Linqing Wang, name: Chunyu Wang, name: Qin Lin, name: Qinglin Lu
Published: 2025-04-16 18:01:59

Current learning-based subject customization approaches, predominantly
relying on U-Net architectures, suffer from limited generalization ability and
compromised image quality. Meanwhile, optimization-based methods require
subject-specific fine-tuning, which inevitably degrades textual
controllability. To address these challenges, we propose InstantCharacter, a
scalable framework for character customization built upon a foundation
diffusion transformer. InstantCharacter demonstrates three fundamental
advantages: first, it achieves open-domain personalization across diverse
character appearances, poses, and styles while maintaining high-fidelity
results. Second, the framework introduces a scalable adapter with stacked
transformer encoders, which effectively processes open-domain character
features and seamlessly interacts with the latent space of modern diffusion
transformers. Third, to effectively train the framework, we construct a
large-scale character dataset containing 10-million-level samples. The dataset
is systematically organized into paired (multi-view character) and unpaired
(text-image combinations) subsets. This dual-data structure enables
simultaneous optimization of identity consistency and textual editability
through distinct learning pathways. Qualitative experiments demonstrate the
advanced capabilities of InstantCharacter in generating high-fidelity,
text-controllable, and character-consistent images, setting a new benchmark for
character-driven image generation. Our source code is available at
https://github.com/Tencent/InstantCharacter.
NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation
Authors: name: Xiangyan Liu, name: Jinjie Ni, name: Zijian Wu, name: Chao Du, name: Longxu Dou, name: Haonan Wang, name: Tianyu Pang, name: Michael Qizhe Shieh
Published: 2025-04-17 16:10:13

Recent advances in reinforcement learning (RL) have strengthened the
reasoning capabilities of vision-language models (VLMs). However, enhancing
policy exploration to more effectively scale test-time compute remains
underexplored in VLMs. In addition, VLMs continue to struggle with imperfect
visual perception, which in turn affects the subsequent reasoning process. To
this end, we propose NoisyRollout, a simple yet effective RL approach that
mixes trajectories from both clean and moderately distorted images to introduce
targeted diversity in visual perception and the resulting reasoning patterns.
Without additional training cost, NoisyRollout enhances the exploration
capabilities of VLMs by incorporating a vision-oriented inductive bias.
Furthermore, NoisyRollout employs a noise annealing schedule that gradually
reduces distortion strength over training, ensuring benefit from noisy signals
early while maintaining training stability and scalability in later stages.
With just 2.1K training samples, NoisyRollout achieves state-of-the-art
performance among open-source RL-tuned models on 5 out-of-domain benchmarks
spanning both reasoning and perception tasks, while preserving comparable or
even better in-domain performance.
A Strategic Coordination Framework of Small LLMs Matches Large LLMs in
  Data Synthesis
Authors: name: Xin Gao, name: Qizhi Pei, name: Zinan Tang, name: Yu Li, name: Honglin Lin, name: Jiang Wu, name: Conghui He, name: Lijun Wu
Published: 2025-04-11 06:13:43

While data synthesis and distillation are promising strategies to enhance
small language models, current approaches heavily rely on Large Language Models
(LLMs), which suffer from high computational costs, environmental inefficiency,
and potential biases inherited from monolithic architectures. In contrast,
smaller LLMs are more accessible and sustainable, but their individual
capabilities often fall short in generating high-quality, diverse, and reliable
data. Inspired by collaborative human processes (e.g., peer review), we propose
a multiple small LLMs involved framework, GRA, that aggregates specialized
roles across small LLMs to iterative refinement and quality control typically
achieved by a single large LLM. In this collaborative framework, multiple small
LLMs assume distinct roles-Generator, Reviewer, and Adjudicator-to simulate a
peer-review-inspired data synthesis pipeline. The Generator proposes initial
data samples, the Reviewer critiques their quality and diversity, and the
Adjudicator resolves conflicts to finalize the output. By decomposing the
synthesis process into specialized sub-tasks, collaborative small LLMs can
achieve data-level parity with large LLM-based distillation. Through
experiments across multiple benchmarks, we demonstrate that GRA-produced data
matches or exceeds the quality of single large LLM outputs, e.g.,
Qwen-2.5-72B-Instruct. Our results challenge the necessity of monolithic large
models for high-quality data synthesis, advocating instead for strategic
coordination of smaller agents. Our datasets, models, and code are publicly
available at https://github.com/GX-XinGao/GRA.
DMM: Building a Versatile Image Generation Model via Distillation-Based
  Model Merging
Authors: name: Tianhui Song, name: Weixin Feng, name: Shuai Wang, name: Xubin Li, name: Tiezheng Ge, name: Bo Zheng, name: Limin Wang
Published: 2025-04-16 15:09:45

The success of text-to-image (T2I) generation models has spurred a
proliferation of numerous model checkpoints fine-tuned from the same base model
on various specialized datasets. This overwhelming specialized model production
introduces new challenges for high parameter redundancy and huge storage cost,
thereby necessitating the development of effective methods to consolidate and
unify the capabilities of diverse powerful models into a single one. A common
practice in model merging adopts static linear interpolation in the parameter
space to achieve the goal of style mixing. However, it neglects the features of
T2I generation task that numerous distinct models cover sundry styles which may
lead to incompatibility and confusion in the merged model. To address this
issue, we introduce a style-promptable image generation pipeline which can
accurately generate arbitrary-style images under the control of style vectors.
Based on this design, we propose the score distillation based model merging
paradigm (DMM), compressing multiple models into a single versatile T2I model.
Moreover, we rethink and reformulate the model merging task in the context of
T2I generation, by presenting new merging goals and evaluation protocols. Our
experiments demonstrate that DMM can compactly reorganize the knowledge from
multiple teacher models and achieve controllable arbitrary-style generation.
CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for
  Language Model Pre-training
Authors: name: Shizhe Diao, name: Yu Yang, name: Yonggan Fu, name: Xin Dong, name: Dan Su, name: Markus Kliegl, name: Zijia Chen, name: Peter Belcak, name: Yoshi Suhara, name: Hongxu Yin, name: Mostofa Patwary, name: Yingyan, name: Lin, name: Jan Kautz, name: Pavlo Molchanov
Published: 2025-04-17 17:58:13

Pre-training datasets are typically collected from web content and lack
inherent domain divisions. For instance, widely used datasets like Common Crawl
do not include explicit domain labels, while manually curating labeled datasets
such as The Pile is labor-intensive. Consequently, identifying an optimal
pre-training data mixture remains a challenging problem, despite its
significant benefits for pre-training performance. To address these challenges,
we propose CLustering-based Iterative Data Mixture Bootstrapping (CLIMB), an
automated framework that discovers, evaluates, and refines data mixtures in a
pre-training setting. Specifically, CLIMB embeds and clusters large-scale
datasets in a semantic space and then iteratively searches for optimal mixtures
using a smaller proxy model and a predictor. When continuously trained on 400B
tokens with this mixture, our 1B model exceeds the state-of-the-art
Llama-3.2-1B by 2.0%. Moreover, we observe that optimizing for a specific
domain (e.g., Social Sciences) yields a 5% improvement over random sampling.
Finally, we introduce ClimbLab, a filtered 1.2-trillion-token corpus with 20
clusters as a research playground, and ClimbMix, a compact yet powerful
400-billion-token dataset designed for efficient pre-training that delivers
superior performance under an equal token budget. We analyze the final data
mixture, elucidating the characteristics of an optimal data mixture. Our data
is available at: https://research.nvidia.com/labs/lpr/climb/
FocusedAD: Character-centric Movie Audio Description
Authors: name: Xiaojun Ye, name: Chun Wang, name: Yiren Song, name: Sheng Zhou, name: Liangcheng Li, name: Jiajun Bu
Published: 2025-04-16 15:04:14

Movie Audio Description (AD) aims to narrate visual content during
dialogue-free segments, particularly benefiting blind and visually impaired
(BVI) audiences. Compared with general video captioning, AD demands
plot-relevant narration with explicit character name references, posing unique
challenges in movie understanding.To identify active main characters and focus
on storyline-relevant regions, we propose FocusedAD, a novel framework that
delivers character-centric movie audio descriptions. It includes: (i) a
Character Perception Module(CPM) for tracking character regions and linking
them to names; (ii) a Dynamic Prior Module(DPM) that injects contextual cues
from prior ADs and subtitles via learnable soft prompts; and (iii) a Focused
Caption Module(FCM) that generates narrations enriched with plot-relevant
details and named characters. To overcome limitations in character
identification, we also introduce an automated pipeline for building character
query banks. FocusedAD achieves state-of-the-art performance on multiple
benchmarks, including strong zero-shot results on MAD-eval-Named and our newly
proposed Cinepile-AD dataset. Code and data will be released at
https://github.com/Thorin215/FocusedAD .
Exploring Expert Failures Improves LLM Agent Tuning
Authors: name: Li-Cheng Lan, name: Andrew Bai, name: Minhao Cheng, name: Cho-Jui Hsieh, name: Tianyi Zhou
Published: 2025-04-17 17:53:54

Large Language Models (LLMs) have shown tremendous potential as agents,
excelling at tasks that require multiple rounds of reasoning and interactions.
Rejection Sampling Fine-Tuning (RFT) has emerged as an effective method for
finetuning LLMs as agents: it first imitates expert-generated successful
trajectories and further improves agentic skills through iterative fine-tuning
on successful, self-generated trajectories. However, since the expert (e.g.,
GPT-4) succeeds primarily on simpler subtasks and RFT inherently favors simpler
scenarios, many complex subtasks remain unsolved and persistently
out-of-distribution (OOD). Upon investigating these challenging subtasks, we
discovered that previously failed expert trajectories can often provide
valuable guidance, e.g., plans and key actions, that can significantly improve
agent exploration efficiency and acquisition of critical skills. Motivated by
these observations, we propose Exploring Expert Failures (EEF), which
identifies beneficial actions from failed expert trajectories and integrates
them into the training dataset. Potentially harmful actions are meticulously
excluded to prevent contamination of the model learning process. By leveraging
the beneficial actions in expert failures, EEF successfully solves some
previously unsolvable subtasks and improves agent tuning performance.
Remarkably, our approach achieved a 62\% win rate in WebShop, outperforming RFT
(53. 6\%) and GPT-4 (35. 6\%), and to the best of our knowledge, setting a new
state-of-the-art as the first method to surpass a score of 0.81 in WebShop and
exceed 81 in SciWorld.