Orionnnn/hub-stats huggingface dataset:
TinyLLaVA-Video-R1: Towards Smaller LMMs for Video Reasoning
Authors: name: Xingjian Zhang, name: Siwei Wen, name: Wenjun Wu, name: Lei Huang
Published: 2025-04-13 16:32:49

Recently, improving the reasoning ability of large multimodal models (LMMs)
through reinforcement learning has made great progress. However, most existing
works are based on highly reasoning-intensive datasets such as mathematics and
code, and researchers generally choose large-scale models as the foundation. We
argue that exploring small-scale models' reasoning capabilities remains
valuable for researchers with limited computational resources. Moreover,
enabling models to explain their reasoning processes on general
question-answering datasets is equally meaningful. Therefore, we present the
small-scale video reasoning model TinyLLaVA-Video-R1. Based on TinyLLaVA-Video,
a traceably trained video understanding model with no more than 4B parameters,
it not only demonstrates significantly improved reasoning and thinking
capabilities after using reinforcement learning on general Video-QA datasets,
but also exhibits the emergent characteristic of "aha moments". Furthermore, we
share a series of experimental findings, aiming to provide practical insights
for future exploration of video reasoning (thinking) abilities in small-scale
models. It is available at https://github.com/ZhangXJ199/TinyLLaVA-Video-R1.
EmoAgent: Assessing and Safeguarding Human-AI Interaction for Mental
  Health Safety
Authors: name: Jiahao Qiu, name: Yinghui He, name: Xinzhe Juan, name: Yiming Wang, name: Yuhan Liu, name: Zixin Yao, name: Yue Wu, name: Xun Jiang, name: Ling Yang, name: Mengdi Wang
Published: 2025-04-13 18:47:22

The rise of LLM-driven AI characters raises safety concerns, particularly for
vulnerable human users with psychological disorders. To address these risks, we
propose EmoAgent, a multi-agent AI framework designed to evaluate and mitigate
mental health hazards in human-AI interactions. EmoAgent comprises two
components: EmoEval simulates virtual users, including those portraying
mentally vulnerable individuals, to assess mental health changes before and
after interactions with AI characters. It uses clinically proven psychological
and psychiatric assessment tools (PHQ-9, PDI, PANSS) to evaluate mental risks
induced by LLM. EmoGuard serves as an intermediary, monitoring users' mental
status, predicting potential harm, and providing corrective feedback to
mitigate risks. Experiments conducted in popular character-based chatbots show
that emotionally engaging dialogues can lead to psychological deterioration in
vulnerable users, with mental state deterioration in more than 34.4% of the
simulations. EmoGuard significantly reduces these deterioration rates,
underscoring its role in ensuring safer AI-human interactions. Our code is
available at: https://github.com/1akaman/EmoAgent
SocioVerse: A World Model for Social Simulation Powered by LLM Agents
  and A Pool of 10 Million Real-World Users
Authors: name: Xinnong Zhang, name: Jiayu Lin, name: Xinyi Mou, name: Shiyue Yang, name: Xiawei Liu, name: Libo Sun, name: Hanjia Lyu, name: Yihang Yang, name: Weihong Qi, name: Yue Chen, name: Guanying Li, name: Ling Yan, name: Yao Hu, name: Siming Chen, name: Yu Wang, name: Jingxuan Huang, name: Jiebo Luo, name: Shiping Tang, name: Libo Wu, name: Baohua Zhou, name: Zhongyu Wei
Published: 2025-04-14 12:12:52

Social simulation is transforming traditional social science research by
modeling human behavior through interactions between virtual individuals and
their environments. With recent advances in large language models (LLMs), this
approach has shown growing potential in capturing individual differences and
predicting group behaviors. However, existing methods face alignment challenges
related to the environment, target users, interaction mechanisms, and
behavioral patterns. To this end, we introduce SocioVerse, an LLM-agent-driven
world model for social simulation. Our framework features four powerful
alignment components and a user pool of 10 million real individuals. To
validate its effectiveness, we conducted large-scale simulation experiments
across three distinct domains: politics, news, and economics. Results
demonstrate that SocioVerse can reflect large-scale population dynamics while
ensuring diversity, credibility, and representativeness through standardized
procedures and minimal manual adjustments.
VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models
  with Reinforcement Learning
Authors: name: Haozhe Wang, name: Chao Qu, name: Zuming Huang, name: Wei Chu, name: Fangzhen Lin, name: Wenhu Chen
Published: 2025-04-10 17:41:56

Recently, slow-thinking systems like GPT-o1 and DeepSeek-R1 have demonstrated
great potential in solving challenging problems through explicit reflection.
They significantly outperform the best fast-thinking models, such as GPT-4o, on
various math and science benchmarks. However, their multimodal reasoning
capabilities remain on par with fast-thinking models. For instance, GPT-o1's
performance on benchmarks like MathVista, MathVerse, and MathVision is similar
to fast-thinking models. In this paper, we aim to enhance the slow-thinking
capabilities of vision-language models using reinforcement learning (without
relying on distillation) to advance the state of the art. First, we adapt the
GRPO algorithm with a novel technique called Selective Sample Replay (SSR) to
address the vanishing advantages problem. While this approach yields strong
performance, the resulting RL-trained models exhibit limited self-reflection or
self-verification. To further encourage slow-thinking, we introduce Forced
Rethinking, which appends a textual rethinking trigger to the end of initial
rollouts in RL training, explicitly enforcing a self-reflection reasoning step.
By combining these two techniques, our model, VL-Rethinker, advances
state-of-the-art scores on MathVista, MathVerse, and MathVision to achieve
80.3%, 61.8%, and 43.9% respectively. VL-Rethinker also achieves open-source
SoTA on multi-disciplinary benchmarks such as MMMU-Pro, EMMA, and MEGA-Bench,
narrowing the gap with GPT-o1.
AgentRewardBench: Evaluating Automatic Evaluations of Web Agent
  Trajectories
Authors: name: Xing Han Lù, name: Amirhossein Kazemnejad, name: Nicholas Meade, name: Arkil Patel, name: Dongchan Shin, name: Alejandra Zambrano, name: Karolina Stańczak, name: Peter Shaw, name: Christopher J. Pal, name: Siva Reddy
Published: 2025-04-11 19:49:22

Web agents enable users to perform tasks on web browsers through natural
language interaction. Evaluating web agents trajectories is an important
problem, since it helps us determine whether the agent successfully completed
the tasks. Rule-based methods are widely used for this purpose, but they are
challenging to extend to new tasks and may not always recognize successful
trajectories. We may achieve higher accuracy through human evaluation, but the
process would be substantially slower and more expensive. Automatic evaluations
with LLMs may avoid the challenges of designing new rules and manually
annotating trajectories, enabling faster and cost-effective evaluation.
However, it is unclear how effective they are at evaluating web agents. To this
end, we propose AgentRewardBench, the first benchmark to assess the
effectiveness of LLM judges for evaluating web agents. AgentRewardBench
contains 1302 trajectories across 5 benchmarks and 4 LLMs. Each trajectory in
AgentRewardBench is reviewed by an expert, who answers questions pertaining to
the success, side effects, and repetitiveness of the agent. Using our
benchmark, we evaluate 12 LLM judges and find that no single LLM excels across
all benchmarks. We also find that the rule-based evaluation used by common
benchmarks tends to underreport the success rate of web agents, highlighting a
key weakness of rule-based evaluation and the need to develop more flexible
automatic evaluations. We release the benchmark at:
https://agent-reward-bench.github.io
Have we unified image generation and understanding yet? An empirical
  study of GPT-4o's image generation ability
Authors: name: Ning Li, name: Jingran Zhang, name: Justin Cui
Published: 2025-04-09 16:10:15

OpenAI's multimodal GPT-4o has demonstrated remarkable capabilities in image
generation and editing, yet its ability to achieve world knowledge-informed
semantic synthesis--seamlessly integrating domain knowledge, contextual
reasoning, and instruction adherence--remains unproven. In this study, we
systematically evaluate these capabilities across three critical dimensions:
(1) Global Instruction Adherence, (2) Fine-Grained Editing Precision, and (3)
Post-Generation Reasoning. While existing benchmarks highlight GPT-4o's strong
capabilities in image generation and editing, our evaluation reveals GPT-4o's
persistent limitations: the model frequently defaults to literal
interpretations of instructions, inconsistently applies knowledge constraints,
and struggles with conditional reasoning tasks. These findings challenge
prevailing assumptions about GPT-4o's unified understanding and generation
capabilities, exposing significant gaps in its dynamic knowledge integration.
Our study calls for the development of more robust benchmarks and training
strategies that go beyond surface-level alignment, emphasizing context-aware
and reasoning-grounded multimodal generation.
PRIMA.CPP: Speeding Up 70B-Scale LLM Inference on Low-Resource Everyday
  Home Clusters
Authors: name: Zonghang Li, name: Tao Li, name: Wenjiao Feng, name: Mohsen Guizani, name: Hongfang Yu
Published: 2025-04-07 13:46:21

Emergency of DeepSeek R1 and QwQ 32B have broken through performance barriers
for running frontier large language models (LLMs) on home devices. While
consumer hardware is getting stronger and model quantization is improving,
existing end-side solutions still demand GPU clusters, large RAM/VRAM, and high
bandwidth, far beyond what a common home cluster can handle. This paper
introduces prima.cpp, a distributed inference system that runs 70B-scale models
on everyday home devices using a mix of CPU/GPU, low RAM/VRAM, Wi-Fi, and
cross-platform support. It uses mmap to manage model weights and introduces
piped-ring parallelism with prefetching to hide disk loading. By modeling
heterogeneity in computation, communication, disk, memory (and its management
behavior), and OS, it optimally assigns model layers to each device's CPU and
GPU, further reducing token latency. An elegant algorithm named Halda is
proposed to solve this NP-hard assignment problem. We evaluate prima.cpp on a
common four-node home cluster. It outperforms llama.cpp, exo, and dllama on
30B+ models while keeping memory pressure below 6%. This brings frontier
30B-70B models, such as Llama 3, DeepSeek R1, Qwen 2.5, and QwQ to home
assistants, making advanced AI truly accessible to individuals. The code is
open source and available at https://github.com/Lizonghang/prima.cpp.
FUSION: Fully Integration of Vision-Language Representations for Deep
  Cross-Modal Understanding
Authors: name: Zheng Liu, name: Mengjie Liu, name: Jingzhou Chen, name: Jingwei Xu, name: Bin Cui, name: Conghui He, name: Wentao Zhang
Published: 2025-04-14 06:33:29

We introduce FUSION, a family of multimodal large language models (MLLMs)
with a fully vision-language alignment and integration paradigm. Unlike
existing methods that primarily rely on late-stage modality interaction during
LLM decoding, our approach achieves deep, dynamic integration throughout the
entire processing pipeline. To this end, we propose Text-Guided Unified Vision
Encoding, incorporating textual information in vision encoding to achieve
pixel-level integration. We further design Context-Aware Recursive Alignment
Decoding that recursively aggregates visual features conditioned on textual
context during decoding, enabling fine-grained, question-level semantic
integration. To guide feature mapping and mitigate modality discrepancies, we
develop Dual-Supervised Semantic Mapping Loss. Additionally, we construct a
Synthesized Language-Driven Question-Answer (QA) dataset through a new data
synthesis method, prioritizing high-quality QA pairs to optimize text-guided
feature integration. Building on these foundations, we train FUSION at two
scales-3B, 8B-and demonstrate that our full-modality integration approach
significantly outperforms existing methods with only 630 vision tokens.
Notably, FUSION 3B surpasses Cambrian-1 8B and Florence-VL 8B on most
benchmarks. FUSION 3B continues to outperform Cambrian-1 8B even when limited
to 300 vision tokens. Our ablation studies show that FUSION outperforms
LLaVA-NeXT on over half of the benchmarks under same configuration without
dynamic resolution, highlighting the effectiveness of our approach. We release
our code, model weights, and dataset. https://github.com/starriver030515/FUSION
DUMP: Automated Distribution-Level Curriculum Learning for RL-based LLM
  Post-training
Authors: name: Zhenting Wang, name: Guofeng Cui, name: Kun Wan, name: Wentian Zhao
Published: 2025-04-13 20:10:27

Recent advances in reinforcement learning (RL)-based post-training have led
to notable improvements in large language models (LLMs), particularly in
enhancing their reasoning capabilities to handle complex tasks. However, most
existing methods treat the training data as a unified whole, overlooking the
fact that modern LLM training often involves a mixture of data from diverse
distributions-varying in both source and difficulty. This heterogeneity
introduces a key challenge: how to adaptively schedule training across
distributions to optimize learning efficiency. In this paper, we present a
principled curriculum learning framework grounded in the notion of
distribution-level learnability. Our core insight is that the magnitude of
policy advantages reflects how much a model can still benefit from further
training on a given distribution. Based on this, we propose a
distribution-level curriculum learning framework for RL-based LLM
post-training, which leverages the Upper Confidence Bound (UCB) principle to
dynamically adjust sampling probabilities for different distrubutions. This
approach prioritizes distributions with either high average advantage
(exploitation) or low sample count (exploration), yielding an adaptive and
theoretically grounded training schedule. We instantiate our curriculum
learning framework with GRPO as the underlying RL algorithm and demonstrate its
effectiveness on logic reasoning datasets with multiple difficulties and
sources. Our experiments show that our framework significantly improves
convergence speed and final performance, highlighting the value of
distribution-aware curriculum strategies in LLM post-training. Code:
https://github.com/ZhentingWang/DUMP.
SpecReason: Fast and Accurate Inference-Time Compute via Speculative
  Reasoning
Authors: name: Rui Pan, name: Yinwei Dai, name: Zhihao Zhang, name: Gabriele Oliaro, name: Zhihao Jia, name: Ravi Netravali
Published: 2025-04-10 16:05:19

Recent advances in inference-time compute have significantly improved
performance on complex tasks by generating long chains of thought (CoTs) using
Large Reasoning Models (LRMs). However, this improved accuracy comes at the
cost of high inference latency due to the length of generated reasoning
sequences and the autoregressive nature of decoding. Our key insight in
tackling these overheads is that LRM inference, and the reasoning that it
embeds, is highly tolerant of approximations: complex tasks are typically
broken down into simpler steps, each of which brings utility based on the
semantic insight it provides for downstream steps rather than the exact tokens
it generates. Accordingly, we introduce SpecReason, a system that automatically
accelerates LRM inference by using a lightweight model to (speculatively) carry
out simpler intermediate reasoning steps and reserving the costly base model
only to assess (and potentially correct) the speculated outputs. Importantly,
SpecReason's focus on exploiting the semantic flexibility of thinking tokens in
preserving final-answer accuracy is complementary to prior speculation
techniques, most notably speculative decoding, which demands token-level
equivalence at each step. Across a variety of reasoning benchmarks, SpecReason
achieves 1.5-2.5times speedup over vanilla LRM inference while improving
accuracy by 1.0-9.9\%. Compared to speculative decoding without SpecReason,
their combination yields an additional 19.4-44.2\% latency reduction. We
open-source SpecReason at https://github.com/ruipeterpan/specreason.
BlenderGym: Benchmarking Foundational Model Systems for Graphics Editing
Authors: name: Yunqi Gu, name: Ian Huang, name: Jihyeon Je, name: Guandao Yang, name: Leonidas Guibas
Published: 2025-04-02 14:51:45

3D graphics editing is crucial in applications like movie production and game
design, yet it remains a time-consuming process that demands highly specialized
domain expertise. Automating this process is challenging because graphical
editing requires performing a variety of tasks, each requiring distinct skill
sets. Recently, vision-language models (VLMs) have emerged as a powerful
framework for automating the editing process, but their development and
evaluation are bottlenecked by the lack of a comprehensive benchmark that
requires human-level perception and presents real-world editing complexity. In
this work, we present BlenderGym, the first comprehensive VLM system benchmark
for 3D graphics editing. BlenderGym evaluates VLM systems through code-based 3D
reconstruction tasks. We evaluate closed- and open-source VLM systems and
observe that even the state-of-the-art VLM system struggles with tasks
relatively easy for human Blender users. Enabled by BlenderGym, we study how
inference scaling techniques impact VLM's performance on graphics editing
tasks. Notably, our findings reveal that the verifier used to guide the scaling
of generation can itself be improved through inference scaling, complementing
recent insights on inference scaling of LLM generation in coding and math
tasks. We further show that inference compute is not uniformly effective and
can be optimized by strategically distributing it between generation and
verification.
UKBOB: One Billion MRI Labeled Masks for Generalizable 3D Medical Image
  Segmentation
Authors: name: Emmanuelle Bourigault, name: Amir Jamaludin, name: Abdullah Hamdi
Published: 2025-04-09 14:10:51

In medical imaging, the primary challenge is collecting large-scale labeled
data due to privacy concerns, logistics, and high labeling costs. In this work,
we present the UK Biobank Organs and Bones (UKBOB), the largest labeled dataset
of body organs, comprising 51,761 MRI 3D samples (equivalent to 17.9 million 2D
images) and more than 1.37 billion 2D segmentation masks of 72 organs, all
based on the UK Biobank MRI dataset. We utilize automatic labeling, introduce
an automated label cleaning pipeline with organ-specific filters, and manually
annotate a subset of 300 MRIs with 11 abdominal classes to validate the quality
(referred to as UKBOB-manual). This approach allows for scaling up the dataset
collection while maintaining confidence in the labels. We further confirm the
validity of the labels by demonstrating zero-shot generalization of trained
models on the filtered UKBOB to other small labeled datasets from similar
domains (e.g., abdominal MRI). To further mitigate the effect of noisy labels,
we propose a novel method called Entropy Test-time Adaptation (ETTA) to refine
the segmentation output. We use UKBOB to train a foundation model, Swin-BOB,
for 3D medical image segmentation based on the Swin-UNetr architecture,
achieving state-of-the-art results in several benchmarks in 3D medical imaging,
including the BRATS brain MRI tumor challenge (with a 0.4% improvement) and the
BTCV abdominal CT scan benchmark (with a 1.3% improvement). The pre-trained
models and the code are available at https://emmanuelleb985.github.io/ukbob ,
and the filtered labels will be made available with the UK Biobank.
Training-free Guidance in Text-to-Video Generation via Multimodal
  Planning and Structured Noise Initialization
Authors: name: Jialu Li, name: Shoubin Yu, name: Han Lin, name: Jaemin Cho, name: Jaehong Yoon, name: Mohit Bansal
Published: 2025-04-11 15:41:43

Recent advancements in text-to-video (T2V) diffusion models have
significantly enhanced the visual quality of the generated videos. However,
even recent T2V models find it challenging to follow text descriptions
accurately, especially when the prompt requires accurate control of spatial
layouts or object trajectories. A recent line of research uses layout guidance
for T2V models that require fine-tuning or iterative manipulation of the
attention map during inference time. This significantly increases the memory
requirement, making it difficult to adopt a large T2V model as a backbone. To
address this, we introduce Video-MSG, a training-free Guidance method for T2V
generation based on Multimodal planning and Structured noise initialization.
Video-MSG consists of three steps, where in the first two steps, Video-MSG
creates Video Sketch, a fine-grained spatio-temporal plan for the final video,
specifying background, foreground, and object trajectories, in the form of
draft video frames. In the last step, Video-MSG guides a downstream T2V
diffusion model with Video Sketch through noise inversion and denoising.
Notably, Video-MSG does not need fine-tuning or attention manipulation with
additional memory during inference time, making it easier to adopt large T2V
models. Video-MSG demonstrates its effectiveness in enhancing text alignment
with multiple T2V backbones (VideoCrafter2 and CogVideoX-5B) on popular T2V
generation benchmarks (T2VCompBench and VBench). We provide comprehensive
ablation studies about noise inversion ratio, different background generators,
background object detection, and foreground object segmentation.
ZipIR: Latent Pyramid Diffusion Transformer for High-Resolution Image
  Restoration
Authors: name: Yongsheng Yu, name: Haitian Zheng, name: Zhifei Zhang, name: Jianming Zhang, name: Yuqian Zhou, name: Connelly Barnes, name: Yuchen Liu, name: Wei Xiong, name: Zhe Lin, name: Jiebo Luo
Published: 2025-04-11 14:49:52

Recent progress in generative models has significantly improved image
restoration capabilities, particularly through powerful diffusion models that
offer remarkable recovery of semantic details and local fidelity. However,
deploying these models at ultra-high resolutions faces a critical trade-off
between quality and efficiency due to the computational demands of long-range
attention mechanisms. To address this, we introduce ZipIR, a novel framework
that enhances efficiency, scalability, and long-range modeling for high-res
image restoration. ZipIR employs a highly compressed latent representation that
compresses image 32x, effectively reducing the number of spatial tokens, and
enabling the use of high-capacity models like the Diffusion Transformer (DiT).
Toward this goal, we propose a Latent Pyramid VAE (LP-VAE) design that
structures the latent space into sub-bands to ease diffusion training. Trained
on full images up to 2K resolution, ZipIR surpasses existing diffusion-based
methods, offering unmatched speed and quality in restoring high-resolution
images from severely degraded inputs.