Orionnnn/hub-stats huggingface dataset:
BEATS: Bias Evaluation and Assessment Test Suite for Large Language
  Models
Authors: name: Alok Abhishek, name: Lisa Erickson, name: Tushar Bandopadhyay
Published: 2025-03-31 16:56:52

In this research, we introduce BEATS, a novel framework for evaluating Bias,
Ethics, Fairness, and Factuality in Large Language Models (LLMs). Building upon
the BEATS framework, we present a bias benchmark for LLMs that measure
performance across 29 distinct metrics. These metrics span a broad range of
characteristics, including demographic, cognitive, and social biases, as well
as measures of ethical reasoning, group fairness, and factuality related
misinformation risk. These metrics enable a quantitative assessment of the
extent to which LLM generated responses may perpetuate societal prejudices that
reinforce or expand systemic inequities. To achieve a high score on this
benchmark a LLM must show very equitable behavior in their responses, making it
a rigorous standard for responsible AI evaluation. Empirical results based on
data from our experiment show that, 37.65\% of outputs generated by industry
leading models contained some form of bias, highlighting a substantial risk of
using these models in critical decision making systems. BEATS framework and
benchmark offer a scalable and statistically rigorous methodology to benchmark
LLMs, diagnose factors driving biases, and develop mitigation strategies. With
the BEATS framework, our goal is to help the development of more socially
responsible and ethically aligned AI models.
Comprehensive Relighting: Generalizable and Consistent Monocular Human
  Relighting and Harmonization
Authors: name: Junying Wang, name: Jingyuan Liu, name: Xin Sun, name: Krishna Kumar Singh, name: Zhixin Shu, name: He Zhang, name: Jimei Yang, name: Nanxuan Zhao, name: Tuanfeng Y. Wang, name: Simon S. Chen, name: Ulrich Neumann, name: Jae Shin Yoon
Published: 2025-04-03 20:10:50

This paper introduces Comprehensive Relighting, the first all-in-one approach
that can both control and harmonize the lighting from an image or video of
humans with arbitrary body parts from any scene. Building such a generalizable
model is extremely challenging due to the lack of dataset, restricting existing
image-based relighting models to a specific scenario (e.g., face or static
human). To address this challenge, we repurpose a pre-trained diffusion model
as a general image prior and jointly model the human relighting and background
harmonization in the coarse-to-fine framework. To further enhance the temporal
coherence of the relighting, we introduce an unsupervised temporal lighting
model that learns the lighting cycle consistency from many real-world videos
without any ground truth. In inference time, our temporal lighting module is
combined with the diffusion models through the spatio-temporal feature blending
algorithms without extra training; and we apply a new guided refinement as a
post-processing to preserve the high-frequency details from the input image. In
the experiments, Comprehensive Relighting shows a strong generalizability and
lighting temporal coherence, outperforming existing image-based human
relighting and harmonization methods.
SynWorld: Virtual Scenario Synthesis for Agentic Action Knowledge
  Refinement
Authors: name: Runnan Fang, name: Xiaobin Wang, name: Yuan Liang, name: Shuofei Qiao, name: Jialong Wu, name: Zekun Xi, name: Ningyu Zhang, name: Yong Jiang, name: Pengjun Xie, name: Fei Huang, name: Huajun Chen
Published: 2025-04-04 16:10:57

In the interaction between agents and their environments, agents expand their
capabilities by planning and executing actions. However, LLM-based agents face
substantial challenges when deployed in novel environments or required to
navigate unconventional action spaces. To empower agents to autonomously
explore environments, optimize workflows, and enhance their understanding of
actions, we propose SynWorld, a framework that allows agents to synthesize
possible scenarios with multi-step action invocation within the action space
and perform Monte Carlo Tree Search (MCTS) exploration to effectively refine
their action knowledge in the current environment. Our experiments demonstrate
that SynWorld is an effective and general approach to learning action knowledge
in new environments. Code is available at https://github.com/zjunlp/SynWorld.
Agentic Knowledgeable Self-awareness
Authors: name: Shuofei Qiao, name: Zhisong Qiu, name: Baochang Ren, name: Xiaobin Wang, name: Xiangyuan Ru, name: Ningyu Zhang, name: Xiang Chen, name: Yong Jiang, name: Pengjun Xie, name: Fei Huang, name: Huajun Chen
Published: 2025-04-04 16:03:38

Large Language Models (LLMs) have achieved considerable performance across
various agentic planning tasks. However, traditional agent planning approaches
adopt a "flood irrigation" methodology that indiscriminately injects gold
trajectories, external feedback, and domain knowledge into agent models. This
practice overlooks the fundamental human cognitive principle of situational
self-awareness during decision-making-the ability to dynamically assess
situational demands and strategically employ resources during decision-making.
We propose agentic knowledgeable self-awareness to address this gap, a novel
paradigm enabling LLM-based agents to autonomously regulate knowledge
utilization. Specifically, we propose KnowSelf, a data-centric approach that
applies agents with knowledgeable self-awareness like humans. Concretely, we
devise a heuristic situation judgement criterion to mark special tokens on the
agent's self-explored trajectories for collecting training data. Through a
two-stage training process, the agent model can switch between different
situations by generating specific special tokens, achieving optimal planning
effects with minimal costs. Our experiments demonstrate that KnowSelf can
outperform various strong baselines on different tasks and models with minimal
use of external knowledge. Code is available at
https://github.com/zjunlp/KnowSelf.
VARGPT-v1.1: Improve Visual Autoregressive Large Unified Model via
  Iterative Instruction Tuning and Reinforcement Learning
Authors: name: Xianwei Zhuang, name: Yuxin Xie, name: Yufan Deng, name: Dongchao Yang, name: Liming Liang, name: Jinghan Ru, name: Yuguo Yin, name: Yuexian Zou
Published: 2025-04-03 18:06:28

In this work, we present VARGPT-v1.1, an advanced unified visual
autoregressive model that builds upon our previous framework VARGPT. The model
preserves the dual paradigm of next-token prediction for visual understanding
and next-scale generation for image synthesis. Specifically, VARGPT-v1.1
integrates: (1) a novel training strategy combining iterative visual
instruction tuning with reinforcement learning through Direct Preference
Optimization (DPO), (2) an expanded training corpus containing 8.3M
visual-generative instruction pairs, (3) an upgraded language model backbone
using Qwen2, (4) enhanced image generation resolution, and (5) emergent image
editing capabilities without architectural modifications. These advancements
enable VARGPT-v1.1 to achieve state-of-the-art performance in multimodal
understanding and text-to-image instruction-following tasks, demonstrating
significant improvements in both comprehension and generation metrics. Notably,
through visual instruction tuning, the model acquires image editing
functionality while maintaining architectural consistency with its predecessor,
revealing the potential for unified visual understanding, generation, and
editing. Our findings suggest that well-designed unified visual autoregressive
models can effectively adopt flexible training strategies from large language
models (LLMs), exhibiting promising scalability. The codebase and model weights
are publicly available at https://github.com/VARGPT-family/VARGPT-v1.1.
MME-Unify: A Comprehensive Benchmark for Unified Multimodal
  Understanding and Generation Models
Authors: name: Wulin Xie, name: Yi-Fan Zhang, name: Chaoyou Fu, name: Yang Shi, name: Bingyan Nie, name: Hongkai Chen, name: Zhang Zhang, name: Liang Wang, name: Tieniu Tan
Published: 2025-04-04 17:59:55

Existing MLLM benchmarks face significant challenges in evaluating Unified
MLLMs (U-MLLMs) due to: 1) lack of standardized benchmarks for traditional
tasks, leading to inconsistent comparisons; 2) absence of benchmarks for
mixed-modality generation, which fails to assess multimodal reasoning
capabilities. We present a comprehensive evaluation framework designed to
systematically assess U-MLLMs. Our benchmark includes: Standardized Traditional
Task Evaluation. We sample from 12 datasets, covering 10 tasks with 30
subtasks, ensuring consistent and fair comparisons across studies." 2. Unified
Task Assessment. We introduce five novel tasks testing multimodal reasoning,
including image editing, commonsense QA with image generation, and geometric
reasoning. 3. Comprehensive Model Benchmarking. We evaluate 12 leading U-MLLMs,
such as Janus-Pro, EMU3, VILA-U, and Gemini2-flash, alongside specialized
understanding (e.g., Claude-3.5-Sonnet) and generation models (e.g., DALL-E-3).
Our findings reveal substantial performance gaps in existing U-MLLMs,
highlighting the need for more robust models capable of handling mixed-modality
tasks effectively. The code and evaluation data can be found in
https://mme-unify.github.io/.
Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving
Authors: name: Daoguang Zan, name: Zhirong Huang, name: Wei Liu, name: Hanwu Chen, name: Linhao Zhang, name: Shulin Xin, name: Lu Chen, name: Qi Liu, name: Xiaojian Zhong, name: Aoyan Li, name: Siyao Liu, name: Yongsheng Xiao, name: Liangqiang Chen, name: Yuyu Zhang, name: Jing Su, name: Tianyu Liu, name: Rui Long, name: Kai Shen, name: Liang Xiang
Published: 2025-04-03 14:06:17

The task of issue resolving is to modify a codebase to generate a patch that
addresses a given issue. However, existing benchmarks, such as SWE-bench, focus
almost exclusively on Python, making them insufficient for evaluating Large
Language Models (LLMs) across diverse software ecosystems. To address this, we
introduce a multilingual issue-resolving benchmark, called Multi-SWE-bench,
covering Java, TypeScript, JavaScript, Go, Rust, C, and C++. It includes a
total of 1,632 high-quality instances, which were carefully annotated from
2,456 candidates by 68 expert annotators, ensuring that the benchmark can
provide an accurate and reliable evaluation. Based on Multi-SWE-bench, we
evaluate a series of state-of-the-art models using three representative methods
(Agentless, SWE-agent, and OpenHands) and present a comprehensive analysis with
key empirical insights. In addition, we launch a Multi-SWE-RL open-source
community, aimed at building large-scale reinforcement learning (RL) training
datasets for issue-resolving tasks. As an initial contribution, we release a
set of 4,723 well-structured instances spanning seven programming languages,
laying a solid foundation for RL research in this domain. More importantly, we
open-source our entire data production pipeline, along with detailed tutorials,
encouraging the open-source community to continuously contribute and expand the
dataset. We envision our Multi-SWE-bench and the ever-growing Multi-SWE-RL
community as catalysts for advancing RL toward its full potential, bringing us
one step closer to the dawn of AGI.
OpenCodeReasoning: Advancing Data Distillation for Competitive Coding
Authors: name: Wasi Uddin Ahmad, name: Sean Narenthiran, name: Somshubra Majumdar, name: Aleksander Ficek, name: Siddhartha Jain, name: Jocelyn Huang, name: Vahid Noroozi, name: Boris Ginsburg
Published: 2025-04-02 17:50:31

Since the advent of reasoning-based large language models, many have found
great success from distilling reasoning capabilities into student models. Such
techniques have significantly bridged the gap between reasoning and standard
LLMs on coding tasks. Despite this, much of the progress on distilling
reasoning models remains locked behind proprietary datasets or lacks details on
data curation, filtering and subsequent training. To address this, we construct
a superior supervised fine-tuning (SFT) dataset that we use to achieve
state-of-the-art coding capability results in models of various sizes. Our
distilled models use only SFT to achieve 61.8% on LiveCodeBench and 24.6% on
CodeContests, surpassing alternatives trained with reinforcement learning. We
then perform analysis on the data sources used to construct our dataset, the
impact of code execution filtering, and the importance of instruction/solution
diversity. We observe that execution filtering negatively affected benchmark
accuracy, leading us to prioritize instruction diversity over solution
correctness. Finally, we also analyze the token efficiency and reasoning
patterns utilized by these models. We will open-source these datasets and
distilled models to the community.
Scene-Centric Unsupervised Panoptic Segmentation
Authors: name: Oliver Hahn, name: Christoph Reich, name: Nikita Araslanov, name: Daniel Cremers, name: Christian Rupprecht, name: Stefan Roth
Published: 2025-04-02 17:58:46

Unsupervised panoptic segmentation aims to partition an image into
semantically meaningful regions and distinct object instances without training
on manually annotated data. In contrast to prior work on unsupervised panoptic
scene understanding, we eliminate the need for object-centric training data,
enabling the unsupervised understanding of complex scenes. To that end, we
present the first unsupervised panoptic method that directly trains on
scene-centric imagery. In particular, we propose an approach to obtain
high-resolution panoptic pseudo labels on complex scene-centric data, combining
visual representations, depth, and motion cues. Utilizing both pseudo-label
training and a panoptic self-training strategy yields a novel approach that
accurately predicts panoptic segmentation of complex scenes without requiring
any human annotations. Our approach significantly improves panoptic quality,
e.g., surpassing the recent state of the art in unsupervised panoptic
segmentation on Cityscapes by 9.4% points in PQ.
WikiVideo: Article Generation from Multiple Videos
Authors: name: Alexander Martin, name: Reno Kriz, name: William Gantt Walden, name: Kate Sanders, name: Hannah Recknor, name: Eugene Yang, name: Francis Ferraro, name: Benjamin Van Durme
Published: 2025-04-01 16:22:15

We present the challenging task of automatically creating a high-level
Wikipedia-style article that aggregates information from multiple diverse
videos about real-world events, such as natural disasters or political
elections. Videos are intuitive sources for retrieval-augmented generation
(RAG), but most contemporary RAG workflows focus heavily on text and existing
methods for video-based summarization focus on low-level scene understanding
rather than high-level event semantics. To close this gap, we introduce
WikiVideo, a benchmark consisting of expert-written articles and densely
annotated videos that provide evidence for articles' claims, facilitating the
integration of video into RAG pipelines and enabling the creation of in-depth
content that is grounded in multimodal sources. We further propose
Collaborative Article Generation (CAG), a novel interactive method for article
creation from multiple videos. CAG leverages an iterative interaction between
an r1-style reasoning model and a VideoLLM to draw higher level inferences
about the target event than is possible with VideoLLMs alone, which fixate on
low-level visual features. We benchmark state-of-the-art VideoLLMs and CAG in
both oracle retrieval and RAG settings and find that CAG consistently
outperforms alternative methods, while suggesting intriguing avenues for future
work.
JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical
  Spatio-Temporal Prior Synchronization
Authors: name: Kai Liu, name: Wei Li, name: Lai Chen, name: Shengqiong Wu, name: Yanhao Zheng, name: Jiayi Ji, name: Fan Zhou, name: Rongxin Jiang, name: Jiebo Luo, name: Hao Fei, name: Tat-Seng Chua
Published: 2025-03-30 09:40:42

This paper introduces JavisDiT, a novel Joint Audio-Video Diffusion
Transformer designed for synchronized audio-video generation (JAVG). Built upon
the powerful Diffusion Transformer (DiT) architecture, JavisDiT is able to
generate high-quality audio and video content simultaneously from open-ended
user prompts. To ensure optimal synchronization, we introduce a fine-grained
spatio-temporal alignment mechanism through a Hierarchical Spatial-Temporal
Synchronized Prior (HiST-Sypo) Estimator. This module extracts both global and
fine-grained spatio-temporal priors, guiding the synchronization between the
visual and auditory components. Furthermore, we propose a new benchmark,
JavisBench, consisting of 10,140 high-quality text-captioned sounding videos
spanning diverse scenes and complex real-world scenarios. Further, we
specifically devise a robust metric for evaluating the synchronization between
generated audio-video pairs in real-world complex content. Experimental results
demonstrate that JavisDiT significantly outperforms existing methods by
ensuring both high-quality generation and precise synchronization, setting a
new standard for JAVG tasks. Our code, model, and dataset will be made publicly
available at https://javisdit.github.io/.
FreSca: Unveiling the Scaling Space in Diffusion Models
Authors: name: Chao Huang, name: Susan Liang, name: Yunlong Tang, name: Li Ma, name: Yapeng Tian, name: Chenliang Xu
Published: 2025-04-02 22:03:11

Diffusion models offer impressive controllability for image tasks, primarily
through noise predictions that encode task-specific information and
classifier-free guidance enabling adjustable scaling. This scaling mechanism
implicitly defines a ``scaling space'' whose potential for fine-grained
semantic manipulation remains underexplored. We investigate this space,
starting with inversion-based editing where the difference between
conditional/unconditional noise predictions carries key semantic information.
Our core contribution stems from a Fourier analysis of noise predictions,
revealing that its low- and high-frequency components evolve differently
throughout diffusion. Based on this insight, we introduce FreSca, a
straightforward method that applies guidance scaling independently to different
frequency bands in the Fourier domain. FreSca demonstrably enhances existing
image editing methods without retraining. Excitingly, its effectiveness extends
to image understanding tasks such as depth estimation, yielding quantitative
gains across multiple datasets.
NeuralGS: Bridging Neural Fields and 3D Gaussian Splatting for Compact
  3D Representations
Authors: name: Zhenyu Tang, name: Chaoran Feng, name: Xinhua Cheng, name: Wangbo Yu, name: Junwu Zhang, name: Yuan Liu, name: Xiaoxiao Long, name: Wenping Wang, name: Li Yuan
Published: 2025-03-29 17:36:53

3D Gaussian Splatting (3DGS) demonstrates superior quality and rendering
speed, but with millions of 3D Gaussians and significant storage and
transmission costs. Recent 3DGS compression methods mainly concentrate on
compressing Scaffold-GS, achieving impressive performance but with an
additional voxel structure and a complex encoding and quantization strategy. In
this paper, we aim to develop a simple yet effective method called NeuralGS
that explores in another way to compress the original 3DGS into a compact
representation without the voxel structure and complex quantization strategies.
Our observation is that neural fields like NeRF can represent complex 3D scenes
with Multi-Layer Perceptron (MLP) neural networks using only a few megabytes.
Thus, NeuralGS effectively adopts the neural field representation to encode the
attributes of 3D Gaussians with MLPs, only requiring a small storage size even
for a large-scale scene. To achieve this, we adopt a clustering strategy and
fit the Gaussians with different tiny MLPs for each cluster, based on
importance scores of Gaussians as fitting weights. We experiment on multiple
datasets, achieving a 45-times average model size reduction without harming the
visual quality. The compression performance of our method on original 3DGS is
comparable to the dedicated Scaffold-GS-based compression methods, which
demonstrate the huge potential of directly compressing original 3DGS with
neural fields.
Whisper-LM: Improving ASR Models with Language Models for Low-Resource
  Languages
Authors: name: Xabier de Zuazo, name: Eva Navas, name: Ibon Saratxaga, name: Inma Hernáez Rioja
Published: 2025-03-30 18:03:52

Automatic speech recognition systems have undoubtedly advanced with the
integration of multilingual and multitask models such as Whisper, which have
shown a promising ability to understand and process speech across a wide range
of languages. Despite their robustness, these models often fall short in
handling the linguistic distinctions of minority languages. This study
addresses this gap by integrating traditional and novel language models with
fine-tuned Whisper models to raise their performance in less commonly studied
languages. Through rigorous fine-tuning and evaluation across multiple
datasets, we demonstrate substantial improvements in word error rate,
particularly in low-resource scenarios. Our approach not only does take
advantage of the extensive data Whisper was pre-trained on, but also
complements its linguistic adaptability by incorporating language models. We
obtained improvements up to 51\% for in-distribution datasets and up to 34\%
for out-of-distribution sentences using statistical language models, while
large language models provided moderate but consistently robust improvement
across diverse linguistic contexts. The findings reveal that, while the
integration reliably benefits all model sizes, the extent of improvement
varies, highlighting the importance of optimized language model parameters.
Finally, we emphasize the importance of selecting appropriate evaluation
parameters when reporting the results using transformer-based ASR models. In
summary, this research clears the way for more inclusive ASR technologies that
perform better across languages by enriching their linguistic knowledge. For
further implementation details of this study, the technical documentation and
source code are available at http://www.github.com/hitz-zentroa/whisper-lm.
Inference-Time Scaling for Generalist Reward Modeling
Authors: name: Zijun Liu, name: Peiyi Wang, name: Runxin Xu, name: Shirong Ma, name: Chong Ruan, name: Peng Li, name: Yang Liu, name: Yu Wu
Published: 2025-04-03 11:19:49

Reinforcement learning (RL) has been widely adopted in post-training for
large language models (LLMs) at scale. Recently, the incentivization of
reasoning capabilities in LLMs from RL indicates that proper learning
methods could enable effective inference-time scalability. A key challenge of
RL is to obtain accurate reward signals for LLMs in various domains beyond
verifiable questions or artificial rules. In this work, we investigate how to
improve reward modeling (RM) with more inference compute for general queries,
i.e. the inference-time scalability of generalist RM, and further,
how to improve the effectiveness of performance-compute scaling with proper
learning methods. For the RM approach, we adopt pointwise generative reward
modeling (GRM) to enable flexibility for different input types and potential
for inference-time scaling. For the learning method, we propose Self-Principled
Critique Tuning (SPCT) to foster scalable reward generation behaviors in GRMs
through online RL, to generate principles adaptively and critiques accurately,
resulting in DeepSeek-GRM models. Furthermore, for effective
inference-time scaling, we use parallel sampling to expand compute usage, and
introduce a meta RM to guide voting process for better scaling performance.
Empirically, we show that SPCT significantly improves the quality and
scalability of GRMs, outperforming existing methods and models in various RM
benchmarks without severe biases, and could achieve better performance compared
to training-time scaling. DeepSeek-GRM still meets challenges in some tasks,
which we believe can be addressed by future efforts in generalist reward
systems. The models will be released and open-sourced.