Orionnnn/hub-stats huggingface dataset:
SeFAR: Semi-supervised Fine-grained Action Recognition with Temporal
  Perturbation and Learning Stabilization
Authors: name: Yongle Huang, name: Haodong Chen, name: Zhenbang Xu, name: Zihan Jia, name: Haozhou Sun, name: Dian Shao
Published: 2025-01-02 13:12:12

Human action understanding is crucial for the advancement of multimodal
systems. While recent developments, driven by powerful large language models
(LLMs), aim to be general enough to cover a wide range of categories, they
often overlook the need for more specific capabilities. In this work, we
address the more challenging task of Fine-grained Action Recognition (FAR),
which focuses on detailed semantic labels within shorter temporal duration
(e.g., "salto backward tucked with 1 turn"). Given the high costs of annotating
fine-grained labels and the substantial data needed for fine-tuning LLMs, we
propose to adopt semi-supervised learning (SSL). Our framework, SeFAR,
incorporates several innovative designs to tackle these challenges.
Specifically, to capture sufficient visual details, we construct Dual-level
temporal elements as more effective representations, based on which we design a
new strong augmentation strategy for the Teacher-Student learning paradigm
through involving moderate temporal perturbation. Furthermore, to handle the
high uncertainty within the teacher model's predictions for FAR, we propose the
Adaptive Regulation to stabilize the learning process. Experiments show that
SeFAR achieves state-of-the-art performance on two FAR datasets, FineGym and
FineDiving, across various data scopes. It also outperforms other
semi-supervised methods on two classical coarse-grained datasets, UCF101 and
HMDB51. Further analysis and ablation studies validate the effectiveness of our
designs. Additionally, we show that the features extracted by our SeFAR could
largely promote the ability of multimodal foundation models to understand
fine-grained and domain-specific semantics.
2.5 Years in Class: A Multimodal Textbook for Vision-Language
  Pretraining
Authors: name: Wenqi Zhang, name: Hang Zhang, name: Xin Li, name: Jiashuo Sun, name: Yongliang Shen, name: Weiming Lu, name: Deli Zhao, name: Yueting Zhuang, name: Lidong Bing
Published: 2025-01-01 21:29:37

Compared to image-text pair data, interleaved corpora enable Vision-Language
Models (VLMs) to understand the world more naturally like humans. However, such
existing datasets are crawled from webpage, facing challenges like low
knowledge density, loose image-text relations, and poor logical coherence
between images. On the other hand, the internet hosts vast instructional videos
(e.g., online geometry courses) that are widely used by humans to learn
foundational subjects, yet these valuable resources remain underexplored in VLM
training. In this paper, we introduce a high-quality multimodal
textbook corpus with richer foundational knowledge for VLM pretraining. It
collects over 2.5 years of instructional videos, totaling 22,000 class hours.
We first use an LLM-proposed taxonomy to systematically gather instructional
videos. Then we progressively extract and refine visual (keyframes), audio
(ASR), and textual knowledge (OCR) from the videos, and organize as an
image-text interleaved corpus based on temporal order. Compared to its
counterparts, our video-centric textbook offers more coherent context, richer
knowledge, and better image-text alignment. Experiments demonstrate its superb
pretraining performance, particularly in knowledge- and reasoning-intensive
tasks like ScienceQA and MathVista. Moreover, VLMs pre-trained on our textbook
exhibit outstanding interleaved context awareness, leveraging visual and
textual cues in their few-shot context for task solving~Our code are
available at \url{https://github.com/DAMO-NLP-SG/multimodal_textbook}.
CodeElo: Benchmarking Competition-level Code Generation of LLMs with
  Human-comparable Elo Ratings
Authors: name: Shanghaoran Quan, name: Jiaxi Yang, name: Bowen Yu, name: Bo Zheng, name: Dayiheng Liu, name: An Yang, name: Xuancheng Ren, name: Bofei Gao, name: Yibo Miao, name: Yunlong Feng, name: Zekun Wang, name: Jian Yang, name: Zeyu Cui, name: Yang Fan, name: Yichang Zhang, name: Binyuan Hui, name: Junyang Lin
Published: 2025-01-02 13:49:00

With the increasing code reasoning capabilities of existing large language
models (LLMs) and breakthroughs in reasoning models like OpenAI o1 and o3,
there is a growing need to develop more challenging and comprehensive
benchmarks that effectively test their sophisticated competition-level coding
abilities. Existing benchmarks, like LiveCodeBench and USACO, fall short due to
the unavailability of private test cases, lack of support for special judges,
and misaligned execution environments. To bridge this gap, we introduce
CodeElo, a standardized competition-level code generation benchmark that
effectively addresses all these challenges for the first time. CodeElo
benchmark is mainly based on the official CodeForces platform and tries to
align with the platform as much as possible. We compile the recent six months
of contest problems on CodeForces with detailed information such as contest
divisions, problem difficulty ratings, and problem algorithm tags. We introduce
a unique judging method in which problems are submitted directly to the
platform and develop a reliable Elo rating calculation system that aligns with
the platform and is comparable with human participants but has lower variance.
By testing on our CodeElo, we provide the Elo ratings of 30 existing popular
open-source and 3 proprietary LLMs for the first time. The results show that
o1-mini and QwQ-32B-Preview stand out significantly, achieving Elo ratings of
1578 and 1261, respectively, while other models struggle even with the easiest
problems, placing in the lowest 20 percent among all human participants.
Detailed analysis experiments are also conducted to provide insights into
performance across algorithms and comparisons between using C++ and Python,
which can suggest directions for future studies.
VideoRefer Suite: Advancing Spatial-Temporal Object Understanding with
  Video LLM
Authors: name: Yuqian Yuan, name: Hang Zhang, name: Wentong Li, name: Zesen Cheng, name: Boqiang Zhang, name: Long Li, name: Xin Li, name: Deli Zhao, name: Wenqiao Zhang, name: Yueting Zhuang, name: Jianke Zhu, name: Lidong Bing
Published: 2024-12-31 18:56:46

Video Large Language Models (Video LLMs) have recently exhibited remarkable
capabilities in general video understanding. However, they mainly focus on
holistic comprehension and struggle with capturing fine-grained spatial and
temporal details. Besides, the lack of high-quality object-level video
instruction data and a comprehensive benchmark further hinders their
advancements. To tackle these challenges, we introduce the VideoRefer Suite to
empower Video LLM for finer-level spatial-temporal video understanding, i.e.,
enabling perception and reasoning on any objects throughout the video.
Specially, we thoroughly develop VideoRefer Suite across three essential
aspects: dataset, model, and benchmark. Firstly, we introduce a multi-agent
data engine to meticulously curate a large-scale, high-quality object-level
video instruction dataset, termed VideoRefer-700K. Next, we present the
VideoRefer model, which equips a versatile spatial-temporal object encoder to
capture precise regional and sequential representations. Finally, we
meticulously create a VideoRefer-Bench to comprehensively assess the
spatial-temporal understanding capability of a Video LLM, evaluating it across
various aspects. Extensive experiments and analyses demonstrate that our
VideoRefer model not only achieves promising performance on video referring
benchmarks but also facilitates general video understanding capabilities.
Dynamic Scaling of Unit Tests for Code Reward Modeling
Authors: name: Zeyao Ma, name: Xiaokang Zhang, name: Jing Zhang, name: Jifan Yu, name: Sijia Luo, name: Jie Tang
Published: 2025-01-02 04:33:31

Current large language models (LLMs) often struggle to produce accurate
responses on the first attempt for complex reasoning tasks like code
generation. Prior research tackles this challenge by generating multiple
candidate solutions and validating them with LLM-generated unit tests. The
execution results of unit tests serve as reward signals to identify correct
solutions. As LLMs always confidently make mistakes, these unit tests are not
reliable, thereby diminishing the quality of reward signals. Motivated by the
observation that scaling the number of solutions improves LLM performance, we
explore the impact of scaling unit tests to enhance reward signal quality. Our
pioneer experiment reveals a positive correlation between the number of unit
tests and reward signal quality, with greater benefits observed in more
challenging problems. Based on these insights, we propose CodeRM-8B, a
lightweight yet effective unit test generator that enables efficient and
high-quality unit test scaling. Additionally, we implement a dynamic scaling
mechanism that adapts the number of unit tests based on problem difficulty,
further improving efficiency. Experimental results show that our approach
significantly improves performance across various models on three benchmarks
(e.g., with gains of 18.43% for Llama3-8B and 3.42% for GPT-4o-mini on
HumanEval Plus).
VideoAnydoor: High-fidelity Video Object Insertion with Precise Motion
  Control
Authors: name: Yuanpeng Tu, name: Hao Luo, name: Xi Chen, name: Sihui Ji, name: Xiang Bai, name: Hengshuang Zhao
Published: 2025-01-02 18:59:54

Despite significant advancements in video generation, inserting a given
object into videos remains a challenging task. The difficulty lies in
preserving the appearance details of the reference object and accurately
modeling coherent motions at the same time. In this paper, we propose
VideoAnydoor, a zero-shot video object insertion framework with high-fidelity
detail preservation and precise motion control. Starting from a text-to-video
model, we utilize an ID extractor to inject the global identity and leverage a
box sequence to control the overall motion. To preserve the detailed appearance
and meanwhile support fine-grained motion control, we design a pixel warper. It
takes the reference image with arbitrary key-points and the corresponding
key-point trajectories as inputs. It warps the pixel details according to the
trajectories and fuses the warped features with the diffusion U-Net, thus
improving detail preservation and supporting users in manipulating the motion
trajectories. In addition, we propose a training strategy involving both videos
and static images with a reweight reconstruction loss to enhance insertion
quality. VideoAnydoor demonstrates significant superiority over existing
methods and naturally supports various downstream applications (e.g., talking
head generation, video virtual try-on, multi-region editing) without
task-specific fine-tuning.
A3: Android Agent Arena for Mobile GUI Agents
Authors: name: Yuxiang Chai, name: Hanhao Li, name: Jiayu Zhang, name: Liang Liu, name: Guozhi Wang, name: Shuai Ren, name: Siyuan Huang, name: Hongsheng Li
Published: 2025-01-02 09:03:56

AI agents have become increasingly prevalent in recent years, driven by
significant advancements in the field of large language models (LLMs). Mobile
GUI agents, a subset of AI agents, are designed to autonomously perform tasks
on mobile devices. While numerous studies have introduced agents, datasets, and
benchmarks to advance mobile GUI agent research, many existing datasets focus
on static frame evaluations and fail to provide a comprehensive platform for
assessing performance on real-world, in-the-wild tasks. To address this gap, we
present Android Agent Arena (A3), a novel evaluation platform. Unlike existing
in-the-wild systems, A3 offers: (1) meaningful and practical tasks, such as
real-time online information retrieval and operational instructions; (2) a
larger, more flexible action space, enabling compatibility with agents trained
on any dataset; and (3) automated business-level LLM-based evaluation process.
A3 includes 21 widely used general third-party apps and 201 tasks
representative of common user scenarios, providing a robust foundation for
evaluating mobile GUI agents in real-world situations and a new autonomous
evaluation process for less human labor and coding expertise. The project is
available at https://yuxiangchai.github.io/Android-Agent-Arena/.
MLLM-as-a-Judge for Image Safety without Human Labeling
Authors: name: Zhenting Wang, name: Shuming Hu, name: Shiyu Zhao, name: Xiaowen Lin, name: Felix Juefei-Xu, name: Zhuowei Li, name: Ligong Han, name: Harihar Subramanyam, name: Li Chen, name: Jianfa Chen, name: Nan Jiang, name: Lingjuan Lyu, name: Shiqing Ma, name: Dimitris N. Metaxas, name: Ankit Jain
Published: 2024-12-31 00:06:04

Image content safety has become a significant challenge with the rise of
visual media on online platforms. Meanwhile, in the age of AI-generated content
(AIGC), many image generation models are capable of producing harmful content,
such as images containing sexual or violent material. Thus, it becomes crucial
to identify such unsafe images based on established safety rules. Pre-trained
Multimodal Large Language Models (MLLMs) offer potential in this regard, given
their strong pattern recognition abilities. Existing approaches typically
fine-tune MLLMs with human-labeled datasets, which however brings a series of
drawbacks. First, relying on human annotators to label data following intricate
and detailed guidelines is both expensive and labor-intensive. Furthermore,
users of safety judgment systems may need to frequently update safety rules,
making fine-tuning on human-based annotation more challenging. This raises the
research question: Can we detect unsafe images by querying MLLMs in a zero-shot
setting using a predefined safety constitution (a set of safety rules)? Our
research showed that simply querying pre-trained MLLMs does not yield
satisfactory results. This lack of effectiveness stems from factors such as the
subjectivity of safety rules, the complexity of lengthy constitutions, and the
inherent biases in the models. To address these challenges, we propose a
MLLM-based method includes objectifying safety rules, assessing the relevance
between rules and images, making quick judgments based on debiased token
probabilities with logically complete yet simplified precondition chains for
safety rules, and conducting more in-depth reasoning with cascaded
chain-of-thought processes if necessary. Experiment results demonstrate that
our method is highly effective for zero-shot image safety judgment tasks.
Reconstruction vs. Generation: Taming Optimization Dilemma in Latent
  Diffusion Models
Authors: name: Jingfeng Yao, name: Xinggang Wang
Published: 2025-01-02 18:59:40

Latent diffusion models with Transformer architectures excel at generating
high-fidelity images. However, recent studies reveal an optimization dilemma in
this two-stage design: while increasing the per-token feature dimension in
visual tokenizers improves reconstruction quality, it requires substantially
larger diffusion models and more training iterations to achieve comparable
generation performance. Consequently, existing systems often settle for
sub-optimal solutions, either producing visual artifacts due to information
loss within tokenizers or failing to converge fully due to expensive
computation costs. We argue that this dilemma stems from the inherent
difficulty in learning unconstrained high-dimensional latent spaces. To address
this, we propose aligning the latent space with pre-trained vision foundation
models when training the visual tokenizers. Our proposed VA-VAE (Vision
foundation model Aligned Variational AutoEncoder) significantly expands the
reconstruction-generation frontier of latent diffusion models, enabling faster
convergence of Diffusion Transformers (DiT) in high-dimensional latent spaces.
To exploit the full potential of VA-VAE, we build an enhanced DiT baseline with
improved training strategies and architecture designs, termed LightningDiT. The
integrated system achieves state-of-the-art (SOTA) performance on ImageNet
256x256 generation with an FID score of 1.35 while demonstrating remarkable
training efficiency by reaching an FID score of 2.11 in just 64
epochs--representing an over 21 times convergence speedup compared to the
original DiT. Models and codes are available at:
https://github.com/hustvl/LightningDiT.
ProgCo: Program Helps Self-Correction of Large Language Models
Authors: name: Xiaoshuai Song, name: Yanan Wu, name: Weixun Wang, name: Jiaheng Liu, name: Wenbo Su, name: Bo Zheng
Published: 2025-01-02 13:59:20

Self-Correction aims to enable large language models (LLMs) to self-verify
and self-refine their initial responses without external feedback. However,
LLMs often fail to effectively self-verify and generate correct feedback,
further misleading refinement and leading to the failure of self-correction,
especially in complex reasoning tasks. In this paper, we propose Program-driven
Self-Correction (ProgCo). First, program-driven verification (ProgVe) achieves
complex verification logic and extensive validation through self-generated,
self-executing verification pseudo-programs. Then, program-driven refinement
(ProgRe) receives feedback from ProgVe, conducts dual reflection and refinement
on both responses and verification programs to mitigate misleading of incorrect
feedback in complex reasoning tasks. Experiments on three instruction-following
and mathematical benchmarks indicate that ProgCo achieves effective
self-correction, and can be further enhance performance when combined with real
program tools.
Are Vision-Language Models Truly Understanding Multi-vision Sensor?
Authors: name: Sangyun Chung, name: Youngjoon Yu, name: Youngchae Chee, name: Se Yeon Kim, name: Byung-Kwan Lee, name: Yong Man Ro
Published: 2024-12-30 06:44:25

Large-scale Vision-Language Models (VLMs) have advanced by aligning vision
inputs with text, significantly improving performance in computer vision tasks.
Moreover, for VLMs to be effectively utilized in real-world applications, an
understanding of diverse multi-vision sensor data, such as thermal, depth, and
X-ray information, is essential. However, we find that current VLMs process
multi-vision sensor images without deep understanding of sensor information,
disregarding each sensor's unique physical properties. This limitation
restricts their capacity to interpret and respond to complex questions
requiring multi-vision sensor reasoning. To address this, we propose a novel
Multi-vision Sensor Perception and Reasoning (MS-PR) benchmark, assessing VLMs
on their capacity for sensor-specific reasoning. Moreover, we introduce Diverse
Negative Attributes (DNA) optimization to enable VLMs to perform deep reasoning
on multi-vision sensor tasks, helping to bridge the core information gap
between images and sensor data. Extensive experimental results validate that
the proposed DNA method can significantly improve the multi-vision sensor
reasoning for VLMs.
VMix: Improving Text-to-Image Diffusion Model with Cross-Attention
  Mixing Control
Authors: name: Shaojin Wu, name: Fei Ding, name: Mengqi Huang, name: Wei Liu, name: Qian He
Published: 2024-12-30 08:47:25

While diffusion models show extraordinary talents in text-to-image
generation, they may still fail to generate highly aesthetic images. More
specifically, there is still a gap between the generated images and the
real-world aesthetic images in finer-grained dimensions including color,
lighting, composition, etc. In this paper, we propose Cross-Attention Value
Mixing Control (VMix) Adapter, a plug-and-play aesthetics adapter, to upgrade
the quality of generated images while maintaining generality across visual
concepts by (1) disentangling the input text prompt into the content
description and aesthetic description by the initialization of aesthetic
embedding, and (2) integrating aesthetic conditions into the denoising process
through value-mixed cross-attention, with the network connected by
zero-initialized linear layers. Our key insight is to enhance the aesthetic
presentation of existing diffusion models by designing a superior condition
control method, all while preserving the image-text alignment. Through our
meticulous design, VMix is flexible enough to be applied to community models
for better visual performance without retraining. To validate the effectiveness
of our method, we conducted extensive experiments, showing that VMix
outperforms other state-of-the-art methods and is compatible with other
community modules (e.g., LoRA, ControlNet, and IPAdapter) for image generation.
The project page is https://vmix-diffusion.github.io/VMix/.
HUNYUANPROVER: A Scalable Data Synthesis Framework and Guided Tree
  Search for Automated Theorem Proving
Authors: name: Yang Li, name: Dong Du, name: Linfeng Song, name: Chen Li, name: Weikang Wang, name: Tao Yang, name: Haitao Mi
Published: 2024-12-30 06:18:33

We introduce HunyuanProver, an language model finetuned from the Hunyuan 7B
for interactive automatic theorem proving with LEAN4. To alleviate the data
sparsity issue, we design a scalable framework to iterative synthesize data
with low cost. Besides, guided tree search algorithms are designed to enable
effective ``system 2 thinking`` of the prover. HunyuanProver achieves
state-of-the-art (SOTA) performances on major benchmarks. Specifically, it
achieves a pass of 68.4% on the miniF2F-test compared to 65.9%, the current
SOTA results. It proves 4 IMO statements (imo_1960_p2, imo_1962_p2},
imo_1964_p2 and imo_1983_p6) in miniF2F-test. To benefit the community, we will
open-source a dataset of 30k synthesized instances, where each instance
contains the original question in natural language, the converted statement by
autoformalization, and the proof by HunyuanProver.
OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse
  Task Synthesis
Authors: name: Qiushi Sun, name: Kanzhi Cheng, name: Zichen Ding, name: Chuanyang Jin, name: Yian Wang, name: Fangzhi Xu, name: Zhenyu Wu, name: Chengyou Jia, name: Liheng Chen, name: Zhoumianze Liu, name: Ben Kao, name: Guohao Li, name: Junxian He, name: Yu Qiao, name: Zhiyong Wu
Published: 2024-12-27 16:21:58

Graphical User Interface (GUI) agents powered by Vision-Language Models
(VLMs) have demonstrated human-like computer control capability. Despite their
utility in advancing digital automation, a critical bottleneck persists:
collecting high-quality trajectory data for training. Common practices for
collecting such data rely on human supervision or synthetic data generation
through executing pre-defined tasks, which are either resource-intensive or
unable to guarantee data quality. Moreover, these methods suffer from limited
data diversity and significant gaps between synthetic data and real-world
environments. To address these challenges, we propose OS-Genesis, a novel GUI
data synthesis pipeline that reverses the conventional trajectory collection
process. Instead of relying on pre-defined tasks, OS-Genesis enables agents
first to perceive environments and perform step-wise interactions, then
retrospectively derive high-quality tasks to enable trajectory-level
exploration. A trajectory reward model is then employed to ensure the quality
of the generated trajectories. We demonstrate that training GUI agents with
OS-Genesis significantly improves their performance on highly challenging
online benchmarks. In-depth analysis further validates OS-Genesis's efficiency
and its superior data quality and diversity compared to existing synthesis
methods. Our codes, data, and checkpoints are available at
https://qiushisun.github.io/OS-Genesis-Home/{OS-Genesis Homepage}.