Orionnnn/hub-stats huggingface dataset:
InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward
  Model
Authors: name: Yuhang Zang, name: Xiaoyi Dong, name: Pan Zhang, name: Yuhang Cao, name: Ziyu Liu, name: Shengyuan Ding, name: Shenxi Wu, name: Yubo Ma, name: Haodong Duan, name: Wenwei Zhang, name: Kai Chen, name: Dahua Lin, name: Jiaqi Wang
Published: 2025-01-21 18:47:32

Despite the promising performance of Large Vision Language Models (LVLMs) in
visual understanding, they occasionally generate incorrect outputs. While
reward models (RMs) with reinforcement learning or test-time scaling offer the
potential for improving generation quality, a critical gap remains: publicly
available multi-modal RMs for LVLMs are scarce, and the implementation details
of proprietary models are often unclear. We bridge this gap with
InternLM-XComposer2.5-Reward (IXC-2.5-Reward), a simple yet effective
multi-modal reward model that aligns LVLMs with human preferences. To ensure
the robustness and versatility of IXC-2.5-Reward, we set up a high-quality
multi-modal preference corpus spanning text, image, and video inputs across
diverse domains, such as instruction following, general understanding,
text-rich documents, mathematical reasoning, and video understanding.
IXC-2.5-Reward achieves excellent results on the latest multi-modal reward
model benchmark and shows competitive performance on text-only reward model
benchmarks. We further demonstrate three key applications of IXC-2.5-Reward:
(1) Providing a supervisory signal for RL training. We integrate IXC-2.5-Reward
with Proximal Policy Optimization (PPO) yields IXC-2.5-Chat, which shows
consistent improvements in instruction following and multi-modal open-ended
dialogue; (2) Selecting the best response from candidate responses for
test-time scaling; and (3) Filtering outlier or noisy samples from existing
image and video instruction tuning training data. To ensure reproducibility and
facilitate further research, we have open-sourced all model weights and
training recipes at https://github.com/InternLM/InternLM-XComposer
TokenVerse: Versatile Multi-concept Personalization in Token Modulation
  Space
Authors: name: Daniel Garibi, name: Shahar Yadin, name: Roni Paiss, name: Omer Tov, name: Shiran Zada, name: Ariel Ephrat, name: Tomer Michaeli, name: Inbar Mosseri, name: Tali Dekel
Published: 2025-01-21 15:49:29

We present TokenVerse -- a method for multi-concept personalization,
leveraging a pre-trained text-to-image diffusion model. Our framework can
disentangle complex visual elements and attributes from as little as a single
image, while enabling seamless plug-and-play generation of combinations of
concepts extracted from multiple images. As opposed to existing works,
TokenVerse can handle multiple images with multiple concepts each, and supports
a wide-range of concepts, including objects, accessories, materials, pose, and
lighting. Our work exploits a DiT-based text-to-image model, in which the input
text affects the generation through both attention and modulation (shift and
scale). We observe that the modulation space is semantic and enables localized
control over complex concepts. Building on this insight, we devise an
optimization-based framework that takes as input an image and a text
description, and finds for each word a distinct direction in the modulation
space. These directions can then be used to generate new images that combine
the learned concepts in a desired configuration. We demonstrate the
effectiveness of TokenVerse in challenging personalization settings, and
showcase its advantages over existing methods. project's webpage in
https://token-verse.github.io/
MSTS: A Multimodal Safety Test Suite for Vision-Language Models
Authors: name: Paul Röttger, name: Giuseppe Attanasio, name: Felix Friedrich, name: Janis Goldzycher, name: Alicia Parrish, name: Rishabh Bhardwaj, name: Chiara Di Bonaventura, name: Roman Eng, name: Gaia El Khoury Geagea, name: Sujata Goswami, name: Jieun Han, name: Dirk Hovy, name: Seogyeong Jeong, name: Paloma Jeretič, name: Flor Miriam Plaza-del-Arco, name: Donya Rooein, name: Patrick Schramowski, name: Anastassia Shaitarova, name: Xudong Shen, name: Richard Willats, name: Andrea Zugarini, name: Bertie Vidgen
Published: 2025-01-17 09:22:35

Vision-language models (VLMs), which process image and text inputs, are
increasingly integrated into chat assistants and other consumer AI
applications. Without proper safeguards, however, VLMs may give harmful advice
(e.g. how to self-harm) or encourage unsafe behaviours (e.g. to consume drugs).
Despite these clear hazards, little work so far has evaluated VLM safety and
the novel risks created by multimodal inputs. To address this gap, we introduce
MSTS, a Multimodal Safety Test Suite for VLMs. MSTS comprises 400 test prompts
across 40 fine-grained hazard categories. Each test prompt consists of a text
and an image that only in combination reveal their full unsafe meaning. With
MSTS, we find clear safety issues in several open VLMs. We also find some VLMs
to be safe by accident, meaning that they are safe because they fail to
understand even simple test prompts. We translate MSTS into ten languages,
showing non-English prompts to increase the rate of unsafe model responses. We
also show models to be safer when tested with text only rather than multimodal
prompts. Finally, we explore the automation of VLM safety assessments, finding
even the best safety classifiers to be lacking.
Go-with-the-Flow: Motion-Controllable Video Diffusion Models Using
  Real-Time Warped Noise
Authors: name: Ryan Burgert, name: Yuancheng Xu, name: Wenqi Xian, name: Oliver Pilarski, name: Pascal Clausen, name: Mingming He, name: Li Ma, name: Yitong Deng, name: Lingxiao Li, name: Mohsen Mousavi, name: Michael Ryoo, name: Paul Debevec, name: Ning Yu
Published: 2025-01-14 18:59:10

Generative modeling aims to transform random noise into structured outputs.
In this work, we enhance video diffusion models by allowing motion control via
structured latent noise sampling. This is achieved by just a change in data: we
pre-process training videos to yield structured noise. Consequently, our method
is agnostic to diffusion model design, requiring no changes to model
architectures or training pipelines. Specifically, we propose a novel noise
warping algorithm, fast enough to run in real time, that replaces random
temporal Gaussianity with correlated warped noise derived from optical flow
fields, while preserving the spatial Gaussianity. The efficiency of our
algorithm enables us to fine-tune modern video diffusion base models using
warped noise with minimal overhead, and provide a one-stop solution for a wide
range of user-friendly motion control: local object motion control, global
camera movement control, and motion transfer. The harmonization between
temporal coherence and spatial Gaussianity in our warped noise leads to
effective motion control while maintaining per-frame pixel quality. Extensive
experiments and user studies demonstrate the advantages of our method, making
it a robust and scalable approach for controlling motion in video diffusion
models. Video results are available on our webpage:
https://vgenai-netflix-eyeline-research.github.io/Go-with-the-Flow. Source code
and model checkpoints are available on GitHub:
https://github.com/VGenAI-Netflix-Eyeline-Research/Go-with-the-Flow.
Video Depth Anything: Consistent Depth Estimation for Super-Long Videos
Authors: name: Sili Chen, name: Hengkai Guo, name: Shengnan Zhu, name: Feihu Zhang, name: Zilong Huang, name: Jiashi Feng, name: Bingyi Kang
Published: 2025-01-21 18:53:30

Depth Anything has achieved remarkable success in monocular depth estimation
with strong generalization ability. However, it suffers from temporal
inconsistency in videos, hindering its practical applications. Various methods
have been proposed to alleviate this issue by leveraging video generation
models or introducing priors from optical flow and camera poses. Nonetheless,
these methods are only applicable to short videos (< 10 seconds) and require a
trade-off between quality and computational efficiency. We propose Video Depth
Anything for high-quality, consistent depth estimation in super-long videos
(over several minutes) without sacrificing efficiency. We base our model on
Depth Anything V2 and replace its head with an efficient spatial-temporal head.
We design a straightforward yet effective temporal consistency loss by
constraining the temporal depth gradient, eliminating the need for additional
geometric priors. The model is trained on a joint dataset of video depth and
unlabeled images, similar to Depth Anything V2. Moreover, a novel
key-frame-based strategy is developed for long video inference. Experiments
show that our model can be applied to arbitrarily long videos without
compromising quality, consistency, or generalization ability. Comprehensive
evaluations on multiple video benchmarks demonstrate that our approach sets a
new state-of-the-art in zero-shot video depth estimation. We offer models of
different scales to support a range of scenarios, with our smallest model
capable of real-time performance at 30 FPS.
EMO2: End-Effector Guided Audio-Driven Avatar Video Generation
Authors: name: Linrui Tian, name: Siqi Hu, name: Qi Wang, name: Bang Zhang, name: Liefeng Bo
Published: 2025-01-18 07:51:29

In this paper, we propose a novel audio-driven talking head method capable of
simultaneously generating highly expressive facial expressions and hand
gestures. Unlike existing methods that focus on generating full-body or
half-body poses, we investigate the challenges of co-speech gesture generation
and identify the weak correspondence between audio features and full-body
gestures as a key limitation. To address this, we redefine the task as a
two-stage process. In the first stage, we generate hand poses directly from
audio input, leveraging the strong correlation between audio signals and hand
movements. In the second stage, we employ a diffusion model to synthesize video
frames, incorporating the hand poses generated in the first stage to produce
realistic facial expressions and body movements. Our experimental results
demonstrate that the proposed method outperforms state-of-the-art approaches,
such as CyberHost and Vlogger, in terms of both visual quality and
synchronization accuracy. This work provides a new perspective on audio-driven
gesture generation and a robust framework for creating expressive and natural
talking head animations.
Hunyuan3D 2.0: Scaling Diffusion Models for High Resolution Textured 3D
  Assets Generation
Authors: name: Zibo Zhao, name: Zeqiang Lai, name: Qingxiang Lin, name: Yunfei Zhao, name: Haolin Liu, name: Shuhui Yang, name: Yifei Feng, name: Mingxin Yang, name: Sheng Zhang, name: Xianghui Yang, name: Huiwen Shi, name: Sicong Liu, name: Junta Wu, name: Yihang Lian, name: Fan Yang, name: Ruining Tang, name: Zebin He, name: Xinzhou Wang, name: Jian Liu, name: Xuhui Zuo, name: Zhuo Chen, name: Biwen Lei, name: Haohan Weng, name: Jing Xu, name: Yiling Zhu, name: Xinhai Liu, name: Lixin Xu, name: Changrong Hu, name: Tianyu Huang, name: Lifu Wang, name: Jihong Zhang, name: Meng Chen, name: Liang Dong, name: Yiwen Jia, name: Yulin Cai, name: Jiaao Yu, name: Yixuan Tang, name: Hao Zhang, name: Zheng Ye, name: Peng He, name: Runzhou Wu, name: Chao Zhang, name: Yonghao Tan, name: Jie Xiao, name: Yangyu Tao, name: Jianchen Zhu, name: Jinbao Xue, name: Kai Liu, name: Chongqing Zhao, name: Xinming Wu, name: Zhichao Hu, name: Lei Qin, name: Jianbing Peng, name: Zhan Li, name: Minghui Chen, name: Xipeng Zhang, name: Lin Niu, name: Paige Wang, name: Yingkai Wang, name: Haozhao Kuang, name: Zhongyi Fan, name: Xu Zheng, name: Weihao Zhuang, name: YingPing He, name: Tian Liu, name: Yong Yang, name: Di Wang, name: Yuhong Liu, name: Jie Jiang, name: Jingwei Huang, name: Chunchao Guo
Published: 2025-01-21 15:16:54

We present Hunyuan3D 2.0, an advanced large-scale 3D synthesis system for
generating high-resolution textured 3D assets. This system includes two
foundation components: a large-scale shape generation model -- Hunyuan3D-DiT,
and a large-scale texture synthesis model -- Hunyuan3D-Paint. The shape
generative model, built on a scalable flow-based diffusion transformer, aims to
create geometry that properly aligns with a given condition image, laying a
solid foundation for downstream applications. The texture synthesis model,
benefiting from strong geometric and diffusion priors, produces high-resolution
and vibrant texture maps for either generated or hand-crafted meshes.
Furthermore, we build Hunyuan3D-Studio -- a versatile, user-friendly production
platform that simplifies the re-creation process of 3D assets. It allows both
professional and amateur users to manipulate or even animate their meshes
efficiently. We systematically evaluate our models, showing that Hunyuan3D 2.0
outperforms previous state-of-the-art models, including the open-source models
and closed-source models in geometry details, condition alignment, texture
quality, and etc. Hunyuan3D 2.0 is publicly released in order to fill the gaps
in the open-source 3D community for large-scale foundation generative models.
The code and pre-trained weights of our models are available at:
https://github.com/Tencent/Hunyuan3D-2
Agent-R: Training Language Model Agents to Reflect via Iterative
  Self-Training
Authors: name: Siyu Yuan, name: Zehui Chen, name: Zhiheng Xi, name: Junjie Ye, name: Zhengyin Du, name: Jiecao Chen
Published: 2025-01-20 11:46:04

Large Language Models (LLMs) agents are increasingly pivotal for addressing
complex tasks in interactive environments. Existing work mainly focuses on
enhancing performance through behavior cloning from stronger experts, yet such
approaches often falter in real-world applications, mainly due to the inability
to recover from errors. However, step-level critique data is difficult and
expensive to collect. Automating and dynamically constructing self-critique
datasets is thus crucial to empowering models with intelligent agent
capabilities. In this work, we propose an iterative self-training framework,
Agent-R, that enables language Agent to Reflect on the fly. Unlike traditional
methods that reward or penalize actions based on correctness, Agent-R leverages
MCTS to construct training data that recover correct trajectories from
erroneous ones. A key challenge of agent reflection lies in the necessity for
timely revision rather than waiting until the end of a rollout. To address
this, we introduce a model-guided critique construction mechanism: the actor
model identifies the first error step (within its current capability) in a
failed trajectory. Starting from it, we splice it with the adjacent correct
path, which shares the same parent node in the tree. This strategy enables the
model to learn reflection based on its current policy, therefore yielding
better learning efficiency. To further explore the scalability of this
self-improvement paradigm, we investigate iterative refinement of both error
correction capabilities and dataset construction. Our findings demonstrate that
Agent-R continuously improves the model's ability to recover from errors and
enables timely error correction. Experiments on three interactive environments
show that Agent-R effectively equips agents to correct erroneous actions while
avoiding loops, achieving superior performance compared to baseline methods
(+5.59%).
Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks
Authors: name: Zhenhailong Wang, name: Haiyang Xu, name: Junyang Wang, name: Xi Zhang, name: Ming Yan, name: Ji Zhang, name: Fei Huang, name: Heng Ji
Published: 2025-01-20 20:35:46

Smartphones have become indispensable in modern life, yet navigating complex
tasks on mobile devices often remains frustrating. Recent advancements in large
multimodal model (LMM)-based mobile agents have demonstrated the ability to
perceive and act in mobile environments. However, current approaches face
significant limitations: they fall short in addressing real-world human needs,
struggle with reasoning-intensive and long-horizon tasks, and lack mechanisms
to learn and improve from prior experiences. To overcome these challenges, we
introduce Mobile-Agent-E, a hierarchical multi-agent framework capable of
self-evolution through past experience. By hierarchical, we mean an explicit
separation of high-level planning and low-level action execution. The framework
comprises a Manager, responsible for devising overall plans by breaking down
complex tasks into subgoals, and four subordinate agents--Perceptor, Operator,
Action Reflector, and Notetaker--which handle fine-grained visual perception,
immediate action execution, error verification, and information aggregation,
respectively. Mobile-Agent-E also features a novel self-evolution module which
maintains a persistent long-term memory comprising Tips and Shortcuts. Tips are
general guidance and lessons learned from prior tasks on how to effectively
interact with the environment. Shortcuts are reusable, executable sequences of
atomic operations tailored for specific subroutines. The inclusion of Tips and
Shortcuts facilitates continuous refinement in performance and efficiency.
Alongside this framework, we introduce Mobile-Eval-E, a new benchmark featuring
complex mobile tasks requiring long-horizon, multi-app interactions. Empirical
results show that Mobile-Agent-E achieves a 22% absolute improvement over
previous state-of-the-art approaches across three foundation model backbones.
Project page: https://x-plug.github.io/MobileAgent.
Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in
  Realistic Environments
Authors: name: Hongjin Su, name: Ruoxi Sun, name: Jinsung Yoon, name: Pengcheng Yin, name: Tao Yu, name: Sercan Ö. Arık
Published: 2025-01-18 22:34:41

Autonomous agents powered by large language models (LLMs) have the potential
to enhance human capabilities, assisting with digital tasks from sending emails
to performing data analysis. The abilities of existing LLMs at such tasks are
often hindered by the lack of high-quality agent data from the corresponding
environments they interact with. We propose Learn-by-interact, a data-centric
framework to adapt LLM agents to any given environments without human
annotations. Learn-by-interact synthesizes trajectories of agent-environment
interactions based on documentations, and constructs instructions by
summarizing or abstracting the interaction histories, a process called backward
construction. We assess the quality of our synthetic data by using them in both
training-based scenarios and training-free in-context learning (ICL), where we
craft innovative retrieval approaches optimized for agents. Extensive
experiments on SWE-bench, WebArena, OSWorld and Spider2-V spanning across
realistic coding, web, and desktop environments show the effectiveness of
Learn-by-interact in various downstream agentic tasks -- baseline results are
improved by up to 12.2\% for ICL with Claude-3.5 and 19.5\% for training with
Codestral-22B. We further demonstrate the critical role of backward
construction, which provides up to 14.0\% improvement for training. Our
ablation studies demonstrate the efficiency provided by our synthesized data in
ICL and the superiority of our retrieval pipeline over alternative approaches
like conventional retrieval-augmented generation (RAG). We expect that
Learn-by-interact will serve as a foundation for agent data synthesis as LLMs
are increasingly deployed at real-world environments.
UI-TARS: Pioneering Automated GUI Interaction with Native Agents
Authors: name: Yujia Qin, name: Yining Ye, name: Junjie Fang, name: Haoming Wang, name: Shihao Liang, name: Shizuo Tian, name: Junda Zhang, name: Jiahao Li, name: Yunxin Li, name: Shijue Huang, name: Wanjun Zhong, name: Kuanye Li, name: Jiale Yang, name: Yu Miao, name: Woyu Lin, name: Longxiang Liu, name: Xu Jiang, name: Qianli Ma, name: Jingyu Li, name: Xiaojun Xiao, name: Kai Cai, name: Chuang Li, name: Yaowei Zheng, name: Chaolin Jin, name: Chen Li, name: Xiao Zhou, name: Minchao Wang, name: Haoli Chen, name: Zhaojian Li, name: Haihua Yang, name: Haifeng Liu, name: Feng Lin, name: Tao Peng, name: Xin Liu, name: Guang Shi
Published: 2025-01-21 17:48:10

This paper introduces UI-TARS, a native GUI agent model that solely perceives
the screenshots as input and performs human-like interactions (e.g., keyboard
and mouse operations). Unlike prevailing agent frameworks that depend on
heavily wrapped commercial models (e.g., GPT-4o) with expert-crafted prompts
and workflows, UI-TARS is an end-to-end model that outperforms these
sophisticated frameworks. Experiments demonstrate its superior performance:
UI-TARS achieves SOTA performance in 10+ GUI agent benchmarks evaluating
perception, grounding, and GUI task execution. Notably, in the OSWorld
benchmark, UI-TARS achieves scores of 24.6 with 50 steps and 22.7 with 15
steps, outperforming Claude (22.0 and 14.9 respectively). In AndroidWorld,
UI-TARS achieves 46.6, surpassing GPT-4o (34.5). UI-TARS incorporates several
key innovations: (1) Enhanced Perception: leveraging a large-scale dataset of
GUI screenshots for context-aware understanding of UI elements and precise
captioning; (2) Unified Action Modeling, which standardizes actions into a
unified space across platforms and achieves precise grounding and interaction
through large-scale action traces; (3) System-2 Reasoning, which incorporates
deliberate reasoning into multi-step decision making, involving multiple
reasoning patterns such as task decomposition, reflection thinking, milestone
recognition, etc. (4) Iterative Training with Reflective Online Traces, which
addresses the data bottleneck by automatically collecting, filtering, and
reflectively refining new interaction traces on hundreds of virtual machines.
Through iterative training and reflection tuning, UI-TARS continuously learns
from its mistakes and adapts to unforeseen situations with minimal human
intervention. We also analyze the evolution path of GUI agents to guide the
further development of this domain.