Orionnnn/hub-stats huggingface dataset:
STEVE: AStep Verification Pipeline for Computer-use Agent Training
Authors: name: Fanbin Lu, name: Zhisheng Zhong, name: Ziqin Wei, name: Shu Liu, name: Chi-Wing Fu, name: Jiaya Jia
Published: 2025-03-16 14:53:43

Developing AI agents to autonomously manipulate graphical user interfaces is
a long challenging task. Recent advances in data scaling law inspire us to
train computer-use agents with a scaled instruction set, yet using behavior
cloning to train agents still requires immense high-quality trajectories. To
meet the scalability need, we designed STEVE, a step verification pipeline for
computer-use agent training. First, we establish a large instruction set for
computer-use agents and collect trajectory data with some suboptimal agents.
GPT-4o is used to verify the correctness of each step in the trajectories based
on the screens before and after the action execution, assigning each step with
a binary label. Last, we adopt the Kahneman and Tversky Optimization to
optimize the agent from the binary stepwise labels. Extensive experiments
manifest that our agent outperforms supervised finetuning by leveraging both
positive and negative actions within a trajectory. Also, STEVE enables us to
train a 7B vision-language model as a computer-use agent, achieving leading
performance in the challenging live desktop environment WinAgentArena with
great efficiency at a reduced cost. Code and data:
https://github.com/FanbinLu/STEVE.
TULIP: Towards Unified Language-Image Pretraining
Authors: name: Zineng Tang, name: Long Lian, name: Seun Eisape, name: XuDong Wang, name: Roei Herzig, name: Adam Yala, name: Alane Suhr, name: Trevor Darrell, name: David M. Chan
Published: 2025-03-19 17:58:57

Despite the recent success of image-text contrastive models like CLIP and
SigLIP, these models often struggle with vision-centric tasks that demand
high-fidelity image understanding, such as counting, depth estimation, and
fine-grained object recognition. These models, by performing language
alignment, tend to prioritize high-level semantics over visual understanding,
weakening their image understanding. On the other hand, vision-focused models
are great at processing visual information but struggle to understand language,
limiting their flexibility for language-driven tasks. In this work, we
introduce TULIP, an open-source, drop-in replacement for existing CLIP-like
models. Our method leverages generative data augmentation, enhanced image-image
and text-text contrastive learning, and image/text reconstruction
regularization to learn fine-grained visual features while preserving global
semantic alignment. Our approach, scaling to over 1B parameters, outperforms
existing state-of-the-art (SOTA) models across multiple benchmarks,
establishing a new SOTA zero-shot performance on ImageNet-1K, delivering up to
a 2times enhancement over SigLIP on RxRx1 in linear probing for few-shot
classification, and improving vision-language models, achieving over 3times
higher scores than SigLIP on MMVP. Our code/checkpoints are available at
https://tulip-berkeley.github.io
Cube: A Roblox View of 3D Intelligence
Authors: name: Foundation AI Team, name: Kiran Bhat, name: Nishchaie Khanna, name: Karun Channa, name: Tinghui Zhou, name: Yiheng Zhu, name: Xiaoxia Sun, name: Charles Shang, name: Anirudh Sudarshan, name: Maurice Chu, name: Daiqing Li, name: Kangle Deng, name: Jean-Philippe Fauconnier, name: Tijmen Verhulsdonck, name: Maneesh Agrawala, name: Kayvon Fatahalian, name: Alexander Weiss, name: Christian Reiser, name: Ravi Kiran Chirravuri, name: Ravali Kandur, name: Alejandro Pelaez, name: Akash Garg, name: Michael Palleschi, name: Jessica Wang, name: Skylar Litz, name: Leon Liu, name: Anying Li, name: David Harmon, name: Derek Liu, name: Liangjun Feng, name: Denis Goupil, name: Lukas Kuczynski, name: Jihyun Yoon, name: Naveen Marri, name: Peiye Zhuang, name: Yinan Zhang, name: Brian Yin, name: Haomiao Jiang, name: Marcel van Workum, name: Thomas Lane, name: Bryce Erickson, name: Salil Pathare, name: Kyle Price, name: Anupam Singh, name: David Baszucki
Published: 2025-03-19 17:52:17

Foundation models trained on vast amounts of data have demonstrated
remarkable reasoning and generation capabilities in the domains of text,
images, audio and video. Our goal at Roblox is to build such a foundation model
for 3D intelligence, a model that can support developers in producing all
aspects of a Roblox experience, from generating 3D objects and scenes to
rigging characters for animation to producing programmatic scripts describing
object behaviors. We discuss three key design requirements for such a 3D
foundation model and then present our first step towards building such a model.
We expect that 3D geometric shapes will be a core data type and describe our
solution for 3D shape tokenizer. We show how our tokenization scheme can be
used in applications for text-to-shape generation, shape-to-text generation and
text-to-scene generation. We demonstrate how these applications can collaborate
with existing large language models (LLMs) to perform scene analysis and
reasoning. We conclude with a discussion outlining our path to building a fully
unified foundation model for 3D intelligence.
ViSpeak: Visual Instruction Feedback in Streaming Videos
Authors: name: Shenghao Fu, name: Qize Yang, name: Yuan-Ming Li, name: Yi-Xing Peng, name: Kun-Yu Lin, name: Xihan Wei, name: Jian-Fang Hu, name: Xiaohua Xie, name: Wei-Shi Zheng
Published: 2025-03-17 03:05:31

Recent advances in Large Multi-modal Models (LMMs) are primarily focused on
offline video understanding. Instead, streaming video understanding poses great
challenges to recent models due to its time-sensitive, omni-modal and
interactive characteristics. In this work, we aim to extend the streaming video
understanding from a new perspective and propose a novel task named Visual
Instruction Feedback in which models should be aware of visual contents and
learn to extract instructions from them. For example, when users wave their
hands to agents, agents should recognize the gesture and start conversations
with welcome information. Thus, following instructions in visual modality
greatly enhances user-agent interactions. To facilitate research, we define
seven key subtasks highly relevant to visual modality and collect the
ViSpeak-Instruct dataset for training and the ViSpeak-Bench for evaluation.
Further, we propose the ViSpeak model, which is a SOTA streaming video
understanding LMM with GPT-4o-level performance on various streaming video
understanding benchmarks. After finetuning on our ViSpeak-Instruct dataset,
ViSpeak is equipped with basic visual instruction feedback ability, serving as
a solid baseline for future research.
PyGDA: A Python Library for Graph Domain Adaptation
Authors: name: Zhen Zhang, name: Meihan Liu, name: Bingsheng He
Published: 2025-03-13 11:52:23

Graph domain adaptation has emerged as a promising approach to facilitate
knowledge transfer across different domains. Recently, numerous models have
been proposed to enhance their generalization capabilities in this field.
However, there is still no unified library that brings together existing
techniques and simplifies their implementation. To fill this gap, we introduce
PyGDA, an open-source Python library tailored for graph domain adaptation. As
the first comprehensive library in this area, PyGDA covers more than 20 widely
used graph domain adaptation methods together with different types of graph
datasets. Specifically, PyGDA offers modular components, enabling users to
seamlessly build custom models with a variety of commonly used utility
functions. To handle large-scale graphs, PyGDA includes support for features
such as sampling and mini-batch processing, ensuring efficient computation. In
addition, PyGDA also includes comprehensive performance benchmarks and
well-documented user-friendly API for both researchers and practitioners. To
foster convenient accessibility, PyGDA is released under the MIT license at
https://github.com/pygda-team/pygda, and the API documentation is
https://pygda.readthedocs.io/en/stable/.
Atlas: Multi-Scale Attention Improves Long Context Image Modeling
Authors: name: Kumar Krishna Agrawal, name: Long Lian, name: Longchao Liu, name: Natalia Harguindeguy, name: Boyi Li, name: Alexander Bick, name: Maggie Chung, name: Trevor Darrell, name: Adam Yala
Published: 2025-03-16 04:52:13

Efficiently modeling massive images is a long-standing challenge in machine
learning. To this end, we introduce Multi-Scale Attention (MSA). MSA relies on
two key ideas, (i) multi-scale representations (ii) bi-directional cross-scale
communication. MSA creates O(log N) scales to represent the image across
progressively coarser features and leverages cross-attention to propagate
information across scales. We then introduce Atlas, a novel neural network
architecture based on MSA. We demonstrate that Atlas significantly improves the
compute-performance tradeoff of long-context image modeling in a
high-resolution variant of ImageNet 100. At 1024px resolution, Atlas-B achieves
91.04% accuracy, comparable to ConvNext-B (91.92%) while being 4.3x faster.
Atlas is 2.95x faster and 7.38% better than FasterViT, 2.25x faster and 4.96%
better than LongViT. In comparisons against MambaVision-S, we find Atlas-S
achieves 5%, 16% and 32% higher accuracy at 1024px, 2048px and 4096px
respectively, while obtaining similar runtimes. Code for reproducing our
experiments and pretrained models is available at
https://github.com/yalalab/atlas.
AudioX: Diffusion Transformer for Anything-to-Audio Generation
Authors: name: Zeyue Tian, name: Yizhu Jin, name: Zhaoyang Liu, name: Ruibin Yuan, name: Xu Tan, name: Qifeng Chen, name: Wei Xue, name: Yike Guo
Published: 2025-03-13 16:30:59

Audio and music generation have emerged as crucial tasks in many
applications, yet existing approaches face significant limitations: they
operate in isolation without unified capabilities across modalities, suffer
from scarce high-quality, multi-modal training data, and struggle to
effectively integrate diverse inputs. In this work, we propose AudioX, a
unified Diffusion Transformer model for Anything-to-Audio and Music Generation.
Unlike previous domain-specific models, AudioX can generate both general audio
and music with high quality, while offering flexible natural language control
and seamless processing of various modalities including text, video, image,
music, and audio. Its key innovation is a multi-modal masked training strategy
that masks inputs across modalities and forces the model to learn from masked
inputs, yielding robust and unified cross-modal representations. To address
data scarcity, we curate two comprehensive datasets: vggsound-caps with 190K
audio captions based on the VGGSound dataset, and V2M-caps with 6 million music
captions derived from the V2M dataset. Extensive experiments demonstrate that
AudioX not only matches or outperforms state-of-the-art specialized models, but
also offers remarkable versatility in handling diverse input modalities and
generation tasks within a unified architecture. The code and datasets will be
available at https://zeyuet.github.io/AudioX/
Learning to Inference Adaptively for Multimodal Large Language Models
Authors: name: Zhuoyan Xu, name: Khoi Duc Nguyen, name: Preeti Mukherjee, name: Saurabh Bagchi, name: Somali Chaterji, name: Yingyu Liang, name: Yin Li
Published: 2025-03-13 21:39:38

Multimodal Large Language Models (MLLMs) have shown impressive capabilities
in reasoning, yet come with substantial computational cost, limiting their
deployment in resource-constrained settings. Despite recent efforts on
improving the efficiency of MLLMs, prior solutions fall short in responding to
varying runtime conditions, in particular changing resource availability (e.g.,
contention due to the execution of other programs on the device). To bridge
this gap, we introduce AdaLLaVA, an adaptive inference framework that learns to
dynamically reconfigure operations in an MLLM during inference, accounting for
the input data and a latency budget. We conduct extensive experiments across
benchmarks involving question-answering, reasoning, and hallucination. Our
results show that AdaLLaVA effectively adheres to input latency budget,
achieving varying accuracy and latency tradeoffs at runtime. Further, we
demonstrate that AdaLLaVA adapts to both input latency and content, can be
integrated with token selection for enhanced efficiency, and generalizes across
MLLMs. Our project webpage with code release is at
https://zhuoyan-xu.github.io/ada-llava/.
CoLMDriver: LLM-based Negotiation Benefits Cooperative Autonomous
  Driving
Authors: name: Changxing Liu, name: Genjia Liu, name: Zijun Wang, name: Jinchang Yang, name: Siheng Chen
Published: 2025-03-11 17:58:42

Vehicle-to-vehicle (V2V) cooperative autonomous driving holds great promise
for improving safety by addressing the perception and prediction uncertainties
inherent in single-agent systems. However, traditional cooperative methods are
constrained by rigid collaboration protocols and limited generalization to
unseen interactive scenarios. While LLM-based approaches offer generalized
reasoning capabilities, their challenges in spatial planning and unstable
inference latency hinder their direct application in cooperative driving. To
address these limitations, we propose CoLMDriver, the first full-pipeline
LLM-based cooperative driving system, enabling effective language-based
negotiation and real-time driving control. CoLMDriver features a parallel
driving pipeline with two key components: (i) an LLM-based negotiation module
under an actor-critic paradigm, which continuously refines cooperation policies
through feedback from previous decisions of all vehicles; and (ii) an
intention-guided waypoint generator, which translates negotiation outcomes into
executable waypoints. Additionally, we introduce InterDrive, a CARLA-based
simulation benchmark comprising 10 challenging interactive driving scenarios
for evaluating V2V cooperation. Experimental results demonstrate that
CoLMDriver significantly outperforms existing approaches, achieving an 11%
higher success rate across diverse highly interactive V2V driving scenarios.
Code will be released on https://github.com/cxliu0314/CoLMDriver.
EvalTree: Profiling Language Model Weaknesses via Hierarchical
  Capability Trees
Authors: name: Zhiyuan Zeng, name: Yizhong Wang, name: Hannaneh Hajishirzi, name: Pang Wei Koh
Published: 2025-03-11 21:12:48

An ideal model evaluation should achieve two goals: identifying where the
model fails and providing actionable improvement guidance. Toward these goals
for Language Model (LM) evaluations, we formulate the problem of generating a
weakness profile, a set of weaknesses expressed in natural language, given an
LM's performance on every individual instance in a benchmark. We introduce a
suite of quantitative assessments to compare different weakness profiling
methods. We also propose a weakness profiling method EvalTree. It constructs a
capability tree where each node represents a capability described in natural
language and is linked to a subset of benchmark instances that specifically
evaluate this capability; it then extracts nodes where the LM performs poorly
to generate a weakness profile. On the MATH and WildChat benchmarks, we show
that EvalTree outperforms baseline weakness profiling methods by identifying
weaknesses more precisely and comprehensively. Weakness profiling further
enables weakness-guided data collection, and training data collection guided by
EvalTree-identified weaknesses improves LM performance more than other data
collection strategies. We also show how EvalTree exposes flaws in Chatbot
Arena's human-voter-based evaluation practice. To facilitate future work, we
release our code and an interface that allows practitioners to interactively
explore the capability trees built by EvalTree.
MeshFleet: Filtered and Annotated 3D Vehicle Dataset for Domain Specific
  Generative Modeling
Authors: name: Damian Boborzi, name: Phillip Mueller, name: Jonas Emrich, name: Dominik Schmid, name: Sebastian Mueller, name: Lars Mikelsons
Published: 2025-03-18 08:09:24

Generative models have recently made remarkable progress in the field of 3D
objects. However, their practical application in fields like engineering
remains limited since they fail to deliver the accuracy, quality, and
controllability needed for domain-specific tasks. Fine-tuning large generative
models is a promising perspective for making these models available in these
fields. Creating high-quality, domain-specific 3D datasets is crucial for
fine-tuning large generative models, yet the data filtering and annotation
process remains a significant bottleneck. We present MeshFleet, a filtered and
annotated 3D vehicle dataset extracted from Objaverse-XL, the most extensive
publicly available collection of 3D objects. Our approach proposes a pipeline
for automated data filtering based on a quality classifier. This classifier is
trained on a manually labeled subset of Objaverse, incorporating DINOv2 and
SigLIP embeddings, refined through caption-based analysis and uncertainty
estimation. We demonstrate the efficacy of our filtering method through a
comparative analysis against caption and image aesthetic score-based techniques
and fine-tuning experiments with SV3D, highlighting the importance of targeted
data selection for domain-specific 3D generative modeling.
Towards Self-Improving Systematic Cognition for Next-Generation
  Foundation MLLMs
Authors: name: Xiaoying Zhang, name: Da Peng, name: Yipeng Zhang, name: Zonghao Guo, name: Chengyue Wu, name: Chi Chen, name: Wei Ke, name: Helen Meng, name: Maosong Sun
Published: 2025-03-16 00:25:13

Despite their impressive capabilities, Multimodal Large Language Models
(MLLMs) face challenges with fine-grained perception and complex reasoning.
Prevalent multimodal pre-training approaches focus on enhancing perception by
training on high-quality image captions due to the extremely high cost of
collecting chain-of-thought (CoT) reasoning data for improving reasoning. While
leveraging advanced MLLMs for caption generation enhances scalability, the
outputs often lack comprehensiveness and accuracy. In this paper, we introduce
Self-Improving cognition (SIcog), a self-learning framework designed to
construct next-generation foundation MLLMs by enhancing their systematic
cognitive capabilities through multimodal pre-training with self-generated
data. Specifically, we propose Chain-of-Description, an approach that improves
an MLLM's systematic perception by enabling step-by-step visual understanding,
ensuring greater comprehensiveness and accuracy. Additionally, we adopt a
structured CoT reasoning technique to enable MLLMs to integrate in-depth
multimodal reasoning. To construct a next-generation foundation MLLM with
self-improved cognition, SIcog first equips an MLLM with systematic perception
and reasoning abilities using minimal external annotations. The enhanced models
then generate detailed captions and CoT reasoning data, which are further
curated through self-consistency. This curated data is ultimately used for
multimodal pre-training to develop next-generation foundation models. Extensive
experiments on both low- and high-resolution MLLMs across diverse benchmarks
demonstrate that, with merely 213K self-generated pre-training samples, SIcog
produces next-generation foundation MLLMs with significantly improved
cognition, achieving benchmark-leading performance compared to prevalent
pre-training approaches.
FlexWorld: Progressively Expanding 3D Scenes for Flexiable-View
  Synthesis
Authors: name: Luxi Chen, name: Zihan Zhou, name: Min Zhao, name: Yikai Wang, name: Ge Zhang, name: Wenhao Huang, name: Hao Sun, name: Ji-Rong Wen, name: Chongxuan Li
Published: 2025-03-17 15:18:38

Generating flexible-view 3D scenes, including 360{\deg} rotation and zooming,
from single images is challenging due to a lack of 3D data. To this end, we
introduce FlexWorld, a novel framework consisting of two key components: (1) a
strong video-to-video (V2V) diffusion model to generate high-quality novel view
images from incomplete input rendered from a coarse scene, and (2) a
progressive expansion process to construct a complete 3D scene. In particular,
leveraging an advanced pre-trained video model and accurate depth-estimated
training pairs, our V2V model can generate novel views under large camera pose
variations. Building upon it, FlexWorld progressively generates new 3D content
and integrates it into the global scene through geometry-aware scene fusion.
Extensive experiments demonstrate the effectiveness of FlexWorld in generating
high-quality novel view videos and flexible-view 3D scenes from single images,
achieving superior visual quality under multiple popular metrics and datasets
compared to existing state-of-the-art methods. Qualitatively, we highlight that
FlexWorld can generate high-fidelity scenes with flexible views like 360{\deg}
rotations and zooming. Project page: https://ml-gsai.github.io/FlexWorld.
MM-Spatial: Exploring 3D Spatial Understanding in Multimodal LLMs
Authors: name: Erik Daxberger, name: Nina Wenzel, name: David Griffiths, name: Haiming Gang, name: Justin Lazarow, name: Gefen Kohavi, name: Kai Kang, name: Marcin Eichner, name: Yinfei Yang, name: Afshin Dehghan, name: Peter Grasch
Published: 2025-03-17 12:34:22

Multimodal large language models (MLLMs) excel at 2D visual understanding but
remain limited in their ability to reason about 3D space. In this work, we
leverage large-scale high-quality 3D scene data with open-set annotations to
introduce 1) a novel supervised fine-tuning dataset and 2) a new evaluation
benchmark, focused on indoor scenes. Our Cubify Anything VQA (CA-VQA) data
covers diverse spatial tasks including spatial relationship prediction, metric
size and distance estimation, and 3D grounding. We show that CA-VQA enables us
to train MM-Spatial, a strong generalist MLLM that also achieves
state-of-the-art performance on 3D spatial understanding benchmarks, including
our own. We show how incorporating metric depth and multi-view inputs (provided
in CA-VQA) can further improve 3D understanding, and demonstrate that data
alone allows our model to achieve depth perception capabilities comparable to
dedicated monocular depth estimation models. We will publish our SFT dataset
and benchmark.
Florenz: Scaling Laws for Systematic Generalization in Vision-Language
  Models
Authors: name: Julian Spravil, name: Sebastian Houben, name: Sven Behnke
Published: 2025-03-12 14:41:10

Cross-lingual transfer enables vision-language models (VLMs) to perform
vision tasks in various languages with training data only in one language.
Current approaches rely on large pre-trained multilingual language models.
However, they face the curse of multilinguality, sacrificing downstream task
performance for multilingual capabilities, struggling with lexical ambiguities,
and falling behind recent advances. In this work, we study the scaling laws of
systematic generalization with monolingual VLMs for multilingual tasks,
focusing on the impact of model size and seen training samples. We propose
Florenz, a monolingual encoder-decoder VLM with 0.4B to 11.2B parameters
combining the pre-trained VLM Florence-2 and the large language model Gemma-2.
Florenz is trained with varying compute budgets on a synthetic dataset that
features intentionally incomplete language coverage for image captioning, thus,
testing generalization from the fully covered translation task. We show that
not only does indirectly learning unseen task-language pairs adhere to a
scaling law, but also that with our data generation pipeline and the proposed
Florenz model family, image captioning abilities can emerge in a specific
language even when only data for the translation task is available. Fine-tuning
on a mix of downstream datasets yields competitive performance and demonstrates
promising scaling trends in multimodal machine translation (Multi30K, CoMMuTE),
lexical disambiguation (CoMMuTE), and image captioning (Multi30K, XM3600, COCO
Karpathy).