Orionnnn/hub-stats huggingface dataset:
V-STaR: Benchmarking Video-LLMs on Video Spatio-Temporal Reasoning
Authors: name: Zixu Cheng, name: Jian Hu, name: Ziquan Liu, name: Chenyang Si, name: Wei Li, name: Shaogang Gong
Published: 2025-03-14 15:21:44

Human processes video reasoning in a sequential spatio-temporal reasoning
logic, we first identify the relevant frames ("when") and then analyse the
spatial relationships ("where") between key objects, and finally leverage these
relationships to draw inferences ("what"). However, can Video Large Language
Models (Video-LLMs) also "reason through a sequential spatio-temporal logic" in
videos? Existing Video-LLM benchmarks primarily focus on assessing object
presence, neglecting relational reasoning. Consequently, it is difficult to
measure whether a model truly comprehends object interactions (actions/events)
in videos or merely relies on pre-trained "memory" of co-occurrences as biases
in generating answers. In this work, we introduce a Video Spatio-Temporal
Reasoning (V-STaR) benchmark to address these shortcomings. The key idea is to
decompose video understanding into a Reverse Spatio-Temporal Reasoning (RSTR)
task that simultaneously evaluates what objects are present, when events occur,
and where they are located while capturing the underlying Chain-of-thought
(CoT) logic. To support this evaluation, we construct a dataset to elicit the
spatial-temporal reasoning process of Video-LLMs. It contains coarse-to-fine
CoT questions generated by a semi-automated GPT-4-powered pipeline, embedding
explicit reasoning chains to mimic human cognition. Experiments from 14
Video-LLMs on our V-STaR reveal significant gaps between current Video-LLMs and
the needs for robust and consistent spatio-temporal reasoning.
MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based
  Scientific Research
Authors: name: James Burgess, name: Jeffrey J Nirschl, name: Laura Bravo-Sánchez, name: Alejandro Lozano, name: Sanket Rajan Gupte, name: Jesus G. Galaz-Montoya, name: Yuhui Zhang, name: Yuchang Su, name: Disha Bhowmik, name: Zachary Coman, name: Sarina M. Hasan, name: Alexandra Johannesson, name: William D. Leineweber, name: Malvika G Nair, name: Ridhi Yarlagadda, name: Connor Zuraski, name: Wah Chiu, name: Sarah Cohen, name: Jan N. Hansen, name: Manuel D Leonetti, name: Chad Liu, name: Emma Lundberg, name: Serena Yeung-Levy
Published: 2025-03-17 17:33:10

Scientific research demands sophisticated reasoning over multimodal data, a
challenge especially prevalent in biology. Despite recent advances in
multimodal large language models (MLLMs) for AI-assisted research, existing
multimodal reasoning benchmarks only target up to college-level difficulty,
while research-level benchmarks emphasize lower-level perception, falling short
of the complex multimodal reasoning needed for scientific discovery. To bridge
this gap, we introduce MicroVQA, a visual-question answering (VQA) benchmark
designed to assess three reasoning capabilities vital in research workflows:
expert image understanding, hypothesis generation, and experiment proposal.
MicroVQA consists of 1,042 multiple-choice questions (MCQs) curated by biology
experts across diverse microscopy modalities, ensuring VQA samples represent
real scientific practice. In constructing the benchmark, we find that standard
MCQ generation methods induce language shortcuts, motivating a new two-stage
pipeline: an optimized LLM prompt structures question-answer pairs into MCQs;
then, an agent-based `RefineBot' updates them to remove shortcuts. Benchmarking
on state-of-the-art MLLMs reveal a peak performance of 53\%; models with
smaller LLMs only slightly underperform top models, suggesting that
language-based reasoning is less challenging than multimodal reasoning; and
tuning with scientific articles enhances performance. Expert analysis of
chain-of-thought responses shows that perception errors are the most frequent,
followed by knowledge errors and then overgeneralization errors. These insights
highlight the challenges in multimodal scientific reasoning, showing MicroVQA
is a valuable resource advancing AI-driven biomedical research. MicroVQA is
available at https://huggingface.co/datasets/jmhb/microvqa, and project page at
https://jmhb0.github.io/microvqa.
CHOrD: Generation of Collision-Free, House-Scale, and Organized Digital
  Twins for 3D Indoor Scenes with Controllable Floor Plans and Optimal Layouts
Authors: name: Chong Su, name: Yingbin Fu, name: Zheyuan Hu, name: Jing Yang, name: Param Hanji, name: Shaojun Wang, name: Xuan Zhao, name: Cengiz Öztireli, name: Fangcheng Zhong
Published: 2025-03-15 02:05:10

We introduce CHOrD, a novel framework for scalable synthesis of 3D indoor
scenes, designed to create house-scale, collision-free, and hierarchically
structured indoor digital twins. In contrast to existing methods that directly
synthesize the scene layout as a scene graph or object list, CHOrD incorporates
a 2D image-based intermediate layout representation, enabling effective
prevention of collision artifacts by successfully capturing them as
out-of-distribution (OOD) scenarios during generation. Furthermore, unlike
existing methods, CHOrD is capable of generating scene layouts that adhere to
complex floor plans with multi-modal controls, enabling the creation of
coherent, house-wide layouts robust to both geometric and semantic variations
in room structures. Additionally, we propose a novel dataset with expanded
coverage of household items and room configurations, as well as significantly
improved data quality. CHOrD demonstrates state-of-the-art performance on both
the 3D-FRONT and our proposed datasets, delivering photorealistic, spatially
coherent indoor scene synthesis adaptable to arbitrary floor plan variations.
Can Large Reasoning Models do Analogical Reasoning under Perceptual
  Uncertainty?
Authors: name: Giacomo Camposampiero, name: Michael Hersche, name: Roger Wattenhofer, name: Abu Sebastian, name: Abbas Rahimi
Published: 2025-03-14 08:52:25

This work presents a first evaluation of two state-of-the-art Large Reasoning
Models (LRMs), OpenAI's o3-mini and DeepSeek R1, on analogical reasoning,
focusing on well-established nonverbal human IQ tests based on Raven's
progressive matrices. We benchmark with the I-RAVEN dataset and its more
difficult extension, I-RAVEN-X, which tests the ability to generalize to longer
reasoning rules and ranges of the attribute values. To assess the influence of
visual uncertainties on these nonverbal analogical reasoning tests, we extend
the I-RAVEN-X dataset, which otherwise assumes an oracle perception. We adopt a
two-fold strategy to simulate this imperfect visual perception: 1) we introduce
confounding attributes which, being sampled at random, do not contribute to the
prediction of the correct answer of the puzzles and 2) smoothen the
distributions of the input attributes' values. We observe a sharp decline in
OpenAI's o3-mini task accuracy, dropping from 86.6% on the original I-RAVEN to
just 17.0% -- approaching random chance -- on the more challenging I-RAVEN-X,
which increases input length and range and emulates perceptual uncertainty.
This drop occurred despite spending 3.4x more reasoning tokens. A similar trend
is also observed for DeepSeek R1: from 80.6% to 23.2%. On the other hand, a
neuro-symbolic probabilistic abductive model, ARLC, that achieves
state-of-the-art performances on I-RAVEN, can robustly reason under all these
out-of-distribution tests, maintaining strong accuracy with only a modest
reduction from 98.6% to 88.0%. Our code is available at
https://github.com/IBM/raven-large-language-models.
SmolDocling: An ultra-compact vision-language model for end-to-end
  multi-modal document conversion
Authors: name: Ahmed Nassar, name: Andres Marafioti, name: Matteo Omenetti, name: Maksym Lysak, name: Nikolaos Livathinos, name: Christoph Auer, name: Lucas Morin, name: Rafael Teixeira de Lima, name: Yusik Kim, name: A. Said Gurbuz, name: Michele Dolfi, name: Miquel Farré, name: Peter W. J. Staar
Published: 2025-03-14 16:44:14

We introduce SmolDocling, an ultra-compact vision-language model targeting
end-to-end document conversion. Our model comprehensively processes entire
pages by generating DocTags, a new universal markup format that captures all
page elements in their full context with location. Unlike existing approaches
that rely on large foundational models, or ensemble solutions that rely on
handcrafted pipelines of multiple specialized models, SmolDocling offers an
end-to-end conversion for accurately capturing content, structure and spatial
location of document elements in a 256M parameters vision-language model.
SmolDocling exhibits robust performance in correctly reproducing document
features such as code listings, tables, equations, charts, lists, and more
across a diverse range of document types including business documents, academic
papers, technical reports, patents, and forms -- significantly extending beyond
the commonly observed focus on scientific papers. Additionally, we contribute
novel publicly sourced datasets for charts, tables, equations, and code
recognition. Experimental results demonstrate that SmolDocling competes with
other Vision Language Models that are up to 27 times larger in size, while
reducing computational requirements substantially. The model is currently
available, datasets will be publicly available soon.
Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformers
Authors: name: Weiming Ren, name: Wentao Ma, name: Huan Yang, name: Cong Wei, name: Ge Zhang, name: Wenhu Chen
Published: 2025-03-14 16:45:23

State-of-the-art transformer-based large multimodal models (LMMs) struggle to
handle hour-long video inputs due to the quadratic complexity of the causal
self-attention operations, leading to high computational costs during training
and inference. Existing token compression-based methods reduce the number of
video tokens but often incur information loss and remain inefficient for
extremely long sequences. In this paper, we explore an orthogonal direction to
build a hybrid Mamba-Transformer model (VAMBA) that employs Mamba-2 blocks to
encode video tokens with linear complexity. Without any token reduction, VAMBA
can encode more than 1024 frames (640times360) on a single GPU, while
transformer-based models can only encode 256 frames. On long video input, VAMBA
achieves at least 50% reduction in GPU memory usage during training and
inference, and nearly doubles the speed per training step compared to
transformer-based LMMs. Our experimental results demonstrate that VAMBA
improves accuracy by 4.3% on the challenging hour-long video understanding
benchmark LVBench over prior efficient video LMMs, and maintains strong
performance on a broad spectrum of long and short video understanding tasks.
TreeMeshGPT: Artistic Mesh Generation with Autoregressive Tree
  Sequencing
Authors: name: Stefan Lionar, name: Jiabin Liang, name: Gim Hee Lee
Published: 2025-03-14 17:48:06

We introduce TreeMeshGPT, an autoregressive Transformer designed to generate
high-quality artistic meshes aligned with input point clouds. Instead of the
conventional next-token prediction in autoregressive Transformer, we propose a
novel Autoregressive Tree Sequencing where the next input token is retrieved
from a dynamically growing tree structure that is built upon the triangle
adjacency of faces within the mesh. Our sequencing enables the mesh to extend
locally from the last generated triangular face at each step, and therefore
reduces training difficulty and improves mesh quality. Our approach represents
each triangular face with two tokens, achieving a compression rate of
approximately 22% compared to the naive face tokenization. This efficient
tokenization enables our model to generate highly detailed artistic meshes with
strong point cloud conditioning, surpassing previous methods in both capacity
and fidelity. Furthermore, our method generates mesh with strong normal
orientation constraints, minimizing flipped normals commonly encountered in
previous methods. Our experiments show that TreeMeshGPT enhances the mesh
generation quality with refined details and normal orientation consistency.
VGGT: Visual Geometry Grounded Transformer
Authors: name: Jianyuan Wang, name: Minghao Chen, name: Nikita Karaev, name: Andrea Vedaldi, name: Christian Rupprecht, name: David Novotny
Published: 2025-03-14 17:59:47

We present VGGT, a feed-forward neural network that directly infers all key
3D attributes of a scene, including camera parameters, point maps, depth maps,
and 3D point tracks, from one, a few, or hundreds of its views. This approach
is a step forward in 3D computer vision, where models have typically been
constrained to and specialized for single tasks. It is also simple and
efficient, reconstructing images in under one second, and still outperforming
alternatives that require post-processing with visual geometry optimization
techniques. The network achieves state-of-the-art results in multiple 3D tasks,
including camera parameter estimation, multi-view depth estimation, dense point
cloud reconstruction, and 3D point tracking. We also show that using pretrained
VGGT as a feature backbone significantly enhances downstream tasks, such as
non-rigid point tracking and feed-forward novel view synthesis. Code and models
are publicly available at https://github.com/facebookresearch/vggt.
Open-World Skill Discovery from Unsegmented Demonstrations
Authors: name: Jingwen Deng, name: Zihao Wang, name: Shaofei Cai, name: Anji Liu, name: Yitao Liang
Published: 2025-03-11 18:51:40

Learning skills in open-world environments is essential for developing agents
capable of handling a variety of tasks by combining basic skills. Online
demonstration videos are typically long but unsegmented, making them difficult
to segment and label with skill identifiers. Unlike existing methods that rely
on sequence sampling or human labeling, we have developed a self-supervised
learning-based approach to segment these long videos into a series of
semantic-aware and skill-consistent segments. Drawing inspiration from human
cognitive event segmentation theory, we introduce Skill Boundary Detection
(SBD), an annotation-free temporal video segmentation algorithm. SBD detects
skill boundaries in a video by leveraging prediction errors from a pretrained
unconditional action-prediction model. This approach is based on the assumption
that a significant increase in prediction error indicates a shift in the skill
being executed. We evaluated our method in Minecraft, a rich open-world
simulator with extensive gameplay videos available online. Our SBD-generated
segments improved the average performance of conditioned policies by 63.7% and
52.1% on short-term atomic skill tasks, and their corresponding hierarchical
agents by 11.3% and 20.8% on long-horizon tasks. Our method can leverage the
diverse YouTube videos to train instruction-following agents. The project page
can be found in https://craftjarvis.github.io/SkillDiscovery.
From TOWER to SPIRE: Adding the Speech Modality to a Text-Only LLM
Authors: name: Kshitij Ambilduke, name: Ben Peters, name: Sonal Sannigrahi, name: Anil Keshwani, name: Tsz Kin Lam, name: Bruno Martins, name: Marcely Zanon Boito, name: André F. T. Martins
Published: 2025-03-13 17:57:32

Large language models (LLMs) have shown remarkable performance and
generalization capabilities across multiple languages and tasks, making them
very attractive targets for multi-modality integration (e.g., images or
speech). In this work, we extend an existing LLM to the speech modality via
speech discretization and continued pre-training. In particular, we are
interested in multilingual LLMs, such as TOWER, as their pre-training setting
allows us to treat discretized speech input as an additional translation
language. The resulting open-source model, SPIRE, is able to transcribe and
translate English speech input while maintaining TOWER's original performance
on translation-related tasks, showcasing that discretized speech input
integration as an additional language is feasible during LLM adaptation. We
make our code and models available to the community.
Group-robust Machine Unlearning
Authors: name: Thomas De Min, name: Subhankar Roy, name: Stéphane Lathuilière, name: Elisa Ricci, name: Massimiliano Mancini
Published: 2025-03-12 12:24:05

Machine unlearning is an emerging paradigm to remove the influence of
specific training data (i.e., the forget set) from a model while preserving its
knowledge of the rest of the data (i.e., the retain set). Previous approaches
assume the forget data to be uniformly distributed from all training
datapoints. However, if the data to unlearn is dominant in one group, we
empirically show that performance for this group degrades, leading to fairness
issues. This work tackles the overlooked problem of non-uniformly distributed
forget sets, which we call group-robust machine unlearning, by presenting a
simple, effective strategy that mitigates the performance loss in dominant
groups via sample distribution reweighting. Moreover, we present MIU (Mutual
Information-aware Machine Unlearning), the first approach for group robustness
in approximate machine unlearning. MIU minimizes the mutual information between
model features and group information, achieving unlearning while reducing
performance degradation in the dominant group of the forget set. Additionally,
MIU exploits sample distribution reweighting and mutual information calibration
with the original model to preserve group robustness. We conduct experiments on
three datasets and show that MIU outperforms standard methods, achieving
unlearning without compromising model robustness. Source code available at
https://github.com/tdemin16/group-robust_machine_unlearning.
Neighboring Autoregressive Modeling for Efficient Visual Generation
Authors: name: Yefei He, name: Yuanyu He, name: Shaoxuan He, name: Feng Chen, name: Hong Zhou, name: Kaipeng Zhang, name: Bohan Zhuang
Published: 2025-03-12 05:52:27

Visual autoregressive models typically adhere to a raster-order ``next-token
prediction" paradigm, which overlooks the spatial and temporal locality
inherent in visual content. Specifically, visual tokens exhibit significantly
stronger correlations with their spatially or temporally adjacent tokens
compared to those that are distant. In this paper, we propose Neighboring
Autoregressive Modeling (NAR), a novel paradigm that formulates autoregressive
visual generation as a progressive outpainting procedure, following a
near-to-far ``next-neighbor prediction" mechanism. Starting from an initial
token, the remaining tokens are decoded in ascending order of their Manhattan
distance from the initial token in the spatial-temporal space, progressively
expanding the boundary of the decoded region. To enable parallel prediction of
multiple adjacent tokens in the spatial-temporal space, we introduce a set of
dimension-oriented decoding heads, each predicting the next token along a
mutually orthogonal dimension. During inference, all tokens adjacent to the
decoded tokens are processed in parallel, substantially reducing the model
forward steps for generation. Experiments on ImageNet256times 256 and UCF101
demonstrate that NAR achieves 2.4times and 8.6times higher throughput
respectively, while obtaining superior FID/FVD scores for both image and video
generation tasks compared to the PAR-4X approach. When evaluating on
text-to-image generation benchmark GenEval, NAR with 0.8B parameters
outperforms Chameleon-7B while using merely 0.4 of the training data. Code is
available at https://github.com/ThisisBillhe/NAR.
MaRI: Material Retrieval Integration across Domains
Authors: name: Jianhui Wang, name: Zhifei Yang, name: Yangfan He, name: Huixiong Zhang, name: Yuxuan Chen, name: Jingwei Huang
Published: 2025-03-11 07:23:11

Accurate material retrieval is critical for creating realistic 3D assets.
Existing methods rely on datasets that capture shape-invariant and
lighting-varied representations of materials, which are scarce and face
challenges due to limited diversity and inadequate real-world generalization.
Most current approaches adopt traditional image search techniques. They fall
short in capturing the unique properties of material spaces, leading to
suboptimal performance in retrieval tasks. Addressing these challenges, we
introduce MaRI, a framework designed to bridge the feature space gap between
synthetic and real-world materials. MaRI constructs a shared embedding space
that harmonizes visual and material attributes through a contrastive learning
strategy by jointly training an image and a material encoder, bringing similar
materials and images closer while separating dissimilar pairs within the
feature space. To support this, we construct a comprehensive dataset comprising
high-quality synthetic materials rendered with controlled shape variations and
diverse lighting conditions, along with real-world materials processed and
standardized using material transfer techniques. Extensive experiments
demonstrate the superior performance, accuracy, and generalization capabilities
of MaRI across diverse and complex material retrieval tasks, outperforming
existing methods.
ETCH: Generalizing Body Fitting to Clothed Humans via Equivariant
  Tightness
Authors: name: Boqian Li, name: Haiwen Feng, name: Zeyu Cai, name: Michael J. Black, name: Yuliang Xiu
Published: 2025-03-13 17:59:14

Fitting a body to a 3D clothed human point cloud is a common yet challenging
task. Traditional optimization-based approaches use multi-stage pipelines that
are sensitive to pose initialization, while recent learning-based methods often
struggle with generalization across diverse poses and garment types. We propose
Equivariant Tightness Fitting for Clothed Humans, or ETCH, a novel pipeline
that estimates cloth-to-body surface mapping through locally approximate SE(3)
equivariance, encoding tightness as displacement vectors from the cloth surface
to the underlying body. Following this mapping, pose-invariant body features
regress sparse body markers, simplifying clothed human fitting into an
inner-body marker fitting task. Extensive experiments on CAPE and 4D-Dress show
that ETCH significantly outperforms state-of-the-art methods -- both
tightness-agnostic and tightness-aware -- in body fitting accuracy on loose
clothing (16.7% ~ 69.5%) and shape accuracy (average 49.9%). Our equivariant
tightness design can even reduce directional errors by (67.2% ~ 89.8%) in
one-shot (or out-of-distribution) settings. Qualitative results demonstrate
strong generalization of ETCH, regardless of challenging poses, unseen shapes,
loose clothing, and non-rigid dynamics. We will release the code and models
soon for research purposes at https://boqian-li.github.io/ETCH/.