Orionnnn/hub-stats huggingface dataset:
One-Minute Video Generation with Test-Time Training
Authors: name: Karan Dalal, name: Daniel Koceja, name: Gashon Hussein, name: Jiarui Xu, name: Yue Zhao, name: Youjin Song, name: Shihao Han, name: Ka Chun Cheung, name: Jan Kautz, name: Carlos Guestrin, name: Tatsunori Hashimoto, name: Sanmi Koyejo, name: Yejin Choi, name: Yu Sun, name: Xiaolong Wang
Published: 2025-04-07 17:56:31

Transformers today still struggle to generate one-minute videos because
self-attention layers are inefficient for long context. Alternatives such as
Mamba layers struggle with complex multi-scene stories because their hidden
states are less expressive. We experiment with Test-Time Training (TTT) layers,
whose hidden states themselves can be neural networks, therefore more
expressive. Adding TTT layers into a pre-trained Transformer enables it to
generate one-minute videos from text storyboards. For proof of concept, we
curate a dataset based on Tom and Jerry cartoons. Compared to baselines such as
Mamba~2, Gated DeltaNet, and sliding-window attention layers, TTT layers
generate much more coherent videos that tell complex stories, leading by 34 Elo
points in a human evaluation of 100 videos per method. Although promising,
results still contain artifacts, likely due to the limited capability of the
pre-trained 5B model. The efficiency of our implementation can also be
improved. We have only experimented with one-minute videos due to resource
constraints, but the approach can be extended to longer videos and more complex
stories. Sample videos, code and annotations are available at:
https://test-time-training.github.io/video-dit
URECA: Unique Region Caption Anything
Authors: name: Sangbeom Lim, name: Junwan Kim, name: Heeji Yoon, name: Jaewoo Jung, name: Seungryong Kim
Published: 2025-04-07 17:59:44

Region-level captioning aims to generate natural language descriptions for
specific image regions while highlighting their distinguishing features.
However, existing methods struggle to produce unique captions across
multi-granularity, limiting their real-world applicability. To address the need
for detailed region-level understanding, we introduce URECA dataset, a
large-scale dataset tailored for multi-granularity region captioning. Unlike
prior datasets that focus primarily on salient objects, URECA dataset ensures a
unique and consistent mapping between regions and captions by incorporating a
diverse set of objects, parts, and background elements. Central to this is a
stage-wise data curation pipeline, where each stage incrementally refines
region selection and caption generation. By leveraging Multimodal Large
Language Models (MLLMs) at each stage, our pipeline produces distinctive and
contextually grounded captions with improved accuracy and semantic diversity.
Building upon this dataset, we present URECA, a novel captioning model designed
to effectively encode multi-granularity regions. URECA maintains essential
spatial properties such as position and shape through simple yet impactful
modifications to existing MLLMs, enabling fine-grained and semantically rich
region descriptions. Our approach introduces dynamic mask modeling and a
high-resolution mask encoder to enhance caption uniqueness. Experiments show
that URECA achieves state-of-the-art performance on URECA dataset and
generalizes well to existing region-level captioning benchmarks.
Gaussian Mixture Flow Matching Models
Authors: name: Hansheng Chen, name: Kai Zhang, name: Hao Tan, name: Zexiang Xu, name: Fujun Luan, name: Leonidas Guibas, name: Gordon Wetzstein, name: Sai Bi
Published: 2025-04-07 17:59:42

Diffusion models approximate the denoising distribution as a Gaussian and
predict its mean, whereas flow matching models reparameterize the Gaussian mean
as flow velocity. However, they underperform in few-step sampling due to
discretization error and tend to produce over-saturated colors under
classifier-free guidance (CFG). To address these limitations, we propose a
novel Gaussian mixture flow matching (GMFlow) model: instead of predicting the
mean, GMFlow predicts dynamic Gaussian mixture (GM) parameters to capture a
multi-modal flow velocity distribution, which can be learned with a KL
divergence loss. We demonstrate that GMFlow generalizes previous diffusion and
flow matching models where a single Gaussian is learned with an L_2 denoising
loss. For inference, we derive GM-SDE/ODE solvers that leverage analytic
denoising distributions and velocity fields for precise few-step sampling.
Furthermore, we introduce a novel probabilistic guidance scheme that mitigates
the over-saturation issues of CFG and improves image generation quality.
Extensive experiments demonstrate that GMFlow consistently outperforms flow
matching baselines in generation quality, achieving a Precision of 0.942 with
only 6 sampling steps on ImageNet 256times256.
ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning
Authors: name: Zhaorun Chen, name: Mintong Kang, name: Bo Li
Published: 2025-03-26 17:58:40

Autonomous agents powered by foundation models have seen widespread adoption
across various real-world applications. However, they remain highly vulnerable
to malicious instructions and attacks, which can result in severe consequences
such as privacy breaches and financial losses. More critically, existing
guardrails for LLMs are not applicable due to the complex and dynamic nature of
agents. To tackle these challenges, we propose ShieldAgent, the first guardrail
agent designed to enforce explicit safety policy compliance for the action
trajectory of other protected agents through logical reasoning. Specifically,
ShieldAgent first constructs a safety policy model by extracting verifiable
rules from policy documents and structuring them into a set of action-based
probabilistic rule circuits. Given the action trajectory of the protected
agent, ShieldAgent retrieves relevant rule circuits and generates a shielding
plan, leveraging its comprehensive tool library and executable code for formal
verification. In addition, given the lack of guardrail benchmarks for agents,
we introduce ShieldAgent-Bench, a dataset with 3K safety-related pairs of agent
instructions and action trajectories, collected via SOTA attacks across 6 web
environments and 7 risk categories. Experiments show that ShieldAgent achieves
SOTA on ShieldAgent-Bench and three existing benchmarks, outperforming prior
methods by 11.3% on average with a high recall of 90.1%. Additionally,
ShieldAgent reduces API queries by 64.7% and inference time by 58.2%,
demonstrating its high precision and efficiency in safeguarding agents.
Delineate Anything: Resolution-Agnostic Field Boundary Delineation on
  Satellite Imagery
Authors: name: Mykola Lavreniuk, name: Nataliia Kussul, name: Andrii Shelestov, name: Bohdan Yailymov, name: Yevhenii Salii, name: Volodymyr Kuzin, name: Zoltan Szantoi
Published: 2025-04-03 12:37:04

The accurate delineation of agricultural field boundaries from satellite
imagery is vital for land management and crop monitoring. However, current
methods face challenges due to limited dataset sizes, resolution discrepancies,
and diverse environmental conditions. We address this by reformulating the task
as instance segmentation and introducing the Field Boundary Instance
Segmentation - 22M dataset (FBIS-22M), a large-scale, multi-resolution dataset
comprising 672,909 high-resolution satellite image patches (ranging from 0.25 m
to 10 m) and 22,926,427 instance masks of individual fields, significantly
narrowing the gap between agricultural datasets and those in other computer
vision domains. We further propose Delineate Anything, an instance segmentation
model trained on our new FBIS-22M dataset. Our proposed model sets a new
state-of-the-art, achieving a substantial improvement of 88.5% in mAP@0.5 and
103% in mAP@0.5:0.95 over existing methods, while also demonstrating
significantly faster inference and strong zero-shot generalization across
diverse image resolutions and unseen geographic regions. Code, pre-trained
models, and the FBIS-22M dataset are available at
https://lavreniuk.github.io/Delineate-Anything.
Real-is-Sim: Bridging the Sim-to-Real Gap with a Dynamic Digital Twin
  for Real-World Robot Policy Evaluation
Authors: name: Jad Abou-Chakra, name: Lingfeng Sun, name: Krishan Rana, name: Brandon May, name: Karl Schmeckpeper, name: Maria Vittoria Minniti, name: Laura Herlant
Published: 2025-04-04 17:05:56

Recent advancements in behavior cloning have enabled robots to perform
complex manipulation tasks. However, accurately assessing training performance
remains challenging, particularly for real-world applications, as behavior
cloning losses often correlate poorly with actual task success. Consequently,
researchers resort to success rate metrics derived from costly and
time-consuming real-world evaluations, making the identification of optimal
policies and detection of overfitting or underfitting impractical. To address
these issues, we propose real-is-sim, a novel behavior cloning framework that
incorporates a dynamic digital twin (based on Embodied Gaussians) throughout
the entire policy development pipeline: data collection, training, and
deployment. By continuously aligning the simulated world with the physical
world, demonstrations can be collected in the real world with states extracted
from the simulator. The simulator enables flexible state representations by
rendering image inputs from any viewpoint or extracting low-level state
information from objects embodied within the scene. During training, policies
can be directly evaluated within the simulator in an offline and highly
parallelizable manner. Finally, during deployment, policies are run within the
simulator where the real robot directly tracks the simulated robot's joints,
effectively decoupling policy execution from real hardware and mitigating
traditional domain-transfer challenges. We validate real-is-sim on the PushT
manipulation task, demonstrating strong correlation between success rates
obtained in the simulator and real-world evaluations. Videos of our system can
be found at https://realissim.rai-inst.com.
Slow-Fast Architecture for Video Multi-Modal Large Language Models
Authors: name: Min Shi, name: Shihao Wang, name: Chieh-Yun Chen, name: Jitesh Jain, name: Kai Wang, name: Junjun Xiong, name: Guilin Liu, name: Zhiding Yu, name: Humphrey Shi
Published: 2025-04-02 03:24:58

Balancing temporal resolution and spatial detail under limited compute budget
remains a key challenge for video-based multi-modal large language models
(MLLMs). Existing methods typically compress video representations using
predefined rules before feeding them into the LLM, resulting in irreversible
information loss and often ignoring input instructions. To address this, we
propose a novel slow-fast architecture that naturally circumvents this
trade-off, enabling the use of more input frames while preserving spatial
details. Inspired by how humans first skim a video before focusing on relevant
parts, our slow-fast design employs a dual-token strategy: 1) "fast" visual
tokens -- a compact set of compressed video features -- are fed into the LLM
alongside text embeddings to provide a quick overview; 2) "slow" visual tokens
-- uncompressed video features -- are cross-attended by text embeddings through
specially designed hybrid decoder layers, enabling instruction-aware extraction
of relevant visual details with linear complexity. We conduct systematic
exploration to optimize both the overall architecture and key components.
Experiments show that our model significantly outperforms self-attention-only
baselines, extending the input capacity from 16 to 128 frames with just a 3%
increase in computation, and achieving a 16% average performance improvement
across five video understanding benchmarks. Our 7B model achieves
state-of-the-art performance among models of similar size. Furthermore, our
slow-fast architecture is a plug-and-play design that can be integrated into
other video MLLMs to improve efficiency and scalability.
SPF-Portrait: Towards Pure Portrait Customization with Semantic
  Pollution-Free Fine-tuning
Authors: name: Xiaole Xian, name: Zhichao Liao, name: Qingyu Li, name: Wenyu Qin, name: Pengfei Wan, name: Weicheng Xie, name: Long Zeng, name: Linlin Shen, name: Pingfa Feng
Published: 2025-04-01 03:37:30

Fine-tuning a pre-trained Text-to-Image (T2I) model on a tailored portrait
dataset is the mainstream method for text-driven customization of portrait
attributes. Due to Semantic Pollution during fine-tuning, existing methods
struggle to maintain the original model's behavior and achieve incremental
learning while customizing target attributes. To address this issue, we propose
SPF-Portrait, a pioneering work to purely understand customized semantics while
eliminating semantic pollution in text-driven portrait customization. In our
SPF-Portrait, we propose a dual-path pipeline that introduces the original
model as a reference for the conventional fine-tuning path. Through contrastive
learning, we ensure adaptation to target attributes and purposefully align
other unrelated attributes with the original portrait. We introduce a novel
Semantic-Aware Fine Control Map, which represents the precise response regions
of the target semantics, to spatially guide the alignment process between the
contrastive paths. This alignment process not only effectively preserves the
performance of the original model but also avoids over-alignment. Furthermore,
we propose a novel response enhancement mechanism to reinforce the performance
of target attributes, while mitigating representation discrepancy inherent in
direct cross-modal supervision. Extensive experiments demonstrate that
SPF-Portrait achieves state-of-the-art performance. Project webpage:
https://spf-portrait.github.io/SPF-Portrait/
MedSAM2: Segment Anything in 3D Medical Images and Videos
Authors: name: Jun Ma, name: Zongxin Yang, name: Sumin Kim, name: Bihui Chen, name: Mohammed Baharoon, name: Adibvafa Fallahpour, name: Reza Asakereh, name: Hongwei Lyu, name: Bo Wang
Published: 2025-04-04 17:13:37

Medical image and video segmentation is a critical task for precision
medicine, which has witnessed considerable progress in developing task or
modality-specific and generalist models for 2D images. However, there have been
limited studies on building general-purpose models for 3D images and videos
with comprehensive user studies. Here, we present MedSAM2, a promptable
segmentation foundation model for 3D image and video segmentation. The model is
developed by fine-tuning the Segment Anything Model 2 on a large medical
dataset with over 455,000 3D image-mask pairs and 76,000 frames, outperforming
previous models across a wide range of organs, lesions, and imaging modalities.
Furthermore, we implement a human-in-the-loop pipeline to facilitate the
creation of large-scale datasets resulting in, to the best of our knowledge,
the most extensive user study to date, involving the annotation of 5,000 CT
lesions, 3,984 liver MRI lesions, and 251,550 echocardiogram video frames,
demonstrating that MedSAM2 can reduce manual costs by more than 85%. MedSAM2 is
also integrated into widely used platforms with user-friendly interfaces for
local and cloud deployment, making it a practical tool for supporting
efficient, scalable, and high-quality segmentation in both research and
healthcare environments.
MegaMath: Pushing the Limits of Open Math Corpora
Authors: name: Fan Zhou, name: Zengzhi Wang, name: Nikhil Ranjan, name: Zhoujun Cheng, name: Liping Tang, name: Guowei He, name: Zhengzhong Liu, name: Eric P. Xing
Published: 2025-04-03 17:52:07

Mathematical reasoning is a cornerstone of human intelligence and a key
benchmark for advanced capabilities in large language models (LLMs). However,
the research community still lacks an open, large-scale, high-quality corpus
tailored to the demands of math-centric LLM pre-training. We present MegaMath,
an open dataset curated from diverse, math-focused sources through following
practices: (1) Revisiting web data: We re-extracted mathematical documents from
Common Crawl with math-oriented HTML optimizations, fasttext-based filtering
and deduplication, all for acquiring higher-quality data on the Internet. (2)
Recalling Math-related code data: We identified high quality math-related code
from large code training corpus, Stack-V2, further enhancing data diversity.
(3) Exploring Synthetic data: We synthesized QA-style text, math-related code,
and interleaved text-code blocks from web data or code data. By integrating
these strategies and validating their effectiveness through extensive
ablations, MegaMath delivers 371B tokens with the largest quantity and top
quality among existing open math pre-training datasets.
TransMamba: Flexibly Switching between Transformer and Mamba
Authors: name: Yixing Li, name: Ruobing Xie, name: Zhen Yang, name: Xingwu Sun, name: Shuaipeng Li, name: Weidong Han, name: Zhanhui Kang, name: Yu Cheng, name: Chengzhong Xu, name: Di Wang, name: Jie Jiang
Published: 2025-03-31 13:26:24

Transformers are the cornerstone of modern large language models, but their
quadratic computational complexity limits efficiency in long-sequence
processing. Recent advancements in Mamba, a state space model (SSM) with linear
complexity, offer promising efficiency gains but suffer from unstable
contextual learning and multitask generalization. This paper proposes
TransMamba, a novel framework that unifies Transformer and Mamba through shared
parameter matrices (e.g., QKV and CBx), and thus could dynamically switch
between attention and SSM mechanisms at different token lengths and layers. We
design the Memory converter to bridge Transformer and Mamba by converting
attention outputs into SSM-compatible states, ensuring seamless information
flow at TransPoints where the transformation happens. The TransPoint scheduling
is also thoroughly explored for further improvements. We conducted extensive
experiments demonstrating that TransMamba achieves superior training efficiency
and performance compared to baselines, and validated the deeper consistency
between Transformer and Mamba paradigms, offering a scalable solution for
next-generation sequence modeling.
EvMic: Event-based Non-contact sound recovery from effective
  spatial-temporal modeling
Authors: name: Hao Yin, name: Shi Guo, name: Xu Jia, name: Xudong XU, name: Lu Zhang, name: Si Liu, name: Dong Wang, name: Huchuan Lu, name: Tianfan Xue
Published: 2025-04-03 08:51:17

When sound waves hit an object, they induce vibrations that produce
high-frequency and subtle visual changes, which can be used for recovering the
sound. Early studies always encounter trade-offs related to sampling rate,
bandwidth, field of view, and the simplicity of the optical path. Recent
advances in event camera hardware show good potential for its application in
visual sound recovery, because of its superior ability in capturing
high-frequency signals. However, existing event-based vibration recovery
methods are still sub-optimal for sound recovery. In this work, we propose a
novel pipeline for non-contact sound recovery, fully utilizing spatial-temporal
information from the event stream. We first generate a large training set using
a novel simulation pipeline. Then we designed a network that leverages the
sparsity of events to capture spatial information and uses Mamba to model
long-term temporal information. Lastly, we train a spatial aggregation block to
aggregate information from different locations to further improve signal
quality. To capture event signals caused by sound waves, we also designed an
imaging system using a laser matrix to enhance the gradient and collected
multiple data sequences for testing. Experimental results on synthetic and
real-world data demonstrate the effectiveness of our method.
APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated
  Agent-Human Interplay
Authors: name: Akshara Prabhakar, name: Zuxin Liu, name: Weiran Yao, name: Jianguo Zhang, name: Ming Zhu, name: Shiyu Wang, name: Zhiwei Liu, name: Tulika Awalgaonkar, name: Haolin Chen, name: Thai Hoang, name: Juan Carlos Niebles, name: Shelby Heinecke, name: Huan Wang, name: Silvio Savarese, name: Caiming Xiong
Published: 2025-04-04 17:13:57

Training effective AI agents for multi-turn interactions requires
high-quality data that captures realistic human-agent dynamics, yet such data
is scarce and expensive to collect manually. We introduce APIGen-MT, a
two-phase framework that generates verifiable and diverse multi-turn agent
data. In the first phase, our agentic pipeline produces detailed task
blueprints with ground-truth actions, leveraging a committee of LLM reviewers
and iterative feedback loops. These blueprints are then transformed into
complete interaction trajectories through simulated human-agent interplay. We
train a family of models -- the xLAM-2-fc-r series with sizes ranging from 1B
to 70B parameters. Our models outperform frontier models such as GPT-4o and
Claude 3.5 on tau-bench and BFCL benchmarks, with the smaller models
surpassing their larger counterparts, particularly in multi-turn settings,
while maintaining superior consistency across multiple trials. Comprehensive
experiments demonstrate that our verified blueprint-to-details approach yields
high-quality training data, enabling the development of more reliable,
efficient, and capable agents. We open-source both the synthetic data collected
and the trained xLAM-2-fc-r models to advance research in AI agents. Models are
available on HuggingFace at
https://huggingface.co/collections/Salesforce/xlam-2-67ef5be12949d8dcdae354c4
and project website is https://apigen-mt.github.io
HumanDreamer-X: Photorealistic Single-image Human Avatars Reconstruction
  via Gaussian Restoration
Authors: name: Boyuan Wang, name: Runqi Ouyang, name: Xiaofeng Wang, name: Zheng Zhu, name: Guosheng Zhao, name: Chaojun Ni, name: Guan Huang, name: Lihong Liu, name: Xingang Wang
Published: 2025-04-04 15:35:14

Single-image human reconstruction is vital for digital human modeling
applications but remains an extremely challenging task. Current approaches rely
on generative models to synthesize multi-view images for subsequent 3D
reconstruction and animation. However, directly generating multiple views from
a single human image suffers from geometric inconsistencies, resulting in
issues like fragmented or blurred limbs in the reconstructed models. To tackle
these limitations, we introduce HumanDreamer-X, a novel framework that
integrates multi-view human generation and reconstruction into a unified
pipeline, which significantly enhances the geometric consistency and visual
fidelity of the reconstructed 3D models. In this framework, 3D Gaussian
Splatting serves as an explicit 3D representation to provide initial geometry
and appearance priority. Building upon this foundation, HumanFixer is
trained to restore 3DGS renderings, which guarantee photorealistic results.
Furthermore, we delve into the inherent challenges associated with attention
mechanisms in multi-view human generation, and propose an attention modulation
strategy that effectively enhances geometric details identity consistency
across multi-view. Experimental results demonstrate that our approach markedly
improves generation and reconstruction PSNR quality metrics by 16.45% and
12.65%, respectively, achieving a PSNR of up to 25.62 dB, while also showing
generalization capabilities on in-the-wild data and applicability to various
human reconstruction backbone models.