Orionnnn/hub-stats huggingface dataset:
OmniMamba: Efficient and Unified Multimodal Understanding and Generation
  via State Space Models
Authors: name: Jialv Zou, name: Bencheng Liao, name: Qian Zhang, name: Wenyu Liu, name: Xinggang Wang
Published: 2025-03-11 17:59:46

Recent advancements in unified multimodal understanding and visual generation
(or multimodal generation) models have been hindered by their quadratic
computational complexity and dependence on large-scale training data. We
present OmniMamba, the first linear-architecture-based multimodal generation
model that generates both text and images through a unified next-token
prediction paradigm. The model fully leverages Mamba-2's high computational and
memory efficiency, extending its capabilities from text generation to
multimodal generation. To address the data inefficiency of existing unified
models, we propose two key innovations: (1) decoupled vocabularies to guide
modality-specific generation, and (2) task-specific LoRA for
parameter-efficient adaptation. Furthermore, we introduce a decoupled two-stage
training strategy to mitigate data imbalance between two tasks. Equipped with
these techniques, OmniMamba achieves competitive performance with JanusFlow
while surpassing Show-o across benchmarks, despite being trained on merely 2M
image-text pairs, which is 1,000 times fewer than Show-o. Notably, OmniMamba
stands out with outstanding inference efficiency, achieving up to a 119.2 times
speedup and 63% GPU memory reduction for long-sequence generation compared to
Transformer-based counterparts. Code and models are released at
https://github.com/hustvl/OmniMamba
UniF^2ace: Fine-grained Face Understanding and Generation
  with Unified Multimodal Models
Authors: name: Junzhe Li, name: Xuerui Qiu, name: Linrui Xu, name: Liya Guo, name: Delin Qu, name: Tingting Long, name: Chun Fan, name: Ming Li
Published: 2025-03-11 07:34:59

Unified multimodal models (UMMs) have emerged as a powerful paradigm in
foundational computer vision research, demonstrating significant potential in
both image understanding and generation. However, existing research in the face
domain primarily focuses on coarse facial attribute understanding,
with limited capacity to handle fine-grained facial attributes and
without addressing generation capabilities. To overcome these limitations, we
propose UniF^2ace, the first UMM tailored specifically for
fine-grained face understanding and generation. In general, we train
UniF^2ace on a self-constructed, specialized dataset utilizing two
mutually beneficial diffusion techniques and a two-level mixture-of-experts
architecture. Specifically, we first build a large-scale facial dataset,
UniF^2ace-130K, which contains 130K image-text pairs with one
million question-answering pairs that span a wide range of facial attributes.
Second, we establish a theoretical connection between discrete diffusion score
matching and masked generative models, optimizing both evidence lower bounds
simultaneously, which significantly improves the model's ability to synthesize
facial details. Finally, we introduce both token-level and sequence-level
mixture-of-experts, enabling efficient fine-grained representation learning for
both understanding and generation tasks. Extensive experiments on
UniF^2ace-130K demonstrate that UniF^2ace outperforms
existing UMMs and generative models, achieving superior performance across both
understanding and generation tasks.
LMM-R1: Empowering 3B LMMs with Strong Reasoning Abilities Through
  Two-Stage Rule-Based RL
Authors: name: Yingzhe Peng, name: Gongrui Zhang, name: Miaosen Zhang, name: Zhiyuan You, name: Jie Liu, name: Qipeng Zhu, name: Kai Yang, name: Xingzhong Xu, name: Xin Geng, name: Xu Yang
Published: 2025-03-10 17:04:14

Enhancing reasoning in Large Multimodal Models (LMMs) faces unique challenges
from the complex interplay between visual perception and logical reasoning,
particularly in compact 3B-parameter architectures where architectural
constraints limit reasoning capacity and modality alignment.
  While rule-based reinforcement learning (RL) excels in text-only domains, its
multimodal extension confronts two critical barriers: (1) data limitations due
to ambiguous answers and scarce complex reasoning examples, and (2) degraded
foundational reasoning induced by multimodal pretraining.
  To address these challenges, we propose \method, a two-stage
framework adapting rule-based RL for multimodal reasoning through
Foundational Reasoning Enhancement (FRE) followed by
Multimodal Generalization Training (MGT). The FRE stage first
strengthens reasoning abilities using text-only data with rule-based RL, then
the MGT stage generalizes these reasoning capabilities to multimodal domains.
  Experiments on Qwen2.5-VL-Instruct-3B demonstrate that \method achieves
4.83\% and 4.5\% average improvements over baselines in multimodal and
text-only benchmarks, respectively, with a 3.63\% gain in complex Football Game
tasks. These results validate that text-based reasoning enhancement enables
effective multimodal generalization, offering a data-efficient paradigm that
bypasses costly high-quality multimodal training data.
"Principal Components" Enable A New Language of Images
Authors: name: Xin Wen, name: Bingchen Zhao, name: Ismail Elezi, name: Jiankang Deng, name: Xiaojuan Qi
Published: 2025-03-11 17:59:41

We introduce a novel visual tokenization framework that embeds a provable
PCA-like structure into the latent token space. While existing visual
tokenizers primarily optimize for reconstruction fidelity, they often neglect
the structural properties of the latent space -- a critical factor for both
interpretability and downstream tasks. Our method generates a 1D causal token
sequence for images, where each successive token contributes non-overlapping
information with mathematically guaranteed decreasing explained variance,
analogous to principal component analysis. This structural constraint ensures
the tokenizer extracts the most salient visual features first, with each
subsequent token adding diminishing yet complementary information.
Additionally, we identified and resolved a semantic-spectrum coupling effect
that causes the unwanted entanglement of high-level semantic content and
low-level spectral details in the tokens by leveraging a diffusion decoder.
Experiments demonstrate that our approach achieves state-of-the-art
reconstruction performance and enables better interpretability to align with
the human vision system. Moreover, auto-regressive models trained on our token
sequences achieve performance comparable to current state-of-the-art methods
while requiring fewer tokens for training and inference.
Seedream 2.0: A Native Chinese-English Bilingual Image Generation
  Foundation Model
Authors: name: Lixue Gong, name: Xiaoxia Hou, name: Fanshi Li, name: Liang Li, name: Xiaochen Lian, name: Fei Liu, name: Liyang Liu, name: Wei Liu, name: Wei Lu, name: Yichun Shi, name: Shiqi Sun, name: Yu Tian, name: Zhi Tian, name: Peng Wang, name: Xun Wang, name: Ye Wang, name: Guofeng Wu, name: Jie Wu, name: Xin Xia, name: Xuefeng Xiao, name: Linjie Yang, name: Zhonghua Zhai, name: Xinyu Zhang, name: Qi Zhang, name: Yuwei Zhang, name: Shijia Zhao, name: Jianchao Yang, name: Weilin Huang
Published: 2025-03-10 17:58:33

Rapid advancement of diffusion models has catalyzed remarkable progress in
the field of image generation. However, prevalent models such as Flux, SD3.5
and Midjourney, still grapple with issues like model bias, limited text
rendering capabilities, and insufficient understanding of Chinese cultural
nuances. To address these limitations, we present Seedream 2.0, a native
Chinese-English bilingual image generation foundation model that excels across
diverse dimensions, which adeptly manages text prompt in both Chinese and
English, supporting bilingual image generation and text rendering. We develop a
powerful data system that facilitates knowledge integration, and a caption
system that balances the accuracy and richness for image description.
Particularly, Seedream is integrated with a self-developed bilingual large
language model as a text encoder, allowing it to learn native knowledge
directly from massive data. This enable it to generate high-fidelity images
with accurate cultural nuances and aesthetic expressions described in either
Chinese or English. Beside, Glyph-Aligned ByT5 is applied for flexible
character-level text rendering, while a Scaled ROPE generalizes well to
untrained resolutions. Multi-phase post-training optimizations, including SFT
and RLHF iterations, further improve the overall capability. Through extensive
experimentation, we demonstrate that Seedream 2.0 achieves state-of-the-art
performance across multiple aspects, including prompt-following, aesthetics,
text rendering, and structural correctness. Furthermore, Seedream 2.0 has been
optimized through multiple RLHF iterations to closely align its output with
human preferences, as revealed by its outstanding ELO score. In addition, it
can be readily adapted to an instruction-based image editing model, such as
SeedEdit, with strong editing capability that balances instruction-following
and image consistency.
Implicit Reasoning in Transformers is Reasoning through Shortcuts
Authors: name: Tianhe Lin, name: Jian Xie, name: Siyu Yuan, name: Deqing Yang
Published: 2025-03-10 17:58:31

Test-time compute is emerging as a new paradigm for enhancing language
models' complex multi-step reasoning capabilities, as demonstrated by the
success of OpenAI's o1 and o3, as well as DeepSeek's R1. Compared to explicit
reasoning in test-time compute, implicit reasoning is more inference-efficient,
requiring fewer generated tokens. However, why does the advanced reasoning
capability fail to emerge in the implicit reasoning style? In this work, we
train GPT-2 from scratch on a curated multi-step mathematical reasoning dataset
and conduct analytical experiments to investigate how language models perform
implicit reasoning in multi-step tasks. Our findings reveal: 1) Language models
can perform step-by-step reasoning and achieve high accuracy in both in-domain
and out-of-domain tests via implicit reasoning. However, this capability only
emerges when trained on fixed-pattern data. 2) Conversely, implicit reasoning
abilities emerging from training on unfixed-pattern data tend to overfit a
specific pattern and fail to generalize further. Notably, this limitation is
also observed in state-of-the-art large language models. These findings suggest
that language models acquire implicit reasoning through shortcut learning,
enabling strong performance on tasks with similar patterns while lacking
generalization.
Tuning-Free Multi-Event Long Video Generation via Synchronized Coupled
  Sampling
Authors: name: Subin Kim, name: Seoung Wug Oh, name: Jui-Hsien Wang, name: Joon-Young Lee, name: Jinwoo Shin
Published: 2025-03-11 16:43:45

While recent advancements in text-to-video diffusion models enable
high-quality short video generation from a single prompt, generating real-world
long videos in a single pass remains challenging due to limited data and high
computational costs. To address this, several works propose tuning-free
approaches, i.e., extending existing models for long video generation,
specifically using multiple prompts to allow for dynamic and controlled content
changes. However, these methods primarily focus on ensuring smooth transitions
between adjacent frames, often leading to content drift and a gradual loss of
semantic coherence over longer sequences. To tackle such an issue, we propose
Synchronized Coupled Sampling (SynCoS), a novel inference framework that
synchronizes denoising paths across the entire video, ensuring long-range
consistency across both adjacent and distant frames. Our approach combines two
complementary sampling strategies: reverse and optimization-based sampling,
which ensure seamless local transitions and enforce global coherence,
respectively. However, directly alternating between these samplings misaligns
denoising trajectories, disrupting prompt guidance and introducing unintended
content changes as they operate independently. To resolve this, SynCoS
synchronizes them through a grounded timestep and a fixed baseline noise,
ensuring fully coupled sampling with aligned denoising paths. Extensive
experiments show that SynCoS significantly improves multi-event long video
generation, achieving smoother transitions and superior long-range coherence,
outperforming previous approaches both quantitatively and qualitatively.
Video Action Differencing
Authors: name: James Burgess, name: Xiaohan Wang, name: Yuhui Zhang, name: Anita Rau, name: Alejandro Lozano, name: Lisa Dunlap, name: Trevor Darrell, name: Serena Yeung-Levy
Published: 2025-03-10 21:18:32

How do two individuals differ when performing the same action? In this work,
we introduce Video Action Differencing (VidDiff), the novel task of identifying
subtle differences between videos of the same action, which has many
applications, such as coaching and skill learning. To enable development on
this new task, we first create VidDiffBench, a benchmark dataset containing 549
video pairs, with human annotations of 4,469 fine-grained action differences
and 2,075 localization timestamps indicating where these differences occur. Our
experiments demonstrate that VidDiffBench poses a significant challenge for
state-of-the-art large multimodal models (LMMs), such as GPT-4o and Qwen2-VL.
By analyzing failure cases of LMMs on VidDiffBench, we highlight two key
challenges for this task: localizing relevant sub-actions over two videos and
fine-grained frame comparison. To overcome these, we propose the VidDiff
method, an agentic workflow that breaks the task into three stages: action
difference proposal, keyframe localization, and frame differencing, each stage
utilizing specialized foundation models. To encourage future research in this
new task, we release the benchmark at
https://huggingface.co/datasets/jmhb/VidDiffBench and code at
http://jmhb0.github.io/viddiff.
Mixture of Experts Made Intrinsically Interpretable
Authors: name: Xingyi Yang, name: Constantin Venhoff, name: Ashkan Khakzar, name: Christian Schroeder de Witt, name: Puneet K. Dokania, name: Adel Bibi, name: Philip Torr
Published: 2025-03-05 17:40:54

Neurons in large language models often exhibit polysemanticity,
simultaneously encoding multiple unrelated concepts and obscuring
interpretability. Instead of relying on post-hoc methods, we present
MoE-X, a Mixture-of-Experts (MoE) language model designed to be
intrinsically interpretable. Our approach is motivated by the
observation that, in language models, wider networks with sparse activations
are more likely to capture interpretable factors. However, directly training
such large sparse networks is computationally prohibitive. MoE architectures
offer a scalable alternative by activating only a subset of experts for any
given input, inherently aligning with interpretability objectives. In MoE-X, we
establish this connection by rewriting the MoE layer as an equivalent sparse,
large MLP. This approach enables efficient scaling of the hidden size while
maintaining sparsity. To further enhance interpretability, we enforce sparse
activation within each expert and redesign the routing mechanism to prioritize
experts with the highest activation sparsity. These designs ensure that only
the most salient features are routed and processed by the experts. We evaluate
MoE-X on chess and natural language tasks, showing that it achieves performance
comparable to dense models while significantly improving interpretability.
MoE-X achieves a perplexity better than GPT-2, with interpretability surpassing
even sparse autoencoder (SAE)-based approaches.
Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning
Authors: name: Yuxiao Qu, name: Matthew Y. R. Yang, name: Amrith Setlur, name: Lewis Tunstall, name: Edward Emanuel Beeching, name: Ruslan Salakhutdinov, name: Aviral Kumar
Published: 2025-03-10 17:40:43

Training models to effectively use test-time compute is crucial for improving
the reasoning performance of LLMs. Current methods mostly do so via fine-tuning
on search traces or running RL with 0/1 outcome reward, but do these approaches
efficiently utilize test-time compute? Would these approaches continue to scale
as the budget improves? In this paper, we try to answer these questions. We
formalize the problem of optimizing test-time compute as a meta-reinforcement
learning (RL) problem, which provides a principled perspective on spending
test-time compute. This perspective enables us to view the long output stream
from the LLM as consisting of several episodes run at test time and leads us to
use a notion of cumulative regret over output tokens as a way to measure the
efficacy of test-time compute. Akin to how RL algorithms can best tradeoff
exploration and exploitation over training, minimizing cumulative regret would
also provide the best balance between exploration and exploitation in the token
stream. While we show that state-of-the-art models do not minimize regret, one
can do so by maximizing a dense reward bonus in conjunction with the outcome
0/1 reward RL. This bonus is the ''progress'' made by each subsequent block in
the output stream, quantified by the change in the likelihood of eventual
success. Using these insights, we develop Meta Reinforcement Fine-Tuning, or
MRT, a new class of fine-tuning methods for optimizing test-time compute. MRT
leads to a 2-3x relative gain in performance and roughly a 1.5x gain in token
efficiency for math reasoning compared to outcome-reward RL.
Next Token Is Enough: Realistic Image Quality and Aesthetic Scoring with
  Multimodal Large Language Model
Authors: name: Mingxing Li, name: Rui Wang, name: Lei Sun, name: Yancheng Bai, name: Xiangxiang Chu
Published: 2025-03-08 09:49:10

The rapid expansion of mobile internet has resulted in a substantial increase
in user-generated content (UGC) images, thereby making the thorough assessment
of UGC images both urgent and essential. Recently, multimodal large language
models (MLLMs) have shown great potential in image quality assessment (IQA) and
image aesthetic assessment (IAA). Despite this progress, effectively scoring
the quality and aesthetics of UGC images still faces two main challenges: 1) A
single score is inadequate to capture the hierarchical human perception. 2) How
to use MLLMs to output numerical scores, such as mean opinion scores (MOS),
remains an open question. To address these challenges, we introduce a novel
dataset, named Realistic image Quality and Aesthetic (RealQA), including 14,715
UGC images, each of which is annoted with 10 fine-grained attributes. These
attributes span three levels: low level (e.g., image clarity), middle level
(e.g., subject integrity) and high level (e.g., composition). Besides, we
conduct a series of in-depth and comprehensive investigations into how to
effectively predict numerical scores using MLLMs. Surprisingly, by predicting
just two extra significant digits, the next token paradigm can achieve SOTA
performance. Furthermore, with the help of chain of thought (CoT) combined with
the learnt fine-grained attributes, the proposed method can outperform SOTA
methods on five public datasets for IQA and IAA with superior interpretability
and show strong zero-shot generalization for video quality assessment (VQA).
The code and dataset will be released.
PhiloBERTA: A Transformer-Based Cross-Lingual Analysis of Greek and
  Latin Lexicons
Authors: name: Rumi A. Allbert, name: Makai L. Allbert
Published: 2025-03-07 09:30:16

We present PhiloBERTA, a cross-lingual transformer model that measures
semantic relationships between ancient Greek and Latin lexicons. Through
analysis of selected term pairs from classical texts, we use contextual
embeddings and angular similarity metrics to identify precise semantic
alignments. Our results show that etymologically related pairs demonstrate
significantly higher similarity scores, particularly for abstract philosophical
concepts such as epist\=em\=e (scientia) and dikaiosyn\=e (iustitia).
Statistical analysis reveals consistent patterns in these relationships (p =
0.012), with etymologically related pairs showing remarkably stable semantic
preservation compared to control pairs. These findings establish a quantitative
framework for examining how philosophical concepts moved between Greek and
Latin traditions, offering new methods for classical philological research.
Feynman-Kac Correctors in Diffusion: Annealing, Guidance, and Product of
  Experts
Authors: name: Marta Skreta, name: Tara Akhound-Sadegh, name: Viktor Ohanesian, name: Roberto Bondesan, name: Alán Aspuru-Guzik, name: Arnaud Doucet, name: Rob Brekelmans, name: Alexander Tong, name: Kirill Neklyudov
Published: 2025-03-04 17:46:51

While score-based generative models are the model of choice across diverse
domains, there are limited tools available for controlling inference-time
behavior in a principled manner, e.g. for composing multiple pretrained models.
Existing classifier-free guidance methods use a simple heuristic to mix
conditional and unconditional scores to approximately sample from conditional
distributions. However, such methods do not approximate the intermediate
distributions, necessitating additional 'corrector' steps. In this work, we
provide an efficient and principled method for sampling from a sequence of
annealed, geometric-averaged, or product distributions derived from pretrained
score-based models. We derive a weighted simulation scheme which we call
Feynman-Kac Correctors (FKCs) based on the celebrated Feynman-Kac formula by
carefully accounting for terms in the appropriate partial differential
equations (PDEs). To simulate these PDEs, we propose Sequential Monte Carlo
(SMC) resampling algorithms that leverage inference-time scaling to improve
sampling quality. We empirically demonstrate the utility of our methods by
proposing amortized sampling via inference-time temperature annealing,
improving multi-objective molecule generation using pretrained models, and
improving classifier-free guidance for text-to-image generation. Our code is
available at https://github.com/martaskrt/fkc-diffusion.
What's in a Latent? Leveraging Diffusion Latent Space for Domain
  Generalization
Authors: name: Xavier Thomas, name: Deepti Ghadiyaram
Published: 2025-03-09 17:29:01

Domain Generalization aims to develop models that can generalize to novel and
unseen data distributions. In this work, we study how model architectures and
pre-training objectives impact feature richness and propose a method to
effectively leverage them for domain generalization. Specifically, given a
pre-trained feature space, we first discover latent domain structures, referred
to as pseudo-domains, that capture domain-specific variations in an
unsupervised manner. Next, we augment existing classifiers with these
complementary pseudo-domain representations making them more amenable to
diverse unseen test domains. We analyze how different pre-training feature
spaces differ in the domain-specific variances they capture. Our empirical
studies reveal that features from diffusion models excel at separating domains
in the absence of explicit domain labels and capture nuanced domain-specific
information. On 5 datasets, we show that our very simple framework improves
generalization to unseen domains by a maximum test accuracy improvement of over
4% compared to the standard baseline Empirical Risk Minimization (ERM).
Crucially, our method outperforms most algorithms that access domain labels
during training.