Orionnnn/hub-stats huggingface dataset:
Distilling Diversity and Control in Diffusion Models
Authors: name: Rohit Gandikota, name: David Bau
Published: 2025-03-13 17:59:56

Distilled diffusion models suffer from a critical limitation: reduced sample
diversity compared to their base counterparts. In this work, we uncover that
despite this diversity loss, distilled models retain the fundamental concept
representations of base models. We demonstrate control distillation - where
control mechanisms like Concept Sliders and LoRAs trained on base models can be
seamlessly transferred to distilled models and vice-versa, effectively
distilling control without any retraining. This preservation of
representational structure prompted our investigation into the mechanisms of
diversity collapse during distillation. To understand how distillation affects
diversity, we introduce Diffusion Target (DT) Visualization, an analysis and
debugging tool that reveals how models predict final outputs at intermediate
steps. Through DT-Visualization, we identify generation artifacts,
inconsistencies, and demonstrate that initial diffusion timesteps
disproportionately determine output diversity, while later steps primarily
refine details. Based on these insights, we introduce diversity distillation -
a hybrid inference approach that strategically employs the base model for only
the first critical timestep before transitioning to the efficient distilled
model. Our experiments demonstrate that this simple modification not only
restores the diversity capabilities from base to distilled models but
surprisingly exceeds it, while maintaining nearly the computational efficiency
of distilled inference, all without requiring additional training or model
modifications. Our code and data are available at
https://distillation.baulab.info
UniGoal: Towards Universal Zero-shot Goal-oriented Navigation
Authors: name: Hang Yin, name: Xiuwei Xu, name: Lingqing Zhao, name: Ziwei Wang, name: Jie Zhou, name: Jiwen Lu
Published: 2025-03-13 17:59:48

In this paper, we propose a general framework for universal zero-shot
goal-oriented navigation. Existing zero-shot methods build inference framework
upon large language models (LLM) for specific tasks, which differs a lot in
overall pipeline and fails to generalize across different types of goal.
Towards the aim of universal zero-shot navigation, we propose a uniform graph
representation to unify different goals, including object category, instance
image and text description. We also convert the observation of agent into an
online maintained scene graph. With this consistent scene and goal
representation, we preserve most structural information compared with pure text
and are able to leverage LLM for explicit graph-based reasoning. Specifically,
we conduct graph matching between the scene graph and goal graph at each time
instant and propose different strategies to generate long-term goal of
exploration according to different matching states. The agent first iteratively
searches subgraph of goal when zero-matched. With partial matching, the agent
then utilizes coordinate projection and anchor pair alignment to infer the goal
location. Finally scene graph correction and goal verification are applied for
perfect matching. We also present a blacklist mechanism to enable robust switch
between stages. Extensive experiments on several benchmarks show that our
UniGoal achieves state-of-the-art zero-shot performance on three studied
navigation tasks with a single model, even outperforming task-specific
zero-shot methods and supervised universal methods.
World Modeling Makes a Better Planner: Dual Preference Optimization for
  Embodied Task Planning
Authors: name: Siyin Wang, name: Zhaoye Fei, name: Qinyuan Cheng, name: Shiduo Zhang, name: Panpan Cai, name: Jinlan Fu, name: Xipeng Qiu
Published: 2025-03-13 15:49:56

Recent advances in large vision-language models (LVLMs) have shown promise
for embodied task planning, yet they struggle with fundamental challenges like
dependency constraints and efficiency. Existing approaches either solely
optimize action selection or leverage world models during inference,
overlooking the benefits of learning to model the world as a way to enhance
planning capabilities. We propose Dual Preference Optimization (D^2PO), a new
learning framework that jointly optimizes state prediction and action selection
through preference learning, enabling LVLMs to understand environment dynamics
for better planning. To automatically collect trajectories and stepwise
preference data without human annotation, we introduce a tree search mechanism
for extensive exploration via trial-and-error. Extensive experiments on
VoTa-Bench demonstrate that our D^2PO-based method significantly outperforms
existing methods and GPT-4o when applied to Qwen2-VL (7B), LLaVA-1.6 (7B), and
LLaMA-3.2 (11B), achieving superior task success rates with more efficient
execution paths.
CoSTAast: Cost-Sensitive Toolpath Agent for Multi-turn Image Editing
Authors: name: Advait Gupta, name: NandaKiran Velaga, name: Dang Nguyen, name: Tianyi Zhou
Published: 2025-03-13 17:55:45

Text-to-image models like stable diffusion and DALLE-3 still struggle with
multi-turn image editing. We decompose such a task as an agentic workflow
(path) of tool use that addresses a sequence of subtasks by AI tools of varying
costs. Conventional search algorithms require expensive exploration to find
tool paths. While large language models (LLMs) possess prior knowledge of
subtask planning, they may lack accurate estimations of capabilities and costs
of tools to determine which to apply in each subtask. Can we combine the
strengths of both LLMs and graph search to find cost-efficient tool paths? We
propose a three-stage approach "CoSTA*" that leverages LLMs to create a subtask
tree, which helps prune a graph of AI tools for the given task, and then
conducts A* search on the small subgraph to find a tool path. To better balance
the total cost and quality, CoSTA* combines both metrics of each tool on every
subtask to guide the A* search. Each subtask's output is then evaluated by a
vision-language model (VLM), where a failure will trigger an update of the
tool's cost and quality on the subtask. Hence, the A* search can recover from
failures quickly to explore other paths. Moreover, CoSTA* can automatically
switch between modalities across subtasks for a better cost-quality trade-off.
We build a novel benchmark of challenging multi-turn image editing, on which
CoSTA* outperforms state-of-the-art image-editing models or agents in terms of
both cost and quality, and performs versatile trade-offs upon user preference.
New Trends for Modern Machine Translation with Large Reasoning Models
Authors: name: Sinuo Liu, name: Chenyang Lyu, name: Minghao Wu, name: Longyue Wang, name: Weihua Luo, name: Kaifu Zhang
Published: 2025-03-13 13:27:53

Recent advances in Large Reasoning Models (LRMs), particularly those
leveraging Chain-of-Thought reasoning (CoT), have opened brand new possibility
for Machine Translation (MT). This position paper argues that LRMs
substantially transformed traditional neural MT as well as LLMs-based MT
paradigms by reframing translation as a dynamic reasoning task that requires
contextual, cultural, and linguistic understanding and reasoning. We identify
three foundational shifts: 1) contextual coherence, where LRMs resolve
ambiguities and preserve discourse structure through explicit reasoning over
cross-sentence and complex context or even lack of context; 2) cultural
intentionality, enabling models to adapt outputs by inferring speaker intent,
audience expectations, and socio-linguistic norms; 3) self-reflection, LRMs can
perform self-reflection during the inference time to correct the potential
errors in translation especially extremely noisy cases, showing better
robustness compared to simply mapping X->Y translation. We explore various
scenarios in translation including stylized translation, document-level
translation and multimodal translation by showcasing empirical examples that
demonstrate the superiority of LRMs in translation. We also identify several
interesting phenomenons for LRMs for MT including auto-pivot translation as
well as the critical challenges such as over-localisation in translation and
inference efficiency. In conclusion, we think that LRMs redefine translation
systems not merely as text converters but as multilingual cognitive agents
capable of reasoning about meaning beyond the text. This paradigm shift reminds
us to think of problems in translation beyond traditional translation scenarios
in a much broader context with LRMs - what we can achieve on top of it.
The Curse of Conditions: Analyzing and Improving Optimal Transport for
  Conditional Flow-Based Generation
Authors: name: Ho Kei Cheng, name: Alexander Schwing
Published: 2025-03-13 17:59:56

Minibatch optimal transport coupling straightens paths in unconditional flow
matching. This leads to computationally less demanding inference as fewer
integration steps and less complex numerical solvers can be employed when
numerically solving an ordinary differential equation at test time. However, in
the conditional setting, minibatch optimal transport falls short. This is
because the default optimal transport mapping disregards conditions, resulting
in a conditionally skewed prior distribution during training. In contrast, at
test time, we have no access to the skewed prior, and instead sample from the
full, unbiased prior distribution. This gap between training and testing leads
to a subpar performance. To bridge this gap, we propose conditional optimal
transport C^2OT that adds a conditional weighting term in the cost matrix when
computing the optimal transport assignment. Experiments demonstrate that this
simple fix works with both discrete and continuous conditions in
8gaussians-to-moons, CIFAR-10, ImageNet-32x32, and ImageNet-256x256. Our method
performs better overall compared to the existing baselines across different
function evaluation budgets. Code is available at
https://hkchengrex.github.io/C2OT
CINEMA: Coherent Multi-Subject Video Generation via MLLM-Based Guidance
Authors: name: Yufan Deng, name: Xun Guo, name: Yizhi Wang, name: Jacob Zhiyuan Fang, name: Angtian Wang, name: Shenghai Yuan, name: Yiding Yang, name: Bo Liu, name: Haibin Huang, name: Chongyang Ma
Published: 2025-03-13 14:07:58

Video generation has witnessed remarkable progress with the advent of deep
generative models, particularly diffusion models. While existing methods excel
in generating high-quality videos from text prompts or single images,
personalized multi-subject video generation remains a largely unexplored
challenge. This task involves synthesizing videos that incorporate multiple
distinct subjects, each defined by separate reference images, while ensuring
temporal and spatial consistency. Current approaches primarily rely on mapping
subject images to keywords in text prompts, which introduces ambiguity and
limits their ability to model subject relationships effectively. In this paper,
we propose CINEMA, a novel framework for coherent multi-subject video
generation by leveraging Multimodal Large Language Model (MLLM). Our approach
eliminates the need for explicit correspondences between subject images and
text entities, mitigating ambiguity and reducing annotation effort. By
leveraging MLLM to interpret subject relationships, our method facilitates
scalability, enabling the use of large and diverse datasets for training.
Furthermore, our framework can be conditioned on varying numbers of subjects,
offering greater flexibility in personalized content creation. Through
extensive evaluations, we demonstrate that our approach significantly improves
subject consistency, and overall video coherence, paving the way for advanced
applications in storytelling, interactive media, and personalized video
generation.
Shifting Long-Context LLMs Research from Input to Output
Authors: name: Yuhao Wu, name: Yushi Bai, name: Zhiqing Hu, name: Shangqing Tu, name: Ming Shan Hee, name: Juanzi Li, name: Roy Ka-Wei Lee
Published: 2025-03-06 18:59:37

Recent advancements in long-context Large Language Models (LLMs) have
primarily concentrated on processing extended input contexts, resulting in
significant strides in long-context comprehension. However, the equally
critical aspect of generating long-form outputs has received comparatively less
attention. This paper advocates for a paradigm shift in NLP research toward
addressing the challenges of long-output generation. Tasks such as novel
writing, long-term planning, and complex reasoning require models to understand
extensive contexts and produce coherent, contextually rich, and logically
consistent extended text. These demands highlight a critical gap in current LLM
capabilities. We underscore the importance of this under-explored domain and
call for focused efforts to develop foundational LLMs tailored for generating
high-quality, long-form outputs, which hold immense potential for real-world
applications.
Quantization for OpenAI's Whisper Models: A Comparative Analysis
Authors: name: Allison Andreyev
Published: 2025-03-12 23:50:35

Automated speech recognition (ASR) models have gained prominence for
applications such as captioning, speech translation, and live transcription.
This paper studies Whisper and two model variants: one optimized for live
speech streaming and another for offline transcription. Notably, these models
have been found to generate hallucinated content, reducing transcription
reliability. Furthermore, larger model variants exhibit increased latency and
pose challenges for deployment on resource-constrained devices. This study
analyzes the similarities and differences between three Whisper models,
qualitatively examining their distinct capabilities. Next, this study
quantifies the impact of model quantization on latency and evaluates its
viability for edge deployment. Using the open source LibriSpeech dataset, this
paper evaluates the word error rate (WER) along with latency analysis of
whispercpp using 3 quantization methods (INT4, INT5, INT8). Results show that
quantization reduces latency by 19\% and model size by 45\%, while preserving
transcription accuracy. These findings provide insights into the optimal use
cases of different Whisper models and edge device deployment possibilities. All
code, datasets, and implementation details are available in a public GitHub
repository: https://github.com/allisonandreyev/WhisperQuantization.git
VisualWebInstruct: Scaling up Multimodal Instruction Data through Web
  Search
Authors: name: Yiming Jia, name: Jiachen Li, name: Xiang Yue, name: Bo Li, name: Ping Nie, name: Kai Zou, name: Wenhu Chen
Published: 2025-03-13 17:32:48

Vision-Language Models have made significant progress on many
perception-focused tasks, however, their progress on reasoning-focused tasks
seem to be limited due to the lack of high-quality and diverse training data.
In this work, we aim to address the scarcity issue of reasoning-focused
multimodal datasets. We propose VisualWebInstruct - a novel approach that
leverages search engine to create a diverse, and high-quality dataset spanning
multiple disciplines like math, physics, finance, chemistry, etc. Starting with
meticulously selected 30,000 seed images, we employ Google Image search to
identify websites containing similar images. We collect and process the HTMLs
from over 700K unique URL sources. Through a pipeline of content extraction,
filtering and synthesis, we build a dataset of approximately 900K
question-answer pairs, with 40% being visual QA pairs and the rest as text QA
pairs. Models fine-tuned on VisualWebInstruct demonstrate significant
performance gains: (1) training from Llava-OV-mid shows 10-20% absolute point
gains across benchmarks, (2) training from MAmmoTH-VL shows 5% absoluate gain.
Our best model MAmmoTH-VL2 shows state-of-the-art performance within the 10B
parameter class on MMMU-Pro-std (40.7%), MathVerse (42.6%), and DynaMath
(55.7%). These remarkable results highlight the effectiveness of our dataset in
enhancing VLMs' reasoning capabilities for complex multimodal tasks.
"Silent Is Not Actually Silent": An Investigation of Toxicity on Bug
  Report Discussion
Authors: name: Mia Mohammad Imran, name: Jaydeb Sarker
Published: 2025-03-13 05:39:29

Toxicity in bug report discussions poses significant challenges to the
collaborative dynamics of open-source software development. Bug reports are
crucial for identifying and resolving defects, yet their inherently
problem-focused nature and emotionally charged context make them susceptible to
toxic interactions. This study explores toxicity in GitHub bug reports through
a qualitative analysis of 203 bug threads, including 81 toxic ones. Our
findings reveal that toxicity frequently arises from misaligned perceptions of
bug severity and priority, unresolved frustrations with tools, and lapses in
professional communication. These toxic interactions not only derail productive
discussions but also reduce the likelihood of actionable outcomes, such as
linking issues with pull requests. Our preliminary findings offer actionable
recommendations to improve bug resolution by mitigating toxicity.
Search-R1: Training LLMs to Reason and Leverage Search Engines with
  Reinforcement Learning
Authors: name: Bowen Jin, name: Hansi Zeng, name: Zhenrui Yue, name: Dong Wang, name: Hamed Zamani, name: Jiawei Han
Published: 2025-03-12 16:26:39

Efficiently acquiring external knowledge and up-to-date information is
essential for effective reasoning and text generation in large language models
(LLMs). Retrieval augmentation and tool-use training approaches where a search
engine is treated as a tool lack complex multi-turn retrieval flexibility or
require large-scale supervised data. Prompting advanced LLMs with reasoning
capabilities during inference to use search engines is not optimal, since the
LLM does not learn how to optimally interact with the search engine. This paper
introduces Search-R1, an extension of the DeepSeek-R1 model where the LLM
learns -- solely through reinforcement learning (RL) -- to autonomously
generate (multiple) search queries during step-by-step reasoning with real-time
retrieval. Search-R1 optimizes LLM rollouts with multi-turn search
interactions, leveraging retrieved token masking for stable RL training and a
simple outcome-based reward function. Experiments on seven question-answering
datasets show that Search-R1 improves performance by 26% (Qwen2.5-7B), 21%
(Qwen2.5-3B), and 10% (LLaMA3.2-3B) over SOTA baselines. This paper further
provides empirical insights into RL optimization methods, LLM choices, and
response length dynamics in retrieval-augmented reasoning. The code and model
checkpoints are available at https://github.com/PeterGriffinJin/Search-R1.
Understanding and Mitigating Distribution Shifts For Machine Learning
  Force Fields
Authors: name: Tobias Kreiman, name: Aditi S. Krishnapriyan
Published: 2025-03-11 17:54:29

Machine Learning Force Fields (MLFFs) are a promising alternative to
expensive ab initio quantum mechanical molecular simulations. Given the
diversity of chemical spaces that are of interest and the cost of generating
new data, it is important to understand how MLFFs generalize beyond their
training distributions. In order to characterize and better understand
distribution shifts in MLFFs, we conduct diagnostic experiments on chemical
datasets, revealing common shifts that pose significant challenges, even for
large foundation models trained on extensive data. Based on these observations,
we hypothesize that current supervised training methods inadequately regularize
MLFFs, resulting in overfitting and learning poor representations of
out-of-distribution systems. We then propose two new methods as initial steps
for mitigating distribution shifts for MLFFs. Our methods focus on test-time
refinement strategies that incur minimal computational cost and do not use
expensive ab initio reference labels. The first strategy, based on spectral
graph theory, modifies the edges of test graphs to align with graph structures
seen during training. Our second strategy improves representations for
out-of-distribution systems at test-time by taking gradient steps using an
auxiliary objective, such as a cheap physical prior. Our test-time refinement
strategies significantly reduce errors on out-of-distribution systems,
suggesting that MLFFs are capable of and can move towards modeling diverse
chemical spaces, but are not being effectively trained to do so. Our
experiments establish clear benchmarks for evaluating the generalization
capabilities of the next generation of MLFFs. Our code is available at
https://tkreiman.github.io/projects/mlff_distribution_shifts/.
PhysicsGen: Can Generative Models Learn from Images to Predict Complex
  Physical Relations?
Authors: name: Martin Spitznagel, name: Jan Vaillant, name: Janis Keuper
Published: 2025-03-07 11:19:13

The image-to-image translation abilities of generative learning models have
recently made significant progress in the estimation of complex (steered)
mappings between image distributions. While appearance based tasks like image
in-painting or style transfer have been studied at length, we propose to
investigate the potential of generative models in the context of physical
simulations. Providing a dataset of 300k image-pairs and baseline evaluations
for three different physical simulation tasks, we propose a benchmark to
investigate the following research questions: i) are generative models able to
learn complex physical relations from input-output image pairs? ii) what
speedups can be achieved by replacing differential equation based simulations?
While baseline evaluations of different current models show the potential for
high speedups (ii), these results also show strong limitations toward the
physical correctness (i). This underlines the need for new methods to enforce
physical correctness. Data, baseline models and evaluation code
http://www.physics-gen.org.
BIMBA: Selective-Scan Compression for Long-Range Video Question
  Answering
Authors: name: Md Mohaiminul Islam, name: Tushar Nagarajan, name: Huiyu Wang, name: Gedas Bertasius, name: Lorenzo Torresani
Published: 2025-03-12 17:57:32

Video Question Answering (VQA) in long videos poses the key challenge of
extracting relevant information and modeling long-range dependencies from many
redundant frames. The self-attention mechanism provides a general solution for
sequence modeling, but it has a prohibitive cost when applied to a massive
number of spatiotemporal tokens in long videos. Most prior methods rely on
compression strategies to lower the computational cost, such as reducing the
input length via sparse frame sampling or compressing the output sequence
passed to the large language model (LLM) via space-time pooling. However, these
naive approaches over-represent redundant information and often miss salient
events or fast-occurring space-time patterns. In this work, we introduce BIMBA,
an efficient state-space model to handle long-form videos. Our model leverages
the selective scan algorithm to learn to effectively select critical
information from high-dimensional video and transform it into a reduced token
sequence for efficient LLM processing. Extensive experiments demonstrate that
BIMBA achieves state-of-the-art accuracy on multiple long-form VQA benchmarks,
including PerceptionTest, NExT-QA, EgoSchema, VNBench, LongVideoBench, and
Video-MME. Code, and models are publicly available at
https://sites.google.com/view/bimba-mllm.