Orionnnn/hub-stats huggingface dataset:
Effectively Controlling Reasoning Models through Thinking Intervention
Authors: name: Tong Wu, name: Chong Xiang, name: Jiachen T. Wang, name: Prateek Mittal
Published: 2025-03-31 17:50:13

Reasoning-enhanced large language models (LLMs) explicitly generate
intermediate reasoning steps prior to generating final answers, helping the
model excel in complex problem-solving. In this paper, we demonstrate that this
emerging generation framework offers a unique opportunity for more fine-grained
control over model behavior. We propose Thinking Intervention, a novel paradigm
designed to explicitly guide the internal reasoning processes of LLMs by
strategically inserting or revising specific thinking tokens. We conduct
comprehensive evaluations across multiple tasks, including instruction
following on IFEval, instruction hierarchy on SEP, and safety alignment on
XSTest and SORRY-Bench. Our results demonstrate that Thinking Intervention
significantly outperforms baseline prompting approaches, achieving up to 6.7%
accuracy gains in instruction-following scenarios, 15.4% improvements in
reasoning about instruction hierarchies, and a 40.0% increase in refusal rates
for unsafe prompts using open-source DeepSeek R1 models. Overall, our work
opens a promising new research avenue for controlling reasoning LLMs.
MoCha: Towards Movie-Grade Talking Character Synthesis
Authors: name: Cong Wei, name: Bo Sun, name: Haoyu Ma, name: Ji Hou, name: Felix Juefei-Xu, name: Zecheng He, name: Xiaoliang Dai, name: Luxin Zhang, name: Kunpeng Li, name: Tingbo Hou, name: Animesh Sinha, name: Peter Vajda, name: Wenhu Chen
Published: 2025-03-30 04:22:09

Recent advancements in video generation have achieved impressive motion
realism, yet they often overlook character-driven storytelling, a crucial task
for automated film, animation generation. We introduce Talking Characters, a
more realistic task to generate talking character animations directly from
speech and text. Unlike talking head, Talking Characters aims at generating the
full portrait of one or more characters beyond the facial region. In this
paper, we propose MoCha, the first of its kind to generate talking characters.
To ensure precise synchronization between video and speech, we propose a
speech-video window attention mechanism that effectively aligns speech and
video tokens. To address the scarcity of large-scale speech-labeled video
datasets, we introduce a joint training strategy that leverages both
speech-labeled and text-labeled video data, significantly improving
generalization across diverse character actions. We also design structured
prompt templates with character tags, enabling, for the first time,
multi-character conversation with turn-based dialogue-allowing AI-generated
characters to engage in context-aware conversations with cinematic coherence.
Extensive qualitative and quantitative evaluations, including human preference
studies and benchmark comparisons, demonstrate that MoCha sets a new standard
for AI-generated cinematic storytelling, achieving superior realism,
expressiveness, controllability and generalization.
Classical Planning with LLM-Generated Heuristics: Challenging the State
  of the Art with Python Code
Authors: name: Augusto B. Corrêa, name: André G. Pereira, name: Jendrik Seipp
Published: 2025-03-24 15:50:20

In recent years, large language models (LLMs) have shown remarkable
capabilities in various artificial intelligence problems. However, they fail to
plan reliably, even when prompted with a detailed definition of the planning
task. Attempts to improve their planning capabilities, such as chain-of-thought
prompting, fine-tuning, and explicit "reasoning" still yield incorrect plans
and usually fail to generalize to larger tasks. In this paper, we show how to
use LLMs to generate correct plans, even for out-of-distribution tasks of
increasing size. For a given planning domain, we ask an LLM to generate several
domain-dependent heuristic functions in the form of Python code, evaluate them
on a set of training tasks within a greedy best-first search, and choose the
strongest one. The resulting LLM-generated heuristics solve many more unseen
test tasks than state-of-the-art domain-independent heuristics for classical
planning. They are even competitive with the strongest learning algorithm for
domain-dependent planning. These findings are especially remarkable given that
our proof-of-concept implementation is based on an unoptimized Python planner
and the baselines all build upon highly optimized C++ code. In some domains,
the LLM-generated heuristics expand fewer states than the baselines, revealing
that they are not only efficiently computable, but sometimes even more
informative than the state-of-the-art heuristics. Overall, our results show
that sampling a set of planning heuristic function programs can significantly
improve the planning capabilities of LLMs.
Bridging Evolutionary Multiobjective Optimization and GPU Acceleration
  via Tensorization
Authors: name: Zhenyu Liang, name: Hao Li, name: Naiwei Yu, name: Kebin Sun, name: Ran Cheng
Published: 2025-03-26 07:30:23

Evolutionary multiobjective optimization (EMO) has made significant strides
over the past two decades. However, as problem scales and complexities
increase, traditional EMO algorithms face substantial performance limitations
due to insufficient parallelism and scalability. While most work has focused on
algorithm design to address these challenges, little attention has been given
to hardware acceleration, thereby leaving a clear gap between EMO algorithms
and advanced computing devices, such as GPUs. To bridge the gap, we propose to
parallelize EMO algorithms on GPUs via the tensorization methodology. By
employing tensorization, the data structures and operations of EMO algorithms
are transformed into concise tensor representations, which seamlessly enables
automatic utilization of GPU computing. We demonstrate the effectiveness of our
approach by applying it to three representative EMO algorithms: NSGA-III,
MOEA/D, and HypE. To comprehensively assess our methodology, we introduce a
multiobjective robot control benchmark using a GPU-accelerated physics engine.
Our experiments show that the tensorized EMO algorithms achieve speedups of up
to 1113x compared to their CPU-based counterparts, while maintaining solution
quality and effectively scaling population sizes to hundreds of thousands.
Furthermore, the tensorized EMO algorithms efficiently tackle complex
multiobjective robot control tasks, producing high-quality solutions with
diverse behaviors. Source codes are available at
https://github.com/EMI-Group/evomo.
Zero4D: Training-Free 4D Video Generation From Single Video Using
  Off-the-Shelf Video Diffusion Model
Authors: name: Jangho Park, name: Taesung Kwon, name: Jong Chul Ye
Published: 2025-03-28 17:14:48

Recently, multi-view or 4D video generation has emerged as a significant
research topic. Nonetheless, recent approaches to 4D generation still struggle
with fundamental limitations, as they primarily rely on harnessing multiple
video diffusion models with additional training or compute-intensive training
of a full 4D diffusion model with limited real-world 4D data and large
computational costs. To address these challenges, here we propose the first
training-free 4D video generation method that leverages the off-the-shelf video
diffusion models to generate multi-view videos from a single input video. Our
approach consists of two key steps: (1) By designating the edge frames in the
spatio-temporal sampling grid as key frames, we first synthesize them using a
video diffusion model, leveraging a depth-based warping technique for guidance.
This approach ensures structural consistency across the generated frames,
preserving spatial and temporal coherence. (2) We then interpolate the
remaining frames using a video diffusion model, constructing a fully populated
and temporally coherent sampling grid while preserving spatial and temporal
consistency. Through this approach, we extend a single video into a multi-view
video along novel camera trajectories while maintaining spatio-temporal
consistency. Our method is training-free and fully utilizes an off-the-shelf
video diffusion model, offering a practical and effective solution for
multi-view video generation.
Challenges and Paths Towards AI for Software Engineering
Authors: name: Alex Gu, name: Naman Jain, name: Wen-Ding Li, name: Manish Shetty, name: Yijia Shao, name: Ziyang Li, name: Diyi Yang, name: Kevin Ellis, name: Koushik Sen, name: Armando Solar-Lezama
Published: 2025-03-28 17:17:57

AI for software engineering has made remarkable progress recently, becoming a
notable success within generative AI. Despite this, there are still many
challenges that need to be addressed before automated software engineering
reaches its full potential. It should be possible to reach high levels of
automation where humans can focus on the critical decisions of what to build
and how to balance difficult tradeoffs while most routine development effort is
automated away. Reaching this level of automation will require substantial
research and engineering efforts across academia and industry. In this paper,
we aim to discuss progress towards this in a threefold manner. First, we
provide a structured taxonomy of concrete tasks in AI for software engineering,
emphasizing the many other tasks in software engineering beyond code generation
and completion. Second, we outline several key bottlenecks that limit current
approaches. Finally, we provide an opinionated list of promising research
directions toward making progress on these bottlenecks, hoping to inspire
future research in this rapidly maturing field.
SWI: Speaking with Intent in Large Language Models
Authors: name: Yuwei Yin, name: EunJeong Hwang, name: Giuseppe Carenini
Published: 2025-03-27 14:34:28

Intent, typically clearly formulated and planned, functions as a cognitive
framework for reasoning and problem-solving. This paper introduces the concept
of Speaking with Intent (SWI) in large language models (LLMs), where the
explicitly generated intent encapsulates the model's underlying intention and
provides high-level planning to guide subsequent analysis and communication. By
emulating deliberate and purposeful thoughts in the human mind, SWI is
hypothesized to enhance the reasoning capabilities and generation quality of
LLMs. Extensive experiments on mathematical reasoning benchmarks consistently
demonstrate the superiority of Speaking with Intent over Baseline (i.e.,
generation without explicit intent). Moreover, SWI outperforms answer-trigger
prompting methods Chain-of-Thought and Plan-and-Solve and maintains competitive
performance with the strong method ARR (Analyzing, Retrieving, and Reasoning).
Additionally, the effectiveness and generalizability of SWI are solidified on
reasoning-intensive question answering (QA) and text summarization benchmarks,
where SWI brings consistent improvement to the Baseline generation. In text
summarization, SWI-generated summaries exhibit greater accuracy, conciseness,
and factual correctness, with fewer hallucinations. Furthermore, human
evaluations verify the coherence, effectiveness, and interpretability of the
intent produced by SWI. This proof-of-concept study creates a novel avenue for
enhancing LLMs' reasoning abilities with cognitive notions.
OThink-MR1: Stimulating multimodal generalized reasoning capabilities
  via dynamic reinforcement learning
Authors: name: Zhiyuan Liu, name: Yuting Zhang, name: Feng Liu, name: Changwang Zhang, name: Ying Sun, name: Jun Wang
Published: 2025-03-20 12:22:18

Multimodal Large Language Models (MLLMs) have gained significant traction for
their ability to process diverse input data types and generate coherent,
contextually relevant outputs across various applications. While supervised
fine-tuning (SFT) has been the predominant approach to enhance MLLM
capabilities in task-specific optimization, it often falls short in fostering
crucial generalized reasoning abilities. Although reinforcement learning (RL)
holds great promise in overcoming these limitations, it encounters two
significant challenges: (1) its generalized capacities in multimodal tasks
remain largely unexplored, and (2) its training constraints, including the
constant Kullback-Leibler divergence or the clamp strategy, often result in
suboptimal bottlenecks. To address these challenges, we propose OThink-MR1, an
advanced MLLM equipped with profound comprehension and reasoning capabilities
across multimodal tasks. Specifically, we introduce Group Relative Policy
Optimization with a dynamic Kullback-Leibler strategy (GRPO-D), which markedly
enhances reinforcement learning (RL) performance. For Qwen2-VL-2B-Instruct,
GRPO-D achieves a relative improvement of more than 5.72% over SFT and more
than 13.59% over GRPO in same-task evaluation on two adapted datasets.
Furthermore, GRPO-D demonstrates remarkable cross-task generalization
capabilities, with an average relative improvement of more than 61.63% over SFT
in cross-task evaluation. These results highlight that the MLLM trained with
GRPO-D on one multimodal task can be effectively transferred to another task,
underscoring the superior generalized reasoning capabilities of our proposed
OThink-MR1 model.
Your ViT is Secretly an Image Segmentation Model
Authors: name: Tommie Kerssies, name: Niccolò Cavagnero, name: Alexander Hermans, name: Narges Norouzi, name: Giuseppe Averta, name: Bastian Leibe, name: Gijs Dubbelman, name: Daan de Geus
Published: 2025-03-24 19:56:02

Vision Transformers (ViTs) have shown remarkable performance and scalability
across various computer vision tasks. To apply single-scale ViTs to image
segmentation, existing methods adopt a convolutional adapter to generate
multi-scale features, a pixel decoder to fuse these features, and a Transformer
decoder that uses the fused features to make predictions. In this paper, we
show that the inductive biases introduced by these task-specific components can
instead be learned by the ViT itself, given sufficiently large models and
extensive pre-training. Based on these findings, we introduce the Encoder-only
Mask Transformer (EoMT), which repurposes the plain ViT architecture to conduct
image segmentation. With large-scale models and pre-training, EoMT obtains a
segmentation accuracy similar to state-of-the-art models that use task-specific
components. At the same time, EoMT is significantly faster than these methods
due to its architectural simplicity, e.g., up to 4x faster with ViT-L. Across a
range of model sizes, EoMT demonstrates an optimal balance between segmentation
accuracy and prediction speed, suggesting that compute resources are better
spent on scaling the ViT itself rather than adding architectural complexity.
Code: https://www.tue-mps.org/eomt/.
Reconstructing Humans with a Biomechanically Accurate Skeleton
Authors: name: Yan Xia, name: Xiaowei Zhou, name: Etienne Vouga, name: Qixing Huang, name: Georgios Pavlakos
Published: 2025-03-27 17:56:24

In this paper, we introduce a method for reconstructing 3D humans from a
single image using a biomechanically accurate skeleton model. To achieve this,
we train a transformer that takes an image as input and estimates the
parameters of the model. Due to the lack of training data for this task, we
build a pipeline to produce pseudo ground truth model parameters for single
images and implement a training procedure that iteratively refines these pseudo
labels. Compared to state-of-the-art methods for 3D human mesh recovery, our
model achieves competitive performance on standard benchmarks, while it
significantly outperforms them in settings with extreme 3D poses and
viewpoints. Additionally, we show that previous reconstruction methods
frequently violate joint angle limits, leading to unnatural rotations. In
contrast, our approach leverages the biomechanically plausible degrees of
freedom making more realistic joint rotation estimates. We validate our
approach across multiple human pose estimation benchmarks. We make the code,
models and data available at: https://isshikihugh.github.io/HSMR/
4D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object
  Understanding
Authors: name: Wenxuan Zhu, name: Bing Li, name: Cheng Zheng, name: Jinjie Mai, name: Jun Chen, name: Letian Jiang, name: Abdullah Hamdi, name: Sara Rojas Martinez, name: Chia-Wen Lin, name: Mohamed Elhoseiny, name: Bernard Ghanem
Published: 2025-03-22 17:55:53

Multimodal Large Language Models (MLLMs) have demonstrated impressive 2D
image/video understanding capabilities. However, there are no publicly
standardized benchmarks to assess the abilities of MLLMs in understanding the
4D objects (3D objects with temporal evolution over time). In this paper, we
introduce 4D-Bench, the first benchmark to evaluate the capabilities of MLLMs
in 4D object understanding, featuring tasks in 4D object Question Answering (4D
object QA) and 4D object captioning. 4D-Bench provides 4D objects with diverse
categories, high-quality annotations, and tasks necessitating multi-view
spatial-temporal understanding, different from existing 2D image/video-based
benchmarks. With 4D-Bench, we evaluate a wide range of open-source and
closed-source MLLMs. The results from the 4D object captioning experiment
indicate that MLLMs generally exhibit weaker temporal understanding compared to
their appearance understanding, notably, while open-source models approach
closed-source performance in appearance understanding, they show larger
performance gaps in temporal understanding. 4D object QA yields surprising
findings: even with simple single-object videos, MLLMs perform poorly, with
state-of-the-art GPT-4o achieving only 63\% accuracy compared to the human
baseline of 91\%. These findings highlight a substantial gap in 4D object
understanding and the need for further advancements in MLLMs.
Perceptually Accurate 3D Talking Head Generation: New Definitions,
  Speech-Mesh Representation, and Evaluation Metrics
Authors: name: Lee Chae-Yeon, name: Oh Hyun-Bin, name: Han EunGi, name: Kim Sung-Bin, name: Suekyeong Nam, name: Tae-Hyun Oh
Published: 2025-03-26 08:18:57

Recent advancements in speech-driven 3D talking head generation have made
significant progress in lip synchronization. However, existing models still
struggle to capture the perceptual alignment between varying speech
characteristics and corresponding lip movements. In this work, we claim that
three criteria -- Temporal Synchronization, Lip Readability, and Expressiveness
-- are crucial for achieving perceptually accurate lip movements. Motivated by
our hypothesis that a desirable representation space exists to meet these three
criteria, we introduce a speech-mesh synchronized representation that captures
intricate correspondences between speech signals and 3D face meshes. We found
that our learned representation exhibits desirable characteristics, and we plug
it into existing models as a perceptual loss to better align lip movements to
the given speech. In addition, we utilize this representation as a perceptual
metric and introduce two other physically grounded lip synchronization metrics
to assess how well the generated 3D talking heads align with these three
criteria. Experiments show that training 3D talking head generation models with
our perceptual loss significantly improve all three aspects of perceptually
accurate lip synchronization. Codes and datasets are available at
https://perceptual-3d-talking-head.github.io/.
On Large Multimodal Models as Open-World Image Classifiers
Authors: name: Alessandro Conti, name: Massimiliano Mancini, name: Enrico Fini, name: Yiming Wang, name: Paolo Rota, name: Elisa Ricci
Published: 2025-03-27 17:03:18

Traditional image classification requires a predefined list of semantic
categories. In contrast, Large Multimodal Models (LMMs) can sidestep this
requirement by classifying images directly using natural language (e.g.,
answering the prompt "What is the main object in the image?"). Despite this
remarkable capability, most existing studies on LMM classification performance
are surprisingly limited in scope, often assuming a closed-world setting with a
predefined set of categories. In this work, we address this gap by thoroughly
evaluating LMM classification performance in a truly open-world setting. We
first formalize the task and introduce an evaluation protocol, defining various
metrics to assess the alignment between predicted and ground truth classes. We
then evaluate 13 models across 10 benchmarks, encompassing prototypical,
non-prototypical, fine-grained, and very fine-grained classes, demonstrating
the challenges LMMs face in this task. Further analyses based on the proposed
metrics reveal the types of errors LMMs make, highlighting challenges related
to granularity and fine-grained capabilities, showing how tailored prompting
and reasoning can alleviate them.
ReFeed: Multi-dimensional Summarization Refinement with Reflective
  Reasoning on Feedback
Authors: name: Taewon Yun, name: Jihwan Oh, name: Hyangsuk Min, name: Yuho Lee, name: Jihwan Bang, name: Jason Cai, name: Hwanjun Song
Published: 2025-03-27 10:11:41

Summarization refinement faces challenges when extending to multi-dimension.
In this paper, we introduce ReFeed, a powerful summarization refinement
pipeline that enhances multiple dimensions through reflective reasoning on
feedback. To achieve this, we release SumFeed-CoT, a large-scale Long-CoT-based
dataset optimized for training a lightweight model with reflective reasoning.
Our experiments reveal how the number of dimensions, feedback exposure, and
reasoning policy influence refinement performance, highlighting reflective
reasoning and simultaneously addressing multiple feedback is crucial to
mitigate trade-off between dimensions. Furthermore, ReFeed is robust to noisy
feedback and feedback order. Lastly, our finding emphasizes that creating data
with a proper goal and guideline constitutes a fundamental pillar of effective
reasoning. The dataset and model will be released.
AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through
  Lightweight Vocabulary Adaptation
Authors: name: Itay Nakash, name: Nitay Calderon, name: Eyal Ben David, name: Elad Hoffer, name: Roi Reichart
Published: 2025-03-25 14:18:21

Large Language Models (LLMs) have shown impressive versatility as general
purpose models. However, their broad applicability comes at a high-cost
computational overhead, particularly in auto-regressive decoding where each
step requires a forward pass. In domain-specific settings, general-purpose
capabilities are unnecessary and can be exchanged for efficiency. In this work,
we take a novel perspective on domain adaptation, reducing latency and
computational costs by adapting the vocabulary to focused domains of interest.
We introduce AdaptiVocab, an end-to-end approach for vocabulary adaptation,
designed to enhance LLM efficiency in low-resource domains. AdaptiVocab can be
applied to any tokenizer and architecture, modifying the vocabulary by
replacing tokens with domain-specific n-gram-based tokens, thereby reducing the
number of tokens required for both input processing and output generation.
AdaptiVocab initializes new n-token embeddings using an exponentially weighted
combination of existing embeddings and employs a lightweight fine-tuning phase
that can be efficiently performed on a single GPU. We evaluate two 7B LLMs
across three niche domains, assessing efficiency, generation quality, and
end-task performance. Our results show that AdaptiVocab reduces token usage by
over 25% without compromising performance