Orionnnn/hub-stats huggingface dataset:
Ola: Pushing the Frontiers of Omni-Modal Language Model with Progressive
  Modality Alignment
Authors: name: Zuyan Liu, name: Yuhao Dong, name: Jiahui Wang, name: Ziwei Liu, name: Winston Hu, name: Jiwen Lu, name: Yongming Rao
Published: 2025-02-06 18:59:55

Recent advances in large language models, particularly following GPT-4o, have
sparked increasing interest in developing omni-modal models capable of
understanding more modalities. While some open-source alternatives have
emerged, there is still a notable lag behind specialized single-modality models
in performance. In this paper, we present Ola, an Omni-modal language model
that achieves competitive performance across image, video, and audio
understanding compared to specialized counterparts. The core design of Ola lies
in its progressive modality alignment strategy that extends the supporting
modality of the language model progressively. Our training pipeline begins with
the most distinct modalities: image and text, then gradually expands the skill
sets of the model using speech data that connects language and audio knowledge,
and video data that connects all modalities. The progressive learning pipeline
also enables us to maintain a relatively small size of the cross-modal
alignment data, making developing omni-modal from existing vision-language
models easy and less costly. Moreover, to unlock an advanced interactive
experience like GPT-4o, we further design a sentence-wise decoding solution for
streaming speech generation. Extensive experiments demonstrate that Ola
surpasses existing open omni-modal LLMs across all modalities while achieving
highly competitive performance compared to state-of-the-art specialized models
of similar sizes. We aim to make Ola a fully open omni-modal understanding
solution to advance future research in this emerging field. Model weights,
code, and data are open-sourced at https://github.com/Ola-Omni/Ola.
DynVFX: Augmenting Real Videos with Dynamic Content
Authors: name: Danah Yatim, name: Rafail Fridman, name: Omer Bar-Tal, name: Tali Dekel
Published: 2025-02-05 21:14:55

We present a method for augmenting real-world videos with newly generated
dynamic content. Given an input video and a simple user-provided text
instruction describing the desired content, our method synthesizes dynamic
objects or complex scene effects that naturally interact with the existing
scene over time. The position, appearance, and motion of the new content are
seamlessly integrated into the original footage while accounting for camera
motion, occlusions, and interactions with other dynamic objects in the scene,
resulting in a cohesive and realistic output video. We achieve this via a
zero-shot, training-free framework that harnesses a pre-trained text-to-video
diffusion transformer to synthesize the new content and a pre-trained Vision
Language Model to envision the augmented scene in detail. Specifically, we
introduce a novel inference-based method that manipulates features within the
attention mechanism, enabling accurate localization and seamless integration of
the new content while preserving the integrity of the original scene. Our
method is fully automated, requiring only a simple user instruction. We
demonstrate its effectiveness on a wide range of edits applied to real-world
videos, encompassing diverse objects and scenarios involving both camera and
object motion.
Towards Physical Understanding in Video Generation: A 3D Point
  Regularization Approach
Authors: name: Yunuo Chen, name: Junli Cao, name: Anil Kag, name: Vidit Goel, name: Sergei Korolev, name: Chenfanfu Jiang, name: Sergey Tulyakov, name: Jian Ren
Published: 2025-02-05 21:49:06

We present a novel video generation framework that integrates 3-dimensional
geometry and dynamic awareness. To achieve this, we augment 2D videos with 3D
point trajectories and align them in pixel space. The resulting 3D-aware video
dataset, PointVid, is then used to fine-tune a latent diffusion model, enabling
it to track 2D objects with 3D Cartesian coordinates. Building on this, we
regularize the shape and motion of objects in the video to eliminate undesired
artifacts, \eg, nonphysical deformation. Consequently, we enhance the quality
of generated RGB videos and alleviate common issues like object morphing, which
are prevalent in current video models due to a lack of shape awareness. With
our 3D augmentation and regularization, our model is capable of handling
contact-rich scenarios such as task-oriented videos. These videos involve
complex interactions of solids, where 3D information is essential for
perceiving deformation and contact. Furthermore, our model improves the overall
quality of video generation by promoting the 3D consistency of moving objects
and reducing abrupt changes in shape and motion.
MotionCanvas: Cinematic Shot Design with Controllable Image-to-Video
  Generation
Authors: name: Jinbo Xing, name: Long Mai, name: Cusuh Ham, name: Jiahui Huang, name: Aniruddha Mahapatra, name: Chi-Wing Fu, name: Tien-Tsin Wong, name: Feng Liu
Published: 2025-02-06 18:41:04

This paper presents a method that allows users to design cinematic video
shots in the context of image-to-video generation. Shot design, a critical
aspect of filmmaking, involves meticulously planning both camera movements and
object motions in a scene. However, enabling intuitive shot design in modern
image-to-video generation systems presents two main challenges: first,
effectively capturing user intentions on the motion design, where both camera
movements and scene-space object motions must be specified jointly; and second,
representing motion information that can be effectively utilized by a video
diffusion model to synthesize the image animations. To address these
challenges, we introduce MotionCanvas, a method that integrates user-driven
controls into image-to-video (I2V) generation models, allowing users to control
both object and camera motions in a scene-aware manner. By connecting insights
from classical computer graphics and contemporary video generation techniques,
we demonstrate the ability to achieve 3D-aware motion control in I2V synthesis
without requiring costly 3D-related training data. MotionCanvas enables users
to intuitively depict scene-space motion intentions, and translates them into
spatiotemporal motion-conditioning signals for video diffusion models. We
demonstrate the effectiveness of our method on a wide range of real-world image
content and shot-design scenarios, highlighting its potential to enhance the
creative workflows in digital content creation and adapt to various image and
video editing applications.
MotionLab: Unified Human Motion Generation and Editing via the
  Motion-Condition-Motion Paradigm
Authors: name: Ziyan Guo, name: Zeyu Hu, name: Na Zhao, name: De Wen Soh
Published: 2025-02-04 14:43:26

Human motion generation and editing are key components of computer graphics
and vision. However, current approaches in this field tend to offer isolated
solutions tailored to specific tasks, which can be inefficient and impractical
for real-world applications. While some efforts have aimed to unify
motion-related tasks, these methods simply use different modalities as
conditions to guide motion generation. Consequently, they lack editing
capabilities, fine-grained control, and fail to facilitate knowledge sharing
across tasks. To address these limitations and provide a versatile, unified
framework capable of handling both human motion generation and editing, we
introduce a novel paradigm: Motion-Condition-Motion, which enables the unified
formulation of diverse tasks with three concepts: source motion, condition, and
target motion. Based on this paradigm, we propose a unified framework,
MotionLab, which incorporates rectified flows to learn the mapping from source
motion to target motion, guided by the specified conditions. In MotionLab, we
introduce the 1) MotionFlow Transformer to enhance conditional generation and
editing without task-specific modules; 2) Aligned Rotational Position Encoding}
to guarantee the time synchronization between source motion and target motion;
3) Task Specified Instruction Modulation; and 4) Motion Curriculum Learning for
effective multi-task learning and knowledge sharing across tasks. Notably, our
MotionLab demonstrates promising generalization capabilities and inference
efficiency across multiple benchmarks for human motion. Our code and additional
video results are available at: https://diouo.github.io/motionlab.github.io/.
Gold-medalist Performance in Solving Olympiad Geometry with
  AlphaGeometry2
Authors: name: Yuri Chervonyi, name: Trieu H. Trinh, name: Miroslav Olšák, name: Xiaomeng Yang, name: Hoang Nguyen, name: Marcelo Menegali, name: Junehyuk Jung, name: Vikas Verma, name: Quoc V. Le, name: Thang Luong
Published: 2025-02-05 19:02:03

We present AlphaGeometry2, a significantly improved version of AlphaGeometry
introduced in Trinh et al. (2024), which has now surpassed an average gold
medalist in solving Olympiad geometry problems. To achieve this, we first
extend the original AlphaGeometry language to tackle harder problems involving
movements of objects, and problems containing linear equations of angles,
ratios, and distances. This, together with other additions, has markedly
improved the coverage rate of the AlphaGeometry language on International Math
Olympiads (IMO) 2000-2024 geometry problems from 66% to 88%. The search process
of AlphaGeometry2 has also been greatly improved through the use of Gemini
architecture for better language modeling, and a novel knowledge-sharing
mechanism that combines multiple search trees. Together with further
enhancements to the symbolic engine and synthetic data generation, we have
significantly boosted the overall solving rate of AlphaGeometry2 to 84% for
all geometry problems over the last 25 years, compared to 54%
previously. AlphaGeometry2 was also part of the system that achieved
silver-medal standard at IMO 2024 https://dpmd.ai/imo-silver. Last but not
least, we report progress towards using AlphaGeometry2 as a part of a fully
automated system that reliably solves geometry problems directly from natural
language input.
Llasa: Scaling Train-Time and Inference-Time Compute for Llama-based
  Speech Synthesis
Authors: name: Zhen Ye, name: Xinfa Zhu, name: Chi-Min Chan, name: Xinsheng Wang, name: Xu Tan, name: Jiahe Lei, name: Yi Peng, name: Haohe Liu, name: Yizhu Jin, name: Zheqi DAI, name: Hongzhan Lin, name: Jianyi Chen, name: Xingjian Du, name: Liumeng Xue, name: Yunlin Chen, name: Zhifei Li, name: Lei Xie, name: Qiuqiang Kong, name: Yike Guo, name: Wei Xue
Published: 2025-02-06 15:04:00

Recent advances in text-based large language models (LLMs), particularly in
the GPT series and the o1 model, have demonstrated the effectiveness of scaling
both training-time and inference-time compute. However, current
state-of-the-art TTS systems leveraging LLMs are often multi-stage, requiring
separate models (e.g., diffusion models after LLM), complicating the decision
of whether to scale a particular model during training or testing. This work
makes the following contributions: First, we explore the scaling of train-time
and inference-time compute for speech synthesis. Second, we propose a simple
framework Llasa for speech synthesis that employs a single-layer vector
quantizer (VQ) codec and a single Transformer architecture to fully align with
standard LLMs such as Llama. Our experiments reveal that scaling train-time
compute for Llasa consistently improves the naturalness of synthesized speech
and enables the generation of more complex and accurate prosody patterns.
Furthermore, from the perspective of scaling inference-time compute, we employ
speech understanding models as verifiers during the search, finding that
scaling inference-time compute shifts the sampling modes toward the preferences
of specific verifiers, thereby improving emotional expressiveness, timbre
consistency, and content accuracy. In addition, we released the checkpoint and
training code for our TTS model (1B, 3B, 8B) and codec model publicly
available.
PILAF: Optimal Human Preference Sampling for Reward Modeling
Authors: name: Yunzhen Feng, name: Ariel Kwiatkowski, name: Kunhao Zheng, name: Julia Kempe, name: Yaqi Duan
Published: 2025-02-06 18:09:00

As large language models increasingly drive real-world applications, aligning
them with human values becomes paramount. Reinforcement Learning from Human
Feedback (RLHF) has emerged as a key technique, translating preference data
into reward models when oracle human values remain inaccessible. In practice,
RLHF mostly relies on approximate reward models, which may not consistently
guide the policy toward maximizing the underlying human values. We propose
Policy-Interpolated Learning for Aligned Feedback (PILAF), a novel response
sampling strategy for preference labeling that explicitly aligns preference
learning with maximizing the underlying oracle reward. PILAF is theoretically
grounded, demonstrating optimality from both an optimization and a statistical
perspective. The method is straightforward to implement and demonstrates strong
performance in iterative and online RLHF settings where feedback curation is
critical.
BOLT: Bootstrap Long Chain-of-Thought in Language Models without
  Distillation
Authors: name: Bo Pang, name: Hanze Dong, name: Jiacheng Xu, name: Silvio Savarese, name: Yingbo Zhou, name: Caiming Xiong
Published: 2025-02-06 08:19:59

Large language models (LLMs), such as o1 from OpenAI, have demonstrated
remarkable reasoning capabilities. o1 generates a long chain-of-thought
(LongCoT) before answering a question. LongCoT allows LLMs to analyze problems,
devise plans, reflect, and backtrack effectively. These actions empower LLM to
solve complex problems. After the release of o1, many teams have attempted to
replicate its LongCoT and reasoning capabilities. In terms of methods, they
primarily rely on knowledge distillation with data from existing models with
LongCoT capacities (e.g., OpenAI-o1, Qwen-QwQ, DeepSeek-R1-Preview), leaving
significant uncertainties on systematically developing such reasoning
abilities. In terms of data domains, these works focus narrowly on math while a
few others include coding, limiting their generalizability. This paper
introduces a novel approach to enable LLM's LongCoT capacity without
distillation from o1-like models or expensive human annotations, where we
bootstrap LongCoT (BOLT) from a standard instruct model. BOLT involves three
stages: 1) LongCoT data bootstrapping with in-context learning on a standard
instruct model; 2) LongCoT supervised finetuning; 3) online training to further
refine LongCoT capacities. In BOLT, only a few in-context examples need to be
constructed during the bootstrapping stage; in our experiments, we created 10
examples, demonstrating the feasibility of this approach. We use
Llama-3.1-70B-Instruct to bootstrap LongCoT and apply our method to various
model scales (7B, 8B, 70B). We achieve impressive performance on a variety of
benchmarks, Arena-Hard, MT-Bench, WildBench, ZebraLogic, MATH500, which
evaluate diverse task-solving and reasoning capabilities.
ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference
  Optimization
Authors: name: Yinjie Wang, name: Ling Yang, name: Guohao Li, name: Mengdi Wang, name: Bryon Aragam
Published: 2025-02-06 18:47:49

Recent research has leveraged large language model multi-agent systems for
complex problem-solving while trying to reduce the manual effort required to
build them, driving the development of automated agent workflow optimization
methods. However, existing methods remain inflexible due to representational
limitations, a lack of adaptability, and poor scalability when relying on
discrete optimization techniques. We address these challenges with ScoreFlow, a
simple yet high-performance framework that leverages efficient gradient-based
optimization in a continuous space. ScoreFlow incorporates Score-DPO, a novel
variant of the direct preference optimization method that accounts for
quantitative feedback. Across six benchmarks spanning question answering,
coding, and mathematical reasoning, ScoreFlow achieves an 8.2% improvement over
existing baselines. Moreover, it empowers smaller models to outperform larger
ones with lower inference costs. Project:
https://github.com/Gen-Verse/ScoreFlow
UltraIF: Advancing Instruction Following from the Wild
Authors: name: Kaikai An, name: Li Sheng, name: Ganqu Cui, name: Shuzheng Si, name: Ning Ding, name: Yu Cheng, name: Baobao Chang
Published: 2025-02-06 15:39:16

Instruction-following made modern large language models (LLMs) helpful
assistants. However, the key to taming LLMs on complex instructions remains
mysterious, for that there are huge gaps between models trained by open-source
community and those trained by leading companies. To bridge the gap, we propose
a simple and scalable approach UltraIF for building LLMs that can follow
complex instructions with open-source data. UltraIF first decomposes real-world
user prompts into simpler queries, constraints, and corresponding evaluation
questions for the constraints. Then, we train an UltraComposer to compose
constraint-associated prompts with evaluation questions. This prompt composer
allows us to synthesize complicated instructions as well as filter responses
with evaluation questions. In our experiment, for the first time, we
successfully align LLaMA-3.1-8B-Base to catch up with its instruct version on 5
instruction-following benchmarks without any benchmark information, using only
8B model as response generator and evaluator. The aligned model also achieved
competitive scores on other benchmarks. Moreover, we also show that UltraIF
could further improve LLaMA-3.1-8B-Instruct through self-alignment, motivating
broader use cases for the method. Our code will be available at
https://github.com/kkk-an/UltraIF.
Beyond Prompt Content: Enhancing LLM Performance via Content-Format
  Integrated Prompt Optimization
Authors: name: Yuanye Liu, name: Jiahang Xu, name: Li Lyna Zhang, name: Qi Chen, name: Xuan Feng, name: Yang Chen, name: Zhongxin Guo, name: Yuqing Yang, name: Cheng Peng
Published: 2025-02-06 18:36:44

Large Language Models (LLMs) have shown significant capability across various
tasks, with their real-world effectiveness often driven by prompt design. While
recent research has focused on optimizing prompt content, the role of prompt
formatting, a critical but often overlooked dimension, has received limited
systematic investigation. In this paper, we introduce Content-Format Integrated
Prompt Optimization (CFPO), an innovative methodology that jointly optimizes
both prompt content and formatting through an iterative refinement process.
CFPO leverages natural language mutations to explore content variations and
employs a dynamic format exploration strategy that systematically evaluates
diverse format options. Our extensive evaluations across multiple tasks and
open-source LLMs demonstrate that CFPO demonstrates measurable performance
improvements compared to content-only optimization methods. This highlights the
importance of integrated content-format optimization and offers a practical,
model-agnostic approach to enhancing LLM performance. Code will be available at
https://github.com/HenryLau7/CFPO.
Learning Real-World Action-Video Dynamics with Heterogeneous Masked
  Autoregression
Authors: name: Lirui Wang, name: Kevin Zhao, name: Chaoqi Liu, name: Xinlei Chen
Published: 2025-02-06 18:38:26

We propose Heterogeneous Masked Autoregression (HMA) for modeling
action-video dynamics to generate high-quality data and evaluation in scaling
robot learning. Building interactive video world models and policies for
robotics is difficult due to the challenge of handling diverse settings while
maintaining computational efficiency to run in real time. HMA uses
heterogeneous pre-training from observations and action sequences across
different robotic embodiments, domains, and tasks. HMA uses masked
autoregression to generate quantized or soft tokens for video predictions.
\ourshort achieves better visual fidelity and controllability than the previous
robotic video generation models with 15 times faster speed in the real world.
After post-training, this model can be used as a video simulator from low-level
action inputs for evaluating policies and generating synthetic data. See this
link https://liruiw.github.io/hma for more information.
HackerRank-ASTRA: Evaluating Correctness & Consistency of Large Language
  Models on cross-domain multi-file project problems
Authors: name: Jun Xing, name: Mayur Bhatia, name: Sahil Phulwani, name: Darshan Suresh, name: Rafik Matta
Published: 2025-01-31 23:47:02

Evaluating the real-world applicability of large language models (LLMs)
provides valuable insights for their development and use in software
development tasks. Existing benchmarks often focus on standalone coding
problems or specific libraries, overlooking multi-file, project-based scenarios
and lacking a rigorous evaluation of consistency. The HackerRank-ASTRA
Benchmark introduces project-based coding problems that mirror real-world
scenarios. It evaluates model consistency through 32 runs (k = 32) and median
standard deviation while incorporating taxonomy-level analysis to assess
sub-skill capabilities. Initial evaluations on 65 problems show that the top
three models -- o1, o1-preview, and Claude-3.5-Sonnet-1022 -- achieved
comparable average scores of 75%, with no statistically significant differences
in performance. Notably, Claude-3.5-Sonnet-1022 demonstrated the highest
consistency across problems, with low variability (SD = 0.0497), which was
statistically significant compared to other models, highlighting its
reliability for real-world software development tasks.
Activation-Informed Merging of Large Language Models
Authors: name: Amin Heyrani Nobari, name: Kaveh Alimohammadi, name: Ali ArjomandBigdeli, name: Akash Srivastava, name: Faez Ahmed, name: Navid Azizan
Published: 2025-02-04 15:42:03

Model merging, a method that combines the parameters and embeddings of
multiple fine-tuned large language models (LLMs), offers a promising approach
to enhance model performance across various tasks while maintaining
computational efficiency. This paper introduces Activation-Informed Merging
(AIM), a technique that integrates the information from the activation space of
LLMs into the merging process to improve performance and robustness. AIM is
designed as a flexible, complementary solution that is applicable to any
existing merging method. It aims to preserve critical weights from the base
model, drawing on principles from continual learning~(CL) and model
compression. Utilizing a task-agnostic calibration set, AIM selectively
prioritizes essential weights during merging. We empirically demonstrate that
AIM significantly enhances the performance of merged models across multiple
benchmarks. Our findings suggest that considering the activation-space
information can provide substantial advancements in the model merging
strategies for LLMs with up to 40\% increase in benchmark performance.