Orionnnn/hub-stats huggingface dataset:
Through-The-Mask: Mask-based Motion Trajectories for Image-to-Video
  Generation
Authors: name: Guy Yariv, name: Yuval Kirstain, name: Amit Zohar, name: Shelly Sheynin, name: Yaniv Taigman, name: Yossi Adi, name: Sagie Benaim, name: Adam Polyak
Published: 2025-01-06 14:49:26

We consider the task of Image-to-Video (I2V) generation, which involves
transforming static images into realistic video sequences based on a textual
description. While recent advancements produce photorealistic outputs, they
frequently struggle to create videos with accurate and consistent object
motion, especially in multi-object scenarios. To address these limitations, we
propose a two-stage compositional framework that decomposes I2V generation
into: (i) An explicit intermediate representation generation stage, followed by
(ii) A video generation stage that is conditioned on this representation. Our
key innovation is the introduction of a mask-based motion trajectory as an
intermediate representation, that captures both semantic object information and
motion, enabling an expressive but compact representation of motion and
semantics. To incorporate the learned representation in the second stage, we
utilize object-level attention objectives. Specifically, we consider a spatial,
per-object, masked-cross attention objective, integrating object-specific
prompts into corresponding latent space regions and a masked spatio-temporal
self-attention objective, ensuring frame-to-frame consistency for each object.
We evaluate our method on challenging benchmarks with multi-object and
high-motion scenarios and empirically demonstrate that the proposed method
achieves state-of-the-art results in temporal coherence, motion realism, and
text-prompt faithfulness. Additionally, we introduce \benchmark, a new
challenging benchmark for single-object and multi-object I2V generation, and
demonstrate our method's superiority on this benchmark. Project page is
available at https://guyyariv.github.io/TTM/.
GS-DiT: Advancing Video Generation with Pseudo 4D Gaussian Fields
  through Efficient Dense 3D Point Tracking
Authors: name: Weikang Bian, name: Zhaoyang Huang, name: Xiaoyu Shi, name: Yijin Li, name: Fu-Yun Wang, name: Hongsheng Li
Published: 2025-01-05 23:55:33

4D video control is essential in video generation as it enables the use of
sophisticated lens techniques, such as multi-camera shooting and dolly zoom,
which are currently unsupported by existing methods. Training a video Diffusion
Transformer (DiT) directly to control 4D content requires expensive multi-view
videos. Inspired by Monocular Dynamic novel View Synthesis (MDVS) that
optimizes a 4D representation and renders videos according to different 4D
elements, such as camera pose and object motion editing, we bring pseudo 4D
Gaussian fields to video generation. Specifically, we propose a novel framework
that constructs a pseudo 4D Gaussian field with dense 3D point tracking and
renders the Gaussian field for all video frames. Then we finetune a pretrained
DiT to generate videos following the guidance of the rendered video, dubbed as
GS-DiT. To boost the training of the GS-DiT, we also propose an efficient Dense
3D Point Tracking (D3D-PT) method for the pseudo 4D Gaussian field
construction. Our D3D-PT outperforms SpatialTracker, the state-of-the-art
sparse 3D point tracking method, in accuracy and accelerates the inference
speed by two orders of magnitude. During the inference stage, GS-DiT can
generate videos with the same dynamic content while adhering to different
camera parameters, addressing a significant limitation of current video
generation models. GS-DiT demonstrates strong generalization capabilities and
extends the 4D controllability of Gaussian splatting to video generation beyond
just camera poses. It supports advanced cinematic effects through the
manipulation of the Gaussian field and camera intrinsics, making it a powerful
tool for creative video production. Demos are available at
https://wkbian.github.io/Projects/GS-DiT/.
Samba-asr state-of-the-art speech recognition leveraging structured
  state-space models
Authors: name: Syed Abdul Gaffar Shakhadri, name: Kruthika KR, name: Kartik Basavaraj Angadi
Published: 2025-01-06 08:16:06

We propose Samba ASR, the first state-of-the-art Automatic Speech Recognition
(ASR) model leveraging the novel Mamba architecture as both encoder and
decoder, built on the foundation of state-space models (SSMs). Unlike
transformer-based ASR models, which rely on self-attention mechanisms to
capture dependencies, Samba ASR effectively models both local and global
temporal dependencies using efficient state-space dynamics, achieving
remarkable performance gains. By addressing the limitations of transformers,
such as quadratic scaling with input length and difficulty in handling
long-range dependencies, Samba ASR achieves superior accuracy and efficiency.
  Experimental results demonstrate that Samba ASR surpasses existing
open-source transformer-based ASR models across various standard benchmarks,
establishing it as the new state of the art in ASR. Extensive evaluations on
benchmark datasets show significant improvements in Word Error Rate (WER), with
competitive performance even in low-resource scenarios. Furthermore, the
computational efficiency and parameter optimization of the Mamba architecture
make Samba ASR a scalable and robust solution for diverse ASR tasks.
  Our contributions include:
  A new Samba ASR architecture demonstrating the superiority of SSMs over
transformer-based models for speech sequence processing. A comprehensive
evaluation on public benchmarks showcasing state-of-the-art performance. An
analysis of computational efficiency, robustness to noise, and sequence
generalization. This work highlights the viability of Mamba SSMs as a
transformer-free alternative for efficient and accurate ASR. By leveraging
state-space modeling advancements, Samba ASR sets a new benchmark for ASR
performance and future research.
Auto-RT: Automatic Jailbreak Strategy Exploration for Red-Teaming Large
  Language Models
Authors: name: Yanjiang Liu, name: Shuhen Zhou, name: Yaojie Lu, name: Huijia Zhu, name: Weiqiang Wang, name: Hongyu Lin, name: Ben He, name: Xianpei Han, name: Le Sun
Published: 2025-01-03 14:30:14

Automated red-teaming has become a crucial approach for uncovering
vulnerabilities in large language models (LLMs). However, most existing methods
focus on isolated safety flaws, limiting their ability to adapt to dynamic
defenses and uncover complex vulnerabilities efficiently. To address this
challenge, we propose Auto-RT, a reinforcement learning framework that
automatically explores and optimizes complex attack strategies to effectively
uncover security vulnerabilities through malicious queries. Specifically, we
introduce two key mechanisms to reduce exploration complexity and improve
strategy optimization: 1) Early-terminated Exploration, which accelerate
exploration by focusing on high-potential attack strategies; and 2) Progressive
Reward Tracking algorithm with intermediate downgrade models, which dynamically
refine the search trajectory toward successful vulnerability exploitation.
Extensive experiments across diverse LLMs demonstrate that, by significantly
improving exploration efficiency and automatically optimizing attack
strategies, Auto-RT detects a boarder range of vulnerabilities, achieving a
faster detection speed and 16.63\% higher success rates compared to existing
methods.
ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models
  in Multi-Hop Tool Use
Authors: name: Junjie Ye, name: Zhengyin Du, name: Xuesong Yao, name: Weijian Lin, name: Yufei Xu, name: Zehui Chen, name: Zaiyuan Wang, name: Sining Zhu, name: Zhiheng Xi, name: Siyu Yuan, name: Tao Gui, name: Qi Zhang, name: Xuanjing Huang, name: Jiechao Chen
Published: 2025-01-05 11:06:55

Effective evaluation of multi-hop tool use is critical for analyzing the
understanding, reasoning, and function-calling capabilities of large language
models (LLMs). However, progress has been hindered by a lack of reliable
evaluation datasets. To address this, we present ToolHop, a dataset comprising
995 user queries and 3,912 associated tools, specifically designed for rigorous
evaluation of multi-hop tool use. ToolHop ensures diverse queries, meaningful
interdependencies, locally executable tools, detailed feedback, and verifiable
answers through a novel query-driven data construction approach that includes
tool creation, document refinement, and code generation. We evaluate 14 LLMs
across five model families (i.e., LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and
GPT), uncovering significant challenges in handling multi-hop tool-use
scenarios. The leading model, GPT-4o, achieves an accuracy of 49.04%,
underscoring substantial room for improvement. Further analysis reveals
variations in tool-use strategies for various families, offering actionable
insights to guide the development of more effective approaches. Code and data
can be found in https://huggingface.co/bytedance-research/ToolHop.
Test-time Computing: from System-1 Thinking to System-2 Thinking
Authors: name: Yixin Ji, name: Juntao Li, name: Hai Ye, name: Kaixin Wu, name: Jia Xu, name: Linjian Mo, name: Min Zhang
Published: 2025-01-05 10:24:20

The remarkable performance of the o1 model in complex reasoning demonstrates
that test-time computing scaling can further unlock the model's potential,
enabling powerful System-2 thinking. However, there is still a lack of
comprehensive surveys for test-time computing scaling. We trace the concept of
test-time computing back to System-1 models. In System-1 models, test-time
computing addresses distribution shifts and improves robustness and
generalization through parameter updating, input modification, representation
editing, and output calibration. In System-2 models, it enhances the model's
reasoning ability to solve complex problems through repeated sampling,
self-correction, and tree search. We organize this survey according to the
trend of System-1 to System-2 thinking, highlighting the key role of test-time
computing in the transition from System-1 models to weak System-2 models, and
then to strong System-2 models. We also point out a few possible future
directions.
Ingredients: Blending Custom Photos with Video Diffusion Transformers
Authors: name: Zhengcong Fei, name: Debang Li, name: Di Qiu, name: Changqian Yu, name: Mingyuan Fan
Published: 2025-01-03 12:45:22

This paper presents a powerful framework to customize video creations by
incorporating multiple specific identity (ID) photos, with video diffusion
Transformers, referred to as Ingredients. Generally, our method
consists of three primary modules: (i) a facial extractor that
captures versatile and precise facial features for each human ID from both
global and local perspectives; (ii) a multi-scale projector that maps
face embeddings into the contextual space of image query in video diffusion
transformers; (iii) an ID router that dynamically combines and
allocates multiple ID embedding to the corresponding space-time regions.
Leveraging a meticulously curated text-video dataset and a multi-stage training
protocol, Ingredients demonstrates superior performance in turning
custom photos into dynamic and personalized video content. Qualitative
evaluations highlight the advantages of proposed method, positioning it as a
significant advancement toward more effective generative video control tools in
Transformer-based architecture, compared to existing methods. The data, code,
and model weights are publicly available at:
https://github.com/feizc/Ingredients.
METAGENE-1: Metagenomic Foundation Model for Pandemic Monitoring
Authors: name: Ollie Liu, name: Sami Jaghouar, name: Johannes Hagemann, name: Shangshang Wang, name: Jason Wiemels, name: Jeff Kaufman, name: Willie Neiswanger
Published: 2025-01-03 18:44:43

We pretrain METAGENE-1, a 7-billion-parameter autoregressive transformer
model, which we refer to as a metagenomic foundation model, on a novel corpus
of diverse metagenomic DNA and RNA sequences comprising over 1.5 trillion base
pairs. This dataset is sourced from a large collection of human wastewater
samples, processed and sequenced using deep metagenomic (next-generation)
sequencing methods. Unlike genomic models that focus on individual genomes or
curated sets of specific species, the aim of METAGENE-1 is to capture the full
distribution of genomic information present within this wastewater, to aid in
tasks relevant to pandemic monitoring and pathogen detection. We carry out
byte-pair encoding (BPE) tokenization on our dataset, tailored for metagenomic
sequences, and then pretrain our model. In this paper, we first detail the
pretraining dataset, tokenization strategy, and model architecture,
highlighting the considerations and design choices that enable the effective
modeling of metagenomic data. We then show results of pretraining this model on
our metagenomic dataset, providing details about our losses, system metrics,
and training stability over the course of pretraining. Finally, we demonstrate
the performance of METAGENE-1, which achieves state-of-the-art results on a set
of genomic benchmarks and new evaluations focused on human-pathogen detection
and genomic sequence embedding, showcasing its potential for public health
applications in pandemic monitoring, biosurveillance, and early detection of
emerging health threats.
STAR: Spatial-Temporal Augmentation with Text-to-Video Models for
  Real-World Video Super-Resolution
Authors: name: Rui Xie, name: Yinhong Liu, name: Penghao Zhou, name: Chen Zhao, name: Jun Zhou, name: Kai Zhang, name: Zhenyu Zhang, name: Jian Yang, name: Zhenheng Yang, name: Ying Tai
Published: 2025-01-06 12:36:21

Image diffusion models have been adapted for real-world video
super-resolution to tackle over-smoothing issues in GAN-based methods. However,
these models struggle to maintain temporal consistency, as they are trained on
static images, limiting their ability to capture temporal dynamics effectively.
Integrating text-to-video (T2V) models into video super-resolution for improved
temporal modeling is straightforward. However, two key challenges remain:
artifacts introduced by complex degradations in real-world scenarios, and
compromised fidelity due to the strong generative capacity of powerful T2V
models (e.g., CogVideoX-5B). To enhance the spatio-temporal quality of
restored videos, we introduce~\name
(Spatial-Temporal Augmentation with T2V models for
Real-world video super-resolution), a novel approach that leverages
T2V models for real-world video super-resolution, achieving realistic spatial
details and robust temporal consistency. Specifically, we introduce a Local
Information Enhancement Module (LIEM) before the global attention block to
enrich local details and mitigate degradation artifacts. Moreover, we propose a
Dynamic Frequency (DF) Loss to reinforce fidelity, guiding the model to focus
on different frequency components across diffusion steps. Extensive experiments
demonstrate~\name~outperforms state-of-the-art methods on both
synthetic and real-world datasets.
Personalized Graph-Based Retrieval for Large Language Models
Authors: name: Steven Au, name: Cameron J. Dimacali, name: Ojasmitha Pedirappagari, name: Namyong Park, name: Franck Dernoncourt, name: Yu Wang, name: Nikos Kanakaris, name: Hanieh Deilamsalehy, name: Ryan A. Rossi, name: Nesreen K. Ahmed
Published: 2025-01-04 01:46:49

As large language models (LLMs) evolve, their ability to deliver personalized
and context-aware responses offers transformative potential for improving user
experiences. Existing personalization approaches, however, often rely solely on
user history to augment the prompt, limiting their effectiveness in generating
tailored outputs, especially in cold-start scenarios with sparse data. To
address these limitations, we propose Personalized Graph-based
Retrieval-Augmented Generation (PGraphRAG), a framework that leverages
user-centric knowledge graphs to enrich personalization. By directly
integrating structured user knowledge into the retrieval process and augmenting
prompts with user-relevant context, PGraphRAG enhances contextual understanding
and output quality. We also introduce the Personalized Graph-based Benchmark
for Text Generation, designed to evaluate personalized text generation tasks in
real-world settings where user history is sparse or unavailable. Experimental
results show that PGraphRAG significantly outperforms state-of-the-art
personalization methods across diverse tasks, demonstrating the unique
advantages of graph-based retrieval for personalization.
TransPixar: Advancing Text-to-Video Generation with Transparency
Authors: name: Luozhou Wang, name: Yijun Li, name: Zhifei Chen, name: Jui-Hsien Wang, name: Zhifei Zhang, name: He Zhang, name: Zhe Lin, name: Yingcong Chen
Published: 2025-01-06 13:32:16

Text-to-video generative models have made significant strides, enabling
diverse applications in entertainment, advertising, and education. However,
generating RGBA video, which includes alpha channels for transparency, remains
a challenge due to limited datasets and the difficulty of adapting existing
models. Alpha channels are crucial for visual effects (VFX), allowing
transparent elements like smoke and reflections to blend seamlessly into
scenes. We introduce TransPixar, a method to extend pretrained video models for
RGBA generation while retaining the original RGB capabilities. TransPixar
leverages a diffusion transformer (DiT) architecture, incorporating
alpha-specific tokens and using LoRA-based fine-tuning to jointly generate RGB
and alpha channels with high consistency. By optimizing attention mechanisms,
TransPixar preserves the strengths of the original RGB model and achieves
strong alignment between RGB and alpha channels despite limited training data.
Our approach effectively generates diverse and consistent RGBA videos,
advancing the possibilities for VFX and interactive content creation.
SDPO: Segment-Level Direct Preference Optimization for Social Agents
Authors: name: Aobo Kong, name: Wentao Ma, name: Shiwan Zhao, name: Yongbin Li, name: Yuchuan Wu, name: Ke Wang, name: Xiaoqian Liu, name: Qicheng Li, name: Yong Qin, name: Fei Huang
Published: 2025-01-03 14:09:46

Social agents powered by large language models (LLMs) can simulate human
social behaviors but fall short in handling complex goal-oriented social
dialogues. Direct Preference Optimization (DPO) has proven effective in
aligning LLM behavior with human preferences across a variety of agent tasks.
Existing DPO-based approaches for multi-turn interactions are divided into
turn-level and session-level methods. The turn-level method is overly
fine-grained, focusing exclusively on individual turns, while session-level
methods are too coarse-grained, often introducing training noise. To address
these limitations, we propose Segment-Level Direct Preference Optimization
(SDPO), which focuses on specific key segments within interactions to optimize
multi-turn agent behavior while minimizing training noise. Evaluations on the
SOTOPIA benchmark demonstrate that SDPO-tuned agents consistently outperform
both existing DPO-based methods and proprietary LLMs like GPT-4o, underscoring
SDPO's potential to advance the social intelligence of LLM-based agents. We
release our code and data at
https://github.com/AlibabaResearch/DAMO-ConvAI/tree/main/SDPO.
VisionReward: Fine-Grained Multi-Dimensional Human Preference Learning
  for Image and Video Generation
Authors: name: Jiazheng Xu, name: Yu Huang, name: Jiale Cheng, name: Yuanming Yang, name: Jiajun Xu, name: Yuan Wang, name: Wenbo Duan, name: Shen Yang, name: Qunlin Jin, name: Shurun Li, name: Jiayan Teng, name: Zhuoyi Yang, name: Wendi Zheng, name: Xiao Liu, name: Ming Ding, name: Xiaohan Zhang, name: Xiaotao Gu, name: Shiyu Huang, name: Minlie Huang, name: Jie Tang, name: Yuxiao Dong
Published: 2024-12-30 16:24:09

We present a general strategy to aligning visual generation models -- both
image and video generation -- with human preference. To start with, we build
VisionReward -- a fine-grained and multi-dimensional reward model. We decompose
human preferences in images and videos into multiple dimensions, each
represented by a series of judgment questions, linearly weighted and summed to
an interpretable and accurate score. To address the challenges of video quality
assessment, we systematically analyze various dynamic features of videos, which
helps VisionReward surpass VideoScore by 17.2% and achieve top performance for
video preference prediction. Based on VisionReward, we develop a
multi-objective preference learning algorithm that effectively addresses the
issue of confounding factors within preference data. Our approach significantly
outperforms existing image and video scoring methods on both machine metrics
and human evaluation. All code and datasets are provided at
https://github.com/THUDM/VisionReward.
BoxingGym: Benchmarking Progress in Automated Experimental Design and
  Model Discovery
Authors: name: Kanishk Gandhi, name: Michael Y. Li, name: Lyle Goodyear, name: Louise Li, name: Aditi Bhaskar, name: Mohammed Zaman, name: Noah D. Goodman
Published: 2025-01-02 21:15:57

Understanding the world and explaining it with scientific theories is a
central aspiration of artificial intelligence research. Proposing theories,
designing experiments to test them, and then revising them based on data are
fundamental to scientific discovery. Despite the significant promise of
LLM-based scientific agents, no benchmarks systematically test LLM's ability to
propose scientific models, collect experimental data, and revise them in light
of new data. We introduce BoxingGym, a benchmark with 10 environments for
systematically evaluating both experimental design (e.g. collecting data to
test a scientific theory) and model discovery (e.g. proposing and revising
scientific theories). To enable tractable and quantitative evaluation, we
implement each environment as a generative probabilistic model with which a
scientific agent can run interactive experiments. These probabilistic models
are drawn from various real-world scientific domains ranging from psychology to
ecology. To quantitatively evaluate a scientific agent's ability to collect
informative experimental data, we compute the expected information gain (EIG),
an information-theoretic quantity which measures how much an experiment reduces
uncertainty about the parameters of a generative model. A good scientific
theory is a concise and predictive explanation. Therefore, to quantitatively
evaluate model discovery, we ask a scientific agent to explain their model and
then assess whether this explanation enables another scientific agent to make
reliable predictions about this environment. In addition to this
explanation-based evaluation, we compute standard model evaluation metrics such
as prediction errors. We find that current LLMs, such as GPT-4o, struggle with
both experimental design and model discovery. We find that augmenting the
LLM-based agent with an explicit statistical model does not reliably improve
these results.