Orionnnn/hub-stats huggingface dataset:
UnCommon Objects in 3D
Authors: name: Xingchen Liu, name: Piyush Tayal, name: Jianyuan Wang, name: Jesus Zarzar, name: Tom Monnier, name: Konstantinos Tertikas, name: Jiali Duan, name: Antoine Toisoul, name: Jason Y. Zhang, name: Natalia Neverova, name: Andrea Vedaldi, name: Roman Shapovalov, name: David Novotny
Published: 2025-01-13 18:59:20

We introduce Uncommon Objects in 3D (uCO3D), a new object-centric dataset for
3D deep learning and 3D generative AI. uCO3D is the largest publicly-available
collection of high-resolution videos of objects with 3D annotations that
ensures full-360^{circ} coverage. uCO3D is significantly more diverse than
MVImgNet and CO3Dv2, covering more than 1,000 object categories. It is also of
higher quality, due to extensive quality checks of both the collected videos
and the 3D annotations. Similar to analogous datasets, uCO3D contains
annotations for 3D camera poses, depth maps and sparse point clouds. In
addition, each object is equipped with a caption and a 3D Gaussian Splat
reconstruction. We train several large 3D models on MVImgNet, CO3Dv2, and uCO3D
and obtain superior results using the latter, showing that uCO3D is better for
learning applications.
Migician: Revealing the Magic of Free-Form Multi-Image Grounding in
  Multimodal Large Language Models
Authors: name: You Li, name: Heyu Huang, name: Chi Chen, name: Kaiyu Huang, name: Chao Huang, name: Zonghao Guo, name: Zhiyuan Liu, name: Jinan Xu, name: Yuhua Li, name: Ruixuan Li, name: Maosong Sun
Published: 2025-01-10 07:56:23

The recent advancement of Multimodal Large Language Models (MLLMs) has
significantly improved their fine-grained perception of single images and
general comprehension across multiple images. However, existing MLLMs still
face challenges in achieving precise grounding in complex multi-image
scenarios. To address this, we first explore a Chain-of-Thought (CoT) framework
that integrates single-image grounding with multi-image comprehension. While
partially effective, it remains unstable and struggles to capture abstract
visual information due to its non-end-to-end nature. Therefore, we introduce
Migician, the first multi-image grounding model capable of performing free-form
and accurate grounding across multiple images. To support this, we present the
MGrounding-630k dataset, which comprises data for several multi-image grounding
tasks derived from existing datasets, along with newly generated free-form
grounding instruction-following data. Furthermore, we propose MIG-Bench, a
comprehensive benchmark specifically designed for evaluating multi-image
grounding capabilities. Experimental results demonstrate that our model
achieves significantly superior multi-image grounding capabilities,
outperforming the best existing MLLMs by 21.61% and even surpassing much larger
70B models. Our code, model, dataset, and benchmark are fully open-sourced.
Generative AI for Cel-Animation: A Survey
Authors: name: Yunlong Tang, name: Junjia Guo, name: Pinxin Liu, name: Zhiyuan Wang, name: Hang Hua, name: Jia-Xing Zhong, name: Yunzhong Xiao, name: Chao Huang, name: Luchuan Song, name: Susan Liang, name: Yizhi Song, name: Liu He, name: Jing Bi, name: Mingqian Feng, name: Xinyang Li, name: Zeliang Zhang, name: Chenliang Xu
Published: 2025-01-08 20:57:39

Traditional Celluloid (Cel) Animation production pipeline encompasses
multiple essential steps, including storyboarding, layout design, keyframe
animation, inbetweening, and colorization, which demand substantial manual
effort, technical expertise, and significant time investment. These challenges
have historically impeded the efficiency and scalability of Cel-Animation
production. The rise of generative artificial intelligence (GenAI),
encompassing large language models, multimodal models, and diffusion models,
offers innovative solutions by automating tasks such as inbetween frame
generation, colorization, and storyboard creation. This survey explores how
GenAI integration is revolutionizing traditional animation workflows by
lowering technical barriers, broadening accessibility for a wider range of
creators through tools like AniDoc, ToonCrafter, and AniSora, and enabling
artists to focus more on creative expression and artistic innovation. Despite
its potential, issues such as maintaining visual consistency, ensuring
stylistic coherence, and addressing ethical considerations continue to pose
challenges. Furthermore, this paper discusses future directions and explores
potential advancements in AI-assisted animation. For further exploration and
resources, please visit our GitHub repository:
https://github.com/yunlong10/Awesome-AI4Animation
Demystifying Domain-adaptive Post-training for Financial LLMs
Authors: name: Zixuan Ke, name: Yifei Ming, name: Xuan-Phi Nguyen, name: Caiming Xiong, name: Shafiq Joty
Published: 2025-01-09 04:26:15

Domain-adaptive post-training of large language models (LLMs) has emerged as
a promising approach for specialized domains such as medicine and finance.
However, significant challenges remain in identifying optimal adaptation
criteria and training strategies across varying data and model configurations.
To address these challenges, we introduce FINDAP, a systematic and fine-grained
investigation into domain-adaptive post-training of LLMs for the finance
domain. Our approach begins by identifying the core capabilities required for
the target domain and designing a comprehensive evaluation suite aligned with
these needs. We then analyze the effectiveness of key post-training stages,
including continual pretraining, instruction tuning, and preference alignment.
Building on these insights, we propose an effective training recipe centered on
a novel preference data distillation method, which leverages process signals
from a generative reward model. The resulting model, Llama-Fin, achieves
state-of-the-art performance across a wide range of financial tasks. Our
analysis also highlights how each post-training stage contributes to distinct
capabilities, uncovering specific challenges and effective solutions, providing
valuable insights for domain adaptation of LLMs. Project page:
https://github.com/SalesforceAIResearch/FinDap
ConceptMaster: Multi-Concept Video Customization on Diffusion
  Transformer Models Without Test-Time Tuning
Authors: name: Yuzhou Huang, name: Ziyang Yuan, name: Quande Liu, name: Qiulin Wang, name: Xintao Wang, name: Ruimao Zhang, name: Pengfei Wan, name: Di Zhang, name: Kun Gai
Published: 2025-01-08 18:59:01

Text-to-video generation has made remarkable advancements through diffusion
models. However, Multi-Concept Video Customization (MCVC) remains a significant
challenge. We identify two key challenges in this task: 1) the identity
decoupling problem, where directly adopting existing customization methods
inevitably mix attributes when handling multiple concepts simultaneously, and
2) the scarcity of high-quality video-entity pairs, which is crucial for
training such a model that represents and decouples various concepts well. To
address these challenges, we introduce ConceptMaster, an innovative framework
that effectively tackles the critical issues of identity decoupling while
maintaining concept fidelity in customized videos. Specifically, we introduce a
novel strategy of learning decoupled multi-concept embeddings that are injected
into the diffusion models in a standalone manner, which effectively guarantees
the quality of customized videos with multiple identities, even for highly
similar visual concepts. To further overcome the scarcity of high-quality MCVC
data, we carefully establish a data construction pipeline, which enables
systematic collection of precise multi-concept video-entity data across diverse
concepts. A comprehensive benchmark is designed to validate the effectiveness
of our model from three critical dimensions: concept fidelity, identity
decoupling ability, and video generation quality across six different concept
composition scenarios. Extensive experiments demonstrate that our ConceptMaster
significantly outperforms previous approaches for this task, paving the way for
generating personalized and semantically accurate videos across multiple
concepts.
Multi-subject Open-set Personalization in Video Generation
Authors: name: Tsai-Shien Chen, name: Aliaksandr Siarohin, name: Willi Menapace, name: Yuwei Fang, name: Kwot Sin Lee, name: Ivan Skorokhodov, name: Kfir Aberman, name: Jun-Yan Zhu, name: Ming-Hsuan Yang, name: Sergey Tulyakov
Published: 2025-01-10 18:59:54

Video personalization methods allow us to synthesize videos with specific
concepts such as people, pets, and places. However, existing methods often
focus on limited domains, require time-consuming optimization per subject, or
support only a single subject. We present Video Alchemist - a video model
with built-in multi-subject, open-set personalization capabilities for both
foreground objects and background, eliminating the need for time-consuming
test-time optimization. Our model is built on a new Diffusion Transformer
module that fuses each conditional reference image and its corresponding
subject-level text prompt with cross-attention layers. Developing such a large
model presents two main challenges: dataset and evaluation. First, as paired
datasets of reference images and videos are extremely hard to collect, we
sample selected video frames as reference images and synthesize a clip of the
target video. However, while models can easily denoise training videos given
reference frames, they fail to generalize to new contexts. To mitigate this
issue, we design a new automatic data construction pipeline with extensive
image augmentations. Second, evaluating open-set video personalization is a
challenge in itself. To address this, we introduce a personalization benchmark
that focuses on accurate subject fidelity and supports diverse personalization
scenarios. Finally, our extensive experiments show that our method
significantly outperforms existing personalization methods in both quantitative
and qualitative evaluations.
OVO-Bench: How Far is Your Video-LLMs from Real-World Online Video
  Understanding?
Authors: name: Yifei Li, name: Junbo Niu, name: Ziyang Miao, name: Chunjiang Ge, name: Yuanhang Zhou, name: Qihao He, name: Xiaoyi Dong, name: Haodong Duan, name: Shuangrui Ding, name: Rui Qian, name: Pan Zhang, name: Yuhang Zang, name: Yuhang Cao, name: Conghui He, name: Jiaqi Wang
Published: 2025-01-09 19:00:01

Temporal Awareness, the ability to reason dynamically based on the timestamp
when a question is raised, is the key distinction between offline and online
video LLMs. Unlike offline models, which rely on complete videos for static,
post hoc analysis, online models process video streams incrementally and
dynamically adapt their responses based on the timestamp at which the question
is posed. Despite its significance, temporal awareness has not been adequately
evaluated in existing benchmarks. To fill this gap, we present OVO-Bench
(Online-VideO-Benchmark), a novel video benchmark that emphasizes the
importance of timestamps for advanced online video understanding capability
benchmarking. OVO-Bench evaluates the ability of video LLMs to reason and
respond to events occurring at specific timestamps under three distinct
scenarios: (1) Backward tracing: trace back to past events to answer the
question. (2) Real-time understanding: understand and respond to events as they
unfold at the current timestamp. (3) Forward active responding: delay the
response until sufficient future information becomes available to answer the
question accurately. OVO-Bench comprises 12 tasks, featuring 644 unique videos
and approximately human-curated 2,800 fine-grained meta-annotations with
precise timestamps. We combine automated generation pipelines with human
curation. With these high-quality samples, we further developed an evaluation
pipeline to systematically query video LLMs along the video timeline.
Evaluations of nine Video-LLMs reveal that, despite advancements on traditional
benchmarks, current models struggle with online video understanding, showing a
significant gap compared to human agents. We hope OVO-Bench will drive progress
in video LLMs and inspire future research in online video reasoning. Our
benchmark and code can be accessed at https://github.com/JoeLeelyf/OVO-Bench.
ReFocus: Visual Editing as a Chain of Thought for Structured Image
  Understanding
Authors: name: Xingyu Fu, name: Minqian Liu, name: Zhengyuan Yang, name: John Corring, name: Yijuan Lu, name: Jianwei Yang, name: Dan Roth, name: Dinei Florencio, name: Cha Zhang
Published: 2025-01-09 18:59:58

Structured image understanding, such as interpreting tables and charts,
requires strategically refocusing across various structures and texts within an
image, forming a reasoning sequence to arrive at the final answer. However,
current multimodal large language models (LLMs) lack this multihop selective
attention capability. In this work, we introduce ReFocus, a simple yet
effective framework that equips multimodal LLMs with the ability to generate
"visual thoughts" by performing visual editing on the input image through code,
shifting and refining their visual focuses. Specifically, ReFocus enables
multimodal LLMs to generate Python codes to call tools and modify the input
image, sequentially drawing boxes, highlighting sections, and masking out
areas, thereby enhancing the visual reasoning process. We experiment upon a
wide range of structured image understanding tasks involving tables and charts.
ReFocus largely improves performance on all tasks over GPT-4o without visual
editing, yielding an average gain of 11.0% on table tasks and 6.8% on chart
tasks. We present an in-depth analysis of the effects of different visual
edits, and reasons why ReFocus can improve the performance without introducing
additional information. Further, we collect a 14k training set using ReFocus,
and prove that such visual chain-of-thought with intermediate information
offers a better supervision than standard VQA data, reaching a 8.0% average
gain over the same model trained with QA pairs and 2.6% over CoT.
Infecting Generative AI With Viruses
Authors: name: David Noever, name: Forrest McKee
Published: 2025-01-09 19:27:29

This study demonstrates a novel approach to testing the security boundaries
of Vision-Large Language Model (VLM/ LLM) using the EICAR test file embedded
within JPEG images. We successfully executed four distinct protocols across
multiple LLM platforms, including OpenAI GPT-4o, Microsoft Copilot, Google
Gemini 1.5 Pro, and Anthropic Claude 3.5 Sonnet. The experiments validated that
a modified JPEG containing the EICAR signature could be uploaded, manipulated,
and potentially executed within LLM virtual workspaces. Key findings include:
1) consistent ability to mask the EICAR string in image metadata without
detection, 2) successful extraction of the test file using Python-based
manipulation within LLM environments, and 3) demonstration of multiple
obfuscation techniques including base64 encoding and string reversal. This
research extends Microsoft Research's "Penetration Testing Rules of Engagement"
framework to evaluate cloud-based generative AI and LLM security boundaries,
particularly focusing on file handling and execution capabilities within
containerized environments.
Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains
Authors: name: Vighnesh Subramaniam, name: Yilun Du, name: Joshua B. Tenenbaum, name: Antonio Torralba, name: Shuang Li, name: Igor Mordatch
Published: 2025-01-10 04:35:46

Large language models (LLMs) have achieved remarkable performance in recent
years but are fundamentally limited by the underlying training data. To improve
models beyond the training data, recent works have explored how LLMs can be
used to generate synthetic data for autonomous self-improvement. However,
successive steps of self-improvement can reach a point of diminishing returns.
In this work, we propose a complementary approach towards self-improvement
where finetuning is applied to a multiagent society of language models. A group
of language models, all starting from the same base model, are independently
specialized by updating each one using data generated through multiagent
interactions among the models. By training each model on independent sets of
data, we illustrate how this approach enables specialization across models and
diversification over the set of models. As a result, our overall system is able
to preserve diverse reasoning chains and autonomously improve over many more
rounds of fine-tuning than single-agent self-improvement methods. We
quantitatively illustrate the efficacy of the approach across a wide suite of
reasoning tasks.
OmniManip: Towards General Robotic Manipulation via Object-Centric
  Interaction Primitives as Spatial Constraints
Authors: name: Mingjie Pan, name: Jiyao Zhang, name: Tianshu Wu, name: Yinghao Zhao, name: Wenlong Gao, name: Hao Dong
Published: 2025-01-07 14:50:33

The development of general robotic systems capable of manipulating in
unstructured environments is a significant challenge. While Vision-Language
Models(VLM) excel in high-level commonsense reasoning, they lack the
fine-grained 3D spatial understanding required for precise manipulation tasks.
Fine-tuning VLM on robotic datasets to create Vision-Language-Action
Models(VLA) is a potential solution, but it is hindered by high data collection
costs and generalization issues. To address these challenges, we propose a
novel object-centric representation that bridges the gap between VLM's
high-level reasoning and the low-level precision required for manipulation. Our
key insight is that an object's canonical space, defined by its functional
affordances, provides a structured and semantically meaningful way to describe
interaction primitives, such as points and directions. These primitives act as
a bridge, translating VLM's commonsense reasoning into actionable 3D spatial
constraints. In this context, we introduce a dual closed-loop, open-vocabulary
robotic manipulation system: one loop for high-level planning through primitive
resampling, interaction rendering and VLM checking, and another for low-level
execution via 6D pose tracking. This design ensures robust, real-time control
without requiring VLM fine-tuning. Extensive experiments demonstrate strong
zero-shot generalization across diverse robotic manipulation tasks,
highlighting the potential of this approach for automating large-scale
simulation data generation.
LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMs
Authors: name: Omkar Thawakar, name: Dinura Dissanayake, name: Ketan More, name: Ritesh Thawkar, name: Ahmed Heakl, name: Noor Ahsan, name: Yuhao Li, name: Mohammed Zumri, name: Jean Lahoud, name: Rao Muhammad Anwer, name: Hisham Cholakkal, name: Ivan Laptev, name: Mubarak Shah, name: Fahad Shahbaz Khan, name: Salman Khan
Published: 2025-01-10 18:59:51

Reasoning is a fundamental capability for solving complex multi-step
problems, particularly in visual contexts where sequential step-wise
understanding is essential. Existing approaches lack a comprehensive framework
for evaluating visual reasoning and do not emphasize step-wise problem-solving.
To this end, we propose a comprehensive framework for advancing step-by-step
visual reasoning in large language models (LMMs) through three key
contributions. First, we introduce a visual reasoning benchmark specifically
designed to evaluate multi-step reasoning tasks. The benchmark presents a
diverse set of challenges with eight different categories ranging from complex
visual perception to scientific reasoning with over 4k reasoning steps in
total, enabling robust evaluation of LLMs' abilities to perform accurate and
interpretable visual reasoning across multiple steps. Second, we propose a
novel metric that assesses visual reasoning quality at the granularity of
individual steps, emphasizing both correctness and logical coherence. The
proposed metric offers deeper insights into reasoning performance compared to
traditional end-task accuracy metrics. Third, we present a new multimodal
visual reasoning model, named LlamaV-o1, trained using a multi-step curriculum
learning approach, where tasks are progressively organized to facilitate
incremental skill acquisition and problem-solving. The proposed LlamaV-o1 is
designed for multi-step reasoning and learns step-by-step through a structured
training paradigm. Extensive experiments show that our LlamaV-o1 outperforms
existing open-source models and performs favorably against close-source
proprietary models. Compared to the recent Llava-CoT, our LlamaV-o1 achieves an
average score of 67.3 with an absolute gain of 3.8\% across six benchmarks
while being 5 times faster during inference scaling. Our benchmark, model, and
code are publicly available.
VideoRAG: Retrieval-Augmented Generation over Video Corpus
Authors: name: Soyeong Jeong, name: Kangsan Kim, name: Jinheon Baek, name: Sung Ju Hwang
Published: 2025-01-10 11:17:15

Retrieval-Augmented Generation (RAG) is a powerful strategy to address the
issue of generating factually incorrect outputs in foundation models by
retrieving external knowledge relevant to queries and incorporating it into
their generation process. However, existing RAG approaches have primarily
focused on textual information, with some recent advancements beginning to
consider images, and they largely overlook videos, a rich source of multimodal
knowledge capable of representing events, processes, and contextual details
more effectively than any other modality. While a few recent studies explore
the integration of videos in the response generation process, they either
predefine query-associated videos without retrieving them according to queries,
or convert videos into the textual descriptions without harnessing their
multimodal richness. To tackle these, we introduce VideoRAG, a novel framework
that not only dynamically retrieves relevant videos based on their relevance
with queries but also utilizes both visual and textual information of videos in
the output generation. Further, to operationalize this, our method revolves
around the recent advance of Large Video Language Models (LVLMs), which enable
the direct processing of video content to represent it for retrieval and
seamless integration of the retrieved videos jointly with queries. We
experimentally validate the effectiveness of VideoRAG, showcasing that it is
superior to relevant baselines.
Enabling Scalable Oversight via Self-Evolving Critic
Authors: name: Zhengyang Tang, name: Ziniu Li, name: Zhenyang Xiao, name: Tian Ding, name: Ruoyu Sun, name: Benyou Wang, name: Dayiheng Liu, name: Fei Huang, name: Tianyu Liu, name: Bowen Yu, name: Junyang Lin
Published: 2025-01-10 05:51:52

Despite their remarkable performance, the development of Large Language
Models (LLMs) faces a critical challenge in scalable oversight: providing
effective feedback for tasks where human evaluation is difficult or where LLMs
outperform humans. While there is growing interest in using LLMs for critique,
current approaches still rely on human annotations or more powerful models,
leaving the issue of enhancing critique capabilities without external
supervision unresolved. We introduce SCRIT (Self-evolving CRITic), a framework
that enables genuine self-evolution of critique abilities. Technically, SCRIT
self-improves by training on synthetic data, generated by a contrastive-based
self-critic that uses reference solutions for step-by-step critique, and a
self-validation mechanism that ensures critique quality through correction
outcomes. Implemented with Qwen2.5-72B-Instruct, one of the most powerful LLMs,
SCRIT achieves up to a 10.3\% improvement on critique-correction and error
identification benchmarks. Our analysis reveals that SCRIT's performance scales
positively with data and model size, outperforms alternative approaches, and
benefits critically from its self-validation component.