Orionnnn/hub-stats huggingface dataset:
TokenFlow: Unified Image Tokenizer for Multimodal Understanding and
  Generation
Authors: name: Liao Qu, name: Huichao Zhang, name: Yiheng Liu, name: Xu Wang, name: Yi Jiang, name: Yiming Gao, name: Hu Ye, name: Daniel K. Du, name: Zehuan Yuan, name: Xinglong Wu
Published: 2024-12-04 06:46:55

We present TokenFlow, a novel unified image tokenizer that bridges the
long-standing gap between multimodal understanding and generation. Prior
research attempt to employ a single reconstruction-targeted Vector Quantization
(VQ) encoder for unifying these two tasks. We observe that understanding and
generation require fundamentally different granularities of visual information.
This leads to a critical trade-off, particularly compromising performance in
multimodal understanding tasks. TokenFlow addresses this challenge through an
innovative dual-codebook architecture that decouples semantic and pixel-level
feature learning while maintaining their alignment via a shared mapping
mechanism. This design enables direct access to both high-level semantic
representations crucial for understanding tasks and fine-grained visual
features essential for generation through shared indices. Our extensive
experiments demonstrate TokenFlow's superiority across multiple dimensions.
Leveraging TokenFlow, we demonstrate for the first time that discrete visual
input can surpass LLaVA-1.5 13B in understanding performance, achieving a 7.2\%
average improvement. For image reconstruction, we achieve a strong FID score of
0.63 at 384*384 resolution. Moreover, TokenFlow establishes state-of-the-art
performance in autoregressive image generation with a GenEval score of 0.55 at
256*256 resolution, achieving comparable results to SDXL.
One Shot, One Talk: Whole-body Talking Avatar from a Single Image
Authors: name: Jun Xiang, name: Yudong Guo, name: Leipeng Hu, name: Boyang Guo, name: Yancheng Yuan, name: Juyong Zhang
Published: 2024-12-02 04:27:41

Building realistic and animatable avatars still requires minutes of
multi-view or monocular self-rotating videos, and most methods lack precise
control over gestures and expressions. To push this boundary, we address the
challenge of constructing a whole-body talking avatar from a single image. We
propose a novel pipeline that tackles two critical issues: 1) complex dynamic
modeling and 2) generalization to novel gestures and expressions. To achieve
seamless generalization, we leverage recent pose-guided image-to-video
diffusion models to generate imperfect video frames as pseudo-labels. To
overcome the dynamic modeling challenge posed by inconsistent and noisy
pseudo-videos, we introduce a tightly coupled 3DGS-mesh hybrid avatar
representation and apply several key regularizations to mitigate
inconsistencies caused by imperfect labels. Extensive experiments on diverse
subjects demonstrate that our method enables the creation of a photorealistic,
precisely animatable, and expressive whole-body talking avatar from just a
single image.
Distilling Diffusion Models to Efficient 3D LiDAR Scene Completion
Authors: name: Shengyuan Zhang, name: An Zhao, name: Ling Yang, name: Zejian Li, name: Chenye Meng, name: Haoran Xu, name: Tianrun Chen, name: AnYang Wei, name: Perry Pengyun GU, name: Lingyun Sun
Published: 2024-12-04 17:57:25

Diffusion models have been applied to 3D LiDAR scene completion due to their
strong training stability and high completion quality. However, the slow
sampling speed limits the practical application of diffusion-based scene
completion models since autonomous vehicles require an efficient perception of
surrounding environments. This paper proposes a novel distillation method
tailored for 3D LiDAR scene completion models, dubbed ScoreLiDAR,
which achieves efficient yet high-quality scene completion. ScoreLiDAR enables
the distilled model to sample in significantly fewer steps after distillation.
To improve completion quality, we also introduce a novel Structural
Loss, which encourages the distilled model to capture the geometric structure
of the 3D LiDAR scene. The loss contains a scene-wise term constraining the
holistic structure and a point-wise term constraining the key landmark points
and their relative configuration. Extensive experiments demonstrate that
ScoreLiDAR significantly accelerates the completion time from 30.55 to 5.37
seconds per frame (>5times) on SemanticKITTI and achieves superior
performance compared to state-of-the-art 3D LiDAR scene completion models. Our
code is publicly available at https://github.com/happyw1nd/ScoreLiDAR.
AIM: Adaptive Inference of Multi-Modal LLMs via Token Merging and
  Pruning
Authors: name: Yiwu Zhong, name: Zhuoming Liu, name: Yin Li, name: Liwei Wang
Published: 2024-12-04 11:47:57

Large language models (LLMs) have enabled the creation of multi-modal LLMs
that exhibit strong comprehension of visual data such as images and videos.
However, these models usually rely on extensive visual tokens from visual
encoders, leading to high computational demands, which limits their
applicability in resource-constrained environments and for long-context tasks.
In this work, we propose a training-free adaptive inference method for
multi-modal LLMs that can accommodate a broad range of efficiency requirements
with a minimum performance drop. Our method consists of a) iterative token
merging based on embedding similarity before LLMs, and b) progressive token
pruning within LLM layers based on multi-modal importance. With a minimalist
design, our method can be applied to both video and image LLMs. Extensive
experiments on diverse video and image benchmarks demonstrate that, our method
substantially reduces computation load (e.g., a 7-fold reduction in
FLOPs) while preserving the performance of video and image LLMs. Further, under
a similar computational cost, our method outperforms the state-of-the-art
methods in long video understanding (e.g., +4.6 on MLVU).
Additionally, our in-depth analysis provides insights into token redundancy and
LLM layer behaviors, offering guidance for future research in designing
efficient multi-modal LLMs. Our code will be available at
https://github.com/LaVi-Lab/AIM.
Generating a Low-code Complete Workflow via Task Decomposition and RAG
Authors: name: Orlando Marquez Ayala, name: Patrice Béchard
Published: 2024-11-29 20:13:56

AI technologies are moving rapidly from research to production. With the
popularity of Foundation Models (FMs) that generate text, images, and video,
AI-based systems are increasing their complexity. Compared to traditional
AI-based software, systems employing FMs, or GenAI-based systems, are more
difficult to design due to their scale and versatility. This makes it necessary
to document best practices, known as design patterns in software engineering,
that can be used across GenAI applications. Our first contribution is to
formalize two techniques, Task Decomposition and Retrieval-Augmented Generation
(RAG), as design patterns for GenAI-based systems. We discuss their trade-offs
in terms of software quality attributes and comment on alternative approaches.
We recommend to AI practitioners to consider these techniques not only from a
scientific perspective but also from the standpoint of desired engineering
properties such as flexibility, maintainability, safety, and security. As a
second contribution, we describe our industry experience applying Task
Decomposition and RAG to build a complex real-world GenAI application for
enterprise users: Workflow Generation. The task of generating workflows entails
generating a specific plan using data from the system environment, taking as
input a user requirement. As these two patterns affect the entire AI
development cycle, we explain how they impacted the dataset creation, model
training, model evaluation, and deployment phases.
Motion Prompting: Controlling Video Generation with Motion Trajectories
Authors: name: Daniel Geng, name: Charles Herrmann, name: Junhwa Hur, name: Forrester Cole, name: Serena Zhang, name: Tobias Pfaff, name: Tatiana Lopez-Guevara, name: Carl Doersch, name: Yusuf Aytar, name: Michael Rubinstein, name: Chen Sun, name: Oliver Wang, name: Andrew Owens, name: Deqing Sun
Published: 2024-12-03 18:59:56

Motion control is crucial for generating expressive and compelling video
content; however, most existing video generation models rely mainly on text
prompts for control, which struggle to capture the nuances of dynamic actions
and temporal compositions. To this end, we train a video generation model
conditioned on spatio-temporally sparse or dense motion trajectories. In
contrast to prior motion conditioning work, this flexible representation can
encode any number of trajectories, object-specific or global scene motion, and
temporally sparse motion; due to its flexibility we refer to this conditioning
as motion prompts. While users may directly specify sparse trajectories, we
also show how to translate high-level user requests into detailed, semi-dense
motion prompts, a process we term motion prompt expansion. We demonstrate the
versatility of our approach through various applications, including camera and
object motion control, "interacting" with an image, motion transfer, and image
editing. Our results showcase emergent behaviors, such as realistic physics,
suggesting the potential of motion prompts for probing video models and
interacting with future generative world models. Finally, we evaluate
quantitatively, conduct a human study, and demonstrate strong performance.
Video results are available on our webpage: https://motion-prompting.github.io/
Truth or Mirage? Towards End-to-End Factuality Evaluation with LLM-OASIS
Authors: name: Alessandro Scirè, name: Andrei Stefan Bejgu, name: Simone Tedeschi, name: Karim Ghonim, name: Federico Martelli, name: Roberto Navigli
Published: 2024-11-29 12:21:15

After the introduction of Large Language Models (LLMs), there have been
substantial improvements in the performance of Natural Language Generation
(NLG) tasks, including Text Summarization and Machine Translation. However,
LLMs still produce outputs containing hallucinations, that is, content not
grounded in factual information. Therefore, developing methods to assess the
factuality of LLMs has become urgent.
  Indeed, resources for factuality evaluation have recently emerged. Although
challenging, these resources face one or more of the following limitations: (i)
they are tailored to a specific task or domain; (ii) they are limited in size,
thereby preventing the training of new factuality evaluators; (iii) they are
designed for simpler verification tasks, such as claim verification.
  To address these issues, we introduce LLM-Oasis, to the best of our knowledge
the largest resource for training end-to-end factuality evaluators. LLM-Oasis
is constructed by extracting claims from Wikipedia, falsifying a subset of
these claims, and generating pairs of factual and unfactual texts. We then rely
on human annotators to both validate the quality of our dataset and to create a
gold standard test set for benchmarking factuality evaluation systems.
  Our experiments demonstrate that LLM-Oasis presents a significant challenge
for state-of-the-art LLMs, with GPT-4o achieving up to 60% accuracy in our
proposed end-to-end factuality evaluation task, highlighting its potential to
drive future research in the field.
MALT: Improving Reasoning with Multi-Agent LLM Training
Authors: name: Sumeet Ramesh Motwani, name: Chandler Smith, name: Rocktim Jyoti Das, name: Markian Rybchuk, name: Philip H. S. Torr, name: Ivan Laptev, name: Fabio Pizzati, name: Ronald Clark, name: Christian Schroeder de Witt
Published: 2024-12-02 19:30:36

Enabling effective collaboration among LLMs is a crucial step toward
developing autonomous systems capable of solving complex problems. While LLMs
are typically used as single-model generators, where humans critique and refine
their outputs, the potential for jointly-trained collaborative models remains
largely unexplored. Despite promising results in multi-agent communication and
debate settings, little progress has been made in training models to work
together on tasks. In this paper, we present a first step toward "Multi-agent
LLM training" (MALT) on reasoning problems. Our approach employs a sequential
multi-agent setup with heterogeneous LLMs assigned specialized roles: a
generator, verifier, and refinement model iteratively solving problems. We
propose a trajectory-expansion-based synthetic data generation process and a
credit assignment strategy driven by joint outcome based rewards. This enables
our post-training setup to utilize both positive and negative trajectories to
autonomously improve each model's specialized capabilities as part of a joint
sequential system. We evaluate our approach across MATH, GSM8k, and CQA, where
MALT on Llama 3.1 8B models achieves relative improvements of 14.14%, 7.12%,
and 9.40% respectively over the same baseline model. This demonstrates an early
advance in multi-agent cooperative capabilities for performance on mathematical
and common sense reasoning questions. More generally, our work provides a
concrete direction for research around multi-agent LLM training approaches.
OmniCreator: Self-Supervised Unified Generation with Universal Editing
Authors: name: Haodong Chen, name: Lan Wang, name: Harry Yang, name: Ser-Nam Lim
Published: 2024-12-03 03:10:19

We introduce OmniCreator, a novel framework that can conduct text-prompted
unified (image+video) generation as well as editing all in one place.
OmniCreator acquires generative and universal editing capabilities in a
self-supervised manner, taking original text-video pairs as conditions while
utilizing the same video as a denoising target to learn the semantic
correspondence between video and text. During inference, when presented with a
text prompt and a video, OmniCreator is capable of generating a target that is
faithful to both, achieving a universal editing effect that is unconstrained as
opposed to existing editing work that primarily focuses on certain editing
types or relies on additional controls (e.g., structural conditions, attention
features, or DDIM inversion). On the other hand, when presented with a text
prompt only, OmniCreator becomes generative, producing high-quality video as a
result of the semantic correspondence learned. Importantly, we found that the
same capabilities extend to images as is, making OmniCreator a truly unified
framework. Further, due to the lack of existing generative video editing
benchmarks, we introduce the OmniBench-99 dataset, designed to evaluate the
performance of generative video editing models comprehensively. Extensive
experiments demonstrate that OmniCreator exhibits substantial superiority over
all other models.
OCR Hinders RAG: Evaluating the Cascading Impact of OCR on
  Retrieval-Augmented Generation
Authors: name: Junyuan Zhang, name: Qintong Zhang, name: Bin Wang, name: Linke Ouyang, name: Zichen Wen, name: Ying Li, name: Ka-Ho Chow, name: Conghui He, name: Wentao Zhang
Published: 2024-12-03 17:23:47

Retrieval-augmented Generation (RAG) enhances Large Language Models (LLMs) by
integrating external knowledge to reduce hallucinations and incorporate
up-to-date information without retraining. As an essential part of RAG,
external knowledge bases are commonly built by extracting structured data from
unstructured PDF documents using Optical Character Recognition (OCR). However,
given the imperfect prediction of OCR and the inherent non-uniform
representation of structured data, knowledge bases inevitably contain various
OCR noises. In this paper, we introduce OHRBench, the first benchmark for
understanding the cascading impact of OCR on RAG systems. OHRBench includes 350
carefully selected unstructured PDF documents from six real-world RAG
application domains, along with Q&As derived from multimodal elements in
documents, challenging existing OCR solutions used for RAG To better understand
OCR's impact on RAG systems, we identify two primary types of OCR noise:
Semantic Noise and Formatting Noise and apply perturbation to generate a set of
structured data with varying degrees of each OCR noise. Using OHRBench, we
first conduct a comprehensive evaluation of current OCR solutions and reveal
that none is competent for constructing high-quality knowledge bases for RAG
systems. We then systematically evaluate the impact of these two noise types
and demonstrate the vulnerability of RAG systems. Furthermore, we discuss the
potential of employing Vision-Language Models (VLMs) without OCR in RAG
systems. Code: https://github.com/opendatalab/OHR-Bench
A dynamic parallel method for performance optimization on hybrid CPUs
Authors: name: Luo Yu, name: Liu Yucheng, name: Shen Haihao
Published: 2024-11-29 08:39:24

The AIPC concept is gaining popularity, and more and more hybrid CPUs will be
running AI models on client devices. However, the current AI inference
framework overlooks the imbalanced hardware capability of hybrid CPUs, leading
to low inference performance. To address this issue, we have introduced a
dynamic parallel method for hybrid CPUs, which significantly increases LLM
inference performance by balancing the workload for each core of a hybrid CPU
before the parallel work starts. This method has enabled Neural Speed to
achieve more than 90% (on average) of memory bandwidth on two hybrid Intel
CPUs.
VideoLights: Feature Refinement and Cross-Task Alignment Transformer for
  Joint Video Highlight Detection and Moment Retrieval
Authors: name: Dhiman Paul, name: Md Rizwan Parvez, name: Nabeel Mohammed, name: Shafin Rahman
Published: 2024-12-02 14:45:53

Video Highlight Detection and Moment Retrieval (HD/MR) are essential in video
analysis. Recent joint prediction transformer models often overlook their
cross-task dynamics and video-text alignment and refinement. Moreover, most
models typically use limited, uni-directional attention mechanisms, resulting
in weakly integrated representations and suboptimal performance in capturing
the interdependence between video and text modalities. Although large-language
and vision-language models (LLM/LVLMs) have gained prominence across various
domains, their application in this field remains relatively underexplored. Here
we propose VideoLights, a novel HD/MR framework addressing these limitations
through (i) Convolutional Projection and Feature Refinement modules with an
alignment loss for better video-text feature alignment, (ii) Bi-Directional
Cross-Modal Fusion network for strongly coupled query-aware clip
representations, and (iii) Uni-directional joint-task feedback mechanism
enhancing both tasks through correlation. In addition, (iv) we introduce hard
positive/negative losses for adaptive error penalization and improved learning,
and (v) leverage LVLMs like BLIP-2 for enhanced multimodal feature integration
and intelligent pretraining using synthetic data generated from LVLMs.
Comprehensive experiments on QVHighlights, TVSum, and Charades-STA benchmarks
demonstrate state-of-the-art performance. Codes and models are available at
https://github.com/dpaul06/VideoLights .
MaskRIS: Semantic Distortion-aware Data Augmentation for Referring Image
  Segmentation
Authors: name: Minhyun Lee, name: Seungho Lee, name: Song Park, name: Dongyoon Han, name: Byeongho Heo, name: Hyunjung Shim
Published: 2024-11-28 11:27:56

Referring Image Segmentation (RIS) is an advanced vision-language task that
involves identifying and segmenting objects within an image as described by
free-form text descriptions. While previous studies focused on aligning visual
and language features, exploring training techniques, such as data
augmentation, remains underexplored. In this work, we explore effective data
augmentation for RIS and propose a novel training framework called Masked
Referring Image Segmentation (MaskRIS). We observe that the conventional image
augmentations fall short of RIS, leading to performance degradation, while
simple random masking significantly enhances the performance of RIS. MaskRIS
uses both image and text masking, followed by Distortion-aware Contextual
Learning (DCL) to fully exploit the benefits of the masking strategy. This
approach can improve the model's robustness to occlusions, incomplete
information, and various linguistic complexities, resulting in a significant
performance improvement. Experiments demonstrate that MaskRIS can easily be
applied to various RIS models, outperforming existing methods in both fully
supervised and weakly supervised settings. Finally, MaskRIS achieves new
state-of-the-art performance on RefCOCO, RefCOCO+, and RefCOCOg datasets. Code
is available at https://github.com/naver-ai/maskris.
Free Process Rewards without Process Labels
Authors: name: Lifan Yuan, name: Wendi Li, name: Huayu Chen, name: Ganqu Cui, name: Ning Ding, name: Kaiyan Zhang, name: Bowen Zhou, name: Zhiyuan Liu, name: Hao Peng
Published: 2024-12-02 21:20:02

Different from its counterpart outcome reward models (ORMs), which evaluate
the entire responses, a process reward model (PRM) scores a reasoning
trajectory step by step, providing denser and more fine grained rewards.
However, training a PRM requires labels annotated at every intermediate step,
presenting significant challenges for both manual and automatic data
collection. This paper aims to address this challenge. Both theoretically and
empirically, we show that an implicit PRM can be obtained at no
additional cost, by simply training an ORM on the cheaper response-level
labels. The only assumption is to parameterize the outcome reward as the
log-likelihood ratios of the policy and reference models, which can be
optimized regardless of the specific choice of loss objectives. In experiments,
we instantiate our implicit PRMs with various objectives and evaluate their
performance on MATH. We show that our implicit PRM outperforms a strong
MCTS-based baseline \'a la Math-Shepherd using less than 1/38 of the
training data. Its performance can be further improved with majority voting. We
further find that scaling up instructions and responses benefits our implicit
PRM, and the latter brings a larger gain. Particularly, we find that our
implicit PRM, when instantiated with the cross-entropy (CE) loss, is more
data-efficient and can keep improving generation models even when trained with
only one response per instruction, the setup that suffers from extreme data
scarcity and imbalance. Further, instructions should be relevant to downstream
tasks while the diversity of responses does not bring gains. Surprisingly,
training on extra Math-Shepherd step labels brings no further improvements to
our implicit PRM trained on only outcome data. We hope that our work will
encourage a rethinking of PRM training approaches and contribute to making
training PRMs more accessible.