Orionnnn/hub-stats huggingface dataset:
Training-free Long Video Generation with Chain of Diffusion Model
  Experts
Authors: name: Wenhao Li, name: Yichao Cao, name: Xie Su, name: Xi Lin, name: Shan You, name: Mingkai Zheng, name: Yi Chen, name: Chang Xu
Published: 2024-08-24 01:33:28

Video generation models hold substantial potential in areas such as
filmmaking. However, current video diffusion models need high computational
costs and produce suboptimal results due to high complexity of video generation
task. In this paper, we propose ConFiner, an efficient high-quality
video generation framework that decouples video generation into easier
subtasks: structure control and spatial-temporal refinement.
It can generate high-quality videos with chain of off-the-shelf diffusion model
experts, each expert responsible for a decoupled subtask. During the
refinement, we introduce coordinated denoising, which can merge multiple
diffusion experts' capabilities into a single sampling. Furthermore, we design
ConFiner-Long framework, which can generate long coherent video with three
constraint strategies on ConFiner. Experimental results indicate that with only
10\% of the inference cost, our ConFiner surpasses representative models like
Lavie and Modelscope across all objective and subjective metrics. And
ConFiner-Long can generate high-quality and coherent videos with up to 600
frames.
TVG: A Training-free Transition Video Generation Method with Diffusion
  Models
Authors: name: Rui Zhang, name: Yaosen Chen, name: Yuegen Liu, name: Wei Wang, name: Xuming Wen, name: Hongxia Wang
Published: 2024-08-24 00:33:14

Transition videos play a crucial role in media production, enhancing the flow
and coherence of visual narratives. Traditional methods like morphing often
lack artistic appeal and require specialized skills, limiting their
effectiveness. Recent advances in diffusion model-based video generation offer
new possibilities for creating transitions but face challenges such as poor
inter-frame relationship modeling and abrupt content changes. We propose a
novel training-free Transition Video Generation (TVG) approach using
video-level diffusion models that addresses these limitations without
additional training. Our method leverages Gaussian Process Regression
(GPR) to model latent representations, ensuring smooth and dynamic
transitions between frames. Additionally, we introduce interpolation-based
conditional controls and a Frequency-aware Bidirectional Fusion (FBiF)
architecture to enhance temporal control and transition reliability.
Evaluations of benchmark datasets and custom image pairs demonstrate the
effectiveness of our approach in generating high-quality smooth transition
videos. The code are provided in https://sobeymil.github.io/tvg.com.
Efficient Detection of Toxic Prompts in Large Language Models
Authors: name: Yi Liu, name: Junzhe Yu, name: Huijia Sun, name: Ling Shi, name: Gelei Deng, name: Yuqi Chen, name: Yang Liu
Published: 2024-08-21 15:54:04

Large language models (LLMs) like ChatGPT and Gemini have significantly
advanced natural language processing, enabling various applications such as
chatbots and automated content generation. However, these models can be
exploited by malicious individuals who craft toxic prompts to elicit harmful or
unethical responses. These individuals often employ jailbreaking techniques to
bypass safety mechanisms, highlighting the need for robust toxic prompt
detection methods. Existing detection techniques, both blackbox and whitebox,
face challenges related to the diversity of toxic prompts, scalability, and
computational efficiency. In response, we propose ToxicDetector, a lightweight
greybox method designed to efficiently detect toxic prompts in LLMs.
ToxicDetector leverages LLMs to create toxic concept prompts, uses embedding
vectors to form feature vectors, and employs a Multi-Layer Perceptron (MLP)
classifier for prompt classification. Our evaluation on various versions of the
LLama models, Gemma-2, and multiple datasets demonstrates that ToxicDetector
achieves a high accuracy of 96.39\% and a low false positive rate of 2.00\%,
outperforming state-of-the-art methods. Additionally, ToxicDetector's
processing time of 0.0780 seconds per prompt makes it highly suitable for
real-time applications. ToxicDetector achieves high accuracy, efficiency, and
scalability, making it a practical method for toxic prompt detection in LLMs.
RoundTable: Leveraging Dynamic Schema and Contextual Autocomplete for
  Enhanced Query Precision in Tabular Question Answering
Authors: name: Pratyush Kumar, name: Kuber Vijaykumar Bellad, name: Bharat Vadlamudi, name: Aman Chadha
Published: 2024-08-22 13:13:06

With advancements in Large Language Models (LLMs), a major use case that has
emerged is querying databases in plain English, translating user questions into
executable database queries, which has improved significantly. However,
real-world datasets often feature a vast array of attributes and complex
values, complicating the LLMs task of accurately identifying relevant columns
or values from natural language queries. Traditional methods cannot fully relay
the datasets size and complexity to the LLM. To address these challenges, we
propose a novel framework that leverages Full-Text Search (FTS) on the input
table. This approach not only enables precise detection of specific values and
columns but also narrows the search space for language models, thereby
enhancing query accuracy. Additionally, it supports a custom auto-complete
feature that suggests queries based on the data in the table. This integration
significantly refines the interaction between the user and complex datasets,
offering a sophisticated solution to the limitations faced by current table
querying capabilities. This work is accompanied by an application for both Mac
and Windows platforms, which readers can try out themselves on their own data.
HiRED: Attention-Guided Token Dropping for Efficient Inference of
  High-Resolution Vision-Language Models in Resource-Constrained Environments
Authors: name: Kazi Hasan Ibn Arif, name: JinYi Yoon, name: Dimitrios S. Nikolopoulos, name: Hans Vandierendonck, name: Deepu John, name: Bo Ji
Published: 2024-08-20 15:34:27

High-resolution Vision-Language Models (VLMs) have been widely used in
multimodal tasks to enhance accuracy by preserving detailed image information.
However, these models often generate excessive visual tokens due to encoding
multiple partitions of the input image. Processing these excessive visual
tokens is computationally challenging, especially in resource-constrained
environments with commodity GPUs. To support high-resolution images while
meeting resource constraints, we propose High-Resolution Early Dropping
(HiRED), a token-dropping scheme that operates within a fixed token budget
before the Large Language Model (LLM) stage. HiRED can be integrated with
existing high-resolution VLMs in a plug-and-play manner, as it requires no
additional training while still maintaining superior accuracy. We strategically
use the vision encoder's attention in the initial layers to assess the visual
content of each image partition and allocate the token budget accordingly.
Then, using the attention in the final layer, we select the most important
visual tokens from each partition within the allocated budget, dropping the
rest. Empirically, when applied to LLaVA-Next-7B on NVIDIA TESLA P40 GPU, HiRED
with a 20% token budget increases token generation throughput by 4.7, reduces
first-token generation latency by 15 seconds, and saves 2.3 GB of GPU memory
for a single inference.
A Web-Based Solution for Federated Learning with LLM-Based Automation
Authors: name: Chamith Mawela, name: Chaouki Ben Issaid, name: Mehdi Bennis
Published: 2024-08-23 11:57:02

Federated Learning (FL) offers a promising approach for collaborative machine
learning across distributed devices. However, its adoption is hindered by the
complexity of building reliable communication architectures and the need for
expertise in both machine learning and network programming. This paper presents
a comprehensive solution that simplifies the orchestration of FL tasks while
integrating intent-based automation. We develop a user-friendly web application
supporting the federated averaging (FedAvg) algorithm, enabling users to
configure parameters through an intuitive interface. The backend solution
efficiently manages communication between the parameter server and edge nodes.
We also implement model compression and scheduling algorithms to optimize FL
performance. Furthermore, we explore intent-based automation in FL using a
fine-tuned Language Model (LLM) trained on a tailored dataset, allowing users
to conduct FL tasks using high-level prompts. We observe that the LLM-based
automated solution achieves comparable test accuracy to the standard web-based
solution while reducing transferred bytes by up to 64% and CPU time by up to
46% for FL tasks. Also, we leverage the neural architecture search (NAS) and
hyperparameter optimization (HPO) using LLM to improve the performance. We
observe that by using this approach test accuracy can be improved by 10-20% for
the carried out FL tasks.
Memory-Efficient LLM Training with Online Subspace Descent
Authors: name: Kaizhao Liang, name: Bo Liu, name: Lizhang Chen, name: Qiang Liu
Published: 2024-08-23 05:54:53

Recently, a wide range of memory-efficient LLM training algorithms have
gained substantial popularity. These methods leverage the low-rank structure of
gradients to project optimizer states into a subspace using projection matrix
found by singular value decomposition (SVD). However, convergence of these
algorithms is highly dependent on the update rules of their projection matrix.
In this work, we provide the first convergence guarantee for arbitrary
update rules of projection matrix. This guarantee is generally applicable to
optimizers that can be analyzed with Hamiltonian Descent, including most common
ones, such as LION, Adam. Inspired by our theoretical understanding, we propose
Online Subspace Descent, a new family of subspace descent optimizer without
SVD. Instead of updating the projection matrix with eigenvectors, Online
Subspace Descent updates the projection matrix with online PCA. Online Subspace
Descent is flexible and introduces only minimum overhead to training. We show
that for the task of pretraining LLaMA models ranging from 60M to 7B parameters
on the C4 dataset, Online Subspace Descent achieves lower perplexity and better
downstream tasks performance than state-of-the-art low-rank training methods
across different settings and narrows the gap with full-rank baselines.
CODE: Confident Ordinary Differential Editing
Authors: name: Bastien van Delft, name: Tommaso Martorella, name: Alexandre Alahi
Published: 2024-08-22 14:12:20

Conditioning image generation facilitates seamless editing and the creation
of photorealistic images. However, conditioning on noisy or Out-of-Distribution
(OoD) images poses significant challenges, particularly in balancing fidelity
to the input and realism of the output. We introduce Confident Ordinary
Differential Editing (CODE), a novel approach for image synthesis that
effectively handles OoD guidance images. Utilizing a diffusion model as a
generative prior, CODE enhances images through score-based updates along the
probability-flow Ordinary Differential Equation (ODE) trajectory. This method
requires no task-specific training, no handcrafted modules, and no assumptions
regarding the corruptions affecting the conditioning image. Our method is
compatible with any diffusion model. Positioned at the intersection of
conditional image generation and blind image restoration, CODE operates in a
fully blind manner, relying solely on a pre-trained generative model. Our
method introduces an alternative approach to blind restoration: instead of
targeting a specific ground truth image based on assumptions about the
underlying corruption, CODE aims to increase the likelihood of the input image
while maintaining fidelity. This results in the most probable in-distribution
image around the input. Our contributions are twofold. First, CODE introduces a
novel editing method based on ODE, providing enhanced control, realism, and
fidelity compared to its SDE-based counterpart. Second, we introduce a
confidence interval-based clipping method, which improves CODE's effectiveness
by allowing it to disregard certain pixels or information, thus enhancing the
restoration process in a blind manner. Experimental results demonstrate CODE's
effectiveness over existing methods, particularly in scenarios involving severe
degradation or OoD inputs.
Multi-Layer Transformers Gradient Can be Approximated in Almost Linear
  Time
Authors: name: Yingyu Liang, name: Zhizhou Sha, name: Zhenmei Shi, name: Zhao Song, name: Yufa Zhou
Published: 2024-08-23 17:16:43

The quadratic computational complexity in the self-attention mechanism of
popular transformer architectures poses significant challenges for training and
inference, particularly in terms of efficiency and memory requirements. Towards
addressing these challenges, this paper introduces a novel fast computation
method for gradient calculation in multi-layer transformer models. Our approach
enables the computation of gradients for the entire multi-layer transformer
model in almost linear time n^{1+o(1)}, where n is the input sequence
length. This breakthrough significantly reduces the computational bottleneck
associated with the traditional quadratic time complexity. Our theory holds for
any loss function and maintains a bounded approximation error across the entire
model. Furthermore, our analysis can hold when the multi-layer transformer
model contains many practical sub-modules, such as residual connection, casual
mask, and multi-head attention. By improving the efficiency of gradient
computation in large language models, we hope that our work will facilitate the
more effective training and deployment of long-context language models based on
our theoretical results.
Building and better understanding vision-language models: insights and
  future directions
Authors: name: Hugo Laurençon, name: Andrés Marafioti, name: Victor Sanh, name: Léo Tronchon
Published: 2024-08-22 17:47:24

The field of vision-language models (VLMs), which take images and texts as
inputs and output texts, is rapidly evolving and has yet to reach consensus on
several key aspects of the development pipeline, including data, architecture,
and training methods. This paper can be seen as a tutorial for building a VLM.
We begin by providing a comprehensive overview of the current state-of-the-art
approaches, highlighting the strengths and weaknesses of each, addressing the
major challenges in the field, and suggesting promising research directions for
underexplored areas. We then walk through the practical steps to build
Idefics3-8B, a powerful VLM that significantly outperforms its predecessor
Idefics2-8B, while being trained efficiently, exclusively on open datasets, and
using a straightforward pipeline. These steps include the creation of Docmatix,
a dataset for improving document understanding capabilities, which is 240 times
larger than previously available datasets. We release the model along with the
datasets created for its training.
LayerPano3D: Layered 3D Panorama for Hyper-Immersive Scene Generation
Authors: name: Shuai Yang, name: Jing Tan, name: Mengchen Zhang, name: Tong Wu, name: Yixuan Li, name: Gordon Wetzstein, name: Ziwei Liu, name: Dahua Lin
Published: 2024-08-23 17:50:23

3D immersive scene generation is a challenging yet critical task in computer
vision and graphics. A desired virtual 3D scene should 1) exhibit
omnidirectional view consistency, and 2) allow for free exploration in complex
scene hierarchies. Existing methods either rely on successive scene expansion
via inpainting or employ panorama representation to represent large FOV scene
environments. However, the generated scene suffers from semantic drift during
expansion and is unable to handle occlusion among scene hierarchies. To tackle
these challenges, we introduce LayerPano3D, a novel framework for full-view,
explorable panoramic 3D scene generation from a single text prompt. Our key
insight is to decompose a reference 2D panorama into multiple layers at
different depth levels, where each layer reveals the unseen space from the
reference views via diffusion prior. LayerPano3D comprises multiple dedicated
designs: 1) we introduce a novel text-guided anchor view synthesis pipeline for
high-quality, consistent panorama generation. 2) We pioneer the Layered 3D
Panorama as underlying representation to manage complex scene hierarchies and
lift it into 3D Gaussians to splat detailed 360-degree omnidirectional scenes
with unconstrained viewing paths. Extensive experiments demonstrate that our
framework generates state-of-the-art 3D panoramic scene in both full view
consistency and immersive exploratory experience. We believe that LayerPano3D
holds promise for advancing 3D panoramic scene creation with numerous
applications.
FLoD: Integrating Flexible Level of Detail into 3D Gaussian Splatting
  for Customizable Rendering
Authors: name: Yunji Seo, name: Young Sun Choi, name: Hyun Seung Son, name: Youngjung Uh
Published: 2024-08-23 07:56:25

3D Gaussian Splatting (3DGS) achieves fast and high-quality renderings by
using numerous small Gaussians, which leads to significant memory consumption.
This reliance on a large number of Gaussians restricts the application of
3DGS-based models on low-cost devices due to memory limitations. However,
simply reducing the number of Gaussians to accommodate devices with less memory
capacity leads to inferior quality compared to the quality that can be achieved
on high-end hardware. To address this lack of scalability, we propose
integrating a Flexible Level of Detail (FLoD) to 3DGS, to allow a scene to be
rendered at varying levels of detail according to hardware capabilities. While
existing 3DGSs with LoD focus on detailed reconstruction, our method provides
reconstructions using a small number of Gaussians for reduced memory
requirements, and a larger number of Gaussians for greater detail. Experiments
demonstrate our various rendering options with tradeoffs between rendering
quality and memory usage, thereby allowing real-time rendering across different
memory constraints. Furthermore, we show that our method generalizes to
different 3DGS frameworks, indicating its potential for integration into future
state-of-the-art developments. Project page:
https://3dgs-flod.github.io/flod.github.io/
MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution
  Real-World Scenarios that are Difficult for Humans?
Authors: name: Yi-Fan Zhang, name: Huanyu Zhang, name: Haochen Tian, name: Chaoyou Fu, name: Shuangqing Zhang, name: Junfei Wu, name: Feng Li, name: Kun Wang, name: Qingsong Wen, name: Zhang Zhang, name: Liang Wang, name: Rong Jin, name: Tieniu Tan
Published: 2024-08-23 17:59:51

Comprehensive evaluation of Multimodal Large Language Models (MLLMs) has
recently garnered widespread attention in the research community. However, we
observe that existing benchmarks present several common barriers that make it
difficult to measure the significant challenges that models face in the real
world, including: 1) small data scale leads to a large performance variance; 2)
reliance on model-based annotations results in restricted data quality; 3)
insufficient task difficulty, especially caused by the limited image
resolution. To tackle these issues, we introduce MME-RealWorld. Specifically,
we collect more than 300K images from public datasets and the Internet,
filtering 13,366 high-quality images for annotation. This involves the
efforts of professional 25 annotators and 7 experts in MLLMs, contributing
to 29,429 question-answer pairs that cover 43 subtasks across 5
real-world scenarios, extremely challenging even for humans. As far as we know,
MME-RealWorld is the largest manually annotated benchmark to date, featuring
the highest resolution and a targeted focus on real-world applications. We
further conduct a thorough evaluation involving 28 prominent MLLMs, such as
GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet. Our results show that even the
most advanced models struggle with our benchmarks, where none of them reach
60% accuracy. The challenges of perceiving high-resolution images and
understanding complex real-world scenarios remain urgent issues to be
addressed. The data and evaluation code are released at
https://mme-realworld.github.io/ .
T3M: Text Guided 3D Human Motion Synthesis from Speech
Authors: name: Wenshuo Peng, name: Kaipeng Zhang, name: Sai Qian Zhang
Published: 2024-08-23 07:37:26

Speech-driven 3D motion synthesis seeks to create lifelike animations based
on human speech, with potential uses in virtual reality, gaming, and the film
production. Existing approaches reply solely on speech audio for motion
generation, leading to inaccurate and inflexible synthesis results. To mitigate
this problem, we introduce a novel text-guided 3D human motion synthesis
method, termed T3M. Unlike traditional approaches, T3M allows precise
control over motion synthesis via textual input, enhancing the degree of
diversity and user customization. The experiment results demonstrate that T3M
can greatly outperform the state-of-the-art methods in both quantitative
metrics and qualitative evaluations. We have publicly released our code at
https://github.com/Gloria2tt/T3M.git{https://github.com/Gloria2tt/T3M.git}
CustomCrafter: Customized Video Generation with Preserving Motion and
  Concept Composition Abilities
Authors: name: Tao Wu, name: Yong Zhang, name: Xintao Wang, name: Xianpan Zhou, name: Guangcong Zheng, name: Zhongang Qi, name: Ying Shan, name: Xi Li
Published: 2024-08-23 17:26:06

Customized video generation aims to generate high-quality videos guided by
text prompts and subject's reference images. However, since it is only trained
on static images, the fine-tuning process of subject learning disrupts
abilities of video diffusion models (VDMs) to combine concepts and generate
motions. To restore these abilities, some methods use additional video similar
to the prompt to fine-tune or guide the model. This requires frequent changes
of guiding videos and even re-tuning of the model when generating different
motions, which is very inconvenient for users. In this paper, we propose
CustomCrafter, a novel framework that preserves the model's motion generation
and conceptual combination abilities without additional video and fine-tuning
to recovery. For preserving conceptual combination ability, we design a
plug-and-play module to update few parameters in VDMs, enhancing the model's
ability to capture the appearance details and the ability of concept
combinations for new subjects. For motion generation, we observed that VDMs
tend to restore the motion of video in the early stage of denoising, while
focusing on the recovery of subject details in the later stage. Therefore, we
propose Dynamic Weighted Video Sampling Strategy. Using the pluggability of our
subject learning modules, we reduce the impact of this module on motion
generation in the early stage of denoising, preserving the ability to generate
motion of VDMs. In the later stage of denoising, we restore this module to
repair the appearance details of the specified subject, thereby ensuring the
fidelity of the subject's appearance. Experimental results show that our method
has a significant improvement compared to previous methods.