Orionnnn/hub-stats huggingface dataset:
VideoLLaMB: Long-context Video Understanding with Recurrent Memory
  Bridges
Authors: name: Yuxuan Wang, name: Cihang Xie, name: Yang Liu, name: Zilong Zheng
Published: 2024-09-02 08:52:58

Recent advancements in large-scale video-language models have shown
significant potential for real-time planning and detailed interactions.
However, their high computational demands and the scarcity of annotated
datasets limit their practicality for academic researchers. In this work, we
introduce VideoLLaMB, a novel framework that utilizes temporal memory tokens
within bridge layers to allow for the encoding of entire video sequences
alongside historical visual data, effectively preserving semantic continuity
and enhancing model performance across various tasks. This approach includes
recurrent memory tokens and a SceneTilling algorithm, which segments videos
into independent semantic units to preserve semantic integrity. Empirically,
VideoLLaMB significantly outstrips existing video-language models,
demonstrating a 5.5 points improvement over its competitors across three
VideoQA benchmarks, and 2.06 points on egocentric planning. Comprehensive
results on the MVBench show that VideoLLaMB-7B achieves markedly better results
than previous 7B models of same LLM. Remarkably, it maintains robust
performance as PLLaVA even as video length increases up to 8 times. Besides,
the frame retrieval results on our specialized Needle in a Video Haystack
(NIAVH) benchmark, further validate VideoLLaMB's prowess in accurately
identifying specific frames within lengthy videos. Our SceneTilling algorithm
also enables the generation of streaming video captions directly, without
necessitating additional training. In terms of efficiency, VideoLLaMB, trained
on 16 frames, supports up to 320 frames on a single Nvidia A100 GPU with linear
GPU memory scaling, ensuring both high performance and cost-effectiveness,
thereby setting a new foundation for long-form video-language models in both
academic and practical applications.
DepthCrafter: Generating Consistent Long Depth Sequences for Open-world
  Videos
Authors: name: Wenbo Hu, name: Xiangjun Gao, name: Xiaoyu Li, name: Sijie Zhao, name: Xiaodong Cun, name: Yong Zhang, name: Long Quan, name: Ying Shan
Published: 2024-09-03 17:52:03

Despite significant advancements in monocular depth estimation for static
images, estimating video depth in the open world remains challenging, since
open-world videos are extremely diverse in content, motion, camera movement,
and length. We present DepthCrafter, an innovative method for generating
temporally consistent long depth sequences with intricate details for
open-world videos, without requiring any supplementary information such as
camera poses or optical flow. DepthCrafter achieves generalization ability to
open-world videos by training a video-to-depth model from a pre-trained
image-to-video diffusion model, through our meticulously designed three-stage
training strategy with the compiled paired video-depth datasets. Our training
approach enables the model to generate depth sequences with variable lengths at
one time, up to 110 frames, and harvest both precise depth details and rich
content diversity from realistic and synthetic datasets. We also propose an
inference strategy that processes extremely long videos through segment-wise
estimation and seamless stitching. Comprehensive evaluations on multiple
datasets reveal that DepthCrafter achieves state-of-the-art performance in
open-world video depth estimation under zero-shot settings. Furthermore,
DepthCrafter facilitates various downstream applications, including depth-based
visual effects and conditional video generation.
OD-VAE: An Omni-dimensional Video Compressor for Improving Latent Video
  Diffusion Model
Authors: name: Liuhan Chen, name: Zongjian Li, name: Bin Lin, name: Bin Zhu, name: Qian Wang, name: Shenghai Yuan, name: Xing Zhou, name: Xinghua Cheng, name: Li Yuan
Published: 2024-09-02 12:20:42

Variational Autoencoder (VAE), compressing videos into latent
representations, is a crucial preceding component of Latent Video Diffusion
Models (LVDMs). With the same reconstruction quality, the more sufficient the
VAE's compression for videos is, the more efficient the LVDMs are. However,
most LVDMs utilize 2D image VAE, whose compression for videos is only in the
spatial dimension and often ignored in the temporal dimension. How to conduct
temporal compression for videos in a VAE to obtain more concise latent
representations while promising accurate reconstruction is seldom explored. To
fill this gap, we propose an omni-dimension compression VAE, named OD-VAE,
which can temporally and spatially compress videos. Although OD-VAE's more
sufficient compression brings a great challenge to video reconstruction, it can
still achieve high reconstructed accuracy by our fine design. To obtain a
better trade-off between video reconstruction quality and compression speed,
four variants of OD-VAE are introduced and analyzed. In addition, a novel tail
initialization is designed to train OD-VAE more efficiently, and a novel
inference strategy is proposed to enable OD-VAE to handle videos of arbitrary
length with limited GPU memory. Comprehensive experiments on video
reconstruction and LVDM-based video generation demonstrate the effectiveness
and efficiency of our proposed methods.
Compositional 3D-aware Video Generation with LLM Director
Authors: name: Hanxin Zhu, name: Tianyu He, name: Anni Tang, name: Junliang Guo, name: Zhibo Chen, name: Jiang Bian
Published: 2024-08-31 23:07:22

Significant progress has been made in text-to-video generation through the
use of powerful generative models and large-scale internet data. However,
substantial challenges remain in precisely controlling individual concepts
within the generated video, such as the motion and appearance of specific
characters and the movement of viewpoints. In this work, we propose a novel
paradigm that generates each concept in 3D representation separately and then
composes them with priors from Large Language Models (LLM) and 2D diffusion
models. Specifically, given an input textual prompt, our scheme consists of
three stages: 1) We leverage LLM as the director to first decompose the complex
query into several sub-prompts that indicate individual concepts within the
video~(e.g., scene, objects, motions), then we let LLM to invoke
pre-trained expert models to obtain corresponding 3D representations of
concepts. 2) To compose these representations, we prompt multi-modal LLM to
produce coarse guidance on the scales and coordinates of trajectories for the
objects. 3) To make the generated frames adhere to natural image distribution,
we further leverage 2D diffusion priors and use Score Distillation Sampling to
refine the composition. Extensive experiments demonstrate that our method can
generate high-fidelity videos from text with diverse motion and flexible
control over each concept. Project page: https://aka.ms/c3v.
LinFusion: 1 GPU, 1 Minute, 16K Image
Authors: name: Songhua Liu, name: Weihao Yu, name: Zhenxiong Tan, name: Xinchao Wang
Published: 2024-09-03 17:54:39

Modern diffusion models, particularly those utilizing a Transformer-based
UNet for denoising, rely heavily on self-attention operations to manage complex
spatial relationships, thus achieving impressive generation performance.
However, this existing paradigm faces significant challenges in generating
high-resolution visual content due to its quadratic time and memory complexity
with respect to the number of spatial tokens. To address this limitation, we
aim at a novel linear attention mechanism as an alternative in this paper.
Specifically, we begin our exploration from recently introduced models with
linear complexity, e.g., Mamba, Mamba2, and Gated Linear Attention, and
identify two key features-attention normalization and non-causal inference-that
enhance high-resolution visual generation performance. Building on these
insights, we introduce a generalized linear attention paradigm, which serves as
a low-rank approximation of a wide spectrum of popular linear token mixers. To
save the training cost and better leverage pre-trained models, we initialize
our models and distill the knowledge from pre-trained StableDiffusion (SD). We
find that the distilled model, termed LinFusion, achieves performance on par
with or superior to the original SD after only modest training, while
significantly reducing time and memory complexity. Extensive experiments on
SD-v1.5, SD-v2.1, and SD-XL demonstrate that LinFusion delivers satisfactory
zero-shot cross-resolution generation performance, generating high-resolution
images like 16K resolution. Moreover, it is highly compatible with pre-trained
SD components, such as ControlNet and IP-Adapter, requiring no adaptation
efforts. Codes are available at https://github.com/Huage001/LinFusion.
LongRecipe: Recipe for Efficient Long Context Generalization in Large
  Languge Models
Authors: name: Zhiyuan Hu, name: Yuliang Liu, name: Jinman Zhao, name: Suyuchen Wang, name: Yan Wang, name: Wei Shen, name: Qing Gu, name: Anh Tuan Luu, name: See-Kiong Ng, name: Zhiwei Jiang, name: Bryan Hooi
Published: 2024-08-31 17:19:30

Large language models (LLMs) face significant challenges in handling
long-context tasks because of their limited effective context window size
during pretraining, which restricts their ability to generalize over extended
sequences. Meanwhile, extending the context window in LLMs through
post-pretraining is highly resource-intensive. To address this, we introduce
**LongRecipe**, an efficient training strategy for extending the context window
of LLMs, including impactful token analysis, position index transformation, and
training optimization strategies. It simulates long-sequence inputs while
maintaining training efficiency and significantly improves the model's
understanding of long-range dependencies. Experiments on three types of LLMs
show that LongRecipe can utilize long sequences while requiring only 30% of the
target context window size, and reduces computational training resource over
85% compared to full sequence training. Furthermore, LongRecipe also preserves
the original LLM's capabilities in general tasks. Ultimately, *we can extend
the effective context window of open-source LLMs from 8k to 128k, achieving
performance close to GPT-4 with just one day of dedicated training using a
single GPU with 80G memory.* Our code is released at the
[link](https://github.com/zhiyuanhubj/LongRecipe).
Density Adaptive Attention-based Speech Network: Enhancing Feature
  Understanding for Mental Health Disorders
Authors: name: Georgios Ioannides, name: Adrian Kieback, name: Aman Chadha, name: Aaron Elkins
Published: 2024-08-31 08:50:28

Speech-based depression detection poses significant challenges for automated
detection due to its unique manifestation across individuals and data scarcity.
Addressing these challenges, we introduce DAAMAudioCNNLSTM and
DAAMAudioTransformer, two parameter efficient and explainable models for audio
feature extraction and depression detection. DAAMAudioCNNLSTM features a novel
CNN-LSTM framework with multi-head Density Adaptive Attention Mechanism (DAAM),
focusing dynamically on informative speech segments. DAAMAudioTransformer,
leveraging a transformer encoder in place of the CNN-LSTM architecture,
incorporates the same DAAM module for enhanced attention and interpretability.
These approaches not only enhance detection robustness and interpretability but
also achieve state-of-the-art performance: DAAMAudioCNNLSTM with an F1 macro
score of 0.702 and DAAMAudioTransformer with an F1 macro score of 0.72 on the
DAIC-WOZ dataset, without reliance on supplementary information such as vowel
positions and speaker information during training/validation as in previous
approaches. Both models' significant explainability and efficiency in
leveraging speech signals for depression detection represent a leap towards
more reliable, clinically useful diagnostic tools, promising advancements in
speech and mental health care. To foster further research in this domain, we
make our code publicly available.
OLMoE: Open Mixture-of-Experts Language Models
Authors: name: Niklas Muennighoff, name: Luca Soldaini, name: Dirk Groeneveld, name: Kyle Lo, name: Jacob Morrison, name: Sewon Min, name: Weijia Shi, name: Pete Walsh, name: Oyvind Tafjord, name: Nathan Lambert, name: Yuling Gu, name: Shane Arora, name: Akshita Bhagia, name: Dustin Schwenk, name: David Wadden, name: Alexander Wettig, name: Binyuan Hui, name: Tim Dettmers, name: Douwe Kiela, name: Ali Farhadi, name: Noah A. Smith, name: Pang Wei Koh, name: Amanpreet Singh, name: Hannaneh Hajishirzi
Published: 2024-09-03 17:08:20

We introduce OLMoE, a fully open, state-of-the-art language model leveraging
sparse Mixture-of-Experts (MoE). OLMoE-1B-7B has 7 billion (B) parameters but
uses only 1B per input token. We pretrain it on 5 trillion tokens and further
adapt it to create OLMoE-1B-7B-Instruct. Our models outperform all available
models with similar active parameters, even surpassing larger ones like
Llama2-13B-Chat and DeepSeekMoE-16B. We present various experiments on MoE
training, analyze routing in our model showing high specialization, and
open-source all aspects of our work: model weights, training data, code, and
logs.
Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
Authors: name: Zhifei Xie, name: Changqiao Wu
Published: 2024-08-29 17:18:53

Recent advances in language models have achieved significant progress.
GPT-4o, as a new milestone, has enabled real-time conversations with humans,
demonstrating near-human natural fluency. Such human-computer interaction
necessitates models with the capability to perform reasoning directly with the
audio modality and generate output in streaming. However, this remains beyond
the reach of current academic models, as they typically depend on extra TTS
systems for speech synthesis, resulting in undesirable latency. This paper
introduces the Mini-Omni, an audio-based end-to-end conversational model,
capable of real-time speech interaction. To achieve this capability, we propose
a text-instructed speech generation method, along with batch-parallel
strategies during inference to further boost the performance. Our method also
helps to retain the original model's language capabilities with minimal
degradation, enabling other works to establish real-time interaction
capabilities. We call this training method "Any Model Can Talk". We also
introduce the VoiceAssistant-400K dataset to fine-tune models optimized for
speech output. To our best knowledge, Mini-Omni is the first fully end-to-end,
open-source model for real-time speech interaction, offering valuable potential
for future research.
VisionTS: Visual Masked Autoencoders Are Free-Lunch Zero-Shot Time
  Series Forecasters
Authors: name: Mouxiang Chen, name: Lefei Shen, name: Zhuo Li, name: Xiaoyun Joy Wang, name: Jianling Sun, name: Chenghao Liu
Published: 2024-08-30 12:51:55

Foundation models have emerged as a promising approach in time series
forecasting (TSF). Existing approaches either fine-tune large language models
(LLMs) or build large-scale time-series datasets to develop TSF foundation
models. However, these methods face challenges due to the severe cross-domain
gap or in-domain heterogeneity. In this paper, we explore a new road to
building a TSF foundation model from rich and high-quality natural images,
based on the intrinsic similarities between images and time series. To bridge
the gap between the two domains, we reformulate the TSF task as an image
reconstruction task, which is further processed by a visual masked autoencoder
(MAE) self-supervised pre-trained on the ImageNet dataset. Surprisingly,
without further adaptation in the time-series domain, the proposed VisionTS
could achieve superior zero-shot forecasting performance compared to existing
TSF foundation models. With minimal fine-tuning, VisionTS could further improve
the forecasting and achieve state-of-the-art performance in most cases. These
findings suggest that visual models could be a free lunch for TSF and highlight
the potential for future cross-domain research between computer vision and TSF.
Our code is publicly available at https://github.com/Keytoyze/VisionTS.
Iterative Graph Alignment
Authors: name: Fangyuan Yu, name: Hardeep Singh Arora, name: Matt Johnson
Published: 2024-08-29 16:15:01

By compressing diverse narratives, LLMs go beyond memorization, achieving
intelligence by capturing generalizable causal relationships. However, they
suffer from local 'representation gaps' due to insufficient training data
diversity, limiting their real-world utility, especially in tasks requiring
strict alignment to rules. Traditional alignment methods relying on heavy human
annotations are inefficient and unscalable. Recent self-alignment techniques
also fall short, as they often depend on self-selection based prompting and
memorization-based learning. To address these issues, we introduce Iterative
Graph Alignment (IGA), an annotation-free rule-based alignment algorithm. A
teacher model (VLM) employs Iterative Graph Prompting (IGP) to create logical
graphs and reference answers. The student model (LLM) identifies local
knowledge gaps by attempting to align its responses with these references,
collaborating with helper models to generate diverse answers. These aligned
responses are then used for iterative supervised fine-tuning (SFT). Our
evaluations across five rule-based scenarios demonstrate IGP's effectiveness,
with a 73.12\% alignment improvement in Claude Sonnet 3.5, and
Llama3-8B-Instruct achieving an 86.20\% improvement, outperforming Claude
Sonnet 3.5 in rule-based alignment.
VQ4DiT: Efficient Post-Training Vector Quantization for Diffusion
  Transformers
Authors: name: Juncan Deng, name: Shuaiting Li, name: Zeyu Wang, name: Hong Gu, name: Kedong Xu, name: Kejie Huang
Published: 2024-08-30 09:15:54

The Diffusion Transformers Models (DiTs) have transitioned the network
architecture from traditional UNets to transformers, demonstrating exceptional
capabilities in image generation. Although DiTs have been widely applied to
high-definition video generation tasks, their large parameter size hinders
inference on edge devices. Vector quantization (VQ) can decompose model weight
into a codebook and assignments, allowing extreme weight quantization and
significantly reducing memory usage. In this paper, we propose VQ4DiT, a fast
post-training vector quantization method for DiTs. We found that traditional VQ
methods calibrate only the codebook without calibrating the assignments. This
leads to weight sub-vectors being incorrectly assigned to the same assignment,
providing inconsistent gradients to the codebook and resulting in a suboptimal
result. To address this challenge, VQ4DiT calculates the candidate assignment
set for each weight sub-vector based on Euclidean distance and reconstructs the
sub-vector based on the weighted average. Then, using the zero-data and
block-wise calibration method, the optimal assignment from the set is
efficiently selected while calibrating the codebook. VQ4DiT quantizes a DiT
XL/2 model on a single NVIDIA A100 GPU within 20 minutes to 5 hours depending
on the different quantization settings. Experiments show that VQ4DiT
establishes a new state-of-the-art in model size and performance trade-offs,
quantizing weights to 2-bit precision while retaining acceptable image
generation quality.
GIFT-SW: Gaussian noise Injected Fine-Tuning of Salient Weights for LLMs
Authors: name: Maxim Zhelnin, name: Viktor Moskvoretskii, name: Egor Shvetsov, name: Egor Venediktov, name: Mariya Krylova, name: Aleksandr Zuev, name: Evgeny Burnaev
Published: 2024-08-27 14:41:14

Parameter Efficient Fine-Tuning (PEFT) methods have gained popularity and
democratized the usage of Large Language Models (LLMs). Recent studies have
shown that a small subset of weights significantly impacts performance. Based
on this observation, we introduce a novel PEFT method, called Gaussian noise
Injected Fine Tuning of Salient Weights (GIFT-SW). Our method updates only
salient columns, while injecting Gaussian noise into non-salient ones. To
identify these columns, we developeda generalized sensitivity metric that
extends and unifies metrics from previous studies. Experiments with LLaMA
models demonstrate that GIFT-SW outperforms full fine-tuning and modern PEFT
methods under the same computational budget. Moreover, GIFT-SW offers practical
advantages to recover performance of models subjected to mixed-precision
quantization with keeping salient weights in full precision.
CrossViewDiff: A Cross-View Diffusion Model for Satellite-to-Street View
  Synthesis
Authors: name: Weijia Li, name: Jun He, name: Junyan Ye, name: Huaping Zhong, name: Zhimeng Zheng, name: Zilong Huang, name: Dahua Lin, name: Conghui He
Published: 2024-08-27 03:41:44

Satellite-to-street view synthesis aims at generating a realistic street-view
image from its corresponding satellite-view image. Although stable diffusion
models have exhibit remarkable performance in a variety of image generation
applications, their reliance on similar-view inputs to control the generated
structure or texture restricts their application to the challenging cross-view
synthesis task. In this work, we propose CrossViewDiff, a cross-view diffusion
model for satellite-to-street view synthesis. To address the challenges posed
by the large discrepancy across views, we design the satellite scene structure
estimation and cross-view texture mapping modules to construct the structural
and textural controls for street-view image synthesis. We further design a
cross-view control guided denoising process that incorporates the above
controls via an enhanced cross-view attention module. To achieve a more
comprehensive evaluation of the synthesis results, we additionally design a
GPT-based scoring method as a supplement to standard evaluation metrics. We
also explore the effect of different data sources (e.g., text, maps, building
heights, and multi-temporal satellite imagery) on this task. Results on three
public cross-view datasets show that CrossViewDiff outperforms current
state-of-the-art on both standard and GPT-based evaluation metrics, generating
high-quality street-view panoramas with more realistic structures and textures
across rural, suburban, and urban scenes. The code and models of this work will
be released at https://opendatalab.github.io/CrossViewDiff/.