Orionnnn/hub-stats huggingface dataset:
Loong: Generating Minute-level Long Videos with Autoregressive Language
  Models
Authors: name: Yuqing Wang, name: Tianwei Xiong, name: Daquan Zhou, name: Zhijie Lin, name: Yang Zhao, name: Bingyi Kang, name: Jiashi Feng, name: Xihui Liu
Published: 2024-10-03 17:59:02

It is desirable but challenging to generate content-rich long videos in the
scale of minutes. Autoregressive large language models (LLMs) have achieved
great success in generating coherent and long sequences of tokens in the domain
of natural language processing, while the exploration of autoregressive LLMs
for video generation is limited to generating short videos of several seconds.
In this work, we conduct a deep analysis of the challenges that prevent
autoregressive LLM-based video generators from generating long videos. Based on
the observations and analysis, we propose Loong, a new autoregressive LLM-based
video generator that can generate minute-long videos. Specifically, we model
the text tokens and video tokens as a unified sequence for autoregressive LLMs
and train the model from scratch. We propose progressive short-to-long training
with a loss re-weighting scheme to mitigate the loss imbalance problem for long
video training. We further investigate inference strategies, including video
token re-encoding and sampling strategies, to diminish error accumulation
during inference. Our proposed Loong can be trained on 10-second videos and be
extended to generate minute-level long videos conditioned on text prompts, as
demonstrated by the results. More samples are available at:
https://epiphqny.github.io/Loong-video.
Revisit Large-Scale Image-Caption Data in Pre-training Multimodal
  Foundation Models
Authors: name: Zhengfeng Lai, name: Vasileios Saveris, name: Chen Chen, name: Hong-You Chen, name: Haotian Zhang, name: Bowen Zhang, name: Juan Lao Tebar, name: Wenze Hu, name: Zhe Gan, name: Peter Grasch, name: Meng Cao, name: Yinfei Yang
Published: 2024-10-03 17:54:52

Recent advancements in multimodal models highlight the value of rewritten
captions for improving performance, yet key challenges remain. For example,
while synthetic captions often provide superior quality and image-text
alignment, it is not clear whether they can fully replace AltTexts: the role of
synthetic captions and their interaction with original web-crawled AltTexts in
pre-training is still not well understood. Moreover, different multimodal
foundation models may have unique preferences for specific caption formats, but
efforts to identify the optimal captions for each model remain limited. In this
work, we propose a novel, controllable, and scalable captioning pipeline
designed to generate diverse caption formats tailored to various multimodal
models. By examining Short Synthetic Captions (SSC) towards Dense Synthetic
Captions (DSC+) as case studies, we systematically explore their effects and
interactions with AltTexts across models such as CLIP, multimodal LLMs, and
diffusion models. Our findings reveal that a hybrid approach that keeps both
synthetic captions and AltTexts can outperform the use of synthetic captions
alone, improving both alignment and performance, with each model demonstrating
preferences for particular caption formats. This comprehensive analysis
provides valuable insights into optimizing captioning strategies, thereby
advancing the pre-training of multimodal foundation models.
SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference
  Acceleration
Authors: name: Jintao Zhang, name: Jia wei, name: Pengle Zhang, name: Jun Zhu, name: Jianfei Chen
Published: 2024-10-03 10:25:23

The transformer architecture predominates across various models. As the heart
of the transformer, attention has a computational complexity of O(N^2),
compared to O(N) for linear transformations. When handling large sequence
lengths, attention becomes the primary time-consuming component. Although
quantization has proven to be an effective method for accelerating model
inference, existing quantization methods primarily focus on optimizing the
linear layer. In response, we first analyze the feasibility of quantization in
attention detailedly. Following that, we propose SageAttention, a highly
efficient and accurate quantization method for attention. The OPS (operations
per second) of our approach outperforms FlashAttention2 and xformers by about
2.1 times and 2.7 times, respectively. SageAttention also achieves superior
accuracy performance over FlashAttention3. Comprehensive experiments confirm
that our approach incurs almost no end-to-end metrics loss across diverse
models, including those for large language processing, image generation, and
video generation.
Synthio: Augmenting Small-Scale Audio Classification Datasets with
  Synthetic Data
Authors: name: Sreyan Ghosh, name: Sonal Kumar, name: Zhifeng Kong, name: Rafael Valle, name: Bryan Catanzaro, name: Dinesh Manocha
Published: 2024-10-02 22:05:36

We present Synthio, a novel approach for augmenting small-scale audio
classification datasets with synthetic data. Our goal is to improve audio
classification accuracy with limited labeled data. Traditional data
augmentation techniques, which apply artificial transformations (e.g., adding
random noise or masking segments), struggle to create data that captures the
true diversity present in real-world audios. To address this shortcoming, we
propose to augment the dataset with synthetic audio generated from
text-to-audio (T2A) diffusion models. However, synthesizing effective
augmentations is challenging because not only should the generated data be
acoustically consistent with the underlying small-scale dataset, but they
should also have sufficient compositional diversity. To overcome the first
challenge, we align the generations of the T2A model with the small-scale
dataset using preference optimization. This ensures that the acoustic
characteristics of the generated data remain consistent with the small-scale
dataset. To address the second challenge, we propose a novel caption generation
technique that leverages the reasoning capabilities of Large Language Models to
(1) generate diverse and meaningful audio captions and (2) iteratively refine
their quality. The generated captions are then used to prompt the aligned T2A
model. We extensively evaluate Synthio on ten datasets and four simulated
limited-data settings. Results indicate our method consistently outperforms all
baselines by 0.1%-39% using a T2A model trained only on weakly-captioned
AudioSet.
MVGS: Multi-view-regulated Gaussian Splatting for Novel View Synthesis
Authors: name: Xiaobiao Du, name: Yida Wang, name: Xin Yu
Published: 2024-10-02 23:48:31

Recent works in volume rendering, e.g. NeRF and 3D Gaussian
Splatting (3DGS), significantly advance the rendering quality and efficiency
with the help of the learned implicit neural radiance field or 3D Gaussians.
Rendering on top of an explicit representation, the vanilla 3DGS and its
variants deliver real-time efficiency by optimizing the parametric model with
single-view supervision per iteration during training which is adopted from
NeRF. Consequently, certain views are overfitted, leading to unsatisfying
appearance in novel-view synthesis and imprecise 3D geometries. To solve
aforementioned problems, we propose a new 3DGS optimization method embodying
four key novel contributions: 1) We transform the conventional single-view
training paradigm into a multi-view training strategy. With our proposed
multi-view regulation, 3D Gaussian attributes are further optimized without
overfitting certain training views. As a general solution, we improve the
overall accuracy in a variety of scenarios and different Gaussian variants. 2)
Inspired by the benefit introduced by additional views, we further propose a
cross-intrinsic guidance scheme, leading to a coarse-to-fine training procedure
concerning different resolutions. 3) Built on top of our multi-view regulated
training, we further propose a cross-ray densification strategy, densifying
more Gaussian kernels in the ray-intersect regions from a selection of views.
4) By further investigating the densification strategy, we found that the
effect of densification should be enhanced when certain views are distinct
dramatically. As a solution, we propose a novel multi-view augmented
densification strategy, where 3D Gaussians are encouraged to get densified to a
sufficient number accordingly, resulting in improved reconstruction accuracy.
LLaVA-Critic: Learning to Evaluate Multimodal Models
Authors: name: Tianyi Xiong, name: Xiyao Wang, name: Dong Guo, name: Qinghao Ye, name: Haoqi Fan, name: Quanquan Gu, name: Heng Huang, name: Chunyuan Li
Published: 2024-10-03 17:36:33

We introduce LLaVA-Critic, the first open-source large multimodal model (LMM)
designed as a generalist evaluator to assess performance across a wide range of
multimodal tasks. LLaVA-Critic is trained using a high-quality critic
instruction-following dataset that incorporates diverse evaluation criteria and
scenarios. Our experiments demonstrate the model's effectiveness in two key
areas: (1) LMM-as-a-Judge, where LLaVA-Critic provides reliable evaluation
scores, performing on par with or surpassing GPT models on multiple evaluation
benchmarks; and (2) Preference Learning, where it generates reward signals for
preference learning, enhancing model alignment capabilities. This work
underscores the potential of open-source LMMs in self-critique and evaluation,
setting the stage for future research into scalable, superhuman alignment
feedback mechanisms for LMMs.
Video Instruction Tuning With Synthetic Data
Authors: name: Yuanhan Zhang, name: Jinming Wu, name: Wei Li, name: Bo Li, name: Zejun Ma, name: Ziwei Liu, name: Chunyuan Li
Published: 2024-10-03 17:36:49

The development of video large multimodal models (LMMs) has been hindered by
the difficulty of curating large amounts of high-quality raw data from the web.
To address this, we propose an alternative approach by creating a high-quality
synthetic dataset specifically for video instruction-following, namely
LLaVA-Video-178K. This dataset includes key tasks such as detailed captioning,
open-ended question-answering (QA), and multiple-choice QA. By training on this
dataset, in combination with existing visual instruction tuning data, we
introduce LLaVA-Video, a new video LMM. Our experiments demonstrate that
LLaVA-Video achieves strong performance across various video benchmarks,
highlighting the effectiveness of our dataset. We plan to release the dataset,
its generation pipeline, and the model checkpoints.
General Preference Modeling with Preference Representations for Aligning
  Language Models
Authors: name: Yifan Zhang, name: Ge Zhang, name: Yue Wu, name: Kangping Xu, name: Quanquan Gu
Published: 2024-10-03 04:22:55

Modeling human preferences is crucial for aligning foundation models with
human values. Traditional reward modeling methods, such as the Bradley-Terry
(BT) reward model, fall short in expressiveness, particularly in addressing
intransitive preferences. Although supervised pair preference models (PairPM)
can express general preferences, their implementation is highly ad-hoc and
cannot guarantee a consistent preference probability of compared pairs.
Additionally, they impose high computational costs due to their quadratic query
complexity when comparing multiple responses. In this paper, we introduce
preference representation learning, an approach that embeds responses into a
latent space to capture intricate preference structures efficiently, achieving
linear query complexity. Additionally, we propose preference score-based
General Preference Optimization (GPO), which generalizes reward-based
reinforcement learning from human feedback. Experimental results show that our
General Preference representation model (GPM) outperforms the BT reward model
on the RewardBench benchmark with a margin of up to 5.6% and effectively models
cyclic preferences where any BT reward model behaves like a random guess.
Furthermore, evaluations on downstream tasks such as AlpacaEval2.0 and
MT-Bench, following the language model post-training with GPO and our general
preference model, reveal substantial performance improvements with margins up
to 9.3%. These findings indicate that our method may enhance the alignment of
foundation models with nuanced human values. The code is available at
https://github.com/general-preference/general-preference-model.
VLMGuard: Defending VLMs against Malicious Prompts via Unlabeled Data
Authors: name: Xuefeng Du, name: Reshmi Ghosh, name: Robert Sim, name: Ahmed Salem, name: Vitor Carvalho, name: Emily Lawton, name: Yixuan Li, name: Jack W. Stokes
Published: 2024-10-01 00:37:29

Vision-language models (VLMs) are essential for contextual understanding of
both visual and textual information. However, their vulnerability to
adversarially manipulated inputs presents significant risks, leading to
compromised outputs and raising concerns about the reliability in
VLM-integrated applications. Detecting these malicious prompts is thus crucial
for maintaining trust in VLM generations. A major challenge in developing a
safeguarding prompt classifier is the lack of a large amount of labeled benign
and malicious data. To address the issue, we introduce VLMGuard, a novel
learning framework that leverages the unlabeled user prompts in the wild for
malicious prompt detection. These unlabeled prompts, which naturally arise when
VLMs are deployed in the open world, consist of both benign and malicious
information. To harness the unlabeled data, we present an automated
maliciousness estimation score for distinguishing between benign and malicious
samples within this unlabeled mixture, thereby enabling the training of a
binary prompt classifier on top. Notably, our framework does not require extra
human annotations, offering strong flexibility and practicality for real-world
applications. Extensive experiment shows VLMGuard achieves superior detection
results, significantly outperforming state-of-the-art methods. Disclaimer: This
paper may contain offensive examples; reader discretion is advised.
PHI-S: Distribution Balancing for Label-Free Multi-Teacher Distillation
Authors: name: Mike Ranzinger, name: Jon Barker, name: Greg Heinrich, name: Pavlo Molchanov, name: Bryan Catanzaro, name: Andrew Tao
Published: 2024-10-02 15:50:35

Various visual foundation models have distinct strengths and weaknesses, both
of which can be improved through heterogeneous multi-teacher knowledge
distillation without labels, termed "agglomerative models." We build upon this
body of work by studying the effect of the teachers' activation statistics,
particularly the impact of the loss function on the resulting student model
quality. We explore a standard toolkit of statistical normalization techniques
to better align the different distributions and assess their effects. Further,
we examine the impact on downstream teacher-matching metrics, which motivates
the use of Hadamard matrices. With these matrices, we demonstrate useful
properties, showing how they can be used for isotropic standardization, where
each dimension of a multivariate distribution is standardized using the same
scale. We call this technique "PHI Standardization" (PHI-S) and empirically
demonstrate that it produces the best student model across the suite of methods
studied.
EmoKnob: Enhance Voice Cloning with Fine-Grained Emotion Control
Authors: name: Haozhe Chen, name: Run Chen, name: Julia Hirschberg
Published: 2024-10-01 01:29:54

While recent advances in Text-to-Speech (TTS) technology produce natural and
expressive speech, they lack the option for users to select emotion and control
intensity. We propose EmoKnob, a framework that allows fine-grained emotion
control in speech synthesis with few-shot demonstrative samples of arbitrary
emotion. Our framework leverages the expressive speaker representation space
made possible by recent advances in foundation voice cloning models. Based on
the few-shot capability of our emotion control framework, we propose two
methods to apply emotion control on emotions described by open-ended text,
enabling an intuitive interface for controlling a diverse array of nuanced
emotions. To facilitate a more systematic emotional speech synthesis field, we
introduce a set of evaluation metrics designed to rigorously assess the
faithfulness and recognizability of emotion control frameworks. Through
objective and subjective evaluations, we show that our emotion control
framework effectively embeds emotions into speech and surpasses emotion
expressiveness of commercial TTS services.
BordIRlines: A Dataset for Evaluating Cross-lingual Retrieval-Augmented
  Generation
Authors: name: Bryan Li, name: Samar Haider, name: Fiona Luo, name: Adwait Agashe, name: Chris Callison-Burch
Published: 2024-10-02 01:59:07

Large language models excel at creative generation but continue to struggle
with the issues of hallucination and bias. While retrieval-augmented generation
(RAG) provides a framework for grounding LLMs' responses in accurate and
up-to-date information, it still raises the question of bias: which sources
should be selected for inclusion in the context? And how should their
importance be weighted? In this paper, we study the challenge of cross-lingual
RAG and present a dataset to investigate the robustness of existing systems at
answering queries about geopolitical disputes, which exist at the intersection
of linguistic, cultural, and political boundaries. Our dataset is sourced from
Wikipedia pages containing information relevant to the given queries and we
investigate the impact of including additional context, as well as the
composition of this context in terms of language and source, on an LLM's
response. Our results show that existing RAG systems continue to be challenged
by cross-lingual use cases and suffer from a lack of consistency when they are
provided with competing information in multiple languages. We present case
studies to illustrate these issues and outline steps for future research to
address these challenges. We make our dataset and code publicly available at
https://github.com/manestay/bordIRlines.
SonicSim: A customizable simulation platform for speech processing in
  moving sound source scenarios
Authors: name: Kai Li, name: Wendi Sang, name: Chang Zeng, name: Runxuan Yang, name: Guo Chen, name: Xiaolin Hu
Published: 2024-10-02 12:33:59

The systematic evaluation of speech separation and enhancement models under
moving sound source conditions typically requires extensive data comprising
diverse scenarios. However, real-world datasets often contain insufficient data
to meet the training and evaluation requirements of models. Although synthetic
datasets offer a larger volume of data, their acoustic simulations lack
realism. Consequently, neither real-world nor synthetic datasets effectively
fulfill practical needs. To address these issues, we introduce SonicSim, a
synthetic toolkit de-designed to generate highly customizable data for moving
sound sources. SonicSim is developed based on the embodied AI simulation
platform, Habitat-sim, supporting multi-level adjustments, including
scene-level, microphone-level, and source-level, thereby generating more
diverse synthetic data. Leveraging SonicSim, we constructed a moving sound
source benchmark dataset, SonicSet, using the Librispeech, the Freesound
Dataset 50k (FSD50K) and Free Music Archive (FMA), and 90 scenes from the
Matterport3D to evaluate speech separation and enhancement models.
Additionally, to validate the differences between synthetic data and real-world
data, we randomly selected 5 hours of raw data without reverberation from the
SonicSet validation set to record a real-world speech separation dataset, which
was then compared with the corresponding synthetic datasets. Similarly, we
utilized the real-world speech enhancement dataset RealMAN to validate the
acoustic gap between other synthetic datasets and the SonicSet dataset for
speech enhancement. The results indicate that the synthetic data generated by
SonicSim can effectively generalize to real-world scenarios. Demo and code are
publicly available at https://cslikai.cn/SonicSim/.
Old Optimizer, New Norm: An Anthology
Authors: name: Jeremy Bernstein, name: Laker Newhouse
Published: 2024-09-30 14:26:12

Deep learning optimizers are often motivated through a mix of convex and
approximate second-order theory. We select three such methods -- Adam, Shampoo
and Prodigy -- and argue that each method can instead be understood as a
squarely first-order method without convexity assumptions. In fact, after
switching off exponential moving averages, each method is equivalent to
steepest descent under a particular norm. By generalizing this observation, we
chart a new design space for training algorithms. Different operator norms
should be assigned to different tensors based on the role that the tensor plays
within the network. For example, while linear and embedding layers may have the
same weight space of R^{mtimes n}, these layers play different
roles and should be assigned different norms. We hope that this idea of
carefully metrizing the neural architecture might lead to more stable, scalable
and indeed faster training.
FactAlign: Long-form Factuality Alignment of Large Language Models
Authors: name: Chao-Wei Huang, name: Yun-Nung Chen
Published: 2024-10-02 16:03:13

Large language models have demonstrated significant potential as the
next-generation information access engines. However, their reliability is
hindered by issues of hallucination and generating non-factual content. This is
particularly problematic in long-form responses, where assessing and ensuring
factual accuracy is complex. In this paper, we address this gap by proposing
FactAlign, a novel alignment framework designed to enhance the factuality of
LLMs' long-form responses while maintaining their helpfulness. We introduce
fKTO, a fine-grained, sentence-level alignment algorithm that extends the
Kahneman-Tversky Optimization (KTO) alignment method. Leveraging recent
advances in automatic factuality evaluation, FactAlign utilizes fine-grained
factuality assessments to guide the alignment process. Our experiments on
open-domain prompts and information-seeking questions demonstrate that
FactAlign significantly improves the factual accuracy of LLM responses while
also improving their helpfulness. Further analyses identify that FactAlign is
capable of training LLMs to provide more information without losing factual
precision, thus improving the factual F1 score. Our source code, datasets, and
trained models are publicly available at https://github.com/MiuLab/FactAlign
InfiniPot: Infinite Context Processing on Memory-Constrained LLMs
Authors: name: Minsoo Kim, name: Kyuhong Shim, name: Jungwook Choi, name: Simyung Chang
Published: 2024-10-02 13:09:41

Handling long input contexts remains a significant challenge for Large
Language Models (LLMs), particularly in resource-constrained environments such
as mobile devices. Our work aims to address this limitation by introducing
InfiniPot, a novel KV cache control framework designed to enable pre-trained
LLMs to manage extensive sequences within fixed memory constraints efficiently,
without requiring additional training. InfiniPot leverages Continual Context
Distillation (CCD), an iterative process that compresses and retains essential
information through novel importance metrics, effectively maintaining critical
data even without access to future context. Our comprehensive evaluations
indicate that InfiniPot significantly outperforms models trained for long
contexts in various NLP tasks, establishing its efficacy and versatility. This
work represents a substantial advancement toward making LLMs applicable to a
broader range of real-world scenarios.