Orionnnn/hub-stats huggingface dataset:
Q-Refine: A Perceptual Quality Refiner for AI-Generated Image
Authors: name: Chunyi Li, name: Haoning Wu, name: Zicheng Zhang, name: Hongkun Hao, name: Kaiwei Zhang, name: Lei Bai, name: Xiaohong Liu, name: Xiongkuo Min, name: Weisi Lin, name: Guangtao Zhai
Published: 2024-01-02 09:11:23

With the rapid evolution of the Text-to-Image (T2I) model in recent years,
their unsatisfactory generation result has become a challenge. However,
uniformly refining AI-Generated Images (AIGIs) of different qualities not only
limited optimization capabilities for low-quality AIGIs but also brought
negative optimization to high-quality AIGIs. To address this issue, a
quality-award refiner named Q-Refine is proposed. Based on the preference of
the Human Visual System (HVS), Q-Refine uses the Image Quality Assessment (IQA)
metric to guide the refining process for the first time, and modify images of
different qualities through three adaptive pipelines. Experimental shows that
for mainstream T2I models, Q-Refine can perform effective optimization to AIGIs
of different qualities. It can be a general refiner to optimize AIGIs from both
fidelity and aesthetic quality levels, thus expanding the application of the
T2I generation models.
LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning
Authors: name: Hongye Jin, name: Xiaotian Han, name: Jingfeng Yang, name: Zhimeng Jiang, name: Zirui Liu, name: Chia-Yuan Chang, name: Huiyuan Chen, name: Xia Hu
Published: 2024-01-02 18:30:51

This work elicits LLMs' inherent ability to handle long contexts without
fine-tuning. The limited length of the training sequence during training may
limit the application of Large Language Models (LLMs) on long input sequences
for inference. In this work, we argue that existing LLMs themselves have
inherent capabilities for handling long contexts. Based on this argument, we
suggest extending LLMs' context window by themselves to fully utilize the
inherent ability.We propose Self-Extend to stimulate LLMs' long context
handling potential. The basic idea is to construct bi-level attention
information: the group level and the neighbor level. The two levels are
computed by the original model's self-attention, which means the proposed does
not require any training. With only four lines of code modification, the
proposed method can effortlessly extend existing LLMs' context window without
any fine-tuning. We conduct comprehensive experiments and the results show that
the proposed method can effectively extend existing LLMs' context window's
length.
DocLLM: A layout-aware generative language model for multimodal document
  understanding
Authors: name: Dongsheng Wang, name: Natraj Raman, name: Mathieu Sibue, name: Zhiqiang Ma, name: Petr Babkin, name: Simerjot Kaur, name: Yulong Pei, name: Armineh Nourbakhsh, name: Xiaomo Liu
Published: 2023-12-31 22:37:52

Enterprise documents such as forms, invoices, receipts, reports, contracts,
and other similar records, often carry rich semantics at the intersection of
textual and spatial modalities. The visual cues offered by their complex
layouts play a crucial role in comprehending these documents effectively. In
this paper, we present DocLLM, a lightweight extension to traditional large
language models (LLMs) for reasoning over visual documents, taking into account
both textual semantics and spatial layout. Our model differs from existing
multimodal LLMs by avoiding expensive image encoders and focuses exclusively on
bounding box information to incorporate the spatial layout structure.
Specifically, the cross-alignment between text and spatial modalities is
captured by decomposing the attention mechanism in classical transformers to a
set of disentangled matrices. Furthermore, we devise a pre-training objective
that learns to infill text segments. This approach allows us to address
irregular layouts and heterogeneous content frequently encountered in visual
documents. The pre-trained model is fine-tuned using a large-scale instruction
dataset, covering four core document intelligence tasks. We demonstrate that
our solution outperforms SotA LLMs on 14 out of 16 datasets across all tasks,
and generalizes well to 4 out of 5 previously unseen datasets.
VideoDrafter: Content-Consistent Multi-Scene Video Generation with LLM
Authors: name: Fuchen Long, name: Zhaofan Qiu, name: Ting Yao, name: Tao Mei
Published: 2024-01-02 15:56:48

The recent innovations and breakthroughs in diffusion models have
significantly expanded the possibilities of generating high-quality videos for
the given prompts. Most existing works tackle the single-scene scenario with
only one video event occurring in a single background. Extending to generate
multi-scene videos nevertheless is not trivial and necessitates to nicely
manage the logic in between while preserving the consistent visual appearance
of key content across video scenes. In this paper, we propose a novel
framework, namely VideoDrafter, for content-consistent multi-scene video
generation. Technically, VideoDrafter leverages Large Language Models (LLM) to
convert the input prompt into comprehensive multi-scene script that benefits
from the logical knowledge learnt by LLM. The script for each scene includes a
prompt describing the event, the foreground/background entities, as well as
camera movement. VideoDrafter identifies the common entities throughout the
script and asks LLM to detail each entity. The resultant entity description is
then fed into a text-to-image model to generate a reference image for each
entity. Finally, VideoDrafter outputs a multi-scene video by generating each
scene video via a diffusion process that takes the reference images, the
descriptive prompt of the event and camera movement into account. The diffusion
model incorporates the reference images as the condition and alignment to
strengthen the content consistency of multi-scene videos. Extensive experiments
demonstrate that VideoDrafter outperforms the SOTA video generation models in
terms of visual quality, content consistency, and user preference.
TrailBlazer: Trajectory Control for Diffusion-Based Video Generation
Authors: name: Wan-Duo Kurt Ma, name: J. P. Lewis, name: W. Bastiaan Kleijn
Published: 2023-12-31 10:51:52

Within recent approaches to text-to-video (T2V) generation, achieving
controllability in the synthesized video is often a challenge. Typically, this
issue is addressed by providing low-level per-frame guidance in the form of
edge maps, depth maps, or an existing video to be altered. However, the process
of obtaining such guidance can be labor-intensive. This paper focuses on
enhancing controllability in video synthesis by employing straightforward
bounding boxes to guide the subject in various ways, all without the need for
neural network training, finetuning, optimization at inference time, or the use
of pre-existing videos. Our algorithm, TrailBlazer, is constructed upon a
pre-trained (T2V) model, and easy to implement. The subject is directed by a
bounding box through the proposed spatial and temporal attention map editing.
Moreover, we introduce the concept of keyframing, allowing the subject
trajectory and overall appearance to be guided by both a moving bounding box
and corresponding prompts, without the need to provide a detailed mask. The
method is efficient, with negligible additional computation relative to the
underlying pre-trained model. Despite the simplicity of the bounding box
guidance, the resulting motion is surprisingly natural, with emergent effects
including perspective and movement toward the virtual camera as the box size
increases.
SteinDreamer: Variance Reduction for Text-to-3D Score Distillation via
  Stein Identity
Authors: name: Peihao Wang, name: Zhiwen Fan, name: Dejia Xu, name: Dilin Wang, name: Sreyas Mohan, name: Forrest Iandola, name: Rakesh Ranjan, name: Yilei Li, name: Qiang Liu, name: Zhangyang Wang, name: Vikas Chandra
Published: 2023-12-31 23:04:25

Score distillation has emerged as one of the most prevalent approaches for
text-to-3D asset synthesis. Essentially, score distillation updates 3D
parameters by lifting and back-propagating scores averaged over different
views. In this paper, we reveal that the gradient estimation in score
distillation is inherent to high variance. Through the lens of variance
reduction, the effectiveness of SDS and VSD can be interpreted as applications
of various control variates to the Monte Carlo estimator of the distilled
score. Motivated by this rethinking and based on Stein's identity, we propose a
more general solution to reduce variance for score distillation, termed Stein
Score Distillation (SSD). SSD incorporates control variates constructed by
Stein identity, allowing for arbitrary baseline functions. This enables us to
include flexible guidance priors and network architectures to explicitly
optimize for variance reduction. In our experiments, the overall pipeline,
dubbed SteinDreamer, is implemented by instantiating the control variate with a
monocular depth estimator. The results suggest that SSD can effectively reduce
the distillation variance and consistently improve visual quality for both
object- and scene-level generation. Moreover, we demonstrate that SteinDreamer
achieves faster convergence than existing methods due to more stable gradient
updates.
Boosting Large Language Model for Speech Synthesis: An Empirical Study
Authors: name: Hongkun Hao, name: Long Zhou, name: Shujie Liu, name: Jinyu Li, name: Shujie Hu, name: Rui Wang, name: Furu Wei
Published: 2023-12-30 14:20:04

Large language models (LLMs) have made significant advancements in natural
language processing and are concurrently extending the language ability to
other modalities, such as speech and vision. Nevertheless, most of the previous
work focuses on prompting LLMs with perception abilities like auditory
comprehension, and the effective approach for augmenting LLMs with speech
synthesis capabilities remains ambiguous. In this paper, we conduct a
comprehensive empirical exploration of boosting LLMs with the ability to
generate speech, by combining pre-trained LLM LLaMA/OPT and text-to-speech
synthesis model VALL-E. We compare three integration methods between LLMs and
speech synthesis models, including directly fine-tuned LLMs, superposed layers
of LLMs and VALL-E, and coupled LLMs and VALL-E using LLMs as a powerful text
encoder. Experimental results show that, using LoRA method to fine-tune LLMs
directly to boost the speech synthesis capability does not work well, and
superposed LLMs and VALL-E can improve the quality of generated speech both in
speaker similarity and word error rate (WER). Among these three methods,
coupled methods leveraging LLMs as the text encoder can achieve the best
performance, making it outperform original speech synthesis models with a
consistently better speaker similarity and a significant (10.9%) WER reduction.
COSMO: COntrastive Streamlined MultimOdal Model with Interleaved
  Pre-Training
Authors: name: Alex Jinpeng Wang, name: Linjie Li, name: Kevin Qinghong Lin, name: Jianfeng Wang, name: Kevin Lin, name: Zhengyuan Yang, name: Lijuan Wang, name: Mike Zheng Shou
Published: 2024-01-01 18:58:42

In the evolution of Vision-Language Pre-training, shifting from short-text
comprehension to encompassing extended textual contexts is pivotal. Recent
autoregressive vision-language models like flamingo, palme, leveraging
the long-context capability of Large Language Models, have excelled in few-shot
text generation tasks but face challenges in alignment tasks. Addressing this
gap, we introduce the contrastive loss into text generation models, presenting
the COntrastive-Streamlined MultimOdal framework (\ModelName), strategically
partitioning the language model into dedicated unimodal text processing and
adept multimodal data handling components. \ModelName, our unified framework,
merges unimodal and multimodal elements, enhancing model performance for tasks
involving textual and visual data while notably reducing learnable parameters.
However, these models demand extensive long-text datasets, yet the availability
of high-quality long-text video datasets remains limited. To bridge this gap,
this work introduces \VideoDatasetName, an inaugural interleaved video-text
dataset featuring comprehensive captions, marking a significant step forward.
Demonstrating its impact, we illustrate how  enhances model
performance in image-text tasks. With 34% learnable parameters and utilizing
72\% of the available data, our model demonstrates significant superiority over
OpenFlamingo~openflamingo. For instance, in the 4-shot flickr captioning
task, performance notably improves from 57.2% to 65.\%. The contributions of
 and  are underscored by notable performance
gains across 14 diverse downstream datasets encompassing both image-text and
video-text tasks.
Astraios: Parameter-Efficient Instruction Tuning Code Large Language
  Models
Authors: name: Terry Yue Zhuo, name: Armel Zebaze, name: Nitchakarn Suppattarachai, name: Leandro von Werra, name: Harm de Vries, name: Qian Liu, name: Niklas Muennighoff
Published: 2024-01-01 15:30:19

The high cost of full-parameter fine-tuning (FFT) of Large Language Models
(LLMs) has led to a series of parameter-efficient fine-tuning (PEFT) methods.
However, it remains unclear which methods provide the best cost-performance
trade-off at different model scales. We introduce Astraios, a suite of 28
instruction-tuned OctoCoder models using 7 tuning methods and 4 model sizes up
to 16 billion parameters. Through investigations across 5 tasks and 8 different
datasets encompassing both code comprehension and code generation tasks, we
find that FFT generally leads to the best downstream performance across all
scales, and PEFT methods differ significantly in their efficacy based on the
model scale. LoRA usually offers the most favorable trade-off between cost and
performance. Further investigation into the effects of these methods on both
model robustness and code security reveals that larger models tend to
demonstrate reduced robustness and less security. At last, we explore the
relationships among updated parameters, cross-entropy loss, and task
performance. We find that the tuning effectiveness observed in small models
generalizes well to larger models, and the validation loss in instruction
tuning can be a reliable indicator of overall downstream performance.
Beyond Chinchilla-Optimal: Accounting for Inference in Language Model
  Scaling Laws
Authors: name: Nikhil Sardana, name: Jonathan Frankle
Published: 2023-12-31 10:53:58

Large language model (LLM) scaling laws are empirical formulas that estimate
changes in model quality as a result of increasing parameter count and training
data. However, these formulas, including the popular DeepMind Chinchilla
scaling laws, neglect to include the cost of inference. We modify the
Chinchilla scaling laws to calculate the optimal LLM parameter count and
pre-training data size to train and deploy a model of a given quality and
inference demand. We conduct our analysis both in terms of a compute budget and
real-world costs and find that LLM researchers expecting reasonably large
inference demand (~1B requests) should train models smaller and longer than
Chinchilla-optimal.
GeoGalactica: A Scientific Large Language Model in Geoscience
Authors: name: Zhouhan Lin, name: Cheng Deng, name: Le Zhou, name: Tianhang Zhang, name: Yi Xu, name: Yutong Xu, name: Zhongmou He, name: Yuanyuan Shi, name: Beiya Dai, name: Yunchong Song, name: Boyi Zeng, name: Qiyuan Chen, name: Tao Shi, name: Tianyu Huang, name: Yiwei Xu, name: Shu Wang, name: Luoyi Fu, name: Weinan Zhang, name: Junxian He, name: Chao Ma, name: Yunqiang Zhu, name: Xinbing Wang, name: Chenghu Zhou
Published: 2023-12-31 09:22:54

Large language models (LLMs) have achieved huge success for their general
knowledge and ability to solve a wide spectrum of tasks in natural language
processing (NLP). Due to their impressive abilities, LLMs have shed light on
potential inter-discipline applications to foster scientific discoveries of a
specific domain by using artificial intelligence (AI for science, AI4S). In the
meantime, utilizing NLP techniques in geoscience research and practice is wide
and convoluted, contributing from knowledge extraction and document
classification to question answering and knowledge discovery. In this work, we
take the initial step to leverage LLM for science, through a rather
straightforward approach. We try to specialize an LLM into geoscience, by
further pre-training the model with a vast amount of texts in geoscience, as
well as supervised fine-tuning (SFT) the resulting model with our custom
collected instruction tuning dataset. These efforts result in a model
GeoGalactica consisting of 30 billion parameters. To our best knowledge, it is
the largest language model for the geoscience domain. More specifically,
GeoGalactica is from further pre-training of Galactica. We train GeoGalactica
over a geoscience-related text corpus containing 65 billion tokens curated from
extensive data sources in the big science project Deep-time Digital Earth
(DDE), preserving as the largest geoscience-specific text corpus. Then we
fine-tune the model with 1 million pairs of instruction-tuning data consisting
of questions that demand professional geoscience knowledge to answer. In this
technical report, we will illustrate in detail all aspects of GeoGalactica,
including data collection, data cleaning, base model selection, pre-training,
SFT, and evaluation. We open-source our data curation tools and the checkpoints
of GeoGalactica during the first 3/4 of pre-training.
Improving Text Embeddings with Large Language Models
Authors: name: Liang Wang, name: Nan Yang, name: Xiaolong Huang, name: Linjun Yang, name: Rangan Majumder, name: Furu Wei
Published: 2023-12-31 02:13:18

In this paper, we introduce a novel and simple method for obtaining
high-quality text embeddings using only synthetic data and less than 1k
training steps. Unlike existing methods that often depend on multi-stage
intermediate pre-training with billions of weakly-supervised text pairs,
followed by fine-tuning with a few labeled datasets, our method does not
require building complex training pipelines or relying on manually collected
datasets that are often constrained by task diversity and language coverage. We
leverage proprietary LLMs to generate diverse synthetic data for hundreds of
thousands of text embedding tasks across nearly 100 languages. We then
fine-tune open-source decoder-only LLMs on the synthetic data using standard
contrastive loss. Experiments demonstrate that our method achieves strong
performance on highly competitive text embedding benchmarks without using any
labeled data. Furthermore, when fine-tuned with a mixture of synthetic and
labeled data, our model sets new state-of-the-art results on the BEIR and MTEB
benchmarks.
Unicron: Economizing Self-Healing LLM Training at Scale
Authors: name: Tao He, name: Xue Li, name: Zhibin Wang, name: Kun Qian, name: Jingbo Xu, name: Wenyuan Yu, name: Jingren Zhou
Published: 2023-12-30 04:06:16

Training large-scale language models is increasingly critical in various
domains, but it is hindered by frequent failures, leading to significant time
and economic costs. Current failure recovery methods in cloud-based settings
inadequately address the diverse and complex scenarios that arise, focusing
narrowly on erasing downtime for individual tasks without considering the
overall cost impact on a cluster. We introduce Unicron, a workload manager
designed for efficient self-healing in large-scale language model training.
Unicron optimizes the training process by minimizing failure-related costs
across multiple concurrent tasks within a cluster. Its key features include
in-band error detection for real-time error identification without extra
overhead, a dynamic cost-aware plan generation mechanism for optimal
reconfiguration, and an efficient transition strategy to reduce downtime during
state changes. Deployed on a 128-GPU distributed cluster, Unicron demonstrates
up to a 1.9x improvement in training efficiency over state-of-the-art methods,
significantly reducing failure recovery costs and enhancing the reliability of
large-scale language model training.
Learning Vision from Models Rivals Learning Vision from Data
Authors: name: Yonglong Tian, name: Lijie Fan, name: Kaifeng Chen, name: Dina Katabi, name: Dilip Krishnan, name: Phillip Isola
Published: 2023-12-28 18:59:55

We introduce SynCLR, a novel approach for learning visual representations
exclusively from synthetic images and synthetic captions, without any real
data. We synthesize a large dataset of image captions using LLMs, then use an
off-the-shelf text-to-image model to generate multiple images corresponding to
each synthetic caption. We perform visual representation learning on these
synthetic images via contrastive learning, treating images sharing the same
caption as positive pairs. The resulting representations transfer well to many
downstream tasks, competing favorably with other general-purpose visual
representation learners such as CLIP and DINO v2 in image classification tasks.
Furthermore, in dense prediction tasks such as semantic segmentation, SynCLR
outperforms previous self-supervised methods by a significant margin, e.g.,
improving over MAE and iBOT by 6.2 and 4.3 mIoU on ADE20k for ViT-B/16.
FlowVid: Taming Imperfect Optical Flows for Consistent Video-to-Video
  Synthesis
Authors: name: Feng Liang, name: Bichen Wu, name: Jialiang Wang, name: Licheng Yu, name: Kunpeng Li, name: Yinan Zhao, name: Ishan Misra, name: Jia-Bin Huang, name: Peizhao Zhang, name: Peter Vajda, name: Diana Marculescu
Published: 2023-12-29 16:57:12

Diffusion models have transformed the image-to-image (I2I) synthesis and are
now permeating into videos. However, the advancement of video-to-video (V2V)
synthesis has been hampered by the challenge of maintaining temporal
consistency across video frames. This paper proposes a consistent V2V synthesis
framework by jointly leveraging spatial conditions and temporal optical flow
clues within the source video. Contrary to prior methods that strictly adhere
to optical flow, our approach harnesses its benefits while handling the
imperfection in flow estimation. We encode the optical flow via warping from
the first frame and serve it as a supplementary reference in the diffusion
model. This enables our model for video synthesis by editing the first frame
with any prevalent I2I models and then propagating edits to successive frames.
Our V2V model, FlowVid, demonstrates remarkable properties: (1) Flexibility:
FlowVid works seamlessly with existing I2I models, facilitating various
modifications, including stylization, object swaps, and local edits. (2)
Efficiency: Generation of a 4-second video with 30 FPS and 512x512 resolution
takes only 1.5 minutes, which is 3.1x, 7.2x, and 10.5x faster than CoDeF,
Rerender, and TokenFlow, respectively. (3) High-quality: In user studies, our
FlowVid is preferred 45.7% of the time, outperforming CoDeF (3.5%), Rerender
(10.2%), and TokenFlow (40.4%).