Orionnnn/hub-stats huggingface dataset:
DistriFusion: Distributed Parallel Inference for High-Resolution
  Diffusion Models
Authors: name: Muyang Li, name: Tianle Cai, name: Jiaxin Cao, name: Qinsheng Zhang, name: Han Cai, name: Junjie Bai, name: Yangqing Jia, name: Ming-Yu Liu, name: Kai Li, name: Song Han
Published: 2024-02-29 18:59:58

Diffusion models have achieved great success in synthesizing high-quality
images. However, generating high-resolution images with diffusion models is
still challenging due to the enormous computational costs, resulting in a
prohibitive latency for interactive applications. In this paper, we propose
DistriFusion to tackle this problem by leveraging parallelism across multiple
GPUs. Our method splits the model input into multiple patches and assigns each
patch to a GPU. However, na\"{\i}vely implementing such an algorithm breaks the
interaction between patches and loses fidelity, while incorporating such an
interaction will incur tremendous communication overhead. To overcome this
dilemma, we observe the high similarity between the input from adjacent
diffusion steps and propose displaced patch parallelism, which takes advantage
of the sequential nature of the diffusion process by reusing the pre-computed
feature maps from the previous timestep to provide context for the current
step. Therefore, our method supports asynchronous communication, which can be
pipelined by computation. Extensive experiments show that our method can be
applied to recent Stable Diffusion XL with no quality degradation and achieve
up to a 6.1times speedup on eight NVIDIA A100s compared to one. Our code is
publicly available at https://github.com/mit-han-lab/distrifuser.
Trajectory Consistency Distillation
Authors: name: Jianbin Zheng, name: Minghui Hu, name: Zhongyi Fan, name: Chaoyue Wang, name: Changxing Ding, name: Dacheng Tao, name: Tat-Jen Cham
Published: 2024-02-29 13:44:14

Latent Consistency Model (LCM) extends the Consistency Model to the latent
space and leverages the guided consistency distillation technique to achieve
impressive performance in accelerating text-to-image synthesis. However, we
observed that LCM struggles to generate images with both clarity and detailed
intricacy. To address this limitation, we initially delve into and elucidate
the underlying causes. Our investigation identifies that the primary issue
stems from errors in three distinct areas. Consequently, we introduce
Trajectory Consistency Distillation (TCD), which encompasses trajectory
consistency function and strategic stochastic sampling. The trajectory
consistency function diminishes the distillation errors by broadening the scope
of the self-consistency boundary condition and endowing the TCD with the
ability to accurately trace the entire trajectory of the Probability Flow ODE.
Additionally, strategic stochastic sampling is specifically designed to
circumvent the accumulated errors inherent in multi-step consistency sampling,
which is meticulously tailored to complement the TCD model. Experiments
demonstrate that TCD not only significantly enhances image quality at low NFEs
but also yields more detailed results compared to the teacher model at high
NFEs.
StarCoder 2 and The Stack v2: The Next Generation
Authors: name: Anton Lozhkov, name: Raymond Li, name: Loubna Ben Allal, name: Federico Cassano, name: Joel Lamy-Poirier, name: Nouamane Tazi, name: Ao Tang, name: Dmytro Pykhtar, name: Jiawei Liu, name: Yuxiang Wei, name: Tianyang Liu, name: Max Tian, name: Denis Kocetkov, name: Arthur Zucker, name: Younes Belkada, name: Zijian Wang, name: Qian Liu, name: Dmitry Abulkhanov, name: Indraneil Paul, name: Zhuang Li, name: Wen-Ding Li, name: Megan Risdal, name: Jia Li, name: Jian Zhu, name: Terry Yue Zhuo, name: Evgenii Zheltonozhskii, name: Nii Osae Osae Dade, name: Wenhao Yu, name: Lucas Krauß, name: Naman Jain, name: Yixuan Su, name: Xuanli He, name: Manan Dey, name: Edoardo Abati, name: Yekun Chai, name: Niklas Muennighoff, name: Xiangru Tang, name: Muhtasham Oblokulov, name: Christopher Akiki, name: Marc Marone, name: Chenghao Mou, name: Mayank Mishra, name: Alex Gu, name: Binyuan Hui, name: Tri Dao, name: Armel Zebaze, name: Olivier Dehaene, name: Nicolas Patry, name: Canwen Xu, name: Julian McAuley, name: Han Hu, name: Torsten Scholak, name: Sebastien Paquet, name: Jennifer Robinson, name: Carolyn Jane Anderson, name: Nicolas Chapados, name: Mostofa Patwary, name: Nima Tajbakhsh, name: Yacine Jernite, name: Carlos Muñoz Ferrandis, name: Lingming Zhang, name: Sean Hughes, name: Thomas Wolf, name: Arjun Guha, name: Leandro von Werra, name: Harm de Vries
Published: 2024-02-29 13:53:35

The BigCode project, an open-scientific collaboration focused on the
responsible development of Large Language Models for Code (Code LLMs),
introduces StarCoder2. In partnership with Software Heritage (SWH), we build
The Stack v2 on top of the digital commons of their source code archive.
Alongside the SWH repositories spanning 619 programming languages, we carefully
select other high-quality data sources, such as GitHub pull requests, Kaggle
notebooks, and code documentation. This results in a training set that is 4x
larger than the first StarCoder dataset. We train StarCoder2 models with 3B,
7B, and 15B parameters on 3.3 to 4.3 trillion tokens and thoroughly evaluate
them on a comprehensive set of Code LLM benchmarks. We find that our small
model, StarCoder2-3B, outperforms other Code LLMs of similar size on most
benchmarks, and also outperforms StarCoderBase-15B. Our large model,
StarCoder2- 15B, significantly outperforms other models of comparable size. In
addition, it matches or outperforms CodeLlama-34B, a model more than twice its
size. Although DeepSeekCoder- 33B is the best-performing model at code
completion for high-resource languages, we find that StarCoder2-15B outperforms
it on math and code reasoning benchmarks, as well as several low-resource
languages. We make the model weights available under an OpenRAIL license and
ensure full transparency regarding the training data by releasing the SoftWare
Heritage persistent IDentifiers (SWHIDs) of the source code data.
Beyond Language Models: Byte Models are Digital World Simulators
Authors: name: Shangda Wu, name: Xu Tan, name: Zili Wang, name: Rui Wang, name: Xiaobing Li, name: Maosong Sun
Published: 2024-02-29 13:38:07

Traditional deep learning often overlooks bytes, the basic units of the
digital world, where all forms of information and operations are encoded and
manipulated in binary format. Inspired by the success of next token prediction
in natural language processing, we introduce bGPT, a model with next byte
prediction to simulate the digital world. bGPT matches specialized models in
performance across various modalities, including text, audio, and images, and
offers new possibilities for predicting, simulating, and diagnosing algorithm
or hardware behaviour. It has almost flawlessly replicated the process of
converting symbolic music data, achieving a low error rate of 0.0011 bits per
byte in converting ABC notation to MIDI format. In addition, bGPT demonstrates
exceptional capabilities in simulating CPU behaviour, with an accuracy
exceeding 99.99% in executing various operations. Leveraging next byte
prediction, models like bGPT can directly learn from vast binary data,
effectively simulating the intricate patterns of the digital world.
Priority Sampling of Large Language Models for Compilers
Authors: name: Dejan Grubisic, name: Chris Cummins, name: Volker Seeker, name: Hugh Leather
Published: 2024-02-28 22:27:49

Large language models show great potential in generating and optimizing code.
Widely used sampling methods such as Nucleus Sampling increase the diversity of
generation but often produce repeated samples for low temperatures and
incoherent samples for high temperatures. Furthermore, the temperature
coefficient has to be tuned for each task, limiting its usability. We present
Priority Sampling, a simple and deterministic sampling technique that produces
unique samples ordered by the model's confidence. Each new sample expands the
unexpanded token with the highest probability in the augmented search tree.
Additionally, Priority Sampling supports generation based on regular expression
that provides a controllable and structured exploration process. Priority
Sampling outperforms Nucleus Sampling for any number of samples, boosting the
performance of the original model from 2.87% to 5% improvement over -Oz.
Moreover, it outperforms the autotuner used for the generation of labels for
the training of the original model in just 30 samples.
Simple linear attention language models balance the recall-throughput
  tradeoff
Authors: name: Simran Arora, name: Sabri Eyuboglu, name: Michael Zhang, name: Aman Timalsina, name: Silas Alberti, name: Dylan Zinsley, name: James Zou, name: Atri Rudra, name: Christopher Ré
Published: 2024-02-28 19:28:27

Recent work has shown that attention-based language models excel at recall,
the ability to ground generations in tokens previously seen in context.
However, the efficiency of attention-based models is bottle-necked during
inference by the KV-cache's aggressive memory consumption. In this work, we
explore whether we can improve language model efficiency (e.g. by reducing
memory consumption) without compromising on recall. By applying experiments and
theory to a broad set of architectures, we identify a key tradeoff between a
model's state size and recall ability. We show that efficient alternatives to
attention (e.g. H3, Mamba, RWKV) maintain a fixed-size recurrent state, but
struggle at recall. We propose BASED a simple architecture combining linear and
sliding window attention. By varying BASED window size and linear attention
feature dimension, we can dial the state size and traverse the pareto frontier
of the recall-memory tradeoff curve, recovering the full quality of attention
on one end and the small state size of attention-alternatives on the other. We
train language models up to 1.3b parameters and show that BASED matches the
strongest sub-quadratic models (e.g. Mamba) in perplexity and outperforms them
on real-world recall-intensive tasks by 6.22 accuracy points. Implementations
of linear attention are often less efficient than optimized standard attention
implementations. To make BASED competitive, we develop IO-aware algorithms that
enable 24x higher throughput on language generation than FlashAttention-2, when
generating 1024 tokens using 1.3b parameter models. Code for this work is
provided at: https://github.com/HazyResearch/based.
Griffin: Mixing Gated Linear Recurrences with Local Attention for
  Efficient Language Models
Authors: name: Soham De, name: Samuel L. Smith, name: Anushan Fernando, name: Aleksandar Botev, name: George Cristian-Muraru, name: Albert Gu, name: Ruba Haroun, name: Leonard Berrada, name: Yutian Chen, name: Srivatsan Srinivasan, name: Guillaume Desjardins, name: Arnaud Doucet, name: David Budden, name: Yee Whye Teh, name: Razvan Pascanu, name: Nando De Freitas, name: Caglar Gulcehre
Published: 2024-02-29 18:24:46

Recurrent neural networks (RNNs) have fast inference and scale efficiently on
long sequences, but they are difficult to train and hard to scale. We propose
Hawk, an RNN with gated linear recurrences, and Griffin, a hybrid model that
mixes gated linear recurrences with local attention. Hawk exceeds the reported
performance of Mamba on downstream tasks, while Griffin matches the performance
of Llama-2 despite being trained on over 6 times fewer tokens. We also show
that Griffin can extrapolate on sequences significantly longer than those seen
during training. Our models match the hardware efficiency of Transformers
during training, and during inference they have lower latency and significantly
higher throughput. We scale Griffin up to 14B parameters, and explain how to
shard our models for efficient distributed training.
ViewFusion: Towards Multi-View Consistency via Interpolated Denoising
Authors: name: Xianghui Yang, name: Yan Zuo, name: Sameera Ramasinghe, name: Loris Bazzani, name: Gil Avraham, name: Anton van den Hengel
Published: 2024-02-29 04:21:38

Novel-view synthesis through diffusion models has demonstrated remarkable
potential for generating diverse and high-quality images. Yet, the independent
process of image generation in these prevailing methods leads to challenges in
maintaining multiple-view consistency. To address this, we introduce
ViewFusion, a novel, training-free algorithm that can be seamlessly integrated
into existing pre-trained diffusion models. Our approach adopts an
auto-regressive method that implicitly leverages previously generated views as
context for the next view generation, ensuring robust multi-view consistency
during the novel-view generation process. Through a diffusion process that
fuses known-view information via interpolated denoising, our framework
successfully extends single-view conditioned models to work in multiple-view
conditional settings without any additional fine-tuning. Extensive experimental
results demonstrate the effectiveness of ViewFusion in generating consistent
and detailed novel views.
Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers
Authors: name: Tsai-Shien Chen, name: Aliaksandr Siarohin, name: Willi Menapace, name: Ekaterina Deyneka, name: Hsiang-wei Chao, name: Byung Eun Jeon, name: Yuwei Fang, name: Hsin-Ying Lee, name: Jian Ren, name: Ming-Hsuan Yang, name: Sergey Tulyakov
Published: 2024-02-29 18:59:50

The quality of the data and annotation upper-bounds the quality of a
downstream model. While there exist large text corpora and image-text pairs,
high-quality video-text data is much harder to collect. First of all, manual
labeling is more time-consuming, as it requires an annotator to watch an entire
video. Second, videos have a temporal dimension, consisting of several scenes
stacked together, and showing multiple actions. Accordingly, to establish a
video dataset with high-quality captions, we propose an automatic approach
leveraging multimodal inputs, such as textual video description, subtitles, and
individual video frames. Specifically, we curate 3.8M high-resolution videos
from the publicly available HD-VILA-100M dataset. We then split them into
semantically consistent video clips, and apply multiple cross-modality teacher
models to obtain captions for each video. Next, we finetune a retrieval model
on a small subset where the best caption of each video is manually selected and
then employ the model in the whole dataset to select the best caption as the
annotation. In this way, we get 70M videos paired with high-quality text
captions. We dub the dataset as Panda-70M. We show the value of the proposed
dataset on three downstream tasks: video captioning, video and text retrieval,
and text-driven video generation. The models trained on the proposed data score
substantially better on the majority of metrics across all the tasks.
DiffuseKronA: A Parameter Efficient Fine-tuning Method for Personalized
  Diffusion Model
Authors: name: Shyam Marjit, name: Harshit Singh, name: Nityanand Mathur, name: Sayak Paul, name: Chia-Mu Yu, name: Pin-Yu Chen
Published: 2024-02-27 11:05:34

In the realm of subject-driven text-to-image (T2I) generative models, recent
developments like DreamBooth and BLIP-Diffusion have led to impressive results
yet encounter limitations due to their intensive fine-tuning demands and
substantial parameter requirements. While the low-rank adaptation (LoRA) module
within DreamBooth offers a reduction in trainable parameters, it introduces a
pronounced sensitivity to hyperparameters, leading to a compromise between
parameter efficiency and the quality of T2I personalized image synthesis.
Addressing these constraints, we introduce \textit{DiffuseKronA}, a
novel Kronecker product-based adaptation module that not only significantly
reduces the parameter count by 35\% and 99.947\% compared to LoRA-DreamBooth
and the original DreamBooth, respectively, but also enhances the quality of
image synthesis. Crucially, DiffuseKronA mitigates the issue of
hyperparameter sensitivity, delivering consistent high-quality generations
across a wide range of hyperparameters, thereby diminishing the necessity for
extensive fine-tuning. Furthermore, a more controllable decomposition makes
DiffuseKronA more interpretable and even can achieve up to a 50\%
reduction with results comparable to LoRA-Dreambooth. Evaluated against diverse
and complex input images and text prompts, DiffuseKronA consistently
outperforms existing models, producing diverse images of higher quality with
improved fidelity and a more accurate color distribution of objects, all the
while upholding exceptional parameter efficiency, thus presenting a substantial
advancement in the field of T2I generative modeling. Our project page,
consisting of links to the code, and pre-trained checkpoints, is available at
https://diffusekrona.github.io/{https://diffusekrona.github.io/}.
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
Authors: name: Shuming Ma, name: Hongyu Wang, name: Lingxiao Ma, name: Lei Wang, name: Wenhui Wang, name: Shaohan Huang, name: Li Dong, name: Ruiping Wang, name: Jilong Xue, name: Furu Wei
Published: 2024-02-27 18:56:19

Recent research, such as BitNet, is paving the way for a new era of 1-bit
Large Language Models (LLMs). In this work, we introduce a 1-bit LLM variant,
namely BitNet b1.58, in which every single parameter (or weight) of the LLM is
ternary {-1, 0, 1}. It matches the full-precision (i.e., FP16 or BF16)
Transformer LLM with the same model size and training tokens in terms of both
perplexity and end-task performance, while being significantly more
cost-effective in terms of latency, memory, throughput, and energy consumption.
More profoundly, the 1.58-bit LLM defines a new scaling law and recipe for
training new generations of LLMs that are both high-performance and
cost-effective. Furthermore, it enables a new computation paradigm and opens
the door for designing specific hardware optimized for 1-bit LLMs.
Towards Optimal Learning of Language Models
Authors: name: Yuxian Gu, name: Li Dong, name: Yaru Hao, name: Qingxiu Dong, name: Minlie Huang, name: Furu Wei
Published: 2024-02-27 18:52:19

This work studies the general principles of improving the learning of
language models (LMs), which aims at reducing the necessary training steps for
achieving superior performance. Specifically, we present a theory for the
optimal learning of LMs. We first propose an objective that optimizes LM
learning by maximizing the data compression ratio in an
"LM-training-as-lossless-compression" view. Then, we derive a theorem, named
Learning Law, to reveal the properties of the dynamics in the optimal learning
process under our objective. The theorem is then validated by experiments on a
linear classification and a real-world language modeling task. Finally, we
empirically verify that the optimal learning of LMs essentially stems from the
improvement of the coefficients in the scaling law of LMs, indicating great
promise and significance for designing practical learning acceleration methods.
Our code can be found at https://aka.ms/LearningLaw.
Evaluating Very Long-Term Conversational Memory of LLM Agents
Authors: name: Adyasha Maharana, name: Dong-Ho Lee, name: Sergey Tulyakov, name: Mohit Bansal, name: Francesco Barbieri, name: Yuwei Fang
Published: 2024-02-27 18:42:31

Existing works on long-term open-domain dialogues focus on evaluating model
responses within contexts spanning no more than five chat sessions. Despite
advancements in long-context large language models (LLMs) and retrieval
augmented generation (RAG) techniques, their efficacy in very long-term
dialogues remains unexplored. To address this research gap, we introduce a
machine-human pipeline to generate high-quality, very long-term dialogues by
leveraging LLM-based agent architectures and grounding their dialogues on
personas and temporal event graphs. Moreover, we equip each agent with the
capability of sharing and reacting to images. The generated conversations are
verified and edited by human annotators for long-range consistency and
grounding to the event graphs. Using this pipeline, we collect LoCoMo, a
dataset of very long-term conversations, each encompassing 300 turns and 9K
tokens on avg., over up to 35 sessions. Based on LoCoMo, we present a
comprehensive evaluation benchmark to measure long-term memory in models,
encompassing question answering, event summarization, and multi-modal dialogue
generation tasks. Our experimental results indicate that LLMs exhibit
challenges in understanding lengthy conversations and comprehending long-range
temporal and causal dynamics within dialogues. Employing strategies like
long-context LLMs or RAG can offer improvements but these models still
substantially lag behind human performance.
Training-Free Long-Context Scaling of Large Language Models
Authors: name: Chenxin An, name: Fei Huang, name: Jun Zhang, name: Shansan Gong, name: Xipeng Qiu, name: Chang Zhou, name: Lingpeng Kong
Published: 2024-02-27 12:39:23

The ability of Large Language Models (LLMs) to process and generate coherent
text is markedly weakened when the number of input tokens exceeds their
pretraining length. Given the expensive overhead of finetuning large-scale
models with longer sequences, we propose Dual Chunk Attention (DCA), which
enables Llama2 70B to support context windows of more than 100k tokens without
continual training. By decomposing the attention computation for long sequences
into chunk-based modules, DCA manages to effectively capture the relative
positional information of tokens within the same chunk (Intra-Chunk) and across
distinct chunks (Inter-Chunk), as well as integrates seamlessly with Flash
Attention. In addition to its impressive extrapolation capability, DCA achieves
performance on practical long-context tasks that is comparable to or even
better than that of finetuned models. When compared with proprietary models,
our training-free 70B model attains 94% of the performance of gpt-3.5-16k,
indicating it is a viable open-source alternative. All code and data used in
this work are released at https://github.com/HKUNLP/ChunkLlama.
When Scaling Meets LLM Finetuning: The Effect of Data, Model and
  Finetuning Method
Authors: name: Biao Zhang, name: Zhongtao Liu, name: Colin Cherry, name: Orhan Firat
Published: 2024-02-27 04:18:49

While large language models (LLMs) often adopt finetuning to unlock their
capabilities for downstream applications, our understanding on the inductive
biases (especially the scaling properties) of different finetuning methods is
still limited. To fill this gap, we conduct systematic experiments studying
whether and how different scaling factors, including LLM model size,
pretraining data size, new finetuning parameter size and finetuning data size,
affect the finetuning performance. We consider two types of finetuning --
full-model tuning (FMT) and parameter efficient tuning (PET, including prompt
tuning and LoRA), and explore their scaling behaviors in the data-limited
regime where the LLM model size substantially outweighs the finetuning data
size. Based on two sets of pretrained bilingual LLMs from 1B to 16B and
experiments on bilingual machine translation and multilingual summarization
benchmarks, we find that 1) LLM finetuning follows a powerbased multiplicative
joint scaling law between finetuning data size and each other scaling factor;
2) LLM finetuning benefits more from LLM model scaling than pretraining data
scaling, and PET parameter scaling is generally ineffective; and 3) the optimal
finetuning method is highly task- and finetuning data-dependent. We hope our
findings could shed light on understanding, selecting and developing LLM
finetuning methods.