Orionnnn/hub-stats huggingface dataset:
Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time
Authors: name: Zichang Liu, name: Jue Wang, name: Tri Dao, name: Tianyi Zhou, name: Binhang Yuan, name: Zhao Song, name: Anshumali Shrivastava, name: Ce Zhang, name: Yuandong Tian, name: Christopher Re, name: Beidi Chen
Published: 2023-10-26 05:01:09

Large language models (LLMs) with hundreds of billions of parameters have
sparked a new wave of exciting AI applications. However, they are
computationally expensive at inference time. Sparsity is a natural approach to
reduce this cost, but existing methods either require costly retraining, have
to forgo LLM's in-context learning ability, or do not yield wall-clock time
speedup on modern hardware. We hypothesize that contextual sparsity, which are
small, input-dependent sets of attention heads and MLP parameters that yield
approximately the same output as the dense model for a given input, can address
these issues. We show that contextual sparsity exists, that it can be
accurately predicted, and that we can exploit it to speed up LLM inference in
wall-clock time without compromising LLM's quality or in-context learning
ability. Based on these insights, we propose DejaVu, a system that uses a
low-cost algorithm to predict contextual sparsity on the fly given inputs to
each layer, along with an asynchronous and hardware-aware implementation that
speeds up LLM inference. We validate that DejaVu can reduce the inference
latency of OPT-175B by over 2X compared to the state-of-the-art
FasterTransformer, and over 6X compared to the widely used Hugging Face
implementation, without compromising model quality. The code is available at
https://github.com/FMInference/DejaVu.
Controlled Decoding from Language Models
Authors: name: Sidharth Mudgal, name: Jong Lee, name: Harish Ganapathy, name: YaGuang Li, name: Tao Wang, name: Yanping Huang, name: Zhifeng Chen, name: Heng-Tze Cheng, name: Michael Collins, name: Trevor Strohman, name: Jilin Chen, name: Alex Beutel, name: Ahmad Beirami
Published: 2023-10-25 22:00:05

We propose controlled decoding (CD), a novel off-policy reinforcement
learning method to control the autoregressive generation from language models
towards high reward outcomes. CD solves an off-policy reinforcement learning
problem through a value function for the reward, which we call a prefix scorer.
The prefix scorer is used at inference time to steer the generation towards
higher reward outcomes. We show that the prefix scorer may be trained on
(possibly) off-policy data to predict the expected reward when decoding is
continued from a partially decoded response. We empirically demonstrate that CD
is effective as a control mechanism on Reddit conversations corpus. We also
show that the modularity of the design of CD makes it possible to control for
multiple rewards, effectively solving a multi-objective reinforcement learning
problem with no additional complexity. Finally, we show that CD can be applied
in a novel blockwise fashion at inference-time, again without the need for any
training-time changes, essentially bridging the gap between the popular
best-of-K strategy and token-level reinforcement learning. This makes CD a
promising approach for alignment of language models.
JudgeLM: Fine-tuned Large Language Models are Scalable Judges
Authors: name: Lianghui Zhu, name: Xinggang Wang, name: Xinlong Wang
Published: 2023-10-26 17:48:58

Evaluating Large Language Models (LLMs) in open-ended scenarios is
challenging because existing benchmarks and metrics can not measure them
comprehensively. To address this problem, we propose to fine-tune LLMs as
scalable judges (JudgeLM) to evaluate LLMs efficiently and effectively in
open-ended benchmarks. We first propose a comprehensive, large-scale,
high-quality dataset containing task seeds, LLMs-generated answers, and
GPT-4-generated judgments for fine-tuning high-performance judges, as well as a
new benchmark for evaluating the judges. We train JudgeLM at different scales
from 7B, 13B, to 33B parameters, and conduct a systematic analysis of its
capabilities and behaviors. We then analyze the key biases in fine-tuning LLM
as a judge and consider them as position bias, knowledge bias, and format bias.
To address these issues, JudgeLM introduces a bag of techniques including swap
augmentation, reference support, and reference drop, which clearly enhance the
judge's performance. JudgeLM obtains the state-of-the-art judge performance on
both the existing PandaLM benchmark and our proposed new benchmark. Our JudgeLM
is efficient and the JudgeLM-7B only needs 3 minutes to judge 5K samples with 8
A100 GPUs. JudgeLM obtains high agreement with the teacher judge, achieving an
agreement exceeding 90% that even surpasses human-to-human agreement. JudgeLM
also demonstrates extended capabilities in being judges of the single answer,
multimodal models, multiple answers, and multi-turn chat.
TD-MPC2: Scalable, Robust World Models for Continuous Control
Authors: name: Nicklas Hansen, name: Hao Su, name: Xiaolong Wang
Published: 2023-10-25 17:57:07

TD-MPC is a model-based reinforcement learning (RL) algorithm that performs
local trajectory optimization in the latent space of a learned implicit
(decoder-free) world model. In this work, we present TD-MPC2: a series of
improvements upon the TD-MPC algorithm. We demonstrate that TD-MPC2 improves
significantly over baselines across 104 online RL tasks spanning 4 diverse task
domains, achieving consistently strong results with a single set of
hyperparameters. We further show that agent capabilities increase with model
and data size, and successfully train a single 317M parameter agent to perform
80 tasks across multiple task domains, embodiments, and action spaces. We
conclude with an account of lessons, opportunities, and risks associated with
large TD-MPC2 agents. Explore videos, models, data, code, and more at
https://nicklashansen.github.io/td-mpc2
CLEX: Continuous Length Extrapolation for Large Language Models
Authors: name: Guanzheng Chen, name: Xin Li, name: Zaiqiao Meng, name: Shangsong Liang, name: Lidong Bing
Published: 2023-10-25 08:13:02

Transformer-based Large Language Models (LLMs) are pioneering advances in
many natural language processing tasks, however, their exceptional capabilities
are restricted within the preset context window of Transformer. Position
Embedding (PE) scaling methods, while effective in extending the context window
to a specific length, demonstrate either notable limitations in their
extrapolation abilities or sacrificing partial performance within the context
window. Length extrapolation methods, although theoretically capable of
extending the context window beyond the training sequence length, often
underperform in practical long-context applications. To address these
challenges, we propose Continuous Length EXtrapolation (CLEX) for LLMs. We
generalise the PE scaling approaches to model the continuous dynamics by
ordinary differential equations over the length scaling factor, thereby
overcoming the constraints of current PE scaling methods designed for specific
lengths. Moreover, by extending the dynamics to desired context lengths beyond
the training sequence length, CLEX facilitates the length extrapolation with
impressive performance in practical tasks. We demonstrate that CLEX can be
seamlessly incorporated into LLMs equipped with Rotary Position Embedding, such
as LLaMA and GPT-NeoX, with negligible impact on training and inference
latency. Experimental results reveal that CLEX can effectively extend the
context window to over 4x or almost 8x training length, with no deterioration
in performance. Furthermore, when evaluated on the practical LongBench
benchmark, our model trained on a 4k length exhibits competitive performance
against state-of-the-art open-source models trained on context lengths up to
32k.
CommonCanvas: An Open Diffusion Model Trained with Creative-Commons
  Images
Authors: name: Aaron Gokaslan, name: A. Feder Cooper, name: Jasmine Collins, name: Landan Seguin, name: Austin Jacobson, name: Mihir Patel, name: Jonathan Frankle, name: Cory Stephenson, name: Volodymyr Kuleshov
Published: 2023-10-25 17:56:07

We assemble a dataset of Creative-Commons-licensed (CC) images, which we use
to train a set of open diffusion models that are qualitatively competitive with
Stable Diffusion 2 (SD2). This task presents two challenges: (1)
high-resolution CC images lack the captions necessary to train text-to-image
generative models; (2) CC images are relatively scarce. In turn, to address
these challenges, we use an intuitive transfer learning technique to produce a
set of high-quality synthetic captions paired with curated CC images. We then
develop a data- and compute-efficient training recipe that requires as little
as 3% of the LAION-2B data needed to train existing SD2 models, but obtains
comparable quality. These results indicate that we have a sufficient number of
CC images (~70 million) for training high-quality models. Our training recipe
also implements a variety of optimizations that achieve ~3X training speed-ups,
enabling rapid model iteration. We leverage this recipe to train several
high-quality text-to-image models, which we dub the CommonCanvas family. Our
largest model achieves comparable performance to SD2 on a human evaluation,
despite being trained on our CC dataset that is significantly smaller than
LAION and using synthetic captions for training. We release our models, data,
and code at
https://github.com/mosaicml/diffusion/blob/main/assets/common-canvas.md
ConvNets Match Vision Transformers at Scale
Authors: name: Samuel L. Smith, name: Andrew Brock, name: Leonard Berrada, name: Soham De
Published: 2023-10-25 16:52:13

Many researchers believe that ConvNets perform well on small or moderately
sized datasets, but are not competitive with Vision Transformers when given
access to datasets on the web-scale. We challenge this belief by evaluating a
performant ConvNet architecture pre-trained on JFT-4B, a large labelled dataset
of images often used for training foundation models. We consider pre-training
compute budgets between 0.4k and 110k TPU-v4 core compute hours, and train a
series of networks of increasing depth and width from the NFNet model family.
We observe a log-log scaling law between held out loss and compute budget.
After fine-tuning on ImageNet, NFNets match the reported performance of Vision
Transformers with comparable compute budgets. Our strongest fine-tuned model
achieves a Top-1 accuracy of 90.4%.
QMoE: Practical Sub-1-Bit Compression of Trillion-Parameter Models
Authors: name: Elias Frantar, name: Dan Alistarh
Published: 2023-10-25 17:24:53

Mixture-of-Experts (MoE) architectures offer a general solution to the high
inference costs of large language models (LLMs) via sparse routing, bringing
faster and more accurate models, at the cost of massive parameter counts. For
example, the SwitchTransformer-c2048 model has 1.6 trillion parameters,
requiring 3.2TB of accelerator memory to run efficiently, which makes practical
deployment challenging and expensive. In this paper, we present a solution to
this memory problem, in form of a new compression and execution framework
called QMoE. Specifically, QMoE consists of a scalable algorithm which
accurately compresses trillion-parameter MoEs to less than 1 bit per parameter,
in a custom format co-designed with bespoke GPU decoding kernels to facilitate
efficient end-to-end compressed inference, with minor runtime overheads
relative to uncompressed execution. Concretely, QMoE can compress the 1.6
trillion parameter SwitchTransformer-c2048 model to less than 160GB (20x
compression, 0.8 bits per parameter) at only minor accuracy loss, in less than
a day on a single GPU. This enables, for the first time, the execution of a
trillion-parameter model on affordable commodity hardware, like a single server
with 4x NVIDIA A6000 or 8x NVIDIA 3090 GPUs, at less than 5% runtime overhead
relative to ideal uncompressed inference. The source code and compressed models
are available at github.com/IST-DASLab/qmoe.
Detecting Pretraining Data from Large Language Models
Authors: name: Weijia Shi, name: Anirudh Ajith, name: Mengzhou Xia, name: Yangsibo Huang, name: Daogao Liu, name: Terra Blevins, name: Danqi Chen, name: Luke Zettlemoyer
Published: 2023-10-25 17:21:23

Although large language models (LLMs) are widely deployed, the data used to
train them is rarely disclosed. Given the incredible scale of this data, up to
trillions of tokens, it is all but certain that it includes potentially
problematic text such as copyrighted materials, personally identifiable
information, and test data for widely reported reference benchmarks. However,
we currently have no way to know which data of these types is included or in
what proportions. In this paper, we study the pretraining data detection
problem: given a piece of text and black-box access to an LLM without knowing
the pretraining data, can we determine if the model was trained on the provided
text? To facilitate this study, we introduce a dynamic benchmark WIKIMIA that
uses data created before and after model training to support gold truth
detection. We also introduce a new detection method Min-K% Prob based on a
simple hypothesis: an unseen example is likely to contain a few outlier words
with low probabilities under the LLM, while a seen example is less likely to
have words with such low probabilities. Min-K% Prob can be applied without any
knowledge about the pretraining corpus or any additional training, departing
from previous detection methods that require training a reference model on data
that is similar to the pretraining data. Moreover, our experiments demonstrate
that Min-K% Prob achieves a 7.4% improvement on WIKIMIA over these previous
methods. We apply Min-K% Prob to two real-world scenarios, copyrighted book
detection, and contaminated downstream example detection, and find it a
consistently effective solution.
LLM-FP4: 4-Bit Floating-Point Quantized Transformers
Authors: name: Shih-yang Liu, name: Zechun Liu, name: Xijie Huang, name: Pingcheng Dong, name: Kwang-Ting Cheng
Published: 2023-10-25 17:59:32

We propose LLM-FP4 for quantizing both weights and activations in large
language models (LLMs) down to 4-bit floating-point values, in a post-training
manner. Existing post-training quantization (PTQ) solutions are primarily
integer-based and struggle with bit widths below 8 bits. Compared to integer
quantization, floating-point (FP) quantization is more flexible and can better
handle long-tail or bell-shaped distributions, and it has emerged as a default
choice in many hardware platforms. One characteristic of FP quantization is
that its performance largely depends on the choice of exponent bits and
clipping range. In this regard, we construct a strong FP-PTQ baseline by
searching for the optimal quantization parameters. Furthermore, we observe a
high inter-channel variance and low intra-channel variance pattern in
activation distributions, which adds activation quantization difficulty. We
recognize this pattern to be consistent across a spectrum of transformer models
designed for diverse tasks, such as LLMs, BERT, and Vision Transformer models.
To tackle this, we propose per-channel activation quantization and show that
these additional scaling factors can be reparameterized as exponential biases
of weights, incurring a negligible cost. Our method, for the first time, can
quantize both weights and activations in the LLaMA-13B to only 4-bit and
achieves an average score of 63.1 on the common sense zero-shot reasoning
tasks, which is only 5.8 lower than the full-precision model, significantly
outperforming the previous state-of-the-art by 12.7 points. Code is available
at: https://github.com/nbasyl/LLM-FP4.
LightSpeed: Light and Fast Neural Light Fields on Mobile Devices
Authors: name: Aarush Gupta, name: Junli Cao, name: Chaoyang Wang, name: Ju Hu, name: Sergey Tulyakov, name: Jian Ren, name: László A Jeni
Published: 2023-10-25 17:59:05

Real-time novel-view image synthesis on mobile devices is prohibitive due to
the limited computational power and storage. Using volumetric rendering
methods, such as NeRF and its derivatives, on mobile devices is not suitable
due to the high computational cost of volumetric rendering. On the other hand,
recent advances in neural light field representations have shown promising
real-time view synthesis results on mobile devices. Neural light field methods
learn a direct mapping from a ray representation to the pixel color. The
current choice of ray representation is either stratified ray sampling or
Pl\"{u}cker coordinates, overlooking the classic light slab (two-plane)
representation, the preferred representation to interpolate between light field
views. In this work, we find that using the light slab representation is an
efficient representation for learning a neural light field. More importantly,
it is a lower-dimensional ray representation enabling us to learn the 4D ray
space using feature grids which are significantly faster to train and render.
Although mostly designed for frontal views, we show that the light-slab
representation can be further extended to non-frontal scenes using a
divide-and-conquer strategy. Our method offers superior rendering quality
compared to previous light field methods and achieves a significantly improved
trade-off between rendering quality and speed.
Wonder3D: Single Image to 3D using Cross-Domain Diffusion
Authors: name: Xiaoxiao Long, name: Yuan-Chen Guo, name: Cheng Lin, name: Yuan Liu, name: Zhiyang Dou, name: Lingjie Liu, name: Yuexin Ma, name: Song-Hai Zhang, name: Marc Habermann, name: Christian Theobalt, name: Wenping Wang
Published: 2023-10-23 15:02:23

In this work, we introduce Wonder3D, a novel method for efficiently
generating high-fidelity textured meshes from single-view images.Recent methods
based on Score Distillation Sampling (SDS) have shown the potential to recover
3D geometry from 2D diffusion priors, but they typically suffer from
time-consuming per-shape optimization and inconsistent geometry. In contrast,
certain works directly produce 3D information via fast network inferences, but
their results are often of low quality and lack geometric details. To
holistically improve the quality, consistency, and efficiency of image-to-3D
tasks, we propose a cross-domain diffusion model that generates multi-view
normal maps and the corresponding color images. To ensure consistency, we
employ a multi-view cross-domain attention mechanism that facilitates
information exchange across views and modalities. Lastly, we introduce a
geometry-aware normal fusion algorithm that extracts high-quality surfaces from
the multi-view 2D representations. Our extensive evaluations demonstrate that
our method achieves high-quality reconstruction results, robust generalization,
and reasonably good efficiency compared to prior works.
DreamCraft3D: Hierarchical 3D Generation with Bootstrapped Diffusion
  Prior
Authors: name: Jingxiang Sun, name: Bo Zhang, name: Ruizhi Shao, name: Lizhen Wang, name: Wen Liu, name: Zhenda Xie, name: Yebin Liu
Published: 2023-10-25 17:50:10

We present DreamCraft3D, a hierarchical 3D content generation method that
produces high-fidelity and coherent 3D objects. We tackle the problem by
leveraging a 2D reference image to guide the stages of geometry sculpting and
texture boosting. A central focus of this work is to address the consistency
issue that existing works encounter. To sculpt geometries that render
coherently, we perform score distillation sampling via a view-dependent
diffusion model. This 3D prior, alongside several training strategies,
prioritizes the geometry consistency but compromises the texture fidelity. We
further propose Bootstrapped Score Distillation to specifically boost the
texture. We train a personalized diffusion model, Dreambooth, on the augmented
renderings of the scene, imbuing it with 3D knowledge of the scene being
optimized. The score distillation from this 3D-aware diffusion prior provides
view-consistent guidance for the scene. Notably, through an alternating
optimization of the diffusion prior and 3D scene representation, we achieve
mutually reinforcing improvements: the optimized 3D scene aids in training the
scene-specific diffusion model, which offers increasingly view-consistent
guidance for 3D optimization. The optimization is thus bootstrapped and leads
to substantial texture boosting. With tailored 3D priors throughout the
hierarchical generation, DreamCraft3D generates coherent 3D objects with
photorealistic renderings, advancing the state-of-the-art in 3D content
generation. Code available at https://github.com/deepseek-ai/DreamCraft3D.
A Picture is Worth a Thousand Words: Principled Recaptioning Improves
  Image Generation
Authors: name: Eyal Segalis, name: Dani Valevski, name: Danny Lumen, name: Yossi Matias, name: Yaniv Leviathan
Published: 2023-10-25 14:10:08

Text-to-image diffusion models achieved a remarkable leap in capabilities
over the last few years, enabling high-quality and diverse synthesis of images
from a textual prompt. However, even the most advanced models often struggle to
precisely follow all of the directions in their prompts. The vast majority of
these models are trained on datasets consisting of (image, caption) pairs where
the images often come from the web, and the captions are their HTML alternate
text. A notable example is the LAION dataset, used by Stable Diffusion and
other models. In this work we observe that these captions are often of low
quality, and argue that this significantly affects the model's capability to
understand nuanced semantics in the textual prompts. We show that by relabeling
the corpus with a specialized automatic captioning model and training a
text-to-image model on the recaptioned dataset, the model benefits
substantially across the board. First, in overall image quality: e.g. FID 14.84
vs. the baseline of 17.87, and 64.3% improvement in faithful image generation
according to human evaluation. Second, in semantic alignment, e.g. semantic
object accuracy 84.34 vs. 78.90, counting alignment errors 1.32 vs. 1.44 and
positional alignment 62.42 vs. 57.60. We analyze various ways to relabel the
corpus and provide evidence that this technique, which we call RECAP, both
reduces the train-inference discrepancy and provides the model with more
information per example, increasing sample efficiency and allowing the model to
better understand the relations between captions and images.
TiC-CLIP: Continual Training of CLIP Models
Authors: name: Saurabh Garg, name: Mehrdad Farajtabar, name: Hadi Pouransari, name: Raviteja Vemulapalli, name: Sachin Mehta, name: Oncel Tuzel, name: Vaishaal Shankar, name: Fartash Faghri
Published: 2023-10-24 22:41:14

Keeping large foundation models up to date on latest data is inherently
expensive. To avoid the prohibitive costs of constantly retraining, it is
imperative to continually train these models. This problem is exacerbated by
the lack of any large scale continual learning benchmarks or baselines. We
introduce the first set of web-scale Time-Continual (TiC) benchmarks for
training vision-language models: TiC-DataCompt, TiC-YFCC, and TiC-RedCaps with
over 12.7B timestamped image-text pairs spanning 9 years (2014--2022). We first
use our benchmarks to curate various dynamic evaluations to measure temporal
robustness of existing models. We show OpenAI's CLIP (trained on data up to
2020) loses approx 8% zero-shot accuracy on our curated retrieval task from
2021--2022 compared with more recently trained models in OpenCLIP repository.
We then study how to efficiently train models on time-continuous data. We
demonstrate that a simple rehearsal-based approach that continues training from
the last checkpoint and replays old data reduces compute by 2.5times when
compared to the standard practice of retraining from scratch.