Orionnnn/hub-stats huggingface dataset:
AstroLLaMA: Towards Specialized Foundation Models in Astronomy
Authors: name: Tuan Dung Nguyen, name: Yuan-Sen Ting, name: Ioana Ciucă, name: Charlie O'Neill, name: Ze-Chang Sun, name: Maja Jabłońska, name: Sandor Kruk, name: Ernest Perkowski, name: Jack Miller, name: Jason Li, name: Josh Peek, name: Kartheik Iyer, name: Tomasz Różański, name: Pranav Khetarpal, name: Sharaf Zaman, name: David Brodrick, name: Sergio J. Rodríguez Méndez, name: Thang Bui, name: Alyssa Goodman, name: Alberto Accomazzi, name: Jill Naiman, name: Jesse Cranney, name: Kevin Schawinski, name: UniverseTBD
Published: 2023-09-12 11:02:27

Large language models excel in many human-language tasks but often falter in
highly specialized domains like scholarly astronomy. To bridge this gap, we
introduce AstroLLaMA, a 7-billion-parameter model fine-tuned from LLaMA-2 using
over 300,000 astronomy abstracts from arXiv. Optimized for traditional causal
language modeling, AstroLLaMA achieves a 30% lower perplexity than Llama-2,
showing marked domain adaptation. Our model generates more insightful and
scientifically relevant text completions and embedding extraction than
state-of-the-arts foundation models despite having significantly fewer
parameters. AstroLLaMA serves as a robust, domain-specific model with broad
fine-tuning potential. Its public release aims to spur astronomy-focused
research, including automatic paper summarization and conversational agent
development.
Efficient Memory Management for Large Language Model Serving with
  PagedAttention
Authors: name: Woosuk Kwon, name: Zhuohan Li, name: Siyuan Zhuang, name: Ying Sheng, name: Lianmin Zheng, name: Cody Hao Yu, name: Joseph E. Gonzalez, name: Hao Zhang, name: Ion Stoica
Published: 2023-09-12 12:50:04

High throughput serving of large language models (LLMs) requires batching
sufficiently many requests at a time. However, existing systems struggle
because the key-value cache (KV cache) memory for each request is huge and
grows and shrinks dynamically. When managed inefficiently, this memory can be
significantly wasted by fragmentation and redundant duplication, limiting the
batch size. To address this problem, we propose PagedAttention, an attention
algorithm inspired by the classical virtual memory and paging techniques in
operating systems. On top of it, we build vLLM, an LLM serving system that
achieves (1) near-zero waste in KV cache memory and (2) flexible sharing of KV
cache within and across requests to further reduce memory usage. Our
evaluations show that vLLM improves the throughput of popular LLMs by
2-4times with the same level of latency compared to the state-of-the-art
systems, such as FasterTransformer and Orca. The improvement is more pronounced
with longer sequences, larger models, and more complex decoding algorithms.
vLLM's source code is publicly available at
https://github.com/vllm-project/vllm
Uncovering mesa-optimization algorithms in Transformers
Authors: name: Johannes von Oswald, name: Eyvind Niklasson, name: Maximilian Schlegel, name: Seijin Kobayashi, name: Nicolas Zucchet, name: Nino Scherrer, name: Nolan Miller, name: Mark Sandler, name: Blaise Agüera y Arcas, name: Max Vladymyrov, name: Razvan Pascanu, name: João Sacramento
Published: 2023-09-11 22:42:50

Transformers have become the dominant model in deep learning, but the reason
for their superior performance is poorly understood. Here, we hypothesize that
the strong performance of Transformers stems from an architectural bias towards
mesa-optimization, a learned process running within the forward pass of a model
consisting of the following two steps: (i) the construction of an internal
learning objective, and (ii) its corresponding solution found through
optimization. To test this hypothesis, we reverse-engineer a series of
autoregressive Transformers trained on simple sequence modeling tasks,
uncovering underlying gradient-based mesa-optimization algorithms driving the
generation of predictions. Moreover, we show that the learned forward-pass
optimization algorithm can be immediately repurposed to solve supervised
few-shot tasks, suggesting that mesa-optimization might underlie the in-context
learning capabilities of large language models. Finally, we propose a novel
self-attention layer, the mesa-layer, that explicitly and efficiently solves
optimization problems specified in context. We find that this layer can lead to
improved performance in synthetic and preliminary language modeling
experiments, adding weight to our hypothesis that mesa-optimization is an
important operation hidden within the weights of trained Transformers.
Natural Language Supervision for General-Purpose Audio Representations
Authors: name: Benjamin Elizalde, name: Soham Deshmukh, name: Huaming Wang
Published: 2023-09-11 18:50:21

Audio-Language models jointly learn multimodal text and audio representations
that enable Zero-Shot inference. Models rely on the encoders to create powerful
representations of the input and generalize to multiple tasks ranging from
sounds, music, and speech. Although models have achieved remarkable
performance, there is still a performance gap with task-specific models. In
this paper, we propose a Contrastive Language-Audio Pretraining model that is
pretrained with a diverse collection of 4.6M audio-text pairs employing two
innovative encoders for Zero-Shot inference. To learn audio representations, we
trained an audio encoder on 22 audio tasks, instead of the standard training of
sound event classification. To learn language representations, we trained an
autoregressive decoder-only model instead of the standard encoder-only models.
Then, the audio and language representations are brought into a joint
multimodal space using Contrastive Learning. We used our encoders to improve
the downstream performance by a margin. We extensively evaluated the
generalization of our representations on 26 downstream tasks, the largest in
the literature. Our model achieves state of the art results in several tasks
leading the way towards general-purpose audio representations.
Large Language Model for Science: A Study on P vs. NP
Authors: name: Qingxiu Dong, name: Li Dong, name: Ke Xu, name: Guangyan Zhou, name: Yaru Hao, name: Zhifang Sui, name: Furu Wei
Published: 2023-09-11 17:49:27

In this work, we use large language models (LLMs) to augment and accelerate
research on the P versus NP problem, one of the most important open problems in
theoretical computer science and mathematics. Specifically, we propose Socratic
reasoning, a general framework that promotes in-depth thinking with LLMs for
complex problem-solving. Socratic reasoning encourages LLMs to recursively
discover, solve, and integrate problems while facilitating self-evaluation and
refinement. Our pilot study on the P vs. NP problem shows that GPT-4
successfully produces a proof schema and engages in rigorous reasoning
throughout 97 dialogue turns, concluding "P neq NP", which is in alignment
with (Xu and Zhou, 2023). The investigation uncovers novel insights within the
extensive solution space of LLMs, shedding light on LLM for Science.
Dynamic Mesh-Aware Radiance Fields
Authors: name: Yi-Ling Qiao, name: Alexander Gao, name: Yiran Xu, name: Yue Feng, name: Jia-Bin Huang, name: Ming C. Lin
Published: 2023-09-08 20:18:18

Embedding polygonal mesh assets within photorealistic Neural Radience Fields
(NeRF) volumes, such that they can be rendered and their dynamics simulated in
a physically consistent manner with the NeRF, is under-explored from the system
perspective of integrating NeRF into the traditional graphics pipeline. This
paper designs a two-way coupling between mesh and NeRF during rendering and
simulation. We first review the light transport equations for both mesh and
NeRF, then distill them into an efficient algorithm for updating radiance and
throughput along a cast ray with an arbitrary number of bounces. To resolve the
discrepancy between the linear color space that the path tracer assumes and the
sRGB color space that standard NeRF uses, we train NeRF with High Dynamic Range
(HDR) images. We also present a strategy to estimate light sources and cast
shadows on the NeRF. Finally, we consider how the hybrid surface-volumetric
formulation can be efficiently integrated with a high-performance physics
simulator that supports cloth, rigid and soft bodies. The full rendering and
simulation system can be run on a GPU at interactive rates. We show that a
hybrid system approach outperforms alternatives in visual realism for mesh
insertion, because it allows realistic light transport from volumetric NeRF
media onto surfaces, which affects the appearance of reflective/refractive
surfaces and illumination of diffuse surfaces informed by the dynamic scene.
NExT-GPT: Any-to-Any Multimodal LLM
Authors: name: Shengqiong Wu, name: Hao Fei, name: Leigang Qu, name: Wei Ji, name: Tat-Seng Chua
Published: 2023-09-11 15:02:25

While recently Multimodal Large Language Models (MM-LLMs) have made exciting
strides, they mostly fall prey to the limitation of only input-side multimodal
understanding, without the ability to produce content in multiple modalities.
As we humans always perceive the world and communicate with people through
various modalities, developing any-to-any MM-LLMs capable of accepting and
delivering content in any modality becomes essential to human-level AI. To fill
the gap, we present an end-to-end general-purpose any-to-any MM-LLM system,
NExT-GPT. We connect an LLM with multimodal adaptors and different diffusion
decoders, enabling NExT-GPT to perceive inputs and generate outputs in
arbitrary combinations of text, images, videos, and audio. By leveraging the
existing well-trained highly-performing encoders and decoders, NExT-GPT is
tuned with only a small amount of parameter (1%) of certain projection layers,
which not only benefits low-cost training and also facilitates convenient
expansion to more potential modalities. Moreover, we introduce a
modality-switching instruction tuning (MosIT) and manually curate a
high-quality dataset for MosIT, based on which NExT-GPT is empowered with
complex cross-modal semantic understanding and content generation. Overall, our
research showcases the promising possibility of building an AI agent capable of
modeling universal modalities, paving the way for more human-like AI research
in the community.
Optimize Weight Rounding via Signed Gradient Descent for the
  Quantization of LLMs
Authors: name: Wenhua Cheng, name: Weiwei Zhang, name: Haihao Shen, name: Yiyang Cai, name: Xin He, name: Kaokao Lv
Published: 2023-09-11 14:58:23

Large Language Models (LLMs) have proven their exceptional capabilities in
performing language-related tasks. However, their deployment poses significant
challenges due to their considerable memory and storage requirements. In
response to this issue, weight-only quantization, particularly 3 and 4-bit
weight-only quantization, has emerged as one of the most viable solutions. As
the number of bits decreases, the quantization grid broadens, thus emphasizing
the importance of up and down rounding. While previous studies have
demonstrated that fine-tuning up and down rounding with the addition of
perturbations can enhance accuracy in some scenarios, our study is driven by
the precise and limited boundary of these perturbations, where only the
threshold for altering the rounding value is of significance. Consequently, we
propose a concise and highly effective approach for optimizing the weight
rounding task. Our method, named SignRound, involves lightweight block-wise
tuning using signed gradient descent, enabling us to achieve outstanding
results within 400 steps. SignRound outperforms the established baseline of
rounding-to-nearest (RTN) and competes impressively against recent methods,
without introducing additional inference overhead. The source code will be
publicly available at https://github.com/intel/neural-compressor soon.
Textbooks Are All You Need II: phi-1.5 technical report
Authors: name: Yuanzhi Li, name: Sébastien Bubeck, name: Ronen Eldan, name: Allie Del Giorno, name: Suriya Gunasekar, name: Yin Tat Lee
Published: 2023-09-11 14:01:45

We continue the investigation into the power of smaller Transformer-based
language models as initiated by TinyStories -- a 10 million parameter
model that can produce coherent English -- and the follow-up work on
phi-1, a 1.3 billion parameter model with Python coding performance
close to the state-of-the-art. The latter work proposed to use existing Large
Language Models (LLMs) to generate ``textbook quality" data as a way to enhance
the learning process compared to traditional web data. We follow the
``Textbooks Are All You Need" approach, focusing this time on common sense
reasoning in natural language, and create a new 1.3 billion parameter model
named phi-1.5, with performance on natural language tasks comparable
to models 5x larger, and surpassing most non-frontier LLMs on more complex
reasoning tasks such as grade-school mathematics and basic coding. More
generally, phi-1.5 exhibits many of the traits of much larger LLMs,
both good -- such as the ability to ``think step by step" or perform some
rudimentary in-context learning -- and bad, including hallucinations and the
potential for toxic and biased generations -- encouragingly though, we are
seeing improvement on that front thanks to the absence of web data. We
open-source phi-1.5 to promote further research on these urgent
topics.
Neurons in Large Language Models: Dead, N-gram, Positional
Authors: name: Elena Voita, name: Javier Ferrando, name: Christoforos Nalmpantis
Published: 2023-09-09 15:51:36

We analyze a family of large language models in such a lightweight manner
that can be done on a single GPU. Specifically, we focus on the OPT family of
models ranging from 125m to 66b parameters and rely only on whether an FFN
neuron is activated or not. First, we find that the early part of the network
is sparse and represents many discrete features. Here, many neurons (more than
70% in some layers of the 66b model) are "dead", i.e. they never activate on a
large collection of diverse data. At the same time, many of the alive neurons
are reserved for discrete features and act as token and n-gram detectors.
Interestingly, their corresponding FFN updates not only promote next token
candidates as could be expected, but also explicitly focus on removing the
information about triggering them tokens, i.e., current input. To the best of
our knowledge, this is the first example of mechanisms specialized at removing
(rather than adding) information from the residual stream. With scale, models
become more sparse in a sense that they have more dead neurons and token
detectors. Finally, some neurons are positional: them being activated or not
depends largely (or solely) on position and less so (or not at all) on textual
data. We find that smaller models have sets of neurons acting as position range
indicators while larger models operate in a less explicit manner.
FIAT: Fusing learning paradigms with Instruction-Accelerated Tuning
Authors: name: Xinyi Wang, name: John Wieting, name: Jonathan H. Clark
Published: 2023-09-09 02:43:48

Learning paradigms for large language models (LLMs) currently tend to fall
within either in-context learning (ICL) or full fine-tuning. Each of these
comes with their own trade-offs based on available data, model size, compute
cost, ease-of-use, and final quality with neither solution performing well
across-the-board. In this article, we first describe ICL and fine-tuning
paradigms in a way that highlights their natural connections. Based on these
connections, we propose a new learning paradigm called FIAT that fuses the best
of these paradigms together, enabling prompt-engineered instructions and
chain-of-thought reasoning with the very largest models while also using
similar methods to perform parameter updates on a modestly-sized LLM with
parameter-efficient tuning. We evaluate FIAT's effectiveness on a variety of
multilingual tasks and observe that FIAT performs better than both ICL and
fine-tuning at scales ranging from 100-10,000 training examples. We hope that
FIAT provides a practical way of harnessing the full potential of LLMs without
needing to make a hard choice between learning paradigms.
MADLAD-400: A Multilingual And Document-Level Large Audited Dataset
Authors: name: Sneha Kudugunta, name: Isaac Caswell, name: Biao Zhang, name: Xavier Garcia, name: Christopher A. Choquette-Choo, name: Katherine Lee, name: Derrick Xin, name: Aditya Kusupati, name: Romi Stella, name: Ankur Bapna, name: Orhan Firat
Published: 2023-09-09 02:34:01

We introduce MADLAD-400, a manually audited, general domain 3T token
monolingual dataset based on CommonCrawl, spanning 419 languages. We discuss
the limitations revealed by self-auditing MADLAD-400, and the role data
auditing had in the dataset creation process. We then train and release a
10.7B-parameter multilingual machine translation model on 250 billion tokens
covering over 450 languages using publicly available data, and find that it is
competitive with models that are significantly larger, and report the results
on different domains. In addition, we train a 8B-parameter language model, and
assess the results on few-shot translation. We make the baseline models
available to the research community.
When Less is More: Investigating Data Pruning for Pretraining LLMs at
  Scale
Authors: name: Max Marion, name: Ahmet Üstün, name: Luiza Pozzobon, name: Alex Wang, name: Marzieh Fadaee, name: Sara Hooker
Published: 2023-09-08 19:34:05

Large volumes of text data have contributed significantly to the development
of large language models (LLMs) in recent years. This data is typically
acquired by scraping the internet, leading to pretraining datasets comprised of
noisy web text. To date, efforts to prune these datasets down to a higher
quality subset have relied on hand-crafted heuristics encoded as rule-based
filters. In this work, we take a wider view and explore scalable estimates of
data quality that can be used to systematically measure the quality of
pretraining data. We perform a rigorous comparison at scale of the simple data
quality estimator of perplexity, as well as more sophisticated and
computationally intensive estimates of the Error L2-Norm and memorization.
These metrics are used to rank and prune pretraining corpora, and we
subsequently compare LLMs trained on these pruned datasets. Surprisingly, we
find that the simple technique of perplexity outperforms our more
computationally expensive scoring methods. We improve over our no-pruning
baseline while training on as little as 30% of the original training dataset.
Our work sets the foundation for unexplored strategies in automatically
curating high quality corpora and suggests the majority of pretraining data can
be removed while retaining performance.
Towards Practical Capture of High-Fidelity Relightable Avatars
Authors: name: Haotian Yang, name: Mingwu Zheng, name: Wanquan Feng, name: Haibin Huang, name: Yu-Kun Lai, name: Pengfei Wan, name: Zhongyuan Wang, name: Chongyang Ma
Published: 2023-09-08 10:26:29

In this paper, we propose a novel framework, Tracking-free Relightable Avatar
(TRAvatar), for capturing and reconstructing high-fidelity 3D avatars. Compared
to previous methods, TRAvatar works in a more practical and efficient setting.
Specifically, TRAvatar is trained with dynamic image sequences captured in a
Light Stage under varying lighting conditions, enabling realistic relighting
and real-time animation for avatars in diverse scenes. Additionally, TRAvatar
allows for tracking-free avatar capture and obviates the need for accurate
surface tracking under varying illumination conditions. Our contributions are
two-fold: First, we propose a novel network architecture that explicitly builds
on and ensures the satisfaction of the linear nature of lighting. Trained on
simple group light captures, TRAvatar can predict the appearance in real-time
with a single forward pass, achieving high-quality relighting effects under
illuminations of arbitrary environment maps. Second, we jointly optimize the
facial geometry and relightable appearance from scratch based on image
sequences, where the tracking is implicitly learned. This tracking-free
approach brings robustness for establishing temporal correspondences between
frames under different lighting conditions. Extensive qualitative and
quantitative experiments demonstrate that our framework achieves superior
performance for photorealistic avatar animation and relighting.
Large-Scale Automatic Audiobook Creation
Authors: name: Brendan Walsh, name: Mark Hamilton, name: Greg Newby, name: Xi Wang, name: Serena Ruan, name: Sheng Zhao, name: Lei He, name: Shaofei Zhang, name: Eric Dettinger, name: William T. Freeman, name: Markus Weimer
Published: 2023-09-07 11:41:23

An audiobook can dramatically improve a work of literature's accessibility
and improve reader engagement. However, audiobooks can take hundreds of hours
of human effort to create, edit, and publish. In this work, we present a system
that can automatically generate high-quality audiobooks from online e-books. In
particular, we leverage recent advances in neural text-to-speech to create and
release thousands of human-quality, open-license audiobooks from the Project
Gutenberg e-book collection. Our method can identify the proper subset of
e-book content to read for a wide collection of diversely structured books and
can operate on hundreds of books in parallel. Our system allows users to
customize an audiobook's speaking speed and style, emotional intonation, and
can even match a desired voice using a small amount of sample audio. This work
contributed over five thousand open-license audiobooks and an interactive demo
that allows users to quickly create their own customized audiobooks. To listen
to the audiobook collection visit https://aka.ms/audiobook.
Mobile V-MoEs: Scaling Down Vision Transformers via Sparse
  Mixture-of-Experts
Authors: name: Erik Daxberger, name: Floris Weers, name: Bowen Zhang, name: Tom Gunter, name: Ruoming Pang, name: Marcin Eichner, name: Michael Emmersberger, name: Yinfei Yang, name: Alexander Toshev, name: Xianzhi Du
Published: 2023-09-08 14:24:10

Sparse Mixture-of-Experts models (MoEs) have recently gained popularity due
to their ability to decouple model size from inference efficiency by only
activating a small subset of the model parameters for any given input token. As
such, sparse MoEs have enabled unprecedented scalability, resulting in
tremendous successes across domains such as natural language processing and
computer vision. In this work, we instead explore the use of sparse MoEs to
scale-down Vision Transformers (ViTs) to make them more attractive for
resource-constrained vision applications. To this end, we propose a simplified
and mobile-friendly MoE design where entire images rather than individual
patches are routed to the experts. We also propose a stable MoE training
procedure that uses super-class information to guide the router. We empirically
show that our sparse Mobile Vision MoEs (V-MoEs) can achieve a better trade-off
between performance and efficiency than the corresponding dense ViTs. For
example, for the ViT-Tiny model, our Mobile V-MoE outperforms its dense
counterpart by 3.39% on ImageNet-1k. For an even smaller ViT variant with only
54M FLOPs inference cost, our MoE achieves an improvement of 4.66%.