Orionnnn/hub-stats huggingface dataset:
Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
Authors: name: Yilong Zhao, name: Chien-Yu Lin, name: Kan Zhu, name: Zihao Ye, name: Lequn Chen, name: Size Zheng, name: Luis Ceze, name: Arvind Krishnamurthy, name: Tianqi Chen, name: Baris Kasikci
Published: 2023-10-29 18:33:05

The growing demand for Large Language Models (LLMs) in applications such as
content generation, intelligent chatbots, and sentiment analysis poses
considerable challenges for LLM service providers. To efficiently use GPU
resources and boost throughput, batching multiple requests has emerged as a
popular paradigm; to further speed up batching, LLM quantization techniques
reduce memory consumption and increase computing capacity. However, prevalent
quantization schemes (e.g., 8-bit weight-activation quantization) cannot fully
leverage the capabilities of modern GPUs, such as 4-bit integer operators,
resulting in sub-optimal performance.
  To maximize LLMs' serving throughput, we introduce Atom, a low-bit
quantization method that achieves high throughput improvements with negligible
accuracy loss. Atom significantly boosts serving throughput by using low-bit
operators and considerably reduces memory consumption via low-bit quantization.
It attains high accuracy by applying a novel mixed-precision and fine-grained
quantization process. We evaluate Atom on 4-bit weight-activation quantization
setups in the serving context. Atom improves end-to-end throughput by up to
7.73times compared to the FP16 and by 2.53times compared to INT8
quantization, while maintaining the same latency target.
TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language
  Modeling Likewise
Authors: name: Nan He, name: Hanyu Lai, name: Chenyang Zhao, name: Zirui Cheng, name: Junting Pan, name: Ruoyu Qin, name: Ruofan Lu, name: Rui Lu, name: Yunchen Zhang, name: Gangming Zhao, name: Zhaohui Hou, name: Zhiyuan Huang, name: Shaoqing Lu, name: Ding Liang, name: Mingjie Zhan
Published: 2023-10-29 14:16:54

Large Language Models (LLMs) exhibit impressive reasoning and data
augmentation capabilities in various NLP tasks. However, what about small
models? In this work, we propose TeacherLM-7.1B, capable of annotating relevant
fundamentals, chain of thought, and common mistakes for most NLP samples, which
makes annotation more than just an answer, thus allowing other models to learn
"why" instead of just "what". The TeacherLM-7.1B model achieved a zero-shot
score of 52.3 on MMLU, surpassing most models with over 100B parameters. Even
more remarkable is its data augmentation ability. Based on TeacherLM-7.1B, we
augmented 58 NLP datasets and taught various student models with different
parameters from OPT and BLOOM series in a multi-task setting. The experimental
results indicate that the data augmentation provided by TeacherLM has brought
significant benefits. We will release the TeacherLM series of models and
augmented datasets as open-source.
LoRAShear: Efficient Large Language Model Structured Pruning and
  Knowledge Recovery
Authors: name: Tianyi Chen, name: Tianyu Ding, name: Badal Yadav, name: Ilya Zharkov, name: Luming Liang
Published: 2023-10-24 00:47:26

Large Language Models (LLMs) have transformed the landscape of artificial
intelligence, while their enormous size presents significant challenges in
terms of computational costs. We introduce LoRAShear, a novel efficient
approach to structurally prune LLMs and recover knowledge. Given general LLMs,
LoRAShear first creates the dependency graphs to discover minimally removal
structures and analyze the knowledge distribution. It then proceeds progressive
structured pruning on LoRA adaptors and enables inherent knowledge transfer to
better preserve the information in the redundant structures. To recover the
lost knowledge during pruning, LoRAShear meticulously studies and proposes a
dynamic fine-tuning schemes with dynamic data adaptors to effectively narrow
down the performance gap to the full models. Numerical results demonstrate that
by only using one GPU within a couple of GPU days, LoRAShear effectively
reduced footprint of LLMs by 20% with only 1.0% performance degradation and
significantly outperforms state-of-the-arts. The source code will be available
at https://github.com/microsoft/lorashear.
MM-VID: Advancing Video Understanding with GPT-4V(ision)
Authors: name: Kevin Lin, name: Faisal Ahmed, name: Linjie Li, name: Chung-Ching Lin, name: Ehsan Azarnasab, name: Zhengyuan Yang, name: Jianfeng Wang, name: Lin Liang, name: Zicheng Liu, name: Yumao Lu, name: Ce Liu, name: Lijuan Wang
Published: 2023-10-30 17:44:09

We present MM-VID, an integrated system that harnesses the capabilities of
GPT-4V, combined with specialized tools in vision, audio, and speech, to
facilitate advanced video understanding. MM-VID is designed to address the
challenges posed by long-form videos and intricate tasks such as reasoning
within hour-long content and grasping storylines spanning multiple episodes.
MM-VID uses a video-to-script generation with GPT-4V to transcribe multimodal
elements into a long textual script. The generated script details character
movements, actions, expressions, and dialogues, paving the way for large
language models (LLMs) to achieve video understanding. This enables advanced
capabilities, including audio description, character identification, and
multimodal high-level comprehension. Experimental results demonstrate the
effectiveness of MM-VID in handling distinct video genres with various video
lengths. Additionally, we showcase its potential when applied to interactive
environments, such as video games and graphic user interfaces.
Personalised Distillation: Empowering Open-Sourced LLMs with Adaptive
  Learning for Code Generation
Authors: name: Hailin Chen, name: Amrita Saha, name: Steven Hoi, name: Shafiq Joty
Published: 2023-10-28 07:54:39

With the rise of powerful closed-sourced LLMs (ChatGPT, GPT-4), there are
increasing interests in distilling the capabilies of close-sourced LLMs to
smaller open-sourced LLMs. Previous distillation methods usually prompt ChatGPT
to generate a set of instructions and answers, for the student model to learn.
However, such standard distillation approach neglects the merits and conditions
of the student model. Inspired by modern teaching principles, we design a
personalised distillation process, in which the student attempts to solve a
task first, then the teacher provides an adaptive refinement for the student to
improve. Instead of feeding the student with teacher's prior, personalised
distillation enables personalised learning for the student model, as it only
learns on examples it makes mistakes upon and learns to improve its own
solution. On code generation, personalised distillation consistently
outperforms standard distillation with only one third of the data. With only
2.5-3K personalised examples that incur a data-collection cost of 4-6$, we
boost CodeGen-mono-16B by 7% to achieve 36.4% pass@1 and StarCoder by 12.2% to
achieve 45.8% pass@1 on HumanEval.
Reconstructive Latent-Space Neural Radiance Fields for Efficient 3D
  Scene Representations
Authors: name: Tristan Aumentado-Armstrong, name: Ashkan Mirzaei, name: Marcus A. Brubaker, name: Jonathan Kelly, name: Alex Levinshtein, name: Konstantinos G. Derpanis, name: Igor Gilitschenski
Published: 2023-10-27 03:52:08

Neural Radiance Fields (NeRFs) have proven to be powerful 3D representations,
capable of high quality novel view synthesis of complex scenes. While NeRFs
have been applied to graphics, vision, and robotics, problems with slow
rendering speed and characteristic visual artifacts prevent adoption in many
use cases. In this work, we investigate combining an autoencoder (AE) with a
NeRF, in which latent features (instead of colours) are rendered and then
convolutionally decoded. The resulting latent-space NeRF can produce novel
views with higher quality than standard colour-space NeRFs, as the AE can
correct certain visual artifacts, while rendering over three times faster. Our
work is orthogonal to other techniques for improving NeRF efficiency. Further,
we can control the tradeoff between efficiency and image quality by shrinking
the AE architecture, achieving over 13 times faster rendering with only a small
drop in performance. We hope that our approach can form the basis of an
efficient, yet high-fidelity, 3D scene representation for downstream tasks,
especially when retaining differentiability is useful, as in many robotics
scenarios requiring continual learning.
ZeroNVS: Zero-Shot 360-Degree View Synthesis from a Single Real Image
Authors: name: Kyle Sargent, name: Zizhang Li, name: Tanmay Shah, name: Charles Herrmann, name: Hong-Xing Yu, name: Yunzhi Zhang, name: Eric Ryan Chan, name: Dmitry Lagun, name: Li Fei-Fei, name: Deqing Sun, name: Jiajun Wu
Published: 2023-10-27 09:06:43

We introduce a 3D-aware diffusion model, ZeroNVS, for single-image novel view
synthesis for in-the-wild scenes. While existing methods are designed for
single objects with masked backgrounds, we propose new techniques to address
challenges introduced by in-the-wild multi-object scenes with complex
backgrounds. Specifically, we train a generative prior on a mixture of data
sources that capture object-centric, indoor, and outdoor scenes. To address
issues from data mixture such as depth-scale ambiguity, we propose a novel
camera conditioning parameterization and normalization scheme. Further, we
observe that Score Distillation Sampling (SDS) tends to truncate the
distribution of complex backgrounds during distillation of 360-degree scenes,
and propose "SDS anchoring" to improve the diversity of synthesized novel
views. Our model sets a new state-of-the-art result in LPIPS on the DTU dataset
in the zero-shot setting, even outperforming methods specifically trained on
DTU. We further adapt the challenging Mip-NeRF 360 dataset as a new benchmark
for single-image novel view synthesis, and demonstrate strong performance in
this setting. Our code and data are at http://kylesargent.github.io/zeronvs/
CodeFusion: A Pre-trained Diffusion Model for Code Generation
Authors: name: Mukul Singh, name: José Cambronero, name: Sumit Gulwani, name: Vu Le, name: Carina Negreanu, name: Gust Verbruggen
Published: 2023-10-26 11:06:15

Imagine a developer who can only change their last line of code, how often
would they have to start writing a function from scratch before it is correct?
Auto-regressive models for code generation from natural language have a similar
limitation: they do not easily allow reconsidering earlier tokens generated. We
introduce CodeFusion, a pre-trained diffusion code generation model that
addresses this limitation by iteratively denoising a complete program
conditioned on the encoded natural language. We evaluate CodeFusion on the task
of natural language to code generation for Bash, Python, and Microsoft Excel
conditional formatting (CF) rules. Experiments show that CodeFusion (75M
parameters) performs on par with state-of-the-art auto-regressive systems
(350M-175B parameters) in top-1 accuracy and outperforms them in top-3 and
top-5 accuracy due to its better balance in diversity versus quality.
Personas as a Way to Model Truthfulness in Language Models
Authors: name: Nitish Joishi, name: Javier Rando, name: Abulhair Saparov, name: Najoung Kim, name: He He
Published: 2023-10-27 14:27:43

Large Language Models are trained on vast amounts of text from the internet,
which contains both factual and misleading information about the world. Can
language models discern truth from falsehood in this contradicting data?
Expanding on the view that LLMs can model different agents producing the
corpora, we hypothesize that they can cluster truthful text by modeling a
truthful persona: a group of agents that are likely to produce truthful text
and share similar features. For example, trustworthy sources like Wikipedia and
Science usually use formal writing styles and make consistent claims. By
modeling this persona, LLMs can generalize truthfulness beyond the specific
contexts in which each agent generated the training text. For example, the
model can infer that the agent "Wikipedia" will behave truthfully on topics
that were only generated by "Science" because they share a persona. We first
show evidence for the persona hypothesis via two observations: (1) we can probe
whether a model's answer will be truthful before it is generated; (2)
finetuning a model on a set of facts improves its truthfulness on unseen
topics. Next, using arithmetics as a synthetic environment, we show that
language models can separate true and false statements, and generalize
truthfulness across agents; but only if agents in the training data share a
truthful generative process that enables the creation of a truthful persona.
Overall, our findings suggest that models can exploit hierarchical structures
in the data to learn abstract concepts like truthfulness.
FP8-LM: Training FP8 Large Language Models
Authors: name: Houwen Peng, name: Kan Wu, name: Yixuan Wei, name: Guoshuai Zhao, name: Yuxiang Yang, name: Ze Liu, name: Yifan Xiong, name: Ziyue Yang, name: Bolin Ni, name: Jingcheng Hu, name: Ruihang Li, name: Miaosen Zhang, name: Chen Li, name: Jia Ning, name: Ruizhe Wang, name: Zheng Zhang, name: Shuguang Liu, name: Joe Chau, name: Han Hu, name: Peng Cheng
Published: 2023-10-27 17:59:51

In this paper, we explore FP8 low-bit data formats for efficient training of
large language models (LLMs). Our key insight is that most variables, such as
gradients and optimizer states, in LLM training can employ low-precision data
formats without compromising model accuracy and requiring no changes to
hyper-parameters. Specifically, we propose a new FP8 automatic mixed-precision
framework for training LLMs. This framework offers three levels of FP8
utilization to streamline mixed-precision and distributed parallel training for
LLMs. It gradually incorporates 8-bit gradients, optimizer states, and
distributed learning in an incremental manner. Experiment results show that,
during the training of GPT-175B model on H100 GPU platform, our FP8
mixed-precision training framework not only achieved a remarkable 42% reduction
in real memory usage but also ran 64% faster than the widely adopted BF16
framework (i.e., Megatron-LM), surpassing the speed of Nvidia Transformer
Engine by 17%. This largely reduces the training costs for large foundation
models. Furthermore, our FP8 mixed-precision training methodology is generic.
It can be seamlessly applied to other tasks such as LLM instruction tuning and
reinforcement learning with human feedback, offering savings in fine-tuning
expenses. Our FP8 low-precision training framework is open-sourced at
{https://github.com/Azure/MS-AMP}{aka.ms/MS.AMP}.
ControlLLM: Augment Language Models with Tools by Searching on Graphs
Authors: name: Zhaoyang Liu, name: Zeqiang Lai, name: Zhangwei Gao, name: Erfei Cui, name: Xizhou Zhu, name: Lewei Lu, name: Qifeng Chen, name: Yu Qiao, name: Jifeng Dai, name: Wenhai Wang
Published: 2023-10-26 21:57:21

We present ControlLLM, a novel framework that enables large language models
(LLMs) to utilize multi-modal tools for solving complex real-world tasks.
Despite the remarkable performance of LLMs, they still struggle with tool
invocation due to ambiguous user prompts, inaccurate tool selection and
parameterization, and inefficient tool scheduling. To overcome these
challenges, our framework comprises three key components: (1) a task
decomposer that breaks down a complex task into clear subtasks with
well-defined inputs and outputs; (2) a Thoughts-on-Graph (ToG)
paradigm that searches the optimal solution path on a pre-built tool graph,
which specifies the parameter and dependency relations among different tools;
and (3) an execution engine with a rich toolbox that interprets the
solution path and runs the tools efficiently on different computational
devices. We evaluate our framework on diverse tasks involving image, audio, and
video processing, demonstrating its superior accuracy, efficiency, and
versatility compared to existing methods.
Data-Centric Financial Large Language Models
Authors: name: Zhixuan Chu, name: Huaiyu Guo, name: Xinyuan Zhou, name: Yijia Wang, name: Fei Yu, name: Hong Chen, name: Wanqing Xu, name: Xin Lu, name: Qing Cui, name: Longfei Li, name: Jun Zhou, name: Sheng Li
Published: 2023-10-07 04:53:31

Large language models (LLMs) show promise for natural language tasks but
struggle when applied directly to complex domains like finance. LLMs have
difficulty reasoning about and integrating all relevant information. We propose
a data-centric approach to enable LLMs to better handle financial tasks. Our
key insight is that rather than overloading the LLM with everything at once, it
is more effective to preprocess and pre-understand the data. We create a
financial LLM (FLLM) using multitask prompt-based finetuning to achieve data
pre-processing and pre-understanding. However, labeled data is scarce for each
task. To overcome manual annotation costs, we employ abductive augmentation
reasoning (AAR) to automatically generate training data by modifying the pseudo
labels from FLLM's own outputs. Experiments show our data-centric FLLM with AAR
substantially outperforms baseline financial LLMs designed for raw text,
achieving state-of-the-art on financial analysis and interpretation tasks. We
also open source a new benchmark for financial analysis and interpretation. Our
methodology provides a promising path to unlock LLMs' potential for complex
real-world domains.
PockEngine: Sparse and Efficient Fine-tuning in a Pocket
Authors: name: Ligeng Zhu, name: Lanxiang Hu, name: Ji Lin, name: Wei-Chen Wang, name: Wei-Ming Chen, name: Chuang Gan, name: Song Han
Published: 2023-10-26 19:46:11

On-device learning and efficient fine-tuning enable continuous and
privacy-preserving customization (e.g., locally fine-tuning large language
models on personalized data). However, existing training frameworks are
designed for cloud servers with powerful accelerators (e.g., GPUs, TPUs) and
lack the optimizations for learning on the edge, which faces challenges of
resource limitations and edge hardware diversity. We introduce PockEngine: a
tiny, sparse and efficient engine to enable fine-tuning on various edge
devices. PockEngine supports sparse backpropagation: it prunes the backward
graph and sparsely updates the model with measured memory saving and latency
reduction while maintaining the model quality. Secondly, PockEngine is
compilation first: the entire training graph (including forward, backward and
optimization steps) is derived at compile-time, which reduces the runtime
overhead and brings opportunities for graph transformations. PockEngine also
integrates a rich set of training graph optimizations, thus can further
accelerate the training cost, including operator reordering and backend
switching. PockEngine supports diverse applications, frontends and hardware
backends: it flexibly compiles and tunes models defined in
PyTorch/TensorFlow/Jax and deploys binaries to mobile CPU/GPU/DSPs. We
evaluated PockEngine on both vision models and large language models.
PockEngine achieves up to 15 times speedup over off-the-shelf TensorFlow
(Raspberry Pi), 5.6 times memory saving back-propagation (Jetson AGX Orin).
Remarkably, PockEngine enables fine-tuning LLaMav2-7B on NVIDIA Jetson AGX Orin
at 550 tokens/s, 7.9times faster than the PyTorch.
Large Language Models as Generalizable Policies for Embodied Tasks
Authors: name: Andrew Szot, name: Max Schwarzer, name: Harsh Agrawal, name: Bogdan Mazoure, name: Walter Talbott, name: Katherine Metcalf, name: Natalie Mackraz, name: Devon Hjelm, name: Alexander Toshev
Published: 2023-10-26 18:32:05

We show that large language models (LLMs) can be adapted to be generalizable
policies for embodied visual tasks. Our approach, called Large LAnguage model
Reinforcement Learning Policy (LLaRP), adapts a pre-trained frozen LLM to take
as input text instructions and visual egocentric observations and output
actions directly in the environment. Using reinforcement learning, we train
LLaRP to see and act solely through environmental interactions. We show that
LLaRP is robust to complex paraphrasings of task instructions and can
generalize to new tasks that require novel optimal behavior. In particular, on
1,000 unseen tasks it achieves 42% success rate, 1.7x the success rate of other
common learned baselines or zero-shot applications of LLMs. Finally, to aid the
community in studying language conditioned, massively multi-task, embodied AI
problems we release a novel benchmark, Language Rearrangement, consisting of
150,000 training and 1,000 testing tasks for language-conditioned
rearrangement. Video examples of LLaRP in unseen Language Rearrangement
instructions are at https://llm-rl.github.io.
A Framework for Automated Measurement of Responsible AI Harms in
  Generative AI Applications
Authors: name: Ahmed Magooda, name: Alec Helyar, name: Kyle Jackson, name: David Sullivan, name: Chad Atalla, name: Emily Sheng, name: Dan Vann, name: Richard Edgar, name: Hamid Palangi, name: Roman Lutz, name: Hongliang Kong, name: Vincent Yun, name: Eslam Kamal, name: Federico Zarfati, name: Hanna Wallach, name: Sarah Bird, name: Mei Chen
Published: 2023-10-26 19:45:06

We present a framework for the automated measurement of responsible AI (RAI)
metrics for large language models (LLMs) and associated products and services.
Our framework for automatically measuring harms from LLMs builds on existing
technical and sociotechnical expertise and leverages the capabilities of
state-of-the-art LLMs, such as GPT-4. We use this framework to run through
several case studies investigating how different LLMs may violate a range of
RAI-related principles. The framework may be employed alongside domain-specific
sociotechnical expertise to create measurements for new harm areas in the
future. By implementing this framework, we aim to enable more advanced harm
measurement efforts and further the responsible use of LLMs.
HyperFields: Towards Zero-Shot Generation of NeRFs from Text
Authors: name: Sudarshan Babu, name: Richard Liu, name: Avery Zhou, name: Michael Maire, name: Greg Shakhnarovich, name: Rana Hanocka
Published: 2023-10-26 00:36:03

We introduce HyperFields, a method for generating text-conditioned Neural
Radiance Fields (NeRFs) with a single forward pass and (optionally) some
fine-tuning. Key to our approach are: (i) a dynamic hypernetwork, which learns
a smooth mapping from text token embeddings to the space of NeRFs; (ii) NeRF
distillation training, which distills scenes encoded in individual NeRFs into
one dynamic hypernetwork. These techniques enable a single network to fit over
a hundred unique scenes. We further demonstrate that HyperFields learns a more
general map between text and NeRFs, and consequently is capable of predicting
novel in-distribution and out-of-distribution scenes -- either zero-shot or
with a few finetuning steps. Finetuning HyperFields benefits from accelerated
convergence thanks to the learned general map, and is capable of synthesizing
novel scenes 5 to 10 times faster than existing neural optimization-based
methods. Our ablation experiments show that both the dynamic architecture and
NeRF distillation are critical to the expressivity of HyperFields.