Orionnnn/hub-stats huggingface dataset:
FlashDecoding++: Faster Large Language Model Inference on GPUs
Authors: name: Ke Hong, name: Guohao Dai, name: Jiaming Xu, name: Qiuli Mao, name: Xiuhong Li, name: Jun Liu, name: Kangdi Chen, name: Hanyu Dong, name: Yu Wang
Published: 2023-11-02 14:57:03

As the Large Language Model (LLM) becomes increasingly important in various
domains. However, the following challenges still remain unsolved in
accelerating LLM inference: (1) Synchronized partial softmax update. The
softmax operation requires a synchronized update operation among each partial
softmax result, leading to ~20% overheads for the attention computation in
LLMs. (2) Under-utilized computation of flat GEMM. The shape of matrices
performing GEMM in LLM inference is flat, leading to under-utilized computation
and >50% performance loss after padding zeros in previous designs. (3)
Performance loss due to static dataflow. Kernel performance in LLM depends on
varied input data features, hardware configurations, etc. A single and static
dataflow may lead to a 50.25% performance loss for GEMMs of different shapes in
LLM inference.
  We present FlashDecoding++, a fast LLM inference engine supporting mainstream
LLMs and hardware back-ends. To tackle the above challenges, FlashDecoding++
creatively proposes: (1) Asynchronized softmax with unified max value.
FlashDecoding++ introduces a unified max value technique for different partial
softmax computations to avoid synchronization. (2) Flat GEMM optimization with
double buffering. FlashDecoding++ points out that flat GEMMs with different
shapes face varied bottlenecks. Then, techniques like double buffering are
introduced. (3) Heuristic dataflow with hardware resource adaptation.
FlashDecoding++ heuristically optimizes dataflow using different hardware
resource considering input dynamics. Due to the versatility of optimizations in
FlashDecoding++, FlashDecoding++ can achieve up to 4.86x and 2.18x speedup on
both NVIDIA and AMD GPUs compared to Hugging Face implementations.
FlashDecoding++ also achieves an average speedup of 1.37x compared to
state-of-the-art LLM inference engines on mainstream LLMs.
RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning
  via Generative Simulation
Authors: name: Yufei Wang, name: Zhou Xian, name: Feng Chen, name: Tsun-Hsuan Wang, name: Yian Wang, name: Katerina Fragkiadaki, name: Zackory Erickson, name: David Held, name: Chuang Gan
Published: 2023-11-02 17:59:21

We present RoboGen, a generative robotic agent that automatically learns
diverse robotic skills at scale via generative simulation. RoboGen leverages
the latest advancements in foundation and generative models. Instead of
directly using or adapting these models to produce policies or low-level
actions, we advocate for a generative scheme, which uses these models to
automatically generate diversified tasks, scenes, and training supervisions,
thereby scaling up robotic skill learning with minimal human supervision. Our
approach equips a robotic agent with a self-guided propose-generate-learn
cycle: the agent first proposes interesting tasks and skills to develop, and
then generates corresponding simulation environments by populating pertinent
objects and assets with proper spatial configurations. Afterwards, the agent
decomposes the proposed high-level task into sub-tasks, selects the optimal
learning approach (reinforcement learning, motion planning, or trajectory
optimization), generates required training supervision, and then learns
policies to acquire the proposed skill. Our work attempts to extract the
extensive and versatile knowledge embedded in large-scale models and transfer
them to the field of robotics. Our fully generative pipeline can be queried
repeatedly, producing an endless stream of skill demonstrations associated with
diverse tasks and environments.
RoboVQA: Multimodal Long-Horizon Reasoning for Robotics
Authors: name: Pierre Sermanet, name: Tianli Ding, name: Jeffrey Zhao, name: Fei Xia, name: Debidatta Dwibedi, name: Keerthana Gopalakrishnan, name: Christine Chan, name: Gabriel Dulac-Arnold, name: Sharath Maddineni, name: Nikhil J Joshi, name: Pete Florence, name: Wei Han, name: Robert Baruch, name: Yao Lu, name: Suvir Mirchandani, name: Peng Xu, name: Pannag Sanketi, name: Karol Hausman, name: Izhak Shafran, name: Brian Ichter, name: Yuan Cao
Published: 2023-11-01 23:40:07

We present a scalable, bottom-up and intrinsically diverse data collection
scheme that can be used for high-level reasoning with long and medium horizons
and that has 2.2x higher throughput compared to traditional narrow top-down
step-by-step collection. We collect realistic data by performing any user
requests within the entirety of 3 office buildings and using multiple robot and
human embodiments. With this data, we show that models trained on all
embodiments perform better than ones trained on the robot data only, even when
evaluated solely on robot episodes. We find that for a fixed collection budget
it is beneficial to take advantage of cheaper human collection along with robot
collection. We release a large and highly diverse (29,520 unique instructions)
dataset dubbed RoboVQA containing 829,502 (video, text) pairs for
robotics-focused visual question answering. We also demonstrate how evaluating
real robot experiments with an intervention mechanism enables performing tasks
to completion, making it deployable with human oversight even if imperfect
while also providing a single performance metric. We demonstrate a single
video-conditioned model named RoboVQA-VideoCoCa trained on our dataset that is
capable of performing a variety of grounded high-level reasoning tasks in broad
realistic settings with a cognitive intervention rate 46% lower than the
zero-shot state of the art visual language model (VLM) baseline and is able to
guide real robots through long-horizon tasks. The performance gap with
zero-shot state-of-the-art models indicates that a lot of grounded data remains
to be collected for real-world deployment, emphasizing the critical need for
scalable data collection approaches. Finally, we show that video VLMs
significantly outperform single-image VLMs with an average error rate reduction
of 19% across all VQA tasks. Data and videos available at
https://robovqa.github.io
Controllable Music Production with Diffusion Models and Guidance
  Gradients
Authors: name: Mark Levy, name: Bruno Di Giorgi, name: Floris Weers, name: Angelos Katharopoulos, name: Tom Nickson
Published: 2023-11-01 16:01:01

We demonstrate how conditional generation from diffusion models can be used
to tackle a variety of realistic tasks in the production of music in 44.1kHz
stereo audio with sampling-time guidance. The scenarios we consider include
continuation, inpainting and regeneration of musical audio, the creation of
smooth transitions between two different music tracks, and the transfer of
desired stylistic characteristics to existing audio clips. We achieve this by
applying guidance at sampling time in a simple framework that supports both
reconstruction and classification losses, or any combination of the two. This
approach ensures that generated audio can match its surrounding context, or
conform to a class distribution or latent representation specified relative to
any suitable pre-trained classifier or embedding model.
LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation,
  Generation and Editing
Authors: name: Wei-Ge Chen, name: Irina Spiridonova, name: Jianwei Yang, name: Jianfeng Gao, name: Chunyuan Li
Published: 2023-11-01 15:13:43

LLaVA-Interactive is a research prototype for multimodal human-AI
interaction. The system can have multi-turn dialogues with human users by
taking multimodal user inputs and generating multimodal responses. Importantly,
LLaVA-Interactive goes beyond language prompt, where visual prompt is enabled
to align human intents in the interaction. The development of LLaVA-Interactive
is extremely cost-efficient as the system combines three multimodal skills of
pre-built AI models without additional model training: visual chat of LLaVA,
image segmentation from SEEM, as well as image generation and editing from
GLIGEN. A diverse set of application scenarios is presented to demonstrate the
promises of LLaVA-Interactive and to inspire future research in multimodal
interactive systems.
Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo
  Labelling
Authors: name: Sanchit Gandhi, name: Patrick von Platen, name: Alexander M. Rush
Published: 2023-11-01 10:45:07

As the size of pre-trained speech recognition models increases, running these
large models in low-latency or resource-constrained environments becomes
challenging. In this work, we leverage pseudo-labelling to assemble a
large-scale open-source dataset which we use to distill the Whisper model into
a smaller variant, called Distil-Whisper. Using a simple word error rate (WER)
heuristic, we select only the highest quality pseudo-labels for training. The
distilled model is 5.8 times faster with 51% fewer parameters, while performing
to within 1% WER on out-of-distribution test data in a zero-shot transfer
setting. Distil-Whisper maintains the robustness of the Whisper model to
difficult acoustic conditions, while being less prone to hallucination errors
on long-form audio. Distil-Whisper is designed to be paired with Whisper for
speculative decoding, yielding a 2 times speed-up while mathematically ensuring
the same outputs as the original model. To facilitate further research in this
domain, we make our training code, inference code and models publicly
accessible.
Grounding Visual Illusions in Language: Do Vision-Language Models
  Perceive Illusions Like Humans?
Authors: name: Yichi Zhang, name: Jiayi Pan, name: Yuchen Zhou, name: Rui Pan, name: Joyce Chai
Published: 2023-10-31 18:01:11

Vision-Language Models (VLMs) are trained on vast amounts of data captured by
humans emulating our understanding of the world. However, known as visual
illusions, human's perception of reality isn't always faithful to the physical
world. This raises a key question: do VLMs have the similar kind of illusions
as humans do, or do they faithfully learn to represent reality? To investigate
this question, we build a dataset containing five types of visual illusions and
formulate four tasks to examine visual illusions in state-of-the-art VLMs. Our
findings have shown that although the overall alignment is low, larger models
are closer to human perception and more susceptible to visual illusions. Our
dataset and initial findings will promote a better understanding of visual
illusions in humans and machines and provide a stepping stone for future
computational models that can better align humans and machines in perceiving
and communicating about the shared visual world. The code and data are
available at https://github.com/vl-illusion/dataset.
The Generative AI Paradox: "What It Can Create, It May Not Understand"
Authors: name: Peter West, name: Ximing Lu, name: Nouha Dziri, name: Faeze Brahman, name: Linjie Li, name: Jena D. Hwang, name: Liwei Jiang, name: Jillian Fisher, name: Abhilasha Ravichander, name: Khyathi Chandu, name: Benjamin Newman, name: Pang Wei Koh, name: Allyson Ettinger, name: Yejin Choi
Published: 2023-10-31 18:07:07

The recent wave of generative AI has sparked unprecedented global attention,
with both excitement and concern over potentially superhuman levels of
artificial intelligence: models now take only seconds to produce outputs that
would challenge or exceed the capabilities even of expert humans. At the same
time, models still show basic errors in understanding that would not be
expected even in non-expert humans. This presents us with an apparent paradox:
how do we reconcile seemingly superhuman capabilities with the persistence of
errors that few humans would make? In this work, we posit that this tension
reflects a divergence in the configuration of intelligence in today's
generative models relative to intelligence in humans. Specifically, we propose
and test the Generative AI Paradox hypothesis: generative models, having been
trained directly to reproduce expert-like outputs, acquire generative
capabilities that are not contingent upon -- and can therefore exceed -- their
ability to understand those same types of outputs. This contrasts with humans,
for whom basic understanding almost always precedes the ability to generate
expert-level outputs. We test this hypothesis through controlled experiments
analyzing generation vs. understanding in generative models, across both
language and image modalities. Our results show that although models can
outperform humans in generation, they consistently fall short of human
capabilities in measures of understanding, as well as weaker correlation
between generation and understanding performance, and more brittleness to
adversarial inputs. Our findings support the hypothesis that models' generative
capability may not be contingent upon understanding capability, and call for
caution in interpreting artificial intelligence by analogy to human
intelligence.
ChatCoder: Chat-based Refine Requirement Improves LLMs' Code Generation
Authors: name: Zejun Wang, name: Jia Li, name: Ge Li, name: Zhi Jin
Published: 2023-11-01 03:46:36

Large language models have shown good performances in generating code to meet
human requirements. However, human requirements expressed in natural languages
can be vague, incomplete, and ambiguous, leading large language models to
misunderstand human requirements and make mistakes. Worse, it is difficult for
a human user to refine the requirement. To help human users refine their
requirements and improve large language models' code generation performances,
we propose ChatCoder: a method to refine the requirements via chatting with
large language models. We design a chat scheme in which the large language
models will guide the human users to refine their expression of requirements to
be more precise, unambiguous, and complete than before. Experiments show that
ChatCoder has improved existing large language models' performance by a large
margin. Besides, ChatCoder has the advantage over refine-based methods and LLMs
fine-tuned via human response.
Text Rendering Strategies for Pixel Language Models
Authors: name: Jonas F. Lotz, name: Elizabeth Salesky, name: Phillip Rust, name: Desmond Elliott
Published: 2023-11-01 13:49:31

Pixel-based language models process text rendered as images, which allows
them to handle any script, making them a promising approach to open vocabulary
language modelling. However, recent approaches use text renderers that produce
a large set of almost-equivalent input patches, which may prove sub-optimal for
downstream tasks, due to redundancy in the input representations. In this
paper, we investigate four approaches to rendering text in the PIXEL model
(Rust et al., 2023), and find that simple character bigram rendering brings
improved performance on sentence-level tasks without compromising performance
on token-level or multilingual tasks. This new rendering strategy also makes it
possible to train a more compact model with only 22M parameters that performs
on par with the original 86M parameter model. Our analyses show that character
bigram rendering leads to a consistently better model but with an anisotropic
patch embedding space, driven by a patch frequency bias, highlighting the
connections between image patch- and tokenization-based language models.
AMSP: Super-Scaling LLM Training via Advanced Model States Partitioning
Authors: name: Qiaoling Chen, name: Qinghao Hu, name: Zhisheng Ye, name: Guoteng Wang, name: Peng Sun, name: Yonggang Wen, name: Tianwei Zhang
Published: 2023-11-01 03:14:48

Large Language Models (LLMs) have demonstrated impressive performance across
various downstream tasks. When training these models, there is a growing
inclination to process more tokens on larger training scales but with
relatively smaller model sizes. Zero Redundancy Optimizer (ZeRO), although
effective in conventional training environments, grapples with scaling
challenges when confronted with this emerging paradigm. To this end, we propose
a novel LLM training framework AMSP, which undertakes a granular partitioning
of model states, encompassing parameters (P), gradient (G), and optimizer
states (OS). Specifically, AMSP(1) builds a unified partitioning space,
enabling independent partitioning strategies for P, G, and OS; (2)
incorporates a scale-aware partitioner to autonomously search for optimal
partitioning strategies: (3) designs a dedicated communication optimizer to
ensure proficient management of data placement discrepancies arising from
diverse partitioning strategies. Our evaluations show that AMSP achieves up to
90.3% scaling efficiency across 1024 GPUs.
De-Diffusion Makes Text a Strong Cross-Modal Interface
Authors: name: Chen Wei, name: Chenxi Liu, name: Siyuan Qiao, name: Zhishuai Zhang, name: Alan Yuille, name: Jiahui Yu
Published: 2023-11-01 16:12:40

We demonstrate text as a strong cross-modal interface. Rather than relying on
deep embeddings to connect image and language as the interface representation,
our approach represents an image as text, from which we enjoy the
interpretability and flexibility inherent to natural language. We employ an
autoencoder that uses a pre-trained text-to-image diffusion model for decoding.
The encoder is trained to transform an input image into text, which is then fed
into the fixed text-to-image diffusion decoder to reconstruct the original
input -- a process we term De-Diffusion. Experiments validate both the
precision and comprehensiveness of De-Diffusion text representing images, such
that it can be readily ingested by off-the-shelf text-to-image tools and LLMs
for diverse multi-modal tasks. For example, a single De-Diffusion model can
generalize to provide transferable prompts for different text-to-image tools,
and also achieves a new state of the art on open-ended vision-language tasks by
simply prompting large language models with few-shot examples.
ChipNeMo: Domain-Adapted LLMs for Chip Design
Authors: name: Mingjie Liu, name: Teo Ene, name: Robert Kirby, name: Chris Cheng, name: Nathaniel Pinckney, name: Rongjian Liang, name: Jonah Alben, name: Himyanshu Anand, name: Sanmitra Banerjee, name: Ismet Bayraktaroglu, name: Bonita Bhaskaran, name: Bryan Catanzaro, name: Arjun Chaudhuri, name: Sharon Clay, name: Bill Dally, name: Laura Dang, name: Parikshit Deshpande, name: Siddhanth Dhodhi, name: Sameer Halepete, name: Eric Hill, name: Jiashang Hu, name: Sumit Jain, name: Brucek Khailany, name: Kishor Kunal, name: Xiaowei Li, name: Hao Liu, name: Stuart Oberman, name: Sujeet Omar, name: Sreedhar Pratty, name: Ambar Sarkar, name: Zhengjiang Shao, name: Hanfei Sun, name: Pratik P Suthar, name: Varun Tej, name: Kaizhe Xu, name: Haoxing Ren
Published: 2023-10-31 22:35:58

ChipNeMo aims to explore the applications of large language models (LLMs) for
industrial chip design. Instead of directly deploying off-the-shelf commercial
or open-source LLMs, we instead adopt the following domain adaptation
techniques: custom tokenizers, domain-adaptive continued pretraining,
supervised fine-tuning (SFT) with domain-specific instructions, and
domain-adapted retrieval models. We evaluate these methods on three selected
LLM applications for chip design: an engineering assistant chatbot, EDA script
generation, and bug summarization and analysis. Our results show that these
domain adaptation techniques enable significant LLM performance improvements
over general-purpose base models across the three evaluated applications,
enabling up to 5x model size reduction with similar or better performance on a
range of design tasks. Our findings also indicate that there's still room for
improvement between our current results and ideal outcomes. We believe that
further investigation of domain-adapted LLM approaches will help close this gap
in the future.
Does GPT-4 Pass the Turing Test?
Authors: name: Cameron Jones, name: Benjamin Bergen
Published: 2023-10-31 06:27:52

We evaluated GPT-4 in a public online Turing Test. The best-performing GPT-4
prompt passed in 41% of games, outperforming baselines set by ELIZA (27%) and
GPT-3.5 (14%), but falling short of chance and the baseline set by human
participants (63%). Participants' decisions were based mainly on linguistic
style (35%) and socio-emotional traits (27%), supporting the idea that
intelligence is not sufficient to pass the Turing Test. Participants'
demographics, including education and familiarity with LLMs, did not predict
detection rate, suggesting that even those who understand systems deeply and
interact with them frequently may be susceptible to deception. Despite known
limitations as a test of intelligence, we argue that the Turing Test continues
to be relevant as an assessment of naturalistic communication and deception. AI
models with the ability to masquerade as humans could have widespread societal
consequences, and we analyse the effectiveness of different strategies and
criteria for judging humanlikeness.
Unleashing the Power of Pre-trained Language Models for Offline
  Reinforcement Learning
Authors: name: Ruizhe Shi, name: Yuyao Liu, name: Yanjie Ze, name: Simon S. Du, name: Huazhe Xu
Published: 2023-10-31 16:24:17

Offline reinforcement learning (RL) aims to find a near-optimal policy using
pre-collected datasets. In real-world scenarios, data collection could be
costly and risky; therefore, offline RL becomes particularly challenging when
the in-domain data is limited. Given recent advances in Large Language Models
(LLMs) and their few-shot learning prowess, this paper introduces
Language Models for Motion Control (LaMo), a
general framework based on Decision Transformers to effectively use pre-trained
Language Models (LMs) for offline RL. Our framework highlights four crucial
components: (1) Initializing Decision Transformers with sequentially
pre-trained LMs, (2) employing the LoRA fine-tuning method, in contrast to
full-weight fine-tuning, to combine the pre-trained knowledge from LMs and
in-domain knowledge effectively, (3) using the non-linear MLP transformation
instead of linear projections, to generate embeddings, and (4) integrating an
auxiliary language prediction loss during fine-tuning to stabilize the LMs and
retain their original abilities on languages. Empirical results indicate
LaMo achieves state-of-the-art performance in sparse-reward tasks
and closes the gap between value-based offline RL methods and decision
transformers in dense-reward tasks. In particular, our method demonstrates
superior performance in scenarios with limited data samples. Our project
website is https://lamo2023.github.io