GPT4All: An ecosystem of open-source assistants that run on local
hardware.
Yuvanesh Anandyuvanesh@nomic.ai

Zach Nussbaum
zach@nomic.ai

Brandon Duderstadt
brandon@nomic.ai

Benjamin M. Schmidt
ben@nomic.ai

Adam Treat
treat.adam@gmail.com

Andriy Mulyar
andriy@nomic.ai

Abstract
We release two new models: GPT4All-J v1.3
Groovy an Apache-2 licensed chatbot, andGPT4All-13B-snoozy, a GPL licenced chatbot, trained over a massive curated corpusof assistant interactions including word problems, multi-turn dialogue, code, poems, songs,and stories. It builds on the previous GPT4All
releases by training on a larger and clearnercorpus, We openly release the training data,
data curation procedure, training code, and final model weights to promote open researchand reproducibility. Additionally, we release
Python bindings and support to our Chat UI
with 4 bit-quantized versions of our models,
allowing virtually anyone to run the model onCPU.

1

Data Collection and Curation

We collected roughly one million promptresponse pairs using the GPT-3.5-Turbo OpenAI
API between March 20, 2023 and March 26th,
2023. To do this, we first gathered a diverse sample of questions/prompts by leveraging five publicly available datasets:• The unified chip2 subset of LAION OIG.
• Coding questions with a random sub-sampleof Stackoverflow Questions• Instruction-tuning with a sub-sample of Bigscience/P3• Conversation Data from ShareGPT• Instruction Following Data curated for Dolly
Dolly (Conover et al.)
We additionally curated a creative-style dataset using GPT-3.5-Turbo to generate poems, short stories, and raps in the style of various artists.
Following our previous work in previous versions of GPT4All, we curated and cleaned the data

Figure 1: TSNE visualization of the training data including Stackoverflow, UnifiedChip, ShareGPT, Dolly,
and data generated by Nomic). The yellow clusters are
sections of creative prompts, such as asking for shortstores or raps in the style of.

using Atlas. We similarly filtered examples that
contained phrases like ”I’m sorry, as an AI language model” and responses where the model refused to answer the question. Adding ShareGPT(90k examples) and Dolly (15k examples), our total dataset size was approximately 900,000 datapoints including our previously published dataset
which contained 806,199 examples.
After cleaning the data for exact duplicates,
missing prompts and responses, and poorly formatted data, we realized that there existed many
data points that were semantically similar. We
tagged potential duplicates using Atlas and on inspection found that the clusters accurately contained semantic duplicates. Using these tags, wequickly removed approximately 8% of semantic
duplicates.
Our final dataset, which GPT4All-J-v1.3
Groovy and GPT4All-13b-snoozy were trainedon, contains 739,259 examples which is visualized
in 1

2.1

Reproducibility

We release all data, training code and logs for thecommunity to learn, build and benefit from. Pleasecheck the Git repository for the most up-to-date
data, training details and checkpoints.
2.2
Figure 2: Cluster of Semantically Similar Examples
Identified by Atlas Duplication Detection

Figure 3: TSNE visualization of the final GPT4All
training data, colored by extracted topic.

Running all of our experiments cost about $5000
in GPU costs. We gratefully acknowledge ourcompute sponsor Paperspace for their generosity in making GPT4All-J and GPT4All-13Bsnoozy training possible. Between GPT4All andGPT4All-J, we have spent about $800 in OpenAI API credits so far to generate the training
samples that we openly release to the community.
Our released model, GPT4All-J, can be trained inabout eight hours on a Paperspace DGX A100 8x80GB for a total cost of $200 while GPT4All-13Bsnoozy can be trained in about 1 day for a total
cost of $600. Using a government calculator, weestimate the model training to produce the equivalent of 0.18 and 0.54 metric tons of carbon dioxide
for GPT4All-J and GPT4All-13B-snoozy, roughly
equivalent to that produced by burning 20 gallons(75 liters) and 60 gallons (227 liters) of gasoline
respectively.

3
You can interactively explore the dataset at thefollowing links:• Deduplicated Dataset Indexed on Prompt• Deduplicated Dataset Indexed on Response
The final dataset has been added as a revisionnamed ‘v1.3-groovy‘ to GPT4All-J dataset. Hereyou can also find previous versions of our dataset.

2Model Training

We trained models finetuned from both LLaMA
13B (Touvron et al., 2023) and GPT-J (Wang andKomatsuzaki, 2021) checkpoints. The model associated with our initial public release is trainedwith LoRA (Hu et al., 2021) on the 437,605 postprocessed examples for four epochs while the finetuned GPT-J was trained for one epoch. Thesemodels that we release are full fine-tunes. Detailed
model hyper-parameters and training code can befound in the associated repository and model training log.

Costs

Evaluation

We performed a preliminary evaluation of ouroriginal model using the human evaluation datafrom the Self-Instruct paper (Wang et al., 2022).
We report the ground truth perplexity of our model
against what is, to our knowledge, the best openly
available alpaca-lora model, provided by user
chainyo on huggingface. We find that all models
have very large perplexities on a small number oftasks, and report perplexities clipped to a maximum of 100.
Models fine-tuned on this collected dataset exhibit much lower perplexity in the Self-Instructevaluation compared to Alpaca. This evaluation isin no way exhaustive and further evaluation work
remains. We welcome the reader to run the model
locally on CPU (see Github for files).
3.1

Common Sense Reasoning

Following results from (Conover et al.), we evaluate on 7 standard common sense reasoning tasks:
ARC easy and challenge (Clark et al., 2018),
BoolQ (Clark et al., 2019), HellaSwag (Zellerset al., 2019), OpenBookQA (Mihaylov et al.,ModelGPT4All-J 6B v1.0
GPT4All-J v1.1-breezy
GPT4All-J v1.2-jazzy
GPT4All-J v1.3-groovy
GPT4All-J Lora 6B
GPT4All LLaMa Lora 7B
GPT4All 13B snoozy
Dolly 6B
Dolly 12B
Alpaca 7B
Alpaca Lora 7B
GPT-J 6.7B
LLama 7B
LLama 13B
Pythia 6.7B
Pythia 12B
Fastchat T5
Fastchat Vicuña 7B
Fastchat Vicuña 13B
StableVicuña RLHF
StableLM Tuned
StableLM Base
Koala 13B
Open Assistant Pythia 12B
text-davinci-003

BoolQ
73.4
74.0
74.8
73.6
68.6
73.1
83.3
68.8
56.7
73.9
74.3
65.4
73.1
68.5
63.5
67.7
81.5
76.6
81.5
82.3
62.5
60.1
76.5
67.9
88.1

PIQA
74.8
75.1
74.9
74.3
75.8
77.6
79.2
77.3
75.4
77.2
79.3
76.2
77.4
79.1
76.3
76.6
64.6
77.2
76.8
78.6
71.2
67.4
77.9
78.0
83.8

HellaSwag
63.4
63.2
63.6
63.8
66.2
72.1
75.0
67.6
71.0
73.9
74.0
66.2
73.0
76.2
64.0
67.3
46.3
70.7
73.3
74.1
53.6
41.2
72.6
68.1
83.4

WinoGrande
64.7
63.6
63.8
63.5
63.5
67.8
71.3
63.9
62.2
66.1
68.8
64.1
66.9
70.1
61.1
63.8
61.8
67.3
66.7
70.9
54.8
50.1
68.8
65.0
75.8

ARC-e
54.9
55.4
56.6
57.7
56.4
51.1
60.9
62.9
64.659.8
56.6
62.2
52.5
60.0
61.3
63.9
49.3
53.5
57.4
61.0
52.4
44.9
54.3
64.2
83.9

ARC-c
36.0
34.9
35.3
35.0
35.7
40.4
44.2
38.7
38.5
43.3
43.9
36.6
41.4
44.6
35.2
34.8
33.3
41.2
42.7
43.5
31.1
27.0
41.0
40.4
63.9

OBQA
40.2
38.4
41.0
38.8
40.2
40.2
43.4
41.2
40.4
43.4
42.6
38.2
42.4
42.2
37.2
38
39.4
40.8
43.6
44.4
33.4
32.0
42.8
43.2
51.0

Table 1: Zero-shot performance on Common Sense Reasoning tasks. The highest performing non-OpenAI model
is bolded in each column. Note text-davinci-003 still beats every model on these tasks.

2018), PIQA (Bisk et al., 2020), and Winogrande(Sakaguchi et al., 2019). We evaluate several models: GPT-J (Wang and Komatsuzaki, 2021), Pythia(6B and 12B) (?), Dolly v1 and v2 (Conoveret al.), Alpaca (Taori et al., 2023), LLama (Touvron et al., 2023), Pythia 6B and 12B (Bidermanet al., 2023), FastChat T5 and Vicuña (Chianget al., 2023), StableVicuña and StableLM base andtuned (Stability-AI), Koala (Geng et al., 2023),
OpenAssistant 12B (Laion-Ai), text-davinci-003(Ouyang et al., 2022) and GPT4All using lmeval-harness (Gao et al., 2021). Similar to results in (Ouyang et al., 2022), instruction-tuning
showed some performance regressions over thebase model for. However, we notice in sometasks that the LoRA instruction finetuned models
as well as GPT4All-13B-snoozy trained on a large
cleaned dataset show some performance improvements.

4Use Considerations

The authors release data and training details inhopes that it will accelerate open LLM research,
particularly in the domains of fairness, alignment, interpretability, and transparency. GPT4AllJ model weights and quantized versions are released under an Apache 2 license and are freely
available for use and distribution. Please notethat the less restrictive license does not apply tothe original GPT4All and GPT4All-13B-snoozymodel that is based on LLaMA, which has a noncommercial GPL license. The assistant data wasgathered from OpenAI’s GPT-3.5-Turbo, whoseterms of use prohibit developing models that compete commercially with OpenAI.

References
Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, AviyaAvg.
58.2
57.8
58.6
58.1
58.1
60.3
65.3
60.1
58.4
62.4
62.8
58.4
61.0
63.0
57.0
58.9
53.7
61.0
63.1
65.0
51.3
42.2
62.0
61.0
75.7

Skowron, Lintang Sutawika, and Oskar van der Wal.
2023. Pythia: A suite for analyzing large language
models across training and scaling.

Paul Christiano, Jan Leike, and Ryan Lowe. 2022.
Training language models to follow instructions
with human feedback.

Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning
about physical commonsense in natural language. In
Thirty-Fourth AAAI Conference on Artificial Intelligence.

Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale.

Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng,
Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan
Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion
Stoica, and Eric P. Xing. 2023. Vicuna: An opensource chatbot impressing gpt-4 with 90%* chatgpt
quality.
Christopher Clark, Kenton Lee, Ming-Wei Chang,
Tom Kwiatkowski, Michael Collins, and Kristina
Toutanova. 2019. Boolq: Exploring the surprisingdifficulty of natural yes/no questions.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot,
Ashish Sabharwal, Carissa Schoenick, and Oyvind
Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge.
Mike Conover, Matt Hayes, Ankit Mathur, XiangruiMeng, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and et al.
Free dolly: Introducing the world’s first truly open
instruction-tuned llm.
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black,
Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang,
Anish Thite, Ben Wang, Kevin Wang, and AndyZou. 2021. A framework for few-shot language
model evaluation.
Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song.
2023. Koala: A dialogue model for academic research. Blog post.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan
Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, andWeizhu Chen. 2021. Lora: Low-rank adaptation oflarge language models.
Laion-Ai. Laion-ai/open-assistant: Openassistant is achat-based assistant that understands tasks, can interact with third-party systems, and retrieve information dynamically to do so.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish
Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang,
Sandhini Agarwal, Katarina Slama, Alex Ray, John
Schulman, Jacob Hilton, Fraser Kelton, Luke Miller,
Maddie Simens, Amanda Askell, Peter Welinder,

Stability-AI. Stability-ai/stablelm: Stablelm: Stabilityai language models.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, YannDubois, Xuechen Li, Carlos Guestrin, PercyLiang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca:
An instruction-following llama
model. https://github.com/tatsu-lab/stanford_alpaca.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
Martinet, Marie-Anne Lachaux, Timothée Lacroix,
Baptiste Rozière, Naman Goyal, Eric Hambro,
Faisal Azhar, Aurelien Rodriguez, Armand Joulin,
Edouard Grave, and Guillaume Lample. 2023.
Llama: Open and efficient foundation language
models.
Ben Wang and Aran Komatsuzaki. 2021.
GPTJ-6B: A 6 Billion Parameter Autoregressive
Language Model.
https://github.com/
kingoflolz/mesh-transformer-jax.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022. Self-instruct: Aligning language model with self generated instructions.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali
Farhadi, and Yejin Choi. 2019. Hellaswag: Can amachine really finish your sentence?