tinyChat: Instruction-Based LLM, <1% the size of GPT-3

Introducing tinyChat, the instruction-based Large Language Model (LLM) that’s less than 1% the size of GPT-3.5. tinyChat is an open-source model under the Apache 2.0 license and based on Google’s Flan-T5-Large, a 770m parameter model. By fine tuning on the databricks-dolly-15k dataset, tinyChat demonstrates improved outputs on a range of tasks compared to Flan-T5. Although not as performant as larger models, tinyChat can perform a variety of NLP tasks such as summarization, question answering, and sentiment analysis using instruction prompts.

tinyChat is available on the HuggingFace model hub and the code repository is on GitHub.

Dataset

databricks-dolly-15k - databricks-dolly-15k is an open source dataset of instruction-following records generated by thousands of Databricks employees in several of the behavioral categories outlined in the InstructGPT paper, including brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization.

Benchmark

The following are results from different models tested on the EleutherAI LLM Evaluation Harness. These results indicate that tinyChat is not as performant as the other models. It also shows that tinyChat is only slightly better than Flan-t5-large on openbookqa while performing worse on other datasets. However, tinyChat shows better outputs when provided with creative prompts compared to its base model. See blog post for examples. This shows the limitations of these benchmarks for evaluating generative models.

model	openbookqa	arc_easy	winogrande	hellaswag	arc_challenge	piqa	boolq
cerebras/Cerebras-GPT-13B	0.36	0.598906	0.607735	0.593109	0.325939	0.749728	0.611621
EleutherAI/gpt-j-6B	0.382	0.621633	0.651144	0.662617	0.363481	0.761153	0.655963
dolly-v1-6b (1 epoch)	0.428	0.608586	0.633781	0.650568	0.377133	0.761697	0.69633
dolly-v1-6b (10 epochs)	0.41	0.62963	0.643252	0.676758	0.384812	0.773667	0.687768
EleutherAI/gpt-neox-20b	0.402	0.683923	0.656669	0.7142	0.408703	0.784004	0.695413
google/flan-t5-large	0.3120	0.5724	0.5991	0.4871	0.3072	0.7220	0.8645
leadmatic/tinyChat	0.3320	0.4811	0.5825	0.4519	0.2961	0.7073	0.8358

Limitations

tinyChat is prone to hallucination and displays model bias. It is under active development and is currently intended for research purposes only.

Running the Code

import transformers
from peft import PeftModel

model_name = "google/flan-t5-large"
peft_model_id = "Leadmatic/tinyChat"
tokenizer = transformers.AutoTokenizer.from_pretrained(model_name)
base_model = transformers.AutoModelForSeq2SeqLM.from_pretrained(model_name)
peft_model = PeftModel.from_pretrained(base_model, peft_model_id)

inputs = tokenizer("""[INSERT INSTRUCTION HERE]""", return_tensors="pt")
outputs = peft_model.generate(**inputs, max_length=300, do_sample=True)
print(tokenizer.batch_decode(outputs, skip_special_tokens=True))