Update README.md

README.md

@@ -7,33 +7,34 @@ datasets:
 
 # tinyChat: Instruction-Based LLM, <1% the size of GPT-3
 
-Introducing tinyChat, the instruction-based Large Language Model (LLM) that’s less than 1% the size of GPT-3.5. tinyChat is an open-source model under the Apache 2.0 license and based on Google’s Flan-T5-Large, a 770m parameter model. Although not as performant as larger models, tinyChat can perform a variety of NLP tasks such as summarization, question answering, and sentiment analysis.
+Introducing tinyChat, the instruction-based Large Language Model (LLM) that’s less than 1% the size of GPT-3.5. tinyChat is an open-source model under the Apache 2.0 license and based on Google’s Flan-T5-Large, a 770m parameter model. By fine tuning on the databricks-dolly-15k dataset, tinyChat demonstrates improved outputs on a range of tasks compared to Flan-T5. Although not as performant as larger models, tinyChat can perform a variety of NLP tasks such as summarization, question answering, and sentiment analysis using instruction prompts.
 
-tinyChat is available on the HuggingFace model hub and the code repository is on GitHub. While tinyChat is open-sourced, we do not recommend using it in a production setting in its current state.
+tinyChat is available on the HuggingFace model hub and the code repository is on [GitHub](https://github.com/Leadmatic/tinyChat). 
 
 
-## Use Cases
+## Dataset
 
-- Chatbots
-- Summarization
-- Sentiment analysis
-- Q&A systems
-- Text completion
-- Language modeling
-- Mobile applications
-- Complementing larger LLMs
+[databricks-dolly-15k](https://huggingface.co/datasets/databricks/databricks-dolly-15k) - databricks-dolly-15k is an open source dataset of instruction-following records generated by thousands of Databricks employees in several of the behavioral categories outlined in the InstructGPT paper, including brainstorming, classification, closed QA, generation, information extraction, open QA, and summarization.
 
-## Future Directions
 
-- Improving model accuracy
-- Reducing biases and toxicity
-- Developing new datasets
-- Collaborating with the open-source community
-- Applying tinyChat to new domains
+## Benchmark
 
-## Acknowledgements
+The following are results from different models tested on the EleutherAI LLM Evaluation Harness. These results indicate that tinyChat is not as performant as the other models. It also shows that tinyChat is only slightly better than Flan-t5-large on openbookqa while performing worse on other datasets. However, tinyChat shows better outputs when provided with creative prompts compared to its base model. See [blog post](https://leadmaticv3.webflow.io/blog/tinychat) for examples. This shows the limitations of these benchmarks for evaluating generative models.
 
-We express our gratitude to OpenAI, Hugging Face, Microsoft Research, and the creators of the Pile, Alpaca, and Databricks 15k datasets for their contributions to the landscape of open-source machine learning and the advancement of generative AI.
+| model                    | openbookqa | arc_easy | winogrande | hellaswag | arc_challenge | piqa    | boolq   |
+|--------------------------|-----------|----------|------------|-----------|---------------|---------|---------|
+| cerebras/Cerebras-GPT-13B | 0.36      | 0.598906 | 0.607735   | 0.593109  | 0.325939      | 0.749728 | 0.611621|
+| EleutherAI/gpt-j-6B       | 0.382     | 0.621633 | 0.651144   | 0.662617  | 0.363481      | 0.761153 | 0.655963|
+| dolly-v1-6b (1 epoch)     | 0.428     | 0.608586 | 0.633781   | 0.650568  | 0.377133      | 0.761697 | 0.69633 |
+| dolly-v1-6b (10 epochs)   | 0.41      | 0.62963  | 0.643252   | 0.676758  | 0.384812      | 0.773667 | 0.687768|
+| EleutherAI/gpt-neox-20b   | 0.402     | 0.683923 | 0.656669   | 0.7142    | 0.408703      | 0.784004 | 0.695413|
+| google/flan-t5-large      | 0.3120    | 0.5724   | 0.5991     | 0.4871    | 0.3072        | 0.7220   | 0.8645  |
+| leadmatic/tinyChat        | 0.3320    | 0.4811   | 0.5825     | 0.4519    | 0.2961        | 0.7073   | 0.8358  |
+
+
+## Limitations
+
+tinyChat is prone to hallucination and displays model bias. It is under active development and is currently intended for research purposes only.
 
 ## Running the Code
 
@@ -50,4 +51,4 @@ peft_model = PeftModel.from_pretrained(base_model, peft_model_id)
 inputs = tokenizer("""[INSERT INSTRUCTION HERE]""", return_tensors="pt")
 outputs = peft_model.generate(**inputs, max_length=300, do_sample=True)
 print(tokenizer.batch_decode(outputs, skip_special_tokens=True))
-```
+```