Update README.md

README.md

@@ -7,7 +7,9 @@ datasets:
 - atom-in-the-universe/fanfics-10k-50k
 ---
 
-- **At least Huggingface Transformers [4.31.0](https://pypi.org/project/transformers/4.31.0/) is required to load this model!**
+Note: **At least Huggingface Transformers [4.31.0](https://pypi.org/project/transformers/4.31.0/) is required to load this model!**
+
+
 - base model: [meta-llama/Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b)
 - License: [Llama 2 Community License Agreement](https://ai.meta.com/resources/models-and-libraries/llama-downloads/)
 - wandb: [public-sft/runs/2jfazjt9](https://wandb.ai/open-assistant/public-sft/runs/2jfazjt9)
@@ -23,17 +25,19 @@ HF transformers >=4.31.0 is installed (`pip install transformers>=4.31.0`).
 
 ## Conversation Template
 
+For the initial response use (the system message is optional):
+
 ```
 <|system|>system message</s><|prompter|>user prompt</s><|assistant|>
 ```
 
-For multi-turn conversations:
+For multi-turn conversations use:
 
 ```
 <|system|>system message</s><|prompter|>Q1</s><|assistant|>A1</s><|prompter|>Q2</s><|assistant|>
 ```
 
-The model was trained with the following 16 system messages that were used to generate the training examples (see [ORCA paper](https://arxiv.org/abs/2306.02707)):
+The model was trained with the following 16 system messages used to generate the training examples (see [ORCA paper](https://arxiv.org/abs/2306.02707)):
 
 1. \<empty system message\>
 2. You are an AI assistant. Provide a detailed answer so user don’t need to search outside to understand the answer.
@@ -53,10 +57,19 @@ The model was trained with the following 16 system messages that were used to ge
 16. You are an AI assistant that helps people find information.
 
 
-## Orca-Chat/Dolphin Datasets
+## Datasets: Orca-Chat/Dolphin, RedPajama1T & FanFics
+
+This model was trained on:
 
-This model is trained on []()
-https://huggingface.co/datasets/ehartford/dolphin
+- [shahules786/orca-chat](https://huggingface.co/datasets/shahules786/orca-chat)
+- [togethercomputer/RedPajama-Data-1T-Sample](https://huggingface.co/datasets/togethercomputer/RedPajama-Data-1T)
+- [atom-in-the-universe/fanfics-10k-50k](https://huggingface.co/datasets/atom-in-the-universe/fanfics-10k-50k)
+  
+The dataset [shahules786/orca-chat](https://huggingface.co/datasets/shahules786/orca-chat) combines similar 
+examples of the GPT-4 subset of [ehartford/dolphin](https://huggingface.co/datasets/ehartford/dolphin) to form longer conversations 
+to improve long-context trainig.
+
+RedPajama and FanFics were additionally used for classic language modelling to fine-tune the RoPE scaling for 8k context size.
 
 
 ## Model Configuration
@@ -110,6 +123,10 @@ llama2_13b_orca_8k:
   peft_model: false
 ```
 
+# Special Thanks
+
+We want to especially thank Eric Hardford for replicating ORCA and making it publicly available at [ehartford/dolphin](https://huggingface.co/datasets/ehartford/dolphin)!
+
 # License
 
 - Llama 2 is licensed under the LLAMA 2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved.