Update README.md

README.md

@@ -8,6 +8,10 @@ This is EleutherAI/pythia-410m finetuned on OpenAssistant/oasst_top1_2023-08-25
 # Why 
 Plain and simple. Im experimenting with making instruction LLMs under 1B params. I think we can still squeeze out better performance out of these models. 
 
+# Random Notes
+- Only using OpenAssistant data gives fantastic results becuase of its high quality. I like the top1 dataset becuase of it's lack of prompt refusals.
+- Prompt refusals have been shown to damage the performance of instruction LLMs. My theory is that the model "spends" parameters learning how to refuse prompts rather than learning actually useful information. Adding to this, I think that unlike other tasks, learning prompt refusals most likely has no other value in terms of transfer learning.
+
 # Usage
 ```
 from transformers import pipeline