appvoid/no-prompt-1.3b

no-prompt

a sheared-llama-1.3b fine-tuning

This model uses an 1.3 billion parameters model as base to be further fine-tuned on the same data as palmer. It works pretty good and even surpasses sota model on hellaswag.

evaluation

Model	ARC_C	HellaSwag	PIQA	Winogrande
tinyllama-2t	0.2807	0.5463	0.7067	0.5683
palmer-001	0.2807	0.5524	0.7106	0.5896
sheared-1.3b	0.2910	0.5935	0.7339	0.5809
no-prompt-1.3b	0.3157	0.6022	0.7334	0.5864
falcon-rw-1b-instruct-openorca (sota)	0.3362	0.5997	0.7394	0.6148

This model was trained on less than 25% of the dataset yet achieves competitive performance to current sota on open llm leaderboard.

training

Training took ~5 P100 gpu hours. It was trained on 15,000 gpt-4 shuffled samples. no-prompt was fine-tuned using lower learning rates ensuring it keeps as much general knowledge as possible.

prompt

no prompt

limitations

Hallucinations are frequent, just as any transformer model this size.