Infos

Pythia-1.4b supervised finetuned with Anthropic-hh-rlhf dataset for 1 epoch.

wandb log

See Pythia-1.4b for model details (paper).

Benchmark raw results:

Results for the base model are taken from the Pythia paper.

Zero shot

Task 1.4B_base 1.4B_sft
Lambada (OpenAI) 0.616 ± 0.007 0.5977 ± 0.0068
PIQA 0.711 ± 0.011 0.7133 ± 0.0106
WinoGrande 0.573 ± 0.014 0.5793 ± 0.0139
WSC 0.365 ± 0.047 0.3654 ± 0.0474
ARC - Easy 0.606 ± 0.010 0.6098 ± 0.0100
ARC - Challenge 0.260 ± 0.013 0.2696 ± 0.0130
SciQ 0.865 ± 0.011 0.8540 ± 0.0112
LogiQA 0.210 ± 0.016 NA

Five shot

Task 1.4B_base 1.4B_sft
Lambada (OpenAI) 0.578 ± 0.007 0.5201 ± 0.007
PIQA 0.705 ± 0.011 0.7176 ± 0.0105
WinoGrande 0.580 ± 0.014 0.5793 ± 0.0139
WSC 0.365 ± 0.047 0.5288 ± 0.0492
ARC - Easy 0.643 ± 0.010 0.6376 ± 0.0099
ARC - Challenge 0.290 ± 0.013 0.2935 ± 0.0133
SciQ 0.92 ± 0.009 0.9180 ± 0.0087
LogiQA 0.240 ± 0.017 N/A
Downloads last month
21
Safetensors
Model size
1.52B params
Tensor type
F32
·
BOOL
·
Inference Examples
This model does not have enough activity to be deployed to Inference API (serverless) yet. Increase its social visibility and check back later, or deploy to Inference Endpoints (dedicated) instead.

Dataset used to train Leogrin/eleuther-pythia1.4b-hh-sft