can't access WandB link
Hello, thanks for iterating and providing next version of pythia, which is excellent option to run common consumer hardware.
Cannot access the provided WandB link.
Could you please provide more information on how this model performs compared to other models like llama 30b?
will there be next version with more epochs or data? (considering pre in the title)
This model is only an intremediate step for training a new Pythia model .. it was "pre-trained" on a couple of different instruction-tuning datasets but not yet on the OA dataset. Training on OA is currently running and will be finished in a couple of hours.
Eagerly waiting for the finished model.
We published two SFT-v8 pythia 12b checkpoints (nice readme still has to be written):
To get an impression about the two checkpoints you can use our model comparer tool to view the sampling reports side by side.
Let us know which one you like better.
This is awesome. Thanks for all this effort. I'll be playing with those models right away for various use cases.