What is the base model for Tiny KAI-#B and others? #1
#2
by
ibibek
- opened
Thank you for open-sourcing the models.
Are these models pre-trained from scratch using the RedPajama Dataset or are they derivatives of other base model?
Keynote-Technology/TinyKAI-1B-v0.1
Keynote-Technology/TinyKAI-3B-v0.1
The TinyKAI models are all improved versions of existing models, and they are all pretrained on RedPajamaData. However, we do train them on our own dataset, PLANE-2K, but they are NOT completely trained from scratch.