Is there a guide on contiuing training of Dolly 2.0's model ( not just fine-tuning )?
I would like to continue training the model with a data set of another language. I would like to start with the pre-trained model. The examples I've seen have mostly been around fine-tuning but I want to be able to re-train the base model with new raw data of a new language.
Thanks!
Well, Dolly was created by fine-tuning Pythia. Do you want to fine-tune Pythia? that's what https://github.com/databrickslabs/dolly does. Do you want to tune Pythia from scratch? that would cost hundreds of thousands of dollars. That code is at https://github.com/EleutherAI/pythia
It's possible I'm misunderstanding the training part. The way I understand this, Dolly 2.0 simply fine-tuned the Pythia model and then a new model was generated. I have a bunch of raw data ( not instructions ) that I would like to feed into the model to improve it's understanding of that particular language.
What would be the best path for this? Would it be to re-train Pythia with this new data and then re-run the Dolly 2.0 training in essence? Can't I just continue training dolly's model and feed it raw data rather than specific instructions ?
You would fine-tune Pythia then. That's probably your best bet, not pre-training it from scratch. Yes you could then try instruction-tuning the result.
Fine-tuning dolly without instructions would tend to lose the instruction tuning and there is not much point in that vs fine-tuning Pythia.