How did you increase the size to 11b?
#5
by
froggeric
- opened
Please provide more details on your process, especially how did you increase the size to 11b? Did you do a self-merge? If so, with what parameters? At what stage did you do your additional training? What are the general topics and purpose of your dataset?
I am also working on extending Mistral based model through self-merges followed by additionnal training, and it would be great if we could share this info.
We used passthrough merging to increase the models size, then continued pretraining. The datasets is a general purpose dataset consisting of a wide variety of topics meant to increase the models general knowledge.
rombodawg
changed discussion status to
closed