*drumroll please*
Introducing Tinypus!
I passthrough merged base Tiny Llama Chat with itself, then fine-tuned with around 1/3 of Platypus dataset.
Observations:
It's smarter (I think?)
It sometimes throws "### Instruction:" line. This could be due to the platypus dataset, or the fact that I know jackshit about programming. You can add it to "custom stopping strings" in oobaboga.
It may be possible to train very specialized mini experts and merge them???
Template
Same with TinyLlama/TinyLlama-1.1B-Chat-v1.0
Merge details
slices:
sources:
model: E://text-generation-webui//models//TinyLlama
layer_range: [0, 12]
sources:
model: E://text-generation-webui//models//TinyLlama
layer_range: [4, 22]
merge_method: passthrough
dtype: bfloat16
QLoRA Details
Chunk Length: 1152 R/A: 64/128 Epoch: 1 q-k-v-o
- Downloads last month
- 3,173