如何扩展到16K上下文呢
#8
by
plancktree
- opened
介绍里说的可由4K上下文扩展到16K如何扩展？
Try linear PI by modifying config.json and add
"rope_scaling": {
"factor": 4.0,
"type": "linear"
}
This works without training.Modify config.json and change rope_theta to 100,000 or 1,000,000, which is the ROPE base. Then you should do long context (16k, 32k) finetuning.
chencyudel
changed discussion status to
closed