Context window

by tapos999 - opened May 2

May 2

Hi, really nice fine-tune specially working with German language usecases. Any plan to extend context window or something similar like llama3-gradient?

Thanks

DavidGF

VAGO solutions org Jun 3

Hey @tapos999
you can change the model config.json:

from

rope_scaling": null,

  "rope_scaling": {
     "type": "dynamic",
     "factor": 4.0
  },

and you get a 32K context size with no performance decrease!

HF,
David

DavidGF changed discussion status to closed Jun 3

tapos999

Jul 4

Hi @DavidGF thanks for responding. One more question, I am deploying it via vllm, I see there's also a rope_theta which is None. would you recommend any specific rope_theta value with this factor 4.0?

Thanks again

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment