MaziyarPanahi

/

Yarn-Solar-10b-64k-GGUF

like 1

Hi @Varkoyote

I remember when I was converting this model I had an issue with the rope config.

"rope_scaling": {
    "factor": 16.0,
    "original_max_position_embeddings": 4096,
    "type": "yarn"
  },

So it is yarn and it's supported in Llama.cpp, however, it needed another filed called finetuned:

if rope_scaling is not None and (typ := rope_scaling.get("type")):
            rope_factor = rope_scaling.get("factor")
            f_rope_scale = rope_factor
            if typ == "linear":
                rope_scaling_type = gguf.RopeScalingType.LINEAR
            elif typ == "yarn":
                rope_scaling_type = gguf.RopeScalingType.YARN
                n_orig_ctx = rope_scaling['original_max_position_embeddings']
                rope_finetuned = rope_scaling['finetuned']
            else:
                raise NotImplementedError(f'Unknown rope scaling type: {typ}')

So I added this field and set the value to True and then it worked.

What I can do is to locally test this, change this value to False and test again to see if it improves anything.