beberik/Vertilake-11B

Just 11B RP experemental merge. For some time I wanted to do something similar.

Model used

• Sao10K/Fimbulvetr-11B-v2
• senseable/WestLake-7B-v2
• KatyTheCutie/LemonadeRP-4.5.3
• KoboldAI/Mistral-7B-Holodeck-1
• KoboldAI/Mistral-7B-Erebus-v3

Prompt template

Just use alpaca or chatml. Both works fine.

The secret sauce

lemolake-11B :

slices:
  - sources:
    - model: KatyTheCutie/LemonadeRP-4.5.3
      layer_range: [0, 24]
  - sources:
    - model: senseable/WestLake-7B-v2
      layer_range: [8, 32]

merge_method: passthrough
dtype: bfloat16

holobus-11B :

slices:
  - sources:
    - model: KoboldAI/Mistral-7B-Holodeck-1
      layer_range: [0, 24]
  - sources:
    - model: KoboldAI/Mistral-7B-Erebus-v3
      layer_range: [8, 32]

merge_method: passthrough
dtype: bfloat16

Vertilake-11B :

base_model: "Mistral-11B-v0.1"
models:
  - model: "Mistral-11B-v0.1"
    # no parameters necessary for base model
  - model: "Fimbulvetr-11B-v2"
    parameters:
      weight: 0.43
      density: 0.8
  - model: "lemolake-11b"
    parameters:
      weight: 0.6
      density: 0.8
  - model: "Holobus-11B"
    parameters:
      weight: 0.17
      density: 0.5
merge_method: dare_ties
parameters:
  int8_mask: true
dtype: bfloat16

I use mergekit for all the manipulation told here.

Thanks to the Undi95 for the original 11B mistral merge recipe.