Excalibur-7b

Update: A fine-tuned version of this model is now publicly available, along with benchmark results. If you're looking for a more conversational, assistant-style exchange you won't want to miss it!

Image generated with Envoid's Model9 SDXL model

GGUFs can be found here

Performance Comparison

Name	Avg.	ARC	HellaSwag	MMLU	TruthfulQA	Winogrande	GSM8K
Excalibur-7b	73.6	69.71	87.56	65.66	67.24	82.79	68.61
Magic-Dolphin-7b	67.48	65.78	85.61	64.64	58.01	79.64	51.18
merlinite-7b	64	63.65	84.52	64.91	50.15	79.72	41.09
* Open LLM Leaderboard Dataset

Methodology

Magic-Dolphin-7b was an unexpected surprise. Profoundly satisfied with it as a first attempt. For this follow-up I wanted to target the MMLU benchmark specifically. The challenge this time was placing more weight on Merlinite-7b as an unknown quantity that hasn't been in the spotlight despite its novel LAB tuning method.

Excalibur-7b builds on past success and is the culmination of several learnings:

• Measuring KL-divergences for new quantization types brought a deeper understanding of benchmarking and assessing model performance
• This signifcantly sped up the testing process by using MMLU as a base, narrowing down over 10 candidate linear merges to 1: merliniteX-blockB1
• Reaching the limitations of linear merging necessitated a pivot to reviewing the viability of SLERP, DARE-TIES, and Passthrough methods
• Thus a competing candidate merge pool was tested between different merge algorithms. Once more the list was narrowed from 10 candidates to 1: merliniteX-blockF2
• merliniteX-blockF2 (SLERP of Magic-Dolphin-7B and jaskier-7b-dpo in unorthadox proportions) was originally planned for release earlier this week
• Instead -blockB1 and -blockF2 were merged and the results were placed head to head in a final round of tests. Ultimately a more conventional execution of SLERP showed the best results for the final step.

Sample Question

Bonus Question - Vision Capabilities

Requires additional mistral-7b-mmproj-v1.5-Q4_1.gguf file for vision functionality

Select up the gguf file of your choice in Kobold as usual, then make sure to choose the mmproj file above in the LLaVA mmproj field of the model submenu:

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the SLERP merge method.

Models Merged

The following models were included in the merge:

• ibm/merlinite-7b
• InferenceIllusionist/Magic-Dolphin-7b
• SanjiWatsuki/Kunoichi-DPO-v2-7B
• mlabonne/Monarch-7B
• bardsai/jaskier-7b-dpo-v6.1

Configuration

The following YAML configurations were used to produce this model:

merliniteX-blockB1

models:
  - model: models/merlinite-7b
    parameters:
      weight: 1.0
  - model: models/Kunoichi-DPO-v2-7B
    parameters:
      weight: 0.2
  - model: models/jaskier-7b-dpo-v6.1
    parameters:
      weight: 0.6
  - model: models/Monarch-7b
    parameters:
      weight: 0.4
merge_method: linear
dtype: float16

merliniteX-blockF2

slices:
  - sources:
      - model: models/Magic-Dolphin-7b
        layer_range: [0, 32]
      - model: models/jaskier-7b-dpo-v6.1
        layer_range: [0, 32]
merge_method: slerp
base_model: models/Magic-Dolphin-7b
parameters:
  t:
    - filter: self_attn
      value: [0, 0.5, 0.3, 0.7, 0.5, 1]
    - filter: mlp
      value: [1, 0.5, 0.7, 0.3, 0.5, 0]
    - value: 0.5 # fallback for rest of tensors
dtype: float16

merliniteX-blockH1 (Excalibur-7b)

slices:
  - sources:
      - model: models/merliniteX-blockF2
        layer_range: [0, 32]
      - model: models/merliniteX-blockB1
        layer_range: [0, 32]
merge_method: slerp
base_model: models/merliniteX-blockF2
parameters:
  t:
    - filter: self_attn
      value: [1, 0.7, 0.3, 0.5, 0]
    - filter: mlp
      value: [0, 0.3, 0.7, 0.5, 1]
    - value: 0.5 # fallback for rest of tensors
dtype: float16