labrabbit7b
This is a merge of pre-trained language models created using mergekit.
Another (failed) experiment of mine. The writing style on this model is good, but it suffers from severe wrap-up bias especially in storywriting applications.
Merge Details
Merge Method
This model was merged using SLERP and the task arithmetic merge method using mistralai/Mistral-7B-v0.1 as a base.
Models Merged
The following models were included in the merge:
- sethuiyer/Dr_Samantha_7b_mistral
- teknium/Mistral-Trismegistus-7B
- cognitivecomputations/dolphin-2.6-mistral-7b
- Open-Orca/Mistral-7B-OpenOrca
- lemonilia/LimaRP-Mistral-7B-v0.1
Configuration
The following YAML configuration was used to produce this model:
models:
- model: cognitivecomputations/dolphin-2.6-mistral-7b
- model: Open-Orca/Mistral-7B-OpenOrca
merge_method: slerp
base_model: cognitivecomputations/dolphin-2.6-mistral-7b
parameters:
t:
- filter: self_attn
value: [0, 0.5, 0.3, 0.7, 1]
- filter: mlp
value: [1, 0.5, 0.7, 0.3, 0]
- value: 0.5 # fallback for rest of tensors
dtype: float16
name: cetacean-slerp-7b
---
base_model: mistralai/Mistral-7B-v0.1
dtype: float16
merge_method: task_arithmetic
slices:
- sources:
- layer_range: [0, 32]
model: mistralai/Mistral-7B-v0.1
- layer_range: [0, 32]
model: cetacean-slerp-7b
parameters:
weight: 1.0
- layer_range: [0, 32]
model: sethuiyer/Dr_Samantha_7b_mistral
parameters:
weight: 0.2
- layer_range: [0, 32]
model: teknium/Mistral-Trismegistus-7B
parameters:
weight: 0.15
- layer_range: [0, 32]
model: lemonilia/LimaRP-Mistral-7B-v0.1
parameters:
weight: 0.1
- Downloads last month
- 1