|
--- |
|
license: other |
|
language: |
|
- en |
|
library_name: transformers |
|
pipeline_tag: text-generation |
|
tags: |
|
- causal-lm |
|
- text-generation-inference |
|
- merge |
|
--- |
|
|
|
# FOR EXPERIMENT |
|
|
|
## Description |
|
|
|
[**stabilityai/stablelm-zephyr-3b**](https://huggingface.co/stabilityai/stablelm-zephyr-3b), [**StableMed-3b**](https://huggingface.co/cxllin/StableMed-3b) merged with a new, experimental implementation of "dare ties" via mergekit. See: |
|
|
|
> [Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch](https://github.com/yule-BUAA/MergeLM) |
|
|
|
> https://github.com/cg123/mergekit/tree/dare |
|
|
|
|
|
## Usage |
|
|
|
`StableLM Zephyr 3B` uses the following instruction format: |
|
``` |
|
<|user|> |
|
List 3 synonyms for the word "tiny"<|endoftext|> |
|
<|assistant|> |
|
1. Dwarf |
|
2. Little |
|
3. Petite<|endoftext|> |
|
``` |
|
|
|
*** |
|
## Testing Notes |
|
|
|
Merged in mergekit with the following config, and the tokenizer from chargoddard's Yi-Llama: |
|
|
|
``` |
|
models: |
|
- model: stabilityai/stablelm-zephyr-3b |
|
# no parameters necessary for base model |
|
- model: cxllin/StableMed-3b |
|
parameters: |
|
weight: 0.08 |
|
density: 0.5 |
|
merge_method: dare_ties |
|
base_model: stabilityai/stablelm-zephyr-3b |
|
parameters: |
|
int8_mask: true |
|
dtype: bfloat16 |
|
``` |
|
|
|
## Model Details |
|
- License: [StabilityAI Non-Commercial Research Community License](https://huggingface.co/stabilityai/stablelm-zephyr-3b/raw/main/LICENSE) |
