Undi95/UtopiaXL-13B · Hugging Face

Description

This repo contains fp16 files of UtopiaXL-13B, a merge I have done with the new layer shuffle method from mergekit (thank you Charles for adding my request to your project!)

This is more a proof of concept showing the following:

Llama2 is very flexible
Llama2 don't care about what is finetuned on the layers specifically if you keep them in the same order
Clean merge (no ties, no SLERP, etc...) with only layer is possible without breaking something
Deleting special tokens/using model with special token don't break the model
Alpaca win, always. So use it.

The name "XL" come from the absurd amount of model pushed into it.

Models and loras used

The sauce

!mergekit-layershuffle ./UtopiaXL \
  --model Undi95/Utopia-13B --weight 0.4 \
  --model KoboldAI/LLAMA2-13B-Holodeck-1 --weight 0.1 \
  --model Undi95/PsyMedRP-v1-13B --weight 0.1 \
  --model PygmalionAI/pygmalion-2-13b --weight 0.25 \
  --model Heralax/Cat-0.5 --weight 0.1 \
  --model KoboldAI/LLaMA2-13B-TiefighterLR --weight 0.1 \
  --model Heralax/Augmental-13b-two-epochs --weight 0.1 \
  --write-yaml UtopiaXL.yaml

=========================

merge_method: passthrough
slices:
- sources:
  - layer_range:
    - 0
    - 1
    model: KoboldAI/LLAMA2-13B-Holodeck-1
- sources:
  - layer_range:
    - 1
    - 4
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 4
    - 5
    model: PygmalionAI/pygmalion-2-13b
- sources:
  - layer_range:
    - 5
    - 6
    model: Undi95/PsyMedRP-v1-13B
- sources:
  - layer_range:
    - 6
    - 7
    model: KoboldAI/LLaMA2-13B-TiefighterLR
- sources:
  - layer_range:
    - 7
    - 8
    model: PygmalionAI/pygmalion-2-13b
- sources:
  - layer_range:
    - 8
    - 9
    model: Undi95/PsyMedRP-v1-13B
- sources:
  - layer_range:
    - 9
    - 10
    model: PygmalionAI/pygmalion-2-13b
- sources:
  - layer_range:
    - 10
    - 13
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 13
    - 14
    model: Heralax/Cat-0.5
- sources:
  - layer_range:
    - 14
    - 17
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 17
    - 18
    model: Heralax/Augmental-13b-two-epochs
- sources:
  - layer_range:
    - 18
    - 19
    model: KoboldAI/LLaMA2-13B-TiefighterLR
- sources:
  - layer_range:
    - 19
    - 22
    model: PygmalionAI/pygmalion-2-13b
- sources:
  - layer_range:
    - 22
    - 23
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 23
    - 25
    model: PygmalionAI/pygmalion-2-13b
- sources:
  - layer_range:
    - 25
    - 27
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 27
    - 28
    model: Heralax/Cat-0.5
- sources:
  - layer_range:
    - 28
    - 30
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 30
    - 31
    model: Heralax/Augmental-13b-two-epochs
- sources:
  - layer_range:
    - 31
    - 32
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 32
    - 33
    model: Heralax/Cat-0.5
- sources:
  - layer_range:
    - 33
    - 34
    model: Heralax/Augmental-13b-two-epochs
- sources:
  - layer_range:
    - 34
    - 35
    model: Undi95/PsyMedRP-v1-13B
- sources:
  - layer_range:
    - 35
    - 36
    model: Heralax/Augmental-13b-two-epochs
- sources:
  - layer_range:
    - 36
    - 37
    model: Undi95/Utopia-13B
- sources:
  - layer_range:
    - 37
    - 38
    model: KoboldAI/LLAMA2-13B-Holodeck-1
- sources:
  - layer_range:
    - 38
    - 39
    model: PygmalionAI/pygmalion-2-13b
- sources:
  - layer_range:
    - 39
    - 40
    model: KoboldAI/LLAMA2-13B-Holodeck-1

=========================

=> Applying Undi95/Storytelling-v2.1-13B-lora x 0.1
=> Trained on LimaRP for +2h
=> Applying Undi95/LimaRP-UtopiaXL-13B-v3-lora x 0.35

Prompt template: Alpaca

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{prompt}

### Response:

A big thanks to Charles for adding the layer shuffle method to his tool mergekit and Henky/KoboldAI for the machine he let me use.

If you want to support me, you can here.