LiteMOE-4x460m

LiteMOE-4x460m is a merge of the following models using mergekit:

🧩 Configuration

base_model: ahxt/LiteLlama-460M-1T
gate_mode: random
dtype: bfloat16
experts:
  - source_model: ahxt/LiteLlama-460M-1T
    positive_prompts: [""]
  - source_model: ahxt/LiteLlama-460M-1T
    positive_prompts: [""]
  - source_model: ahxt/LiteLlama-460M-1T
    positive_prompts: [""]
  - source_model: ahxt/LiteLlama-460M-1T
    positive_prompts: [""]

Needs finetuning