Isotonic
/

smol_llama-4x220M-MoE

Text Generation

Mixture of Experts

BEE-spoke-data/smol_llama-220M-openhermes

BEE-spoke-data/beecoder-220M-python

BEE-spoke-data/zephyr-220m-sft-full

BEE-spoke-data/zephyr-220m-dpo-full

Inference Endpoints

text-generation-inference

Model card Files Files and versions Community

Isotonic commited on Feb 27

Commit

416e2cc

•

1 Parent(s): 5d2a0bc

Update config.json

Files changed (1) hide show

config.json +1 -1

config.json CHANGED Viewed

@@ -14,7 +14,7 @@
   "max_position_embeddings": 2048,
   "model_type": "mixtral",
   "num_attention_heads": 32,
-  "num_experts_per_tok": 3,
   "num_hidden_layers": 10,
   "num_key_value_heads": 8,
   "num_local_experts": 4,

   "max_position_embeddings": 2048,
   "model_type": "mixtral",
   "num_attention_heads": 32,
+  "num_experts_per_tok": 2,
   "num_hidden_layers": 10,
   "num_key_value_heads": 8,
   "num_local_experts": 4,