Crystalcareai
/

GemMoE-Beta-1

Text Generation

Model card Files Files and versions Community

Crystalcareai commited on Mar 14

Commit

c28e2ee

•

1 Parent(s): 66c1565

Update configuration_gemmoe.py

Files changed (1) hide show

configuration_gemmoe.py +2 -2

configuration_gemmoe.py CHANGED Viewed

@@ -156,8 +156,8 @@ class GemmoeConfig(PretrainedConfig):
         attention_dropout=0.0,
         num_experts_per_tok=2,
         num_local_experts=8,
-        n_shared_experts=None,
-        n_routed_experts=None,
         moe_layer_freq=1,
         first_k_dense_replace=0,
         norm_topk_prob=False,

         attention_dropout=0.0,
         num_experts_per_tok=2,
         num_local_experts=8,
+        n_shared_experts=8,
+        n_routed_experts=2,
         moe_layer_freq=1,
         first_k_dense_replace=0,
         norm_topk_prob=False,