Crystalcareai
/

GemMoE-Beta-1

Text Generation

Model card Files Files and versions Community

Crystalcareai commited on Mar 13

Commit

7aaee95

•

1 Parent(s): d90f4c5

Update configuration_gemmoe.py

Files changed (1) hide show

configuration_gemmoe.py +1 -1

configuration_gemmoe.py CHANGED Viewed

@@ -132,7 +132,7 @@ class GemmoeConfig(PretrainedConfig):
         attention_dropout=0.0,
         num_experts_per_tok=2,
         num_local_experts=8,
-        router_aux_loss_coef=0.01,
         output_router_logits=False,
         **kwargs,
     ):

         attention_dropout=0.0,
         num_experts_per_tok=2,
         num_local_experts=8,
+        router_aux_loss_coef=0.02,
         output_router_logits=False,
         **kwargs,
     ):