bobox's picture
Training in progress, step 2745, checkpoint
07ba75d verified
raw
history blame
222 Bytes
{
"embed_dim": 768,
"num_heads": 4,
"dropout": 0.025,
"bias": true,
"gate_min": 0.05,
"gate_max": 0.95,
"gate_dropout": 0.01,
"dropout_gate_open": 0.075,
"dropout_gate_close": 0.05,
"CLS_self_attn": 0
}