LongSafari
/

hyenadna-large-1m-seqlen-hf

Text Generation

Model card Files Files and versions Community

Rocketknight1 HF staff commited on Nov 17, 2023

Commit

e0522bc

•

1 Parent(s): 6e84b23

Update modeling_hyena.py

Files changed (1) hide show

modeling_hyena.py +1 -1

modeling_hyena.py CHANGED Viewed

@@ -463,7 +463,7 @@ class HyenaDNAForCausalLM(HyenaDNAPreTrainedModel):
             shift_labels = labels[..., 1:].contiguous()
             # Flatten the tokens
             loss_fct = nn.CrossEntropyLoss()
-            shift_logits = shift_logits.view(-1, self.config.vocab_size)
             shift_labels = shift_labels.view(-1)
             # Enable model parallelism
             shift_labels = shift_labels.to(shift_logits.device)

             shift_labels = labels[..., 1:].contiguous()
             # Flatten the tokens
             loss_fct = nn.CrossEntropyLoss()
+            shift_logits = shift_logits.view(-1, self.vocab_size)
             shift_labels = shift_labels.view(-1)
             # Enable model parallelism
             shift_labels = shift_labels.to(shift_logits.device)