Attention mask
#2
by
tanhevg
- opened
Hi there! Thank you for the great model. Any reason why the attention mask has been removed in the latest version? It's kind of inconsistent with other checkpoints ('medium' and others).
Thanks in advance,
Evgeny
Hi @tanhevg - this is my fault, I'm sorry! We actually plan to propagate this change to all of the HyenaDNA models, since the attention mask doesn't really work for Hyena in the same way that it does in transformers. I'm sorry for the period of incompatibility between 1M and the other sizes, but the others will have the new behaviour very soon!