Attention mask

by tanhevg - opened

Hi there! Thank you for the great model. Any reason why the attention mask has been removed in the latest version? It's kind of inconsistent with other checkpoints ('medium' and others).

Thanks in advance,

LongSafari org

Hi @tanhevg - this is my fault, I'm sorry! We actually plan to propagate this change to all of the HyenaDNA models, since the attention mask doesn't really work for Hyena in the same way that it does in transformers. I'm sorry for the period of incompatibility between 1M and the other sizes, but the others will have the new behaviour very soon!

Sign up or log in to comment