Back to all models

Unable to determine this model’s pipeline type. Check the docs .

Monthly model downloads

allenai/longformer-base-4096-extra.pos.embd.only allenai/longformer-base-4096-extra.pos.embd.only
1,810 downloads
last 30 days

pytorch

tf

Contributed by

Allen Institute for AI non-profit
4 team members · 38 models

How to use this model directly from the 🤗/transformers library:

			
Copy to clipboard
from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("allenai/longformer-base-4096-extra.pos.embd.only") model = AutoModel.from_pretrained("allenai/longformer-base-4096-extra.pos.embd.only")

longformer-base-4096-extra.pos.embd.only

This model is similar to longformer-base-4096 but it was pretrained to preserve RoBERTa weights by freezing all RoBERTa weights and only train the additional position embeddings.

Citing

If you use Longformer in your research, please cite Longformer: The Long-Document Transformer.

@article{Beltagy2020Longformer,
  title={Longformer: The Long-Document Transformer},
  author={Iz Beltagy and Matthew E. Peters and Arman Cohan},
  journal={arXiv:2004.05150},
  year={2020},
}

Longformer is an open-source project developed by the Allen Institute for Artificial Intelligence (AI2). AI2 is a non-profit institute with the mission to contribute to humanity through high-impact AI research and engineering.