---
language: en
license: apache-2.0
datasets:
- billsum
---

# Longformer Encoder-Decoder (LED) fine-tuned on Billsum
This model is a fine-tuned version of led-large-16384 on the billsum dataset.

As described in Longformer: The Long-Document Transformer by Iz Beltagy, Matthew E. Peters, Arman Cohan, led-base-16384 was initialized from bart-base since both models share the exact same architecture. To be able to process 16K tokens, bart-base's position embedding matrix was simply copied 16 times.


# Use In Transformers
```
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("Artifact-AI/led_large_16384_billsum_summarization")

model = AutoModelForSeq2SeqLM.from_pretrained("Artifact-AI/led_large_16384_billsum_summarization")
```