---
license: apache-2.0
datasets: billsum
tags:
- summarization
model-index:
- name: d0r1h/LEDBill
  results:
  - task:
      type: summarization
      name: Summarization
    dataset:
      name: billsum
      type: billsum
      config: default
      split: test
    metrics:
    - name: ROUGE-1
      type: rouge
      value: 38.6502
      verified: true
    - name: ROUGE-2
      type: rouge
      value: 18.5458
      verified: true
    - name: ROUGE-L
      type: rouge
      value: 25.6561
      verified: true
    - name: ROUGE-LSUM
      type: rouge
      value: 33.1575
      verified: true
    - name: loss
      type: loss
      value: 2.1305277347564697
      verified: true
    - name: gen_len
      type: gen_len
      value: 288.372
      verified: true
---

# Longformer Encoder-Decoder (LED) fine-tuned on Billsum



This model is a fine-tuned version of [led-base-16384](https://huggingface.co/allenai/led-base-16384) on the [billsum](https://huggingface.co/datasets/billsum) dataset.

As described in [Longformer: The Long-Document Transformer](https://arxiv.org/pdf/2004.05150.pdf) by Iz Beltagy, Matthew E. Peters, Arman Cohan, *led-base-16384* was initialized from [*bart-base*](https://huggingface.co/facebook/bart-base) since both models share the exact same architecture. To be able to process 16K tokens, *bart-base*'s position embedding matrix was simply copied 16 times.


## How to use

```Python
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained("d0r1h/LEDBill")
model = AutoModelForSeq2SeqLM.from_pretrained("d0r1h/LEDBill", return_dict_in_generate=True).to(device)

case = "......."

input_ids = tokenizer(case, return_tensors="pt").input_ids.to(device)
global_attention_mask = torch.zeros_like(input_ids)
global_attention_mask[:, 0] = 1

sequences = model.generate(input_ids, 
                           global_attention_mask=global_attention_mask).sequences
summary = tokenizer.batch_decode(sequences, 
                                 skip_special_tokens=True)
                                 
```


## Evaluation results

When the model is used for summarizing Billsum documents(10 sample), it achieves the following results:

| Model |            rouge1-f  |  rouge1-p  |  rouge2-f   |  rouge2-p  | rougeL-f   |  rougeL-p  |
|:-----------:|:-----:|:-----:|:------:|:-----:|:------:|:-----:|
| LEDBill           | **34** |  **37** | **15**  | **16** | **30**  | **32** | 
| led-base          | 2     |  15     | 0      | 0     | 2      | 15 |

[This notebook](https://colab.research.google.com/drive/1iEEFbWeTGUSDesmxHIU2QDsPQM85Ka1K?usp=sharing) shows how *led* can effectively be used for downstream task such summarization.