Update README.md

README.md

@@ -1,3 +1,43 @@
 ---
 license: apache-2.0
+datasets: ilc
+tags:
+- summarization
 ---
+
+# Longformer Encoder-Decoder (LED) fine-tuned on ILC
+
+This model is a fine-tuned version of  [led-base-16384](https://huggingface.co/allenai/led-base-16384) on the [ILC](https://huggingface.co/datasets/d0r1h/ILC) dataset.
+
+As described in [Longformer: The Long-Document Transformer](https://arxiv.org/pdf/2004.05150.pdf) by Iz Beltagy, Matthew E. Peters, Arman Cohan, *led-base-16384* was initialized from [*bart-base*](https://huggingface.co/facebook/bart-base) since both models share the exact same architecture. To be able to process 16K tokens, *bart-base*'s position embedding matrix was simply copied 16 times.
+
+```Python
+
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+device = "cuda" if torch.cuda.is_available() else "CPU"
+
+checkpoint = "d0r1h/led-base-ilc"
+
+tokenizer = AutoTokenizer.from_pretrained(checkpoint)
+model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint, return_dict_in_generate=True).to(device)
+case = "......."
+input_ids = tokenizer(case, return_tensors="pt").input_ids.to(device)
+global_attention_mask = torch.zeros_like(input_ids)
+global_attention_mask[:, 0] = 1
+sequences = model.generate(input_ids, 
+                           global_attention_mask=global_attention_mask).sequences
+summary = tokenizer.batch_decode(sequences, 
+                                 skip_special_tokens=True)
+                                 
+```
+
+## Evaluation results
+
+When the model is used for summarizing ILC documents(10 samples), it achieves the following results:
+
+| Model |            rouge1-f  |  rouge1-p  |  rouge2-f   |  rouge2-p  | rougeL-f   |  rougeL-p  |
+|:-----------:|:-----:|:-----:|:------:|:-----:|:------:|:-----:|
+| led-base-ilc      | **42** |  **47** | **22**  | **24** | **39**  | **44** | 
+| led-base          | 3     |  39     | 1      | 21     | 3      | 37 |
+
+[This notebook](https://colab.research.google.com/github/d0r1h/Notebooks/blob/main/NLP/Summarization/led_base_ilc_summarization.ipynb) shows how *led* can effectively be used for downstream tasks such as summarization.