|
--- |
|
language: |
|
- vi |
|
--- |
|
|
|
## Introduction |
|
|
|
This model was initialized from [vinai/bartpho-word-base](https://huggingface.co/vinai/bartpho-word-base) and converted to [Allenai's Longformer Encoder-Decoder (LED)](https://github.com/allenai/longformer#longformer) based on [Longformer: The Long-Document Transformer](https://arxiv.org/pdf/2004.05150.pdf). |
|
|
|
To be able to process 16K tokens, *bartpho-word-base*'s position embedding matrix was simply copied 16 times. |
|
|
|
This model is especially interesting for long-range summarization and question answering. |
|
|
|
|
|
## Fine-tuning for down-stream task |
|
|
|
[This notebook](https://colab.research.google.com/drive/12LjJazBl7Gam0XBPy_y0CTOJZeZ34c2v?usp=sharing) shows how led model can effectively be fine-tuned on a downstream task. |
