AraT5-base-title-generation
We introduce the News Title Generation (NGT) task as a new task for Arabic language generation. Given an article, a title generation model needs to output a short grammatical sequence of words suited to the article content. For NGT, we create a novel dataset from an existing news dataset namely ARGENNTG. We extract 120K articles along with their titles from AraNews (Nagoudi et al., 2020). We only include titles with at least three words in this dataset. We split ARGENNTG data into 80% (93.3K), 10% (11.7K), and 10% (11.7K) for training, development, and test respectively.
We fine-tune ARGENNTG on AraT5-base, more details described in our AraT5: Text-to-Text Transformers for Arabic Language Understanding and Generation
How to use
!pip install transformers sentencepiece
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
tokenizer = AutoTokenizer.from_pretrained("UBC-NLP/AraT5-base-title-generation")
model = AutoModelForSeq2SeqLM.from_pretrained("UBC-NLP/AraT5-base-title-generation")
Document = "تحت رعاية صاحب السمو الملكي الأمير سعود بن نايف بن عبدالعزيز أمير المنطقة الشرقية اختتمت غرفة الشرقية مؤخرا، الثاني من مبادرتها لتأهيل وتدريب أبناء وبنات المملكة ضمن مبادرتها المجانية للعام 2019 حيث قدمت 6 برامج تدريبية نوعية. وثمن رئيس مجلس إدارة الغرفة، عبدالحكيم العمار الخالدي، رعاية سمو أمير المنطقة الشرقية للمبادرة، مؤكدا أن دعم سموه لجميع أنشطة ."
encoding = tokenizer.encode_plus(Document,pad_to_max_length=True, return_tensors="pt")
input_ids, attention_masks = encoding["input_ids"], encoding["attention_mask"]
outputs = model.generate(
input_ids=input_ids, attention_mask=attention_masks,
max_length=256,
do_sample=True,
top_k=120,
top_p=0.95,
early_stopping=True,
num_return_sequences=5
)
for id, output in enumerate(outputs):
title = tokenizer.decode(output, skip_special_tokens=True,clean_up_tokenization_spaces=True)
print("title#"+str(id), title)
The input news document
The generated titles
title#0 غرفة الشرقية تختتم المرحلة الثانية من مبادرتها لتأهيل وتدريب أبناء وبنات المملكة
title#1 غرفة الشرقية تختتم الثاني من مبادرة تأهيل وتأهيل أبناء وبناتنا
title#2 سعود بن نايف يختتم ثانى مبادراتها لتأهيل وتدريب أبناء وبنات المملكة
title#3 أمير الشرقية يرعى اختتام برنامج برنامج تدريب أبناء وبنات المملكة
title#4 سعود بن نايف يرعى اختتام مبادرة تأهيل وتدريب أبناء وبنات المملكة
How to use AraT5 models
This is an example for fine-tuning AraT5-base for News Title Generation on the Aranews dataset
For more details, please visit our own GitHub.
AraT5 Models Checkpoints
AraT5 Pytorch and TensorFlow checkpoints are available on the Huggingface website for direct download and use exclusively for research. For commercial use, please contact the authors via email @ (muhammad.mageed[at]ubc[dot]ca).
| Model | Link |
|---|---|
| AraT5-base | https://huggingface.co/UBC-NLP/AraT5-base |
| AraT5-msa-base | https://huggingface.co/UBC-NLP/AraT5-msa-base |
| AraT5-tweet-base | https://huggingface.co/UBC-NLP/AraT5-tweet-base |
| AraT5-msa-small | https://huggingface.co/UBC-NLP/AraT5-msa-small |
| AraT5-tweet-small | https://huggingface.co/UBC-NLP/AraT5-tweet-small |
BibTex
If you use our models (Arat5-base, Arat5-msa-base, Arat5-tweet-base, Arat5-msa-small, or Arat5-tweet-small ) for your scientific publication, or if you find the resources in this repository useful, please cite our paper as follows (to be updated):
@inproceedings{araT5-2021,
title = "{AraT5: Text-to-Text Transformers for Arabic Language Understanding and Generation",
author = "Nagoudi, El Moatez Billah and
Elmadany, AbdelRahim and
Abdul-Mageed, Muhammad",
booktitle = "https://arxiv.org/abs/2109.12068",
month = aug,
year = "2021"}
Acknowledgments
We gratefully acknowledge support from the Natural Sciences and Engineering Research Council of Canada, the Social Sciences and Humanities Research Council of Canada, Canadian Foundation for Innovation, ComputeCanada and UBC ARC-Sockeye. We also thank the Google TensorFlow Research Cloud (TFRC) program for providing us with free TPU access.