Update README.md

README.md

@@ -1,62 +1,39 @@
 # AraT5-base-title-generation
-<img src="https://raw.githubusercontent.com/UBC-NLP/araT5/main/AraT5_logo.jpg" alt="drawing" width="30%" height="30%" align="right"/>
+# AraT5: Text-to-Text Transformers for Arabic Language Generation
 
-We introduce the News Title Generation (NGT) task as a new task for Arabic language generation. Given an article, a title generation model needs to output a short grammatical sequence of words suited to the article content. For NGT, we create a novel dataset from an existing news dataset namely **ARGEN<sub>NTG</sub>**. We extract 120K articles along with their titles from [AraNews (Nagoudi et al., 2020)](https://arxiv.org/abs/2011.03092). We only include titles with at least three words in this dataset. We split ARGEN<sub>NTG</sub> data into 80% (93.3K), 10% (11.7K), and 10% (11.7K) for training, development, and test respectively.
+<img src="https://huggingface.co/UBC-NLP/AraT5-base/resolve/main/AraT5_CR_new.png" alt="AraT5" width="45%" height="35%" align="right"/>
 
-We fine-tune ARGEN<sub>NTG</sub> on [**AraT5-base**](https://huggingface.co/UBC-NLP/AraT5-base), more details described in our [**AraT5: Text-to-Text Transformers for Arabic Language Understanding and Generation**](https://arxiv.org/abs/2109.12068)
+This is the repository accompanying our paper [AraT5: Text-to-Text Transformers for Arabic Language Understanding and Generation](https://arxiv.org/abs/2109.12068). In this is the repository we introduce:
+* Introduce **AraT5<sub>MSA</sub>**, **AraT5<sub>Tweet</sub>**, and **AraT5**: three powerful Arabic-specific text-to-text Transformer based models;
+* Introduce **ARGEN**:  A new benchmark for Arabic language generation and evaluation for four Arabic NLP tasks, namely, ```machine  translation```,  ```summarization```,  ```news title   generation```,   ```question   generation```, ,   ```paraphrasing```,   ```transliteration```, and  ```code-switched translation```.
+* Evaluate  ```AraT5``` models on ```ARGEN``` and compare against available language models.
 
 
-# How to use
+---
+# How to use AraT5 models
+Below is an example for fine-tuning **AraT5-base** for News Title Generation on the Aranews dataset 
 ``` bash
-!pip install transformers sentencepiece 
+!python run_trainier_seq2seq_huggingface.py \
+        --learning_rate 5e-5 \
+        --max_target_length 128 --max_source_length 128 \
+        --per_device_train_batch_size 8 --per_device_eval_batch_size 8 \
+        --model_name_or_path "UBC-NLP/AraT5-base" \
+        --output_dir "/content/AraT5_FT_title_generation" --overwrite_output_dir \
+        --num_train_epochs 3 \
+        --train_file "/content/ARGEn_title_genration_sample_train.tsv" \
+        --validation_file "/content/ARGEn_title_genration_sample_valid.tsv" \
+        --task "title_generation" --text_column "document" --summary_column "title" \
+        --load_best_model_at_end --metric_for_best_model "eval_bleu" --greater_is_better True --evaluation_strategy epoch --logging_strategy epoch --predict_with_generate\
+        --do_train --do_eval
 ```
+For more details about the fine-tuning example, please read this notebook [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://github.com/UBC-NLP/araT5/blob/main/examples/Fine_tuning_AraT5.ipynb) 
 
-```python
-from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
-tokenizer = AutoTokenizer.from_pretrained("UBC-NLP/AraT5-base-title-generation")  
-model = AutoModelForSeq2SeqLM.from_pretrained("UBC-NLP/AraT5-base-title-generation")
-
-Document = "تحت رعاية صاحب السمو الملكي الأمير سعود بن نايف بن عبدالعزيز أمير المنطقة الشرقية اختتمت غرفة الشرقية مؤخرا، الثاني من مبادرتها لتأهيل وتدريب أبناء وبنات المملكة ضمن مبادرتها المجانية للعام 2019 حيث قدمت 6 برامج تدريبية نوعية. وثمن رئيس مجلس إدارة الغرفة، عبدالحكيم العمار الخالدي، رعاية سمو أمير المنطقة الشرقية للمبادرة، مؤكدا أن دعم سموه لجميع أنشطة ."
-
-encoding = tokenizer.encode_plus(Document,pad_to_max_length=True, return_tensors="pt")
-input_ids, attention_masks = encoding["input_ids"], encoding["attention_mask"]
-
-
-outputs = model.generate(
-    input_ids=input_ids, attention_mask=attention_masks,
-    max_length=256,
-    do_sample=True,
-    top_k=120,
-    top_p=0.95,
-    early_stopping=True,
-    num_return_sequences=5
-)
-
-for id, output in enumerate(outputs):
-    title = tokenizer.decode(output, skip_special_tokens=True,clean_up_tokenization_spaces=True)
-    print("title#"+str(id), title)
-```
-**The input news document**
+In addition, we release the fine-tuned checkpoint of the News Title Generation (NGT) which is described in the paper. The model available at Huggingface ([UBC-NLP/AraT5-base-title-generation](https://huggingface.co/UBC-NLP/AraT5-base-title-generation)).
 
-<div style="white-space : pre-wrap !important;word-break: break-word; direction:rtl; text-align: right">
-تحت رعاية صاحب السمو الملكي الأمير سعود بن نايف بن عبدالعزيز أمير المنطقة الشرقية اختتمت غرفة الشرقية مؤخرا، الثاني من مبادرتها لتأهيل وتدريب أبناء وبنات المملكة ضمن مبادرتها المجانية للعام 2019 حيث قدمت 6 برامج تدريبية نوعية. وثمن رئيس مجلس إدارة الغرفة، عبدالحكيم العمار الخالدي، رعاية سمو أمير المنطقة الشرقية للمبادرة، مؤكدا أن دعم سموه لجميع أنشطة .
-  <br>
-</div>
- 
+For more details, please visit our own [GitHub](https://github.com/UBC-NLP/araT5).
 
-**The generated titles**
-```
-title#0 غرفة الشرقية تختتم المرحلة الثانية من مبادرتها لتأهيل وتدريب أبناء وبنات المملكة
-title#1 غرفة الشرقية تختتم الثاني من مبادرة تأهيل وتأهيل أبناء وبناتنا
-title#2 سعود بن نايف يختتم ثانى مبادراتها لتأهيل وتدريب أبناء وبنات المملكة
-title#3 أمير الشرقية يرعى اختتام برنامج برنامج تدريب أبناء وبنات المملكة
-title#4 سعود بن نايف يرعى اختتام مبادرة تأهيل وتدريب أبناء وبنات المملكة
-```
 
-# How to use AraT5 models
-[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/drive/1GFOGolWPIfDvYdSNdGFrOXwu3Gu28k2b?usp=sharing)This is an example for fine-tuning **AraT5-base** for News Title Generation on the Aranews dataset 
 
-For more details, please visit our own [GitHub](https://github.com/UBC-NLP/araT5).
 
 
 # AraT5 Models Checkpoints 
@@ -75,15 +52,18 @@ AraT5 Pytorch and TensorFlow checkpoints are available on the Huggingface websit
 
 If you use our models (Arat5-base, Arat5-msa-base, Arat5-tweet-base, Arat5-msa-small, or Arat5-tweet-small ) for your scientific publication, or if you find the resources in this repository useful, please cite our paper as follows (to be updated):
 ```bibtex
-@inproceedings{araT5-2021,
-    title = "{AraT5: Text-to-Text Transformers for Arabic Language Understanding and Generation",
+@inproceedings{nagoudi-2022-arat5,
+    title = "{AraT5: Text-to-Text Transformers for Arabic Language Generation",
     author = "Nagoudi, El Moatez Billah  and
       Elmadany, AbdelRahim  and
       Abdul-Mageed, Muhammad",
-    booktitle = "https://arxiv.org/abs/2109.12068",
-    month = aug,
-    year = "2021"}
+    booktitle = "Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics",
+    month = May,
+    year = "2022",
+    address = "Online",
+    publisher = "Association for Computational Linguistics",
+    }
 ```
 
 ## Acknowledgments
-We gratefully acknowledge support from the Natural Sciences and Engineering Research Council  of Canada, the  Social  Sciences and  Humanities  Research  Council  of  Canada, Canadian  Foundation for  Innovation,  [ComputeCanada](www.computecanada.ca) and [UBC ARC-Sockeye](https://doi.org/10.14288/SOCKEYE). We  also  thank  the  [Google TensorFlow Research Cloud (TFRC)](https://www.tensorflow.org/tfrc) program for providing us with free TPU access.
+We gratefully acknowledge support from the Natural Sciences and Engineering Research Council  of Canada, the  Social  Sciences and  Humanities  Research  Council  of  Canada, Canadian  Foundation for  Innovation,  [ComputeCanada](www.computecanada.ca) and [UBC ARC-Sockeye](https://doi.org/10.14288/SOCKEYE). We  also  thank  the  [Google TensorFlow Research Cloud (TFRC)](https://www.tensorflow.org/tfrc) program for providing us with free TPU access.