Create README.md

README.md

@@ -0,0 +1,71 @@
+---
+datasets:
+- masakhane/masakhaner2
+metrics:
+- accuracy
+- f1
+---
+Paper: `FonMTL: Toward Building a Multi-Task Learning Model for Fon Language`, accepted at WiNLP co-located at EMNLP 2023
+
+- Official Github: https://github.com/bonaventuredossou/multitask_fon
+
+- Build Multi-task Learning Model: For the shared layers (encoders) we used the following language model heads:
+
+    - [AfroLM-Large](https://huggingface.co/bonadossou/afrolm_active_learning)
+        - [AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages](https://aclanthology.org/2022.sustainlp-1.11/) (Dossou et.al., EMNLP 2022)        
+    
+    - [XLMR-Large](https://huggingface.co/xlm-roberta-large):
+        - [Unsupervised Cross-lingual Representation Learning at Scale](https://aclanthology.org/2020.acl-main.747) (Conneau et.al., ACL 2020)
+
+- Evaluation:
+
+    - The goal primarily is to explore whether multitask learning improves performance on downstream tasks for Fon. We try two settings: (a) training only on Fon and evaluating on Fon, (b) training on all languages and evaluating on Fon. We evaluate the multi-task learning model on NER and POS tasks, and compare it with baselines (models finetuned and evaluated on single tasks)
+
+# How to get started
+
+- Run the training: `sbatch run.sh`
+
+This command will:
+
+- Set up the environement
+- Install required libraries: `pip install -r requirements.txt -q`
+- Move to the code folder: `cd code`
+- Run the training & evaluate: `python run_train.py`
+
+# NER Results
+Model | Task | Pretraining/Finetuning Dataset | Pretraining/Finetuning Language(s) | Evaluation Dataset | Metric | Metric's Value |
+|:---: |:---: |:---: | :---: |:---: | :---: | :---: |
+`AfroLM-Large` | Single Task | MasakhaNER 2.0 | All | FON NER | F1-Score | 80.48 |
+`AfriBERTa-Large` | Single Task | MasakhaNER 2.0 | All | FON NER | F1-Score | 79.90 |
+`XLMR-Base` | Single Task | MasakhaNER 2.0 | All | FON NER | F1-Score | 81.90 |
+`XLMR-Large` | Single Task | MasakhaNER 2.0 | All | FON NER | F1-Score | 81.60 |
+`AfroXLMR-Base` | Single Task | MasakhaNER 2.0 | All | FON NER | F1-Score | 82.30 |
+`AfroXLMR-Large` | Single Task | MasakhaNER 2.0 | All | FON NER | F1-Score | 82.70 |
+|:---: |:---: |:---: | :---: |:---: | :---: |
+`MTL Sum (ours)` | Multi-Task | MasakhaNER 2.0 & MasakhaPOS | All | FON NER | F1-Score | 79.87 |
+`MTL Weighted (ours)` | Multi-Task | MasakhaNER 2.0 & MasakhaPOS | All | FON NER | F1-Score | 81.92 |
+`MTL Weighted (ours)` | Multi-Task | MasakhaNER 2.0 & MasakhaPOS | Fon Data | FON NER | F1-Score | 64.43 |
+
+
+# POS Results
+Model | Task | Pretraining/Finetuning Dataset | Pretraining/Finetuning Language(s) | Evaluation Dataset | Metric | Metric's Value |
+|:---: |:---: |:---: | :---: |:---: | :---: | :---: |
+`AfroLM-Large` | Single Task | MasakhaPOS | All | FON POS | Accuracy | 82.40 |
+`AfriBERTa-Large` | Single Task | MasakhaPOS | All | FON POS | Accuracy | 88.40 |
+`XLMR-Base` | Single Task | MasakhaPOS | All | FON POS | Accuracy | 90.10 |
+`XLMR-Large` | Single Task | MasakhaPOS | All | FON POS | Accuracy | 90.20 |
+`AfroXLMR-Base` | Single Task | MasakhaPOS | All | FON POS | Accuracy | 90.10 |
+`AfroXLMR-Large` | Single Task | MasakhaPOS | All | FON POS | Accuracy | 90.40 |
+|:---: |:---: |:---: | :---: |:---: | :---: |
+`MTL Sum (ours)` | Multi-Task | MasakhaNER 2.0 & MasakhaPOS | All | FON POS | Accuracy | 82.45 |
+`MTL Weighted (ours)` | Multi-Task | MasakhaNER 2.0 & MasakhaPOS | All | FON POS | Accuracy | 89.20 |
+`MTL Weighted (ours)` | Multi-Task | MasakhaNER 2.0 & MasakhaPOS | Fon Data | FON POS | Accuracy | 80.85 |
+
+# Model End-Points
+
+- [`multitask_model_fon_False_multiplicative.bin`](https://huggingface.co/bonadossou/multitask_model_fon_False_multiplicative) is the MTL Fon Model which has been pre-trained on all MasakhaNER 2.0 and MasakhaPOS datasets, and merging representations in a multiplicative way.
+
+- [`multitask_model_fon_True_multiplicative.bin`](https://huggingface.co/bonadossou/multitask-learning-fon-true-multiplicative) is the MTL Fon Model which has been pre-trained only on Fon data from the MasakhaNER 2.0 and MasakhaPOS datasets, and merging representations in a multiplicative way.
+
+# How to run inference when you have the model
+ To run inference with the model(s), you can use the [testing block](https://github.com/bonaventuredossou/multitask_fon/blob/main/code/run_train.py#L209) defined in our MultitaskFON class.