---
datasets:
- masakhane/masakhaner2
- masakhane/masakhapos
metrics:
- accuracy
- f1
---
- Paper: [`FonMTL: Toward Building a Multi-Task Learning Model for Fon Language`](https://arxiv.org/abs/2308.14280), accepted at WiNLP co-located at EMNLP 2023

- Official Github: https://github.com/bonaventuredossou/multitask_fon

- Build Multi-task Learning Model: For the shared layers (encoders) we used the following language model heads:

    - [AfroLM-Large](https://huggingface.co/bonadossou/afrolm_active_learning)
        - [AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages](https://aclanthology.org/2022.sustainlp-1.11/) (Dossou et.al., EMNLP 2022)        
    
    - [XLMR-Large](https://huggingface.co/xlm-roberta-large):
        - [Unsupervised Cross-lingual Representation Learning at Scale](https://aclanthology.org/2020.acl-main.747) (Conneau et.al., ACL 2020)

- Evaluation:

    - The goal primarily is to explore whether multitask learning improves performance on downstream tasks for Fon. We try two settings: (a) training only on Fon and evaluating on Fon, (b) training on all languages and evaluating on Fon. We evaluate the multi-task learning model on NER and POS tasks, and compare it with baselines (models finetuned and evaluated on single tasks)

# How to get started

- Run the training: `sbatch run.sh`

This command will:

- Set up the environement
- Install required libraries: `pip install -r requirements.txt -q`
- Move to the code folder: `cd code`
- Run the training & evaluate: `python run_train.py`

# NER Results
Model | Task | Pretraining/Finetuning Dataset | Pretraining/Finetuning Language(s) | Evaluation Dataset | Metric | Metric's Value |
|:---: |:---: |:---: | :---: |:---: | :---: | :---: |
`AfroLM-Large` | Single Task | MasakhaNER 2.0 | All | FON NER | F1-Score | 80.48 |
`AfriBERTa-Large` | Single Task | MasakhaNER 2.0 | All | FON NER | F1-Score | 79.90 |
`XLMR-Base` | Single Task | MasakhaNER 2.0 | All | FON NER | F1-Score | 81.90 |
`XLMR-Large` | Single Task | MasakhaNER 2.0 | All | FON NER | F1-Score | 81.60 |
`AfroXLMR-Base` | Single Task | MasakhaNER 2.0 | All | FON NER | F1-Score | 82.30 |
`AfroXLMR-Large` | Single Task | MasakhaNER 2.0 | All | FON NER | F1-Score | 82.70 |
|:---: |:---: |:---: | :---: |:---: | :---: |
`MTL Sum (ours)` | Multi-Task | MasakhaNER 2.0 & MasakhaPOS | All | FON NER | F1-Score | 79.87 |
`MTL Weighted (ours)` | Multi-Task | MasakhaNER 2.0 & MasakhaPOS | All | FON NER | F1-Score | 81.92 |
`MTL Weighted (ours)` | Multi-Task | MasakhaNER 2.0 & MasakhaPOS | Fon Data | FON NER | F1-Score | 64.43 |


# POS Results
Model | Task | Pretraining/Finetuning Dataset | Pretraining/Finetuning Language(s) | Evaluation Dataset | Metric | Metric's Value |
|:---: |:---: |:---: | :---: |:---: | :---: | :---: |
`AfroLM-Large` | Single Task | MasakhaPOS | All | FON POS | Accuracy | 82.40 |
`AfriBERTa-Large` | Single Task | MasakhaPOS | All | FON POS | Accuracy | 88.40 |
`XLMR-Base` | Single Task | MasakhaPOS | All | FON POS | Accuracy | 90.10 |
`XLMR-Large` | Single Task | MasakhaPOS | All | FON POS | Accuracy | 90.20 |
`AfroXLMR-Base` | Single Task | MasakhaPOS | All | FON POS | Accuracy | 90.10 |
`AfroXLMR-Large` | Single Task | MasakhaPOS | All | FON POS | Accuracy | 90.40 |
|:---: |:---: |:---: | :---: |:---: | :---: |
`MTL Sum (ours)` | Multi-Task | MasakhaNER 2.0 & MasakhaPOS | All | FON POS | Accuracy | 82.45 |
`MTL Weighted (ours)` | Multi-Task | MasakhaNER 2.0 & MasakhaPOS | All | FON POS | Accuracy | 89.20 |
`MTL Weighted (ours)` | Multi-Task | MasakhaNER 2.0 & MasakhaPOS | Fon Data | FON POS | Accuracy | 80.85 |

# Importance of Merging Representation Type

Merging Type | Models | Task | Metric | Metric's Value |
| :---: | :---: | :---: | :---: | :---: |
Multiplicative | MTL Weighted (multi-task; ours; *) | NER | F1-Score | **81.92** |
Multiplicative | MTL Weighted (multi-task; ours; +) | NER | F1-Score | 64.43 |
| :---: | :---: | :---: | :---: | :---:|
Multiplicative | MTL Weighted (multi-task; ours; *) | POS | Accuracy | **89.20** |
Multiplicative & MTL Weighted (multi-task; ours; +) | POS | Accuracy | 80.85 | 
| :---: | :---: | :---: | :---: | :---: |
Additive | MTL Weighted (multi-task; ours; *) | NER | F1-Score | 78.91 |
Additive | MTL Weighted (multi-task; ours; +) | NER | F1-Score | 60.93 |
| :---: | :---: | :---: | :---: | :---: |
Additive | MTL Weighted (multi-task; ours; *) | POS | Accuracy | 86.99 |
Additive | MTL Weighted (multi-task; ours; +) | POS | Accuracy | 78.25 |

# Model End-Points

- [`multitask_model_fon_False_multiplicative.bin`](https://huggingface.co/bonadossou/multitask_model_fon_False_multiplicative) is the MTL Fon Model which has been pre-trained on all MasakhaNER 2.0 and MasakhaPOS datasets, and merging representations in a multiplicative way.

- [`multitask_model_fon_True_multiplicative.bin`](https://huggingface.co/bonadossou/multitask-learning-fon-true-multiplicative) is the MTL Fon Model which has been pre-trained only on Fon data from the MasakhaNER 2.0 and MasakhaPOS datasets, and merging representations in a multiplicative way.

# How to run inference when you have the model
 To run inference with the model(s), you can use the [testing block](https://github.com/bonaventuredossou/multitask_fon/blob/main/code/run_train.py#L209) defined in our MultitaskFON class.

 # TODO
 - leverage the impact of `the dynamic weighted average loss`