|
--- |
|
library_name: transformers |
|
license: mit |
|
datasets: |
|
- sem_eval_2020_task_11 |
|
language: |
|
- en |
|
--- |
|
|
|
# Model Card for Model ID |
|
|
|
<!-- Provide a quick summary of what the model is/does. --> |
|
Given a sentence, our model predicts whether or not the sentence contains "persuasive" language, or language designed to elicit emotions or change |
|
readers' opinions. The model was tuned on the SemEval 2020 Task 11 dataset. However, we preprocessed the dataset to adapt it from |
|
multilabel technique classification and span-classification to our binary classification task. |
|
|
|
There are two revisions: |
|
* BERT - we finetuned `bert-large-cased` on our main branch |
|
* XLM-RoBERTa - we finetuned `xlm-roberta-base` on our `roberta` branch. |
|
|
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
<!-- Provide a longer summary of what this model is. --> |
|
|
|
This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated. |
|
|
|
- **Developed by:** Ultraviolet Text |
|
- **Model type:** BERT / RoBERTa |
|
- **Language(s) (NLP):** En |
|
- **License:** MIT |
|
- **Finetuned from model [optional]:** bert-large-cased / xlm-roberta-base |
|
|
|
## How to Get Started with the Model |
|
|
|
Use the code below to get started with the model. |
|
|
|
### Loading from the main branch (BERT) |
|
```py |
|
from transformers import AutoModelForSequenceClassification, AutoTokenizer |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("bert-large-cased") |
|
model = AutoModelForSequenceClassification.from_pretrained("chreh/persuasive_language_detector") |
|
``` |
|
|
|
### Loading from the `roberta` branch (XLM RoBERTa) |
|
```py |
|
from transformers import AutoModelForSequenceClassification, AutoTokenizer |
|
|
|
tokenizer = AutoTokenizer.from_pretrained("xlm-roberta-base") |
|
model = AutoModelForSequenceClassification.from_pretrained("chreh/persuasive_language_detector", revision="roberta") |
|
``` |
|
## Training Details |
|
|
|
### Training Data |
|
|
|
<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. --> |
|
Training data can be downloaded from [the Semeval website](https://propaganda.qcri.org/semeval2020-task11/). |
|
|
|
### Training Procedure |
|
|
|
<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. --> |
|
The training was done using Huggingface Trainer on both our local machines and Intel Developer Cloud kernels, enabling us to prototype multiple models simultaneously. |
|
|
|
#### Preprocessing [optional] |
|
All sentences containing spans of persuasive language techniques were labeled as persuasive language examples, while all others |
|
were labeled as examples of non-persuasive language. |
|
|
|
### Testing Data, Factors & Metrics |
|
|
|
#### Testing Data |
|
|
|
<!-- This should link to a Dataset Card if possible. --> |
|
The test data is from the test data of `sem_eval_2020_task_11`, which can be downloaded from [the original website](https://propaganda.qcri.org/semeval2020-task11/). |
|
The test data contains 38.25% persuasive examples and non-persuasive examples 61.75%. Metrics can be found in the following section |
|
|
|
#### Metrics |
|
|
|
<!-- These are the evaluation metrics being used, ideally with a description of why. --> |
|
Metrics are reported in the format (main_branch), (roberta branch) |
|
* Accuracy - 0.7165140725669719, 0.7326693227091633 |
|
* Recall - 0.6875584658559402, 0.6822916666666666 |
|
* Precision - 0.5941794664510913, 0.6415279138099902 |
|
* F1 - 0.6374674761491761, 0.6612821807168097 |
|
|
|
Overall, the `roberta` branch performs better, and with faster inference times. Thus, we recommend users download from the `roberta` revision. |
|
|
|
|
