chkla /roberta-argument

Welcome to RoBERTArg!

🤖 Model description

This model was trained on ~25k heterogeneous manually annotated sentences (📚 Stab et al. 2018) of controversial topics to classify text into one of two labels: 🏷 NON-ARGUMENT (0) and ARGUMENT (1).

🗃 Dataset

The dataset (📚 Stab et al. 2018) consists of ARGUMENTS (~11k) that either support or oppose a topic if it includes a relevant reason for supporting or opposing the topic, or as a NON-ARGUMENT (~14k) if it does not include reasons. The authors focus on controversial topics, i.e., topics that include "an obvious polarity to the possible outcomes" and compile a final set of eight controversial topics: abortion, school uniforms, death penalty, marijuana legalization, nuclear energy, cloning, gun control, and minimum wage.

TOPIC ARGUMENT NON-ARGUMENT
abortion 2213 2,427
school uniforms 325 1,734
death penalty 325 2,083
marijuana legalization 325 1,262
nuclear energy 325 2,118
cloning 325 1,494
gun control 325 1,889
minimum wage 325 1,346

🏃🏼‍♂️Model training

RoBERTArg was fine-tuned on a RoBERTA (base) pre-trained model from HuggingFace using the HuggingFace trainer with the following hyperparameters:

training_args = TrainingArguments(
num_train_epochs=2,
learning_rate=2.3102e-06,
seed=8,
per_device_train_batch_size=64,
per_device_eval_batch_size=64,
)


📊 Evaluation

The model was evaluated on an evaluation set (20%):

Model Acc F1 R arg R non P arg P non
RoBERTArg 0.8193 0.8021 0.8463 0.7986 0.7623 0.8719

Showing the confusion matrix using again the evaluation set:

ARGUMENT NON-ARGUMENT
ARGUMENT 2213 558
NON-ARGUMENT 325 1790

⚠️ Intended Uses & Potential Limitations

The model can only be a starting point to dive into the exciting field of argument mining. But be aware. An argument is a complex structure, with multiple dependencies. Therefore, the model may perform less well on different topics and text types not included in the training set.

Enjoy and stay tuned! 🚀

1,302
Hosted inference API
Text Classification
This model can be loaded on the Inference API on-demand.