MattiaSangermano/bert-political-leaning-it

This model categorizes the political leaning of an Italian sentence into 4 categories: moderate_left, left, right, moderate_right. The model is a fine-tuned version of neuraly/bert-base-italian-cased-sentiment.

• Developed by: Mattia Sangermano and Fabio Murgese
• Model type: Bert
• Language(s) (NLP): it
• License: Apache 2.0

How to Get Started with the Model

You can use this model directly with a pipeline for text classification:

from transformers import pipeline
classifier = pipeline("text-classification",model='MattiaSangermano/bert-political-leaning-it')
prediction = classifier("Sovranità nazionale e identità forte")
print(prediction)

Here is how to use this model to classify a text in PyTorch:

from transformers import BertForSequenceClassification, AutoTokenizer
import torch
tokenizer = AutoTokenizer.from_pretrained('MattiaSangermano/bert-political-leaning-it')
model = BertForSequenceClassification.from_pretrained('MattiaSangermano/bert-political-leaning-it')
tokens = tokenizer("Uguaglianza e giustizia sociale", return_tensors='pt')
logits = model(**tokens)[0]
prediction = model.config.id2label[torch.argmax(logits).item()]
print(prediction)

and in TensorFlow:

from transformers import AutoTokenizer, TFBertForSequenceClassification
import tensorflow as tf
tokenizer = AutoTokenizer.from_pretrained('MattiaSangermano/bert-political-leaning-it')
model = TFBertForSequenceClassification.from_pretrained('MattiaSangermano/bert-political-leaning-it')
tokens = tokenizer("Ambiente sano, futuro sicuro", padding=True,truncation=True,return_tensors='tf')
logits = model(tokens)[0]
prediction = model.config.id2label[tf.argmax(logits,1)[0].numpy()]
print(prediction)

Out-of-Scope Use

It is important to recognize that political leaning is a personal and complex aspect of an individual's identity and attempting to classify it can be considered unethical and raise significant concerns. Therefore, the model should not be used to identify or classify the political orientation of individual users, nor should it be used for unethical purposes.

Bias, Risks, and Limitations

During the construction of the dataset, deliberate efforts were made to exclude the names of politicians and political parties. As a result, these specific names might not hold relevance to the model.

Dataset

We trained the model using the PoliticIT competition dataset. The dataset was collected during 2020 and 2022 from the Twitter accounts of Italian politicians. These users were selected because their political affiliation can be guessed according to the party to which politicians belong to. The goal of the task was to classify a cluster of tweets, where a cluster is composed of texts written by different users that share the user self-assigned gender and the political ideology.

Preprocessing

According to PoliticIT mantainers, from the dataset were discarded those tweets that contain mentions to news sites or some linguistic clues, such as the pipe symbol, which is used commonly by news sites to categorise their news. Moreover, the Twitter mentions were anonymised by replacing them with the token @user. Therefore the text traits cannot be guessed trivially by reading polititian's name and searching information on them on the Internet. Overall, the dataset consists of 103840 tweets.

Training Procedure

The Dataset was split into train and validation sets with a stratified split having a ratio of 80-20. Although the main task of the original competition was to classify clusters of tweets this model was trained to predict only the political leaning of individual tweets.

Training Hyperparameters

• Optimizer: Adam with learning rate of 4e-5, epsilon of 1e-7
• Loss: Categorical Cross Entropy using balanced class weights
• Max epochs: 10
• Batch size: 64
• Early Stopping: monitoring validation loss with patience = 3
• Training regime: fp16 mixed precision

Evaluation

• test f1-macro: 61.3
• test accuracy: 69.4

Avg Type	Precision	Recall	F1-score	Accuracy
Macro	0.67	0.61	0.61	-
Weighted	0.74	0.69	0.77	0.69