Updated readme to match dataset type name regex

95cacd9 5 months ago

No virus

5.33 kB

	---
	language:
	- it
	tags:
	- twitter
	- political-leaning
	- politics
	datasets:
	- politic-it
	widget:
	- text: >-
	È necessario garantire salari dignitosi e condizioni di lavoro adeguate per
	tutelare i diritti dei lavoratori
	example_title: Left-wing example
	- text: >-
	L'immigrazione deve essere gestita con rigore per preservare l'identità
	nazionale!
	example_title: Right-wing example
	model-index:
	- name: bert-political-leaning-it
	results:
	- task:
	type: text-classification
	name: Text Classification
	dataset:
	type: social-media
	name: politic-it
	metrics:
	- type: f1 macro
	value: 61.3
	- type: accuracy
	value: 69.4
	license: apache-2.0
	metrics:
	- f1
	- accuracy
	pipeline_tag: text-classification
	---

	# MattiaSangermano/bert-political-leaning-it

	This model categorizes the political leaning of an Italian sentence into 4 categories: `moderate_left`, `left`, `right`, `moderate_right`. The model is a fine-tuned version of [neuraly/bert-base-italian-cased-sentiment](https://huggingface.co/neuraly/bert-base-italian-cased-sentiment).

	- Developed by: [Mattia Sangermano](https://www.linkedin.com/in/mattia-sangermano/) and [Fabio Murgese](https://www.linkedin.com/in/fabio-murgese/)
	- Model type: Bert
	- Language(s) (NLP): it
	- License: Apache 2.0

	### How to Get Started with the Model

	You can use this model directly with a pipeline for text classification:

	``` python
	from transformers import pipeline
	classifier = pipeline("text-classification",model='MattiaSangermano/bert-political-leaning-it')
	prediction = classifier("Sovranità nazionale e identità forte")
	print(prediction)
	```

	Here is how to use this model to classify a text in PyTorch:

	``` python
	from transformers import BertForSequenceClassification, AutoTokenizer
	import torch
	tokenizer = AutoTokenizer.from_pretrained('MattiaSangermano/bert-political-leaning-it')
	model = BertForSequenceClassification.from_pretrained('MattiaSangermano/bert-political-leaning-it')
	tokens = tokenizer("Uguaglianza e giustizia sociale", return_tensors='pt')
	logits = model(**tokens)[0]
	prediction = model.config.id2label[torch.argmax(logits).item()]
	print(prediction)
	```

	and in TensorFlow:

	``` python
	from transformers import AutoTokenizer, TFBertForSequenceClassification
	import tensorflow as tf
	tokenizer = AutoTokenizer.from_pretrained('MattiaSangermano/bert-political-leaning-it')
	model = TFBertForSequenceClassification.from_pretrained('MattiaSangermano/bert-political-leaning-it')
	tokens = tokenizer("Ambiente sano, futuro sicuro", padding=True,truncation=True,return_tensors='tf')
	logits = model(tokens)[0]
	prediction = model.config.id2label[tf.argmax(logits,1)[0].numpy()]
	print(prediction)
	```


	### Out-of-Scope Use

	It is important to recognize that political leaning is a personal and complex aspect of an individual's identity and attempting to classify it can be considered unethical and raise significant concerns. Therefore, the model should not be used to identify or classify the political orientation of individual users, nor should it be used for unethical purposes.

	## Bias, Risks, and Limitations
	During the construction of the dataset, deliberate efforts were made to exclude the names of politicians and political parties. As a result, these specific names might not hold relevance to the model.


	## Dataset

	We trained the model using the [PoliticIT](https://codalab.lisn.upsaclay.fr/competitions/8507#learn_the_details) competition dataset. The dataset was collected during 2020 and 2022 from the Twitter accounts of Italian politicians. These users were selected because their political affiliation can be guessed according to the party to which politicians belong to. The goal of the task was to classify a cluster of tweets, where a cluster is composed of texts written by different users that share the user self-assigned gender and the political ideology.

	### Preprocessing

	According to PoliticIT mantainers, from the dataset were discarded those tweets that contain mentions to news sites or some linguistic clues, such as the pipe symbol, which is used commonly by news sites to categorise their news. Moreover, the Twitter mentions were anonymised by replacing them with the token @user. Therefore the text traits cannot be guessed trivially by reading polititian's name and searching information on them on the Internet. Overall, the dataset consists of 103840 tweets.

	#### Training Procedure

	The Dataset was split into train and validation sets with a stratified split having a ratio of 80-20. Although the main task of the original competition was to classify clusters of tweets this model was trained to predict only the political leaning of individual tweets.

	### Training Hyperparameters

	- Optimizer: Adam with learning rate of 4e-5, epsilon of 1e-7
	- Loss: Categorical Cross Entropy using balanced class weights
	- Max epochs: 10
	- Batch size: 64
	- Early Stopping: monitoring validation loss with patience = 3
	- Training regime: fp16 mixed precision

	## Evaluation
	- test f1-macro: 61.3
	- test accuracy: 69.4


	\| Avg Type \| Precision \| Recall \| F1-score \| Accuracy \|
	\| ------ \| ------ \| ------ \| ------ \| ------ \|
	\| Macro \| 0.67 \| 0.61 \| 0.61 \| - \|
	\| Weighted \| 0.74 \| 0.69 \| 0.77 \| 0.69 \|