EUroBerta-xlm

This model is a fine-tuned version of FacebookAI/xlm-roberta-base on national parliamentary speeches in Europe, using ParlSpeech dataset. It achieves the following results on the evaluation set:

• Loss: 0.1751
• Accuracy: 0.9442
• F1: 0.9442
• Precision: 0.9455
• Recall: 0.9442

Model description

EUroBerta-xlm is a fine-tuned multilingual text classification model designed to identify whether a sentence discusses the European Union (EU). Given any sentence as input, the model returns a binary label: EU (the sentence references the European Union) or No EU (the sentence does not discuss the EU).

This model is the first of three steps in a full EU stance classification pipeline:

• EUroBerta-xlm → identifies EU-relevant sentences (EU / No EU) ← this model
• EuOppostion_Classifier → classifies EU sentences as Opposition / Non-Opposition
• EuSupport_Classifier → classifies Non-Opposition sentences as Neutral / Support

The final model achieves an F1 score of 0.94 on the held-out evaluation set, with high and balanced precision and recall. The confusion matrix below reflects the model's strong ability to distinguish EU-relevant from non-EU sentences, including after correction for false positives introduced by geographic or sporting references to "Europe".

Intended uses & limitations

The model was developed to support large-scale computational text analysis of political discourse, enabling researchers to efficiently filter and categorise party manifestos, parliamentary speeches, and other political documents across multiple European languages. Altought the model can be use for any EU-related task, it was specifically designed for assisting annotation pipelines that require downstream classification of EU attitudes (pro-EU, Eurosceptic, neutral).

Training and evaluation data

The training data was constructed through two rounds of annotation, resulting in a final corpus of 5,950 sentences drawn from plenary speeches from 7 European national parliaments (Austria, Denmark, Germany, the Netherlands, Spain, Sweden, and the United Kingdom). These speeches are sourced from the ParlSpeech dataset, covering a 20-year period from 1999 to 2019 (Rauh & Schwalbach, 2020)

The source data originally used five categories of EU stance:

• No EU — sentence does not reference the EU
• Pro-EU — sentence expresses a favourable view of the EU
• Neutral — sentence mentions the EU without taking a stance
• Soft Eurosceptic — sentence expresses mild scepticism toward the EU
• Hard Eurosceptic — sentence expresses strong opposition to the EU

These five categories were consolidated into a binary format for this model: sentences labelled No EU were retained as one class, while Pro-EU, Neutral, Soft Eurosceptic, and Hard Eurosceptic were merged into a single EU class. After an initial training run, the model showed a systematic tendency to overclassify sentences as EU-related, particularly sentences referencing individual countries (including non-EU states) or generic mentions of "Europe" (e.g., UEFA, European geography). To address this, a second round of targeted annotation was conducted:

• 250 additional sentences per country were selected for re-annotation.
• Radical right parties were intentionally oversampled (~50 sentences per country) to improve coverage of oppositional discourse, which differs structurally from mainstream party language.
• The model was then retrained on the full combined corpus of 5,950 sentences.

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 2e-05
• train_batch_size: 24
• eval_batch_size: 24
• seed: 42
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• num_epochs: 3

Training results

Training Loss	Epoch	Step	Validation Loss	Accuracy	F1	Precision	Recall
0.2651	1.0	198	0.2666	0.9154	0.9154	0.9185	0.9154
0.2581	2.0	396	0.2291	0.9171	0.9171	0.9242	0.9171
0.128	3.0	594	0.1751	0.9442	0.9442	0.9455	0.9442

Confusion Matrix

Framework versions

• Transformers 4.52.4
• Pytorch 2.6.0+cu124
• Datasets 3.6.0
• Tokenizers 0.21.1