DOI: 10.5281/zenodo.13957176
title: AgoraSpeech: A multi-annotated comprehensive dataset of political discourse through the lens of humans and AI

description: Political discourse datasets are important for gaining political insights, analyzing communication strategies or social science phenomena. Although numerous political discourse corpora exist, comprehensive, high-quality, annotated datasets are scarce. This is largely due to the substantial manual effort, multidisciplinarity, and expertise required for the nuanced annotation of rhetorical strategies and ideological contexts. In this paper, we present AgoraSpeech, a meticulously curated, high-quality dataset of 171 political speeches from six parties during the Greek national elections in 2023. The dataset includes annotations (per paragraph) for six natural language processing (NLP) tasks: text classification, topic identification, sentiment analysis, named entity recognition, polarization and populism detection. A two-step annotation was employed, starting with ChatGPT-generated annotations and followed by exhaustive human-in-the-loop validation. The dataset was initially used in a case study to provide insights during the pre-election period. However, it has general applicability by serving as a rich source of information for political and social scientists, journalists, or data scientists, while it can be used for benchmarking and fine-tuning NLP and large language models (LLMs).
description: The AgoraSpeech dataset is stored and provided as a CSV file.

Each row in the CSV file corresponds to a paragraph from a campaign speech, totaling 5,279 rows, while each column represents a specific type of information related to that paragraph, with 20 columns in total.

The dataset provides multiple types of information for each paragraph. First, each paragraph is described by metadata, such as the politician's name, the location of the speech, and other relevant details. Additionally, the paragraph is contextualized by its position within the full campaign speech, with the text available in both Greek and English. Finally, the results of ChatGPT's analysis across six NLP tasks are included, with each task creating a dedicated column.




Information type
Feature name
Description
Values


Metadata
elections
The election period of the speech


"1st Elections 2023-05-21" or "2nd Elections 2023-06-25"



Metadata
speech_id
The ID of the speech
string in format: PoliticianName_YYYY_MM_DD_Location


Metadata
politician
The name of the politician that gave the speech
"Androulakis", "Koutsoumpas", "Mitsotakis", "Tsipras", "Varoufakis", or "Velopoulos"


Metadata
date (YYYY-MM-DD)
The date of the speech
string in format: YYYY-MM-DD


Metadata
location
The location where the speech took place
cities in Greece


Content
paragraph
The number of the paragraph of the speech
an integer (starting from 1)


Content
text
The text of the paragraph (in english)
english text


Content
text_el
The text of the paragraph (in Greek)
greek text


Analysis
criticism_or_agenda
Whether the text is identified as "political agenda" or "criticism"
"political agenda" or "criticism"


Analysis
topic
The main topic of the text
string from a predefined list of topics


Analysis
sentiment
The detected sentiment of the text
a float number between -1 (negative) and 1 (positive)


Analysis
polarization
The detected level of polarization in the text
a float number between -1 (none/low) and 1 (high)


Analysis
populism
The detected level of populism in the text
a float number between -1 (none/low) and 1 (high)


Analysis
named entities
The detected entities (people, locations, organizations, etc.) in the text
string values detected as entities along with metadata




 

Remark: The table above presents only 14 features (cf. the dataset contains 20 features). The first 8 features, i.e., metadata and content, are created before any annotations. The remaining 6 features correspond to NLP tasks, each appearing twice in the dataset: once as annotated by ChatGPT and once following verification through the human-in-the-loop process.
description: Code Availability

Exploratory Data Analysis: https://github.com/Datalab-AUTH/AgoraSpeech-EDA 

Related Publications

Pavlos Sermpezis, Stelios Karamanidis, Eva Paraschou, Ilias Dimitriadis, Sofia Yfantidou, Filitsa-Ioanna Kouskouveli, Thanasis Troboukis, Kelly Kiki, Antonis Galanopoulos, and Athena Vakali, 2024, AgoraSpeech: A multi-annotated comprehensive dataset of political discourse through the lens of humans and AI (Submitted for peer-review).

Acknowledgments

We would like to thank Nota Vafea, Katerina Voutsina, Stefania Ibrishimova, Athina Thanasi, Chrysoula Marinou, and Georgios Schinas (iMEdD) for their contributions to the human data annotation process. We also thank Christos Nomikos, Nikos Sarantos (iMEdD), and Dimitrios-Panteleimon Giakatos (Datalab) for their IT support and software development of the online tool, as well as Anatoli Stavroulopoulou for cross-checking the political speeches' translations.