# Limitations of Large Language Models in Propaganda Detection Task
## 3 Datasets
### Propaganda Techniques Corpus  
The International Workshop on Semantic Evaluation in 2020 (SemEval 2020) consisted of twelve tasks. As part of the \"Societal Applications of NLP\" section, Task 11 concerned the \"Detection of Propaganda Techniques in News Articles\". The main goal of this workshop was to develop the automatic tools to detect the aforementioned techniques. The organizers created the PTC-SemEval20 corpus [30; 42], which consisted of 536 news articles in the final version. Table 4 shows the exact distribution of articles.  
All of articles were annotated by experts. The annotation included a span in which a propaganda technique was used--span identification (SI), which is a binary sequence tagging task, as well as the technique's name--technique classification (TC), which is a multiclass classification problem. Initially, there were 18 propaganda techniques, but due  
\begin{table}
\begin{tabular}{l l} \hline \hline
**Dataset** & **News Article Count** \\ \hline Training set & 371 \\ Development set & 75 \\ Test set & 90 \\ \hline \hline \end{tabular}
\end{table}
Table 4: PTC-SemEval2020 corpus distribution.  
\begin{table}
\begin{tabular}{l l l l} \hline \hline
**Organization** & **Year of Creation** & **Website** & **Notes** \\ \hline Fundacja Reporterow & 2010 & www.fundacjareporterow.org & indEX and Ukraine Monitor projects observe \\  & & (accessed on 20 March 2024) & influence of disinformation on \\  & & (accessed on 20 March 2024) & extremist groups \\ \hline Demagog & 2014 & www.demagog.org.pl & First Polish fact-checking organization \\  & & (accessed on 20 March 2024) & Does not allow readers to report \\  & & (accessed on 20 March 2024) & suspicious content \\ \hline Demaskator24 & 2018 & www.web.archive.org/web/20190615000 & Only Facebook account exists, no \\  & & (accessed on 20 March 2024) & longer updated \\ \hline Konkret24.pl & 2018 & www.konkret24.trn24.pl & Created by the TVN group \\  & & (accessed on 20 March 2024) &
\begin{tabular}{l} Part of Agence France-Presse \\ (AFP; a multilingual, multicultural \\ news agency) \\ \end{tabular} \\ \hline AntyFAKE & 2019 & www.antyfake.pl & No longer updated \\  & & (accessed on 20 March 2024) & \\ \hline Odfekuj.info & 2020 & www.odfekuj.info & No longer updated \\  & & (accessed on 20 March 2024) & \\ \hline Pravda & 2020 & www.pravda.org.pl & Fact checking of information, statements \\  & & (accessed on 20 March 2024) & and digital content \\ \hline FakeNews.pl & 2020 & www.fakenews.pl & International Fact-Checking \\  & & (accessed on 20 March 2024) & Network member \\ \hline \#FakeHunter & 2020 & www.fake-hunter.pap.pl & Created to check news regarding \\  & & (accessed on 20 March 2024) & SARS-CoV-2 \\ \hline Zglos Trolla & 2022 & www.web.archive.org/web/201906150000 & Created to check news regarding war \\  & & (accessed on 20 March 2024) & in Ukraine \\ \hline \hline \end{tabular} List may not be exhaustive.  
\end{table}
Table 3: List of Polish organizations that investigate media bias and perform fact checking of news in Polish.  
to the scarce appearance of certain categories, similar underrepresented ones were joined together or removed, leaving 14 categories as a result:  
* Appeal to authority;
* Appeal to fear/prejudice;
* Bandwagon, reductio ad hitlerum;
* Black-and-white fallacy;
* Causal oversimplification;
* Doubt;
* Exaggeration, minimization;
* Flag-waving;
* Loaded language;
* Name calling, labeling;
* Repetition;
* Slogans;
* Thought-terminating cliches;
* Whataboutism, straw men, red herring.  
Table 5 shows columns in training dataset.