# Limitations of Large Language Models in Propaganda Detection Task
## 5 Results
### LLM on SemEval2020--English Data  
First, we ran _gpt-3.5-turbo-0125_ and _gpt-4-0125-preview_ three times on the SemEval2020 Task 11 test set. We present the results of the first subtask (SI) in Table 8.  
None of our attempts came close to the best results from the shared task. _gpt-4-0125-preview_ performed better than _gpt-3.5-turbo-0125_, and both models were better than the baseline.  
Next, we checked our annotation for the second subtask (TC) with the golden set and show the results in Table 9.  
The overall F1 score for _gpt-3.5-turbo-0125_ turned out to be the best and outperformed the baseline's value, but all the values oscillated between 20% and 30%. Again, as in the first experiment, _loaded language_ proved to be the easiest to detect, and the best result was obtained by _gpt-3.5-turbo-0125_. We could also observe a slight improvement in the detection of _appeal to fear/prejudice_ and _causal oversimplification_ for _gpt-4-0125-preview_, and  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Model** & **Precision** & **Recall** & **F1** \\ \hline Hitachi & 0.56544 & 0.47368 & 0.51551 \\ ApplicaAI & 0.59954 & 0.41650 & 0.49153 \\ \hline _gpt-4-0125-preview_, try 1 & 0.16059 & 0.20109 & 0.17857 \\ _gpt-4-0125-preview_, try 2 & 0.16659 & 0.23174 & 0.19384 \\ _gpt-4-0125-preview_, try 3 & 0.16595 & 0.23174 & 0.19340 \\ _gpt-3.5-turbo-0125_, try 1 & 0.15132 & 0.06707 & 0.09294 \\ _gpt-3.5-turbo-0125_, try 2 & 0.17226 & 0.06583 & 0.09525 \\ _gpt-3.5-turbo-0125_, try 3 & 0.16172 & 0.07142 & 0.09098 \\ baseline & 0.00320 & 0.13045 & 0.00162 \\ \hline \hline \end{tabular}  
* Due to limited space, only the top 2 results from the SemEval2020 Task 11 shared task are presented.  
\end{table}
Table 8: Task span identification (SI)—results.  
\begin{table}
\begin{tabular}{l c c c} \hline \hline
**Technique—Model** & **Baseline** & **gpt-4-0125-Preview** & **gpt-3.5-turbo-0125** \\ \hline
**F1 score** & 0.25196 & 0.23352 & 0.30000 \\ \hline
**Appeal to authority** & 0.00000 & 0.00000 & 0.00000 \\ \hline
**Appeal to fear/prejudice** & 0.03681 & 0.05128 & 0.01449 \\ \hline
**Bandwagon, reductio ad hitlerum** & 0.00000 & 0.00000 & 0.00000 \\ \hline
**Black-and-white fallacy** & 0.00000 & 0.04211 & 0.03125 \\ \hline
**Causal oversimplification** & 0.11561 & 0.03922 & 0.00000 \\ \hline
**Doubt** & 0.29143 & 0.03265 & 0.00995 \\ \hline
**Exaggeration, minimization** & 0.14420 & 0.05882 & 0.06329 \\ \hline
**Flag-waying** & 0.06195 & 0.04878 & 0.01869 \\ \hline
**Loaded language** & 0.46477 & 0.43468 & 0.47970 \\ \hline
**Name calling, labeling** & 0.00000 & 0.08458 & 0.02778 \\ \hline
**Repetition** & 0.19262 & 0.02643 & 0.03550 \\ \hline
**Slogans** & 0.00000 & 0.00000 & 0.00000 \\ \hline
**Thought-terminating cliches** & 0.00000 & 0.00000 & 0.00000 \\ \hline
**Whataboutism, straw men, red herring** & 0.00000 & 0.00000 & 0.00000 \\ \hline \hline \end{tabular}
\end{table}
Table 9: Task technique classification (TC)—F1 scores.  
for both LLMs for the _black-and-white fallacy_ and _name calling_, _labeling_ categories. None of the approaches could handle _appeal to authority_; _bandwagon_, _reductio ad hitherum; slogans_; _thought-terminating cliches_; nor _whataboutism_, _straw men, red herring_ techniques. Such F1 scores were the result of unbalanced data, in which some techniques had a larger number of occurrences. In other words, we did not see much improvement in comparison with the baseline results, and they were notably worse than the ones obtained by the shared task participants [42].