# Limitations of Large Language Models in Propaganda Detection Task
## 8 Conclusions and Future Work  
Our work shows the results of various experiments with the use of LLMs in propaganda detection tasks. The results show that the outputs of generative models were unpredictable, even if the parameters were set in such a way that should ensure reproducibility. We believe that at this stage, it is too soon to confidently use LLMs for such complex tasks, and other methods should be used for problems that require deeper reasoning. At the same time, further enhancements of LLMs can bring new capabilities, as just scaling such models have shown visible improvement on various NLP benchmarks [81]. One of the biggest obstacles for current propaganda detection studies is the lack of datasets that have full open access and are reliably annotated. We see potential in the further annotation of the PONC subset that could be beneficial for future studies. In order to reduce the cost and workload of such a task, we think that it is possible to use LLMs, such as _gpt-4-0125-preview_, as a method for selecting more examples for the training and testing of a propaganda detection task. Finally, we believe that this approach at a fine-grained level (detecting the exact span with the propaganda technique) might still be difficult for LLMs, but the coarse-grained approach of the binary detection of propagandist news could already be implemented in organizations that fight against misinformation and disinformation. As part of our findings, we also provide an extensive list of organizations that deal with misinformation in online news in Poland that could potentially be interested in automatic propaganda detection. We additionally discovered that GPT models can generate concise outputs in Python code that is easy to process for analyses. Although we decided to use OpenAI's GPT-4, as it is a popular benchmark in recent studies, there are many other LLMs, both open source and paid, that were not tested in our research due to limited GPU resources, such as Gemini, Llama 2, Bloom, Claude, Falcon 180B, OPT-175B, XGen-7B, GPT-NeoX, GPT-J, Gemma, Mistral 7B, Zephyr-7B, Vicuna 13-B or Polish Llama version--QRA. It would be interesting to see the differences in the quality of results between these models in the future. We also plan to experiment with the BERT-based models mentioned as SOTA in previous works to see their performance on the Polish online news. Another idea is to use the updated SemEval2023 dataset from Task 3 on \"Detecting the Category, the Framing, and the Persuasion Techniques in Online News in a Multi-lingual Setup\" [47]. The third subtask involved detecting persuasion techniques in paragraphs of news articles. It used the findings from SemEval2020 Task 11 and proposed a different approach to the problem, namely, a multi-label task at the paragraph level. In other words, instead of finding very specific spans where the propaganda technique was used, it focused more on a general context and asked to find these techniques in whole paragraphs. Additionally, more than one technique could be found in the paragraph. This method seems to be less detailed, but still focuses on the problem and can possibly yield better results than the initial approach. [PERSON] approach is also worth considering [59]--although the method is simplified, the latest LLMs could be used to do a binary check regarding whether a given article is propaganda or not, estimate a probability of this as a percentage and also name the techniques that are used. Furthermore, we could perform further analysis of labeling the PONC articles by an expert in the field of propaganda. GPT results could be used as a suggestion to be verified by a professional so that the task could be performed in a time-efficient manner. It would also be interesting to investigate the overlap of patterns, as well as the differences between the Polish and English languages. Finally, we believe that further analyses with the use of explainable artificial intelligence (XAI) could help to understand and interpret the generated outputs. Conceptualization, J.S. and R.R.