Leveraging Advanced Prompting Strategies in Llama-8b for Enhanced Hyperpartisan News Detection
Authors: Michele Joshua Maggini, Erik Bran Marino, Pablo Gamallo Otero (University of Santiago de Compostela, Universiti of Évora, 2024)
Topic: Fact Checking and Fake News Detection

Abstract
This paper explores advanced prompting strategies for hyperpartisan news detection using the Llama3-8b-Instruct model, an open-source LLM developed by Meta AI. We evaluate zero-shot, few-shot, and Chain-of-Thought (CoT) techniques on two datasets: SemEval-2019 Task 4 and a headline-specific corpus. Collaborating with a political science expert, we incorporate domain-specific knowledge and structured reasoning steps into our prompts, particularly for the CoT approach. Our findings reveal that some prompting strategies work better than others, specifically on LLaMA, depending on the dataset and the task. This unexpected result challenges assumptions about ICL efficacy on classification tasks. We discuss the implications of these findings for In-Context Learning (ICL) in political text analysis and suggest directions for future research in leveraging large language models for nuanced content classification tasks.

Introduction
%motivations The proliferation of hyperpartisan news content in digital media has become a significant challenge for modern societies, potentially undermining democratic processes and social cohesion. Hypepartisan news consists of politically polarized content presented through the usage of rhetorical bias. In the media landscape, news outlets disseminate information using proprietary websites and social networks. Each news outlet frames the narratives of the facts based on their political leaning, influencing the content with rhetorical biases, emotional purposes, ideology, and reporting the facts while omitting parts AUTHOR. To improve the virality of the news, even mainstream journalists adopted click-bait practices like eye-catching titles AUTHOR. Furthermore, the news not only stands for one opinion but could have an underlying political background that manifests through a specific vocabulary or assumptions against the opposite political leaning AUTHOR. This type of news could radicalize the voters because of their emotional language AUTHOR. When there is a massive usage of these techniques, we can consider news extremely partisan toward a particular political leaning. Although hyperpartisan news can share traits with misinformation and disinformation, it cannot be classified within these domains because the intent is not deceptive. For this reason, hyperpartisan news detection is closer to propaganda. Recent advancements in large language models (LLMs) have opened new avenues for tackling complex NLP tasks, including detecting nuanced linguistic phenomena such as bias and partisanship. Among these models, LLama3 AUTHOR, developed by Meta AI. This research makes use of the new LLM recently released by Meta AI, Llama3-8b-Instruct, fine-tuned and optimized for dialogue/chat use cases, to explore its application in the detection of both hyperpartisan news headlines and articles. LLMs can be prompted with instructions to perform classification tasks. Thus, we intend to use this open source model. In our case, by prompting the model with instructions and context, we are in the In-Context Learning (ICL) domain, a learning approach different from fine-tuning that does not require to update models' weights AUTHOR. The study aims to investigate the efficiency and compare the performances of the following ICL techniques: 0-shot with a general prompt and a specific prompt, few-shot with a different number of examples and Multi-task Guided CoT. We investigate how carefully crafted prompts with the help of a political expert can guide the model to identify subtle indicators of extreme political bias in news articles, leveraging the model's deep understanding of language and context. Our approach aims to overcome the limitations of traditional machine learning methods, which often struggle with the complex and evolving nature of partisan language. Furthermore, we can include definitions of the political phenomena of our interest in the prompt to further define the task and narrow the application domain. By focusing on ICL to provide context and background information, we seek to: \item Develop a flexible and adaptable system that can identify hyperpartisan content across various topics and writing styles without the need for extensive retraining; \item Reduce ambiguity and guide the model towards the desired outcome; \item Minimize the influence of biases in the training data, by incorporating diverse perspectives and examples. This research not only contributes to the field of automated content analysis but also aims to compare the efficiency of prompting techniques and to analyze if LLMs are valuable tools for classification task via ICL. The structure of the paper is as follows. In section RelatedWork we discuss the related literature; section ExperimentalSetting describes the experimental set-up we adopted and the methodology; section Results&Discussion covers the findings of our experiment comparing them based on the method used and highlights the limitation of our approach; section Conclusion reports the main findings and future research. The main contributions of the paper are the following: \item We evaluated the state-of-the-art model Llama3-8b-Instruct on two benchmark datasets in political domain; \item We assessed how well the model performs under different inference approaches: zero-shot learning, few-shot learning, and Multi-task Guided Chain-of-Thought reasoning \item Introduction of external in-domain knowledge in the prompt and segmentation of reasoning steps in the CoT considering the difficulty of the micro-tasks.

Conclusion
In this paper, we study the reliability of a SOTA model like Llama3-8b-Instruct for classification tasks in the political domain, namely to detect hyperpartisan articles and headlines comparing different prompting techniques. We cast the problem of the classification task using the generative capabilities of LLMs. Experiment results contradict the hypothesis that feeding the model with more context could lead to better performances AUTHOR. Indeed, in our case, the 0-shot approach was the most efficient. An interesting future direction would be building a new dataset of instructions to improve models' capability in zero-shot AUTHOR in identifying hyperpartisan news, inspired by datasets used for false information detection, such as Truthful-QA AUTHOR. Indeed, this dataset could be used to fine-tune generative models to enhance their performance. Additionally, we plan to explore more sophisticated prompting techniques in zero-shot and few-shot settings like prompt tuning in the political domain AUTHOR. Finally, we would like to investigate Retrieval-Augmented Generation (RAG) and implement neuro-symbolic strategies, to incorporate retrieved documents or knowledge bases into the process. By pursuing these research directions, we aim to develop more effective and reliable systems for detecting hyperpartisan news and promoting media literacy. This work is supported by the EUHORIZON2021 European Union's Horizon Europe research and innovation programme () the Marie Skłodowska-Curie Grant No.: 101073351. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Executive Agency (REA). Neither the European Union nor the granting authority can be held responsible for them. The authors have no relevant financial or non-financial interests to disclose.