Venue: NAACL 2024

# Media Bias Detection Across Families of Language Models

Iffat Maab1, Edison Marrese-Taylor1,2, Sebastian Pado3, Yutaka Matsuo1

###### Abstract

Bias in reporting can influence the public's opinion on relevant societal issues. Examples include informational bias (selective presentation of content) and lexical bias (specific framing of content through linguistic choices). The recognition of media bias is arguably an area where NLP can contribute to the "social good". Traditional NLP models have shown good performance in classifying media bias, but require careful model design and extensive tuning. In this paper, we ask how well prompting of large language models can recognize media bias. Through an extensive empirical study including a wide selection of pre-trained models, we find that prompt-based techniques can deliver comparable performance to traditional models with greatly reduced effort and that, similar to traditional models, the availability of context substantially improves results. We further show that larger models can leverage different kinds of context simultaneously, obtaining further performance improvements.

The University of Tokyo

2National Institute of Advanced Industrial Science and Technology

3University of Stuttgart

{iffatmaab, emarrese, matsuo}@weblab.t.u-tokyo.ac.jp pado@ims.uni-stuttgart.de

## 1 Introduction

Both mass media and social media are potent channels for expressing viewpoints and shaping decisions. News outlets stand out, exerting a pivotal influence on altering and molding individual and collective perspectives Entman (2007); Hamborg et al. (2019). Clearly, biased media has the ability to influence people, for instance, in a study by Fletcher and Park (2017), it is revealed that there is a negative association between trust in the news media and online news participation. Hamborg et al. (2019) highlight that distinctive contributions can be made by computer scientists to study _bias_. We follow them in defining bias as subjective standpoints manifested variously in, for example, word choice, framing, intentional omission or misrepresentation of specific details Lin et al. (2006); Iyyer et al. (2014); Rashkin et al. (2017).

In NLP, media bias has been scrutinized in different ways using different names Pan et al. (2018); Perez-Rosas et al. (2017). An important step towards a common perspective was the development of the Bias Annotation Spans on the Informational Level (BASIL) dataset, which established the specific concepts of _information bias_ and _lexical bias_Fan et al. (2019). Informational bias refers to selective presentation of content in a factual manner to sway opinion of readers van den Berg and Markert (2020); Fan et al. (2019), whereas lexical bias refers to linguistic attributes like word selection and syntax Hube and Fetahu (2019); Greene and Resnik (2009); Iyyer et al. (2014).

These types of bias have been studied with supervised learning methods Lei et al. (2022); van den Berg and Markert (2020); Lee et al. (2021); Fan et al. (2019); Guo and Zhu (2022a); Maab et al. (2023a). However, so far the task proves challenging and often results in unsatisfactory performance when trained with a limited set of labeled examples. This can be due to the expensive nature of data annotation and extensive variety of domains, languages, and tasks Chen et al. (2020); Akhter et al. (2020). This is particularly true given the importance of context in the identification of bias van den Berg and Markert (2020); Guo and Zhu (2022a); Lei et al. (2022); Lee et al. (2021); Maab et al. (2023a, b). For example, in the BASIL dataset, where political bias is identified at the sentence-level, it has been shown that _informational bias_ depends fundamentally on the context of the sentence Guo and Zhu (2022b); van den Berg and Markert (2020), arises from manipulation of information or selective presentation of content in a factual way, e.g., use of quotes, to evoke reader's emotions towards news entities Fan et al. (2019); van den Berg and Markert (2020).

Thus, recent work on automatic bias detection in news has focused on identifying the right context to present during training. Context ranges from

whole articles (van den Berg and Markert, 2020), to just sentences surrounding the target sentence Guo and Zhu (2022), and other sentences discussing the same entity sampled from the article Maab et al. (2023). Although these approaches lead to substantial improvements, all of them have so far focused on model fine-tuning, which limits their applicability in broader scenarios.

This paper investigates whether bias detection can profit from current developments on large language models (LLMs), which can achieve excellent performance on a wide variety of downstream tasks utilizing zero-shot or few-shot approaches Brown et al. (2020); Kojima et al. (2022), i.e., without fine-tuning. Previous work Beltagy et al. (2022); Shin et al. (2020); Schick and Schutze (2020) shows the significance of in-context learning for a large variety of tasks. Our work continues in this direction, and we propose to test prompting strategies for bias detection using LLMs. Our prompt design is motivated to a large extent by its _flexibility_ as an experimental platform: we can investigate the interaction between various experimental variables, the most important of which are: (a) the style of the prompt (concise vs. detailed); (b) the amount of document context presented to the model; (c) the amount of supervision afforded to the model (zero-shot vs. few-shot). We carry out experiments regarding these variables on a range of current LLMs, empirically evaluating zero-shot and few-shot learning approaches using task-specific prompts to detect political bias across a wide range of LLMs. To the best of our knowledge, our work is the first to attempt this kind of investigation.

Our work shows the importance of the specificity and detail in prompt engineering, which is a highly task-dependent and a potentially cumbersome process. We additionally show that providing an LLM with more appropriate and consistent bias contexts, either in the form of examples, or of related sentences, can approximate conventional supervised learning approaches, setting up a research direction for the future.

## 2 Related Work

The societal consequence of misinformation in various fields are substantial. News articles can contain biased opinion, resulting in misleading views Gentzkow and Shapiro (2010). Political scientists have determined that bias in news reporting can be defined by the choices in content selection and organization of information within articles Prat and Stromberg (2013); Gentzkow et al. (2015). According to Shapiro (2016), bias can also be introduced when less information or facts on news articles is provided because for journalists who try to appear neutral, avoids conveying ample information. In another study, Fletcher and Park (2017) show a negative association between trust in the news media and online news participation.

Since news media advance their interests by devoting resources to control reporting Entman (2007); Chang et al. (2019), the NLP field has devoted attention to predicting the political leaning and trustworthiness of news media outlets Baly et al. (2019); Mehta et al. (2022). Regarding bias in news, the introduction of the BASIL dataset Fan et al. (2019) marks the onset of interesting paradigms. Within BASIL, sentences are annotated with their corresponding bias type, a designated target (the primary entity), and several other labels. Prior studies on BASIL have shown that the incorporation of _contextual information_ enhances supervised learning models: van den Berg and Markert (2020) utilized article and event-level context; Guo and Zhu (2022); Maab et al. (2023) integrate three levels of context, i.e., article, event, and adjacent sentences, and Chen et al. (2020) utilized second order bias features to detect article-level bias.

Generative models are outlined as complex systems by Holtzman et al. (2023) due to their tendency to exhibit emergent behaviors, for example, the ability of LLMs to perform in-context learning Tornberg (2023). Along these lines, instructing LLMs with prompt in zero or few-shot settings enables models to perform tasks with no or minimal task-specific training, which is evidence of the LLMs' generalization capability Gao et al. (2020), and has put LLMs at the center of recent progress in NLP Devlin et al. (2018); Brown et al. (2020); Thoppilan et al. (2022); Rae et al. (2021); Liu et al. (2022).

In essence, LLMs offer a broad spectrum of beneficial applications that extend beyond their generative functionalities. Task division into multiple concise steps and sequentially introducing them to the large language model yields improved performance across an array of reasoning tasks, spanning word problems, arithmetic operations, and code execution Anil et al. (2022); Hao et al. (2022). Notably, GPT models have achieved success in diverse 

language-related endeavors, showcasing their capacity to produce text resembling human-written content [14, 25].

## 3 Method

Recent studies have shown that fine-tuning of LLMs to instruction-style prompts is a common approach to achieve gains in performance [8, 9, 60, 7]. However, language model fine-tuning is also computationally very expensive. Consequently, based on our comprehensive analysis on how models for language representation have been incorporating broader contextual scopes into their predictive processes, for example, models have evolved to consider neighboring words [15], sentences [16, 17, 18], paragraphs [19, 12], and even articles [20, 21], we align analogous contexts with prompts to learn the capacity of context dependence across different large language models.

Our basic setting adopts a zero-shot approach with prompts shown in Table 1. We follow Maab et al. (2023) by modeling the two bias classification tasks of INF/OTH and INF/LEX, where 'INF' refers to sentences with informational bias, 'LEX' refers to sentences with lexical bias and 'OTH' denotes the combination of neutral and lexically biased sentences. We employ two prompts, as detailed in Table 1, a simple one (Concis), and one with a definition of the two bias types (Detailed). Our goal is to assess how the difference between these prompts influences the LLMs' capability for bias classification. We evaluate the two prompts in the following settings.

Zero-shotIn this approach, using zero-shot means we do not give the model any knowledge of question-answer pairs as to how a certain sentence gets classified as a particular label, nor fine-tuned the language models in any aspect.

Context-augmented (+ctx)Given the context-sensitive of the task, we find that a large portion of previous work in detection of political bias on BASIL has focused on introducing contextual information into classification [1, 20, 6], for example, by mixing contexts of informational and lexical bias at both the article-level (entire article encompassing target sentence) and event-level (triplet of articles discussing the same event). In this context, recent work by Maab et al. (2023) proposed a comprehensive framework to generate more consistent and similar bias contexts to improve performance when fine-tuning models. In this paper, we propose to adapt these techniques for the task of zero-shot and few-shot political bias detection. Concretely, we take advantage of three context-augmented techniques:

**Bias-Aware Neighborhood (BANC)**: extends the target sentence with neighboring spans, i.e., combining former and next sentence with target sentence.
**Article-Based Target-Aware (ABTA)**: extends this idea by extending the target sentence with spans from the same article which share both the same bias label (bias type of BASIL, i.e., informational and lexical) and the same target, i.e., main entity or topic of the target sentence already annotated in BASIL (cf. Figure 0(a)).
**Event-Based Target-Aware (EBTA)**: extends the ABTA approach by gathering spans for the same bias label and target across articles, i.e., triplets of BASIL (cf. Figure 0(b)).

Utilizing annotated bias spans gathered through these approaches are added to the input prompt, which effectively mix relevant local and global contexts of informational and lexical bias at the article and event levels.

Few-shot (+k-shots)Our experiments with few-shot learning are parallel to the zero-shot case, except that we present the model with question-answer (i.e., sentence-label) pairs. Since the available data is limited, the choice of parameter \(k\) (the number of examples provided) is dependent on the task. More details about this are given in SS4.

## 4 Experimental Setup

### Tasks and Data

Utilizing the target-aware contextualization techniques by maab2023, we incorporate a total of 4,987 informational and 1,551 lexical contextualized sentences. The remaining 6,250 sentences named "OTH", originating from the foundational work by fan2019, do not exhibit bias. Our approach aligns with the task formulations outlined in maab2023,b for both the INF/OTH and INF/LEX tasks, as mentioned in SS3. For comparison of sentences devoid of bias contexts, we 

employ the original BASIL sentences, comprising 1,221 examples of informational and 462 examples of lexical bias Fan et al. (2019). Table 2 summarizes the details of the examples available for each task and setting.

Regarding k-shots experiments, in the INF/OTH task, we employ 5-shot examples, comprising 3 examples derived from INF sentences and 2 examples from OTH. For the INF/LEX task, we utilize 3 INF examples and 3 LEX examples. The slight variation in the k-shots of the INF/LEX task aims to ensure that the model receives equal exposure to LEX bias information, considering its limited size.

### Models

For all tasks, large language models are used without any fine-tuning or gradient updates, and the zero-shot and few-shot experiments are specified only as prompts. Furthermore, supplementary experiments are also conducted to involve k-shot experiments to delve into the influence of \(k\) examples on model performance. We consider a range of LLMs with various parameter counts and in-context learning abilities as measured by standard benchmarks. We also consider models that are instruction fine-tuned. Within our proposed framework, concretely, we work with the family of FLAN instruction-tuned models which includes FLAN-T5-Base, FLAN-T5-XL, FLAN-T5-XXL, Chung et al. (2022) and FLAN-UL2 Tay et al. (2022). We also consider two publicly-available regular (causal) LLMs, namely GPT-Neo Black et al. (2018),

Figure 1: Two contextualization methods for a target sentence with informational bias on the target “Barack Obama” and three news sources of BASIL i.e., FOX, HPO, and NYT.

 

2021) and OPT (Zhang et al., 2022). Finally, we consider the two popular black-box models ChatGPT/ GPT-3.5-turbo and GPT-4. Table 3 provides detailed properties on the LLMs that we use in our study.

### Answer Parsing

Some models, like the ones from the FLAN family, which are instruction-tuned solely, produce the necessary target label, which makes interpretation trivial. Other model families, like GPT-Neo and OPT, sometimes return long and intricate answers which require additional mapping onto a label. We select the first portion of the response that is interpretable as an answer, and if no such answer exists, we treat the prediction as incorrect. See Appendix A for details.

### Hardware

Our experiments were performed on two kinds of GPUs, 16-GB NVIDIA V100 and 40-GB NVIDIA A100. We access these GPUs by means of nodes on a large cluster, where each node has several such GPUs. Some experiments were performed using data parallelism to speed up inference time.

### Implementation Details

We use PyTorch to implement models, borrowing from HuggingFace (Face, 2021) for FLAN-T5 (Chung et al., 2022), FLAN-UL2 (Ty et al., 2022), GPT-NeO (Black et al., 2021), and OPT (Zhang et al., 2022). For ChatGPT/ GPT-3.5-turbo and GPT-4, we rely on the official OpenAI API. GPT-3.5-turbo and GPT-4 were tested with low and high settings of temperature (0, 0.5, and 0.8), however we did not observe much variation between the generated texts.

## 5 Results

On our first set of experiments, we use the Concise prompt, which we test on regular baseline (zero-shot) and also in the CTX setting. Results are shown in Table 4. After pilot experiments, we observed that models could only provide marginal performance improvements when provided with long contexts, so we did not consider few-shot learning using this prompt. In Table 4, overall we observe that with target-aware contextual information CTX, advance LLMs result in enhancements in F1-scores and accuracy for both INF/OTH and INF/LEX bias tasks. For instance, when context is utilized for FLAN-T5 and its variants, FLAN-UL2, GPT-3.5-turbo, and GPT-4, INF/OTH task shows a rise in INF-F1 scores against regular, respectively. It can be seen that GPT-4 substantially

\begin{table}
\begin{tabular}{l c c c c c c} \hline \multicolumn{7}{c}{Inf/ Oth} & \multicolumn{3}{c}{Iner/ Lex} \\  & **F1 Score** & **F1 Score** & **F1 Score** & **F1 Score** \\
**Model** & **Acc.** & **INF** & **OTH** & **Acc.** & **INF** & **LEX** \\ \hline FLAN-T5-Base & 46.98 & 35.32 & 87.11 & 39.98 & 32.77 & 11.43 \\ + ctx & 59.61 & 36.14 & 67.43 & 51.67 & 38.99 & 19.33 \\ \hline FLAN-T5-XL & 78.36 & 40.66 & 71.23 & 55.98 & 40.45 & 20.12 \\ +ctx & 76.79 & 39.32 & 63.38 & 59.10 & 46.84 & 27.98 \\ \hline FLAN-T5-XXL & 71.13 & 41.87 & 69.43 & 52.55 & 47.09 & 38.39 \\ +ctx & 73.67 & 44.89 & 58.34 & 60.86 & 48.22 & 33.00 \\ \hline FLAN-UL2 & 73.09 & 43.34 & 65.22 & 69.34 & 48.83 & 35.17 \\ +ctx & 73.06 & 44.96 & 60.34 & 68.99 & **51.02** & 38.64 \\ \hline GPT-3.5-turbo & 74.46 & 47.87 & 68.09 & 68.98 & 48.11 & 37.53 \\ +ctx & 75.50 & 49.55 & 70.34 & 70.07 & 48.91 & 36.05 \\ \hline GPT-4 & 77.11 & 51.66 & 71.56 & 72.21 & 47.03 & 38.27 \\ +ctx & 77.20 & **54.04** & 69.89 & **74.45** & 49.90 & **40.71** \\ \hline GPT-Neo & 62.24 & 50.01 & 78.23 & 50.36 & 39.91 & 30.23 \\ +ctx & 71.38 & 47.43 & 63.42 & 58.27 & 42.93 & 28.26 \\ \hline OPT & **83.01** & 44.07 & **79.03** & 48.78 & 43.12 & 31.82 \\ +ctx & 76.24 & 46.11 & 53.98 & 69.33 & 45.25 & 33.15 \\ \hline \end{tabular}
\end{table}
Table 4: Summary of our results with the Concisbprompt in terms of accuracy and micro-F1 scores. Results in bold indicate the best performance on each task and metric across, models and settings.

\begin{table}
\begin{tabular}{r l r} \hline \hline
**Model** & **Training Data** & **\# Par.** \\ \hline Flan-T5 & Based on TS and Raffelt (2019) & \\ (Google) & and later fine-tuned on I,836 tasks by combining four mixtures from prior work, Muffin (80 tasks) Wei et al. (2021), T0-SF (193 tasks) Sanh et al. (2021), NYQ (1,554 tasks) Wang et al. (2022), and CoT manual annotations for reasoning tasks & \\ \multirow{2}{*}{FLAN-UL2 (Google)} & Model pre-trained on hundreds of gigabytes of clean English text named & \\  & “Colossal Clean Crawled Corpus” & \\  & (C4)Raffel et al. (2019), acquired by web scraping, and then same as above. & \\ \multirow{3}{*}{GPP-3.5 (OpenAI)} & Reportedly, trained on hundreds of dilations data from a diverse range of data sources, including Common Crawld, web documents, books, and Wikipedia Brown et al. (2020). & \\ \multirow{3}{*}{GFT-4 (OpenAI)} & Reportedly, a model based on Mixture-of-Experts, which consists of 16 different experts working together, each with approx. 1100 parameters and trained for a specific task/field. & \\ \multirow{3}{*}{OPT} & Causal language model pre-trained on five datasets: three RoBERTA (Meta AI) datasets Liu et al. (2019), the Pile Gao et al. (2020), uPhushrifitto Reddit Baumgartner et al. (2020). & \\ \multirow{3}{*}{GPT-Neo (EleutherAI)} & Causal language model pre-trained on the File Gao et al. (2020) & \\ \cline{1-1} \cline{2-2}  & & 1.3B– \\ \cline{1-1} \cline{2-2}  & & 1.

 

outperforms other models, where INF/OTH task shows INF F1-score of 54.04 against 51.66 of regular, and INF/LEX task shows INF F1-score of 49.90 against 47.03 of regular, respectively. For GPT-Neo, it is found that they perform well even with no context augmented information. However, GPT-Neo encounter challenges when working with extended contexts Yang et al. (2022). OPT also seems to have increased performance with CTX given concise prompts as discussed in section 5.3.

Owing to the fact that contextual information raise performance in LLMs, our findings are well-aligned with prior work, which implies that LLMs are capable of classifying certain kinds of bias. We include additional experiments on context sensitivity to assess LLM capacities in comprehending contextual information. These experiments involve extensive prompts, and additional tests are conducted with k-shot context examples. Table 5 shows a summary of our experiments. Through this study, we aim to determine whether context enhances the model's ability to efficiently process more extensive prompts when combined with k-shot examples.

When we combine k-shot with context examples, i.e., ctx + k-shots, we note that the performance starts improving against zero-shot and simple k-shot experiments. Using an extensive prompt, our results further demonstrate the effectiveness of context-augmented zero-shot experiments. For instance, for the ctx scenario, FLAN-UL2 shows INF F1- score of 58.21 against 53.08 of k-shots in INF/OTH task, whereas we observe INF F1-scores of 59.55 against 58.03 of k-shots for the INF/LEX task, respectively. Similarly, an increase in the LEX F1-score is noticeable in the INF/LEX task for most of the LLMs as we go down from regular (zero-shot) to contextualized information, i.e., following the trend from regular to k-shots to ctx and finally to both ctx + k-shots. Consequently, we note that that best performance in INF F1-score of 69.07 in INF/OTH and 64.55 in INF/LEX, is achieved by GPT-4 when presented with context-augmented k-shot examples.

Most notably, using a more detailed definitions of bias types (cf. Table 5) led to consistent performance improvements over results in Table 4, especially in instruction-tuned models such as FLAN-T5 and its variants, GPT-3.5-turbo, and GPT-4. Our findings also align with the notion that compact models, such as GPT-Neo and OPT, featuring 2.7B parameters, tend to produce more precise results when presented with concise prompts. This is reflected in increased performance of INF/OTH and INF/LEX bias tasks of the concise prompt (Table 4) over the detailed prompt (Table 5).

Overall, it is observed that LLMs benefits the most from informational context examples, i.e., INF bias over non-bias OTH i.e., INF/OTH task. Furthermore, we note that performance improvement of Detailed prompt over Concise prompt is more prominent in instruction-tuned models including FLAN-T5 variants, FLAN-UL2, GPT-3.5-turbo and GPT-4, whereas the Concise prompt holds more significance when presented to GPT-Neo and OPT models. We think this is because smaller language models tend to struggle when presented with long contexts, being essentially unable to parse the information provided, even if it is potentially use

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{2}{c}{INF/OTH} & \multicolumn{2}{c}{Inf/LEx} \\  & **F1 Score** & **FI Score** & **FI Score** & **FI Score** \\
**Model** & **Acc.** & **INV** & **OTH** & **Acc.** & **INVE** & **LEX** \\ \hline FLAN-T5-Base & 60.48 & 35.81 & 87.43 & 42.69 & 39.97 & 23.14 \\   + K-shoots & 72.81 & 40.66 & 87.00 & 49.39 & 45.55 & 31.96 \\   + ctx & 73.03 & 42.81 & 89.28 & 49.67 & 49.88 & 28.90 \\   +ctx + k-shots & 69.61 & 42.14 & 87.43 & 61.67 & 51.18 & 33.88 \\ \hline FLAN-T5-XL & 64.08 & 39.32 & 82.83 & 53.20 & 54.21 & 40.90 \\   + kt-shots & 65.89 & 43.45 & 84.11 & 63.01 & 52.59 & 38.18 \\   + CTX & 61.93 & 44.09 & 80.24 & 69.29 & 57.23 & 36.76 \\   + ctx + k-shots & 71.79 & 53.32 & 82.38 & 67.00 & 54.11 & 40.35 \\ \hline FLAN-T5-XX & 74.47 & 46.56 & 77.34 & 59.16 & 53.83 & 42.42 \\   + sk-shots & 80.34 & 51.68 & 78.77 & 70.82 & 55.35 & 38.65 \\   + ctx & 78.35 & 48.21 & 79.09 & 72.88 & 56.74 & 38.73 \\   + ctx + x-shots & 76.67 & 60.71 & 77.34 & 75.34 & 55.09 & 38.12 \\ \hline FLAN-UL2 & 78.50 & 49.99 & 79.00 & 73.08 & 54.77 & 37.33 \\   + sk-shots & 80.71 & 53.08 & 73.32 & 78.37 & 58.03 & 36.51 \\   + ctx & 77.29 & 58.21 & 74.31 & 71.34 & 95.95 & 39.10 \\   + ctx + k-shots & 78.87 & 61.08 & 69.40 & 81.89 & 62.91 & 41.36 \\ \hline GPT-3.5-turbo & 70.44 & 50.90 & 73.13 & 71.08 & 52.54 & 37.77 \\   + st-shots & 71.71 & 54.54 & 73.43 & 73.47 & 54.88 & 38.87 \\   + ctx & 74.09 & 62.08 & 74.70 & 72.69 & 59.69 & 41.36 \\   + ctx + tx-shots & 75.98 & 65.86 & 75.83 & 76.08 & 60.89 & 40.58 \\ \hline GPT-4 & 73.08 & 53.16 & 75.88 & 70.45 & 57.16 & 40.04 \\   + sk-shots & 71.98 & 55.23 & **76.01** & 24.55 & 59.21 & 38.20 \\   + crx & 75.54 & 67.87 & 73.08 & 75.01 & 63.06 & 40.89 \\   + ctx + k-shots & 78.88 & **69.07** & 73.59 & **82.99** & **64.55** & **42.76** \\ \hline GPT-No & 67.09 & 39.53 & 60.22 & 62.12 & 40.88 & 33.70 \\   + s-shots & 61.22 & 50.08 & 44.16 & 64.48 & 42.09 & 35.19 \\   + ctx & **81.09** & 43.36 & 67.21 & 39.89 & 41.12 & 32.43 \\   + ctx + cls-shots & 56.24 & 44.90 & 56.23 & 57.81 & 40.03 & 35.54 \\ \hline OPT & 68.00 & 42.12 & 54.41 & 64.46 & 41.12 & 30.06 \\   + k-shots & 70.09 & 40.80 & 60.99 & 61.08 & 39.84 & 33.34 \\   + ctx & 72.06 & 45.56 & 65.11 & 74.54 & 40.02 & 31.50 \\   + ctx + k-shots & 68.21 & 43.96 & 45.98 & 68.50 & 42.99 & 30.87 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Results of our experiments with the Detailed prompt, in terms of accuracy and micro-F1 scores, detailizing the performance impact when adding context as discussed in §4. Results in bold indicate the best performance overall, per metric.

 

ful for the task at hand. Finally, we also find that GPT-4 outperforms GPT-3.5-turbo.

### Comparison to prior work

We compare our approach against state-of-the-art works using model fine-tuning. To the best of our knowledge, no previous studies have employed LLMs for zero-shot or few shot political bias classification. Table 6 presents best performance outcomes for all selected models, as determined from the findings in Table 5 against state-of-the-art work. In case of OPT and GPT-Neo, we present the best results achieved per model, taken from both Table 4 and Table 5. We see that GPT-4, GPT-3.5-turbo, FLAN-UL2, and FLAN-T5-XXL achieve higher performance showing increased INF F1-score of 69.07, 65.86, 61.08, and 60.71, respectively, in the INF/OTH task, surpassing the best BERT model by Maab et al. (2023) with INF F1-score of 58.15. We find that LLMs with enhanced capabilities of context utilization hold significance in contributing towards higher performance over baselines including BERT Chen et al. (2020), RoBERTa Lei et al. (2022), MultiCTX that uses multi-contrast learning across BASIL articles Guo and Zhu (2022), and target-aware BERT Maab et al. (2023).

To detect informational bias from lexical bias sentences, i.e., INF/LEX task, the only baseline available is the BERT model by Maab et al. (2023). As can be seen, we find that no LLM model is able to outperform the existing prior work in INF/LEX task, i.e., the top two performance in terms of INF F1-score of 75.46 and 74.01 is achieved by Maab et al. (2023), followed by the INF F1-score of 64.55 with GPT-4 as third, and FLAN-UL2 as fourth with only 62.91. Since distinguishing particular types of bias, i.e., between informational and lexical bias in INF/LEX is a challenging task, and no training is performed while utilizing LLMs, none of our models obtain a performance superior to prior models with dedicated training. See Appendix B for robustness study against baseline BERT models of Maab et al. (2023, 20).

### Role of \(k\)

We also explore the impact of k-shot context examples ctx + k-shots to uncover the significance of context and the extent to which it affects model performance. For this purpose, we choose GPT-4, our overall best performing model, and experiment with variations of \(k\) using k-shot experiments to analyze the changes attributed by contexts examples in INF/OTH and INF/LEX bias tasks. We present \(k\)=2, 4, 6, 8, 10 context augmented examples to GPT-4. For this experiment, we allocate an equal number of examples for \(k\)-shot learning in both INF/OTH and INF/LEX bias tasks. For instance, in INF/OTH task, 2-shot experiment entails one example originating from context-augmented INF bias, and the other from OTH. The same is true for INF/LEX, where one example is sourced from context-augmented INF, while the other is from context-augmented LEX bias.

As shown in Figure 2, the F1-score of INF and LEX bias demonstrates a continuous increase as

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline  & \multicolumn{3}{c}{Inf/Orth} & \multicolumn{2}{c}{INF/Lex} \\ \cline{2-6}  & \multicolumn{3}{c}{**F1 Score**} \\ \cline{2-6} Model & Acc. & INF & OTH & Acc. & INF & LEX \\ \hline FLAN-T5-Base & 73.03\% & 42.81 & **89.28** & 61.67\% & 51.18 & 33.88 \\ FLAN-T5-XL & 71.79\% & 53.32 & 82.38 & 69.29\% & 57.23 & 36.76 \\ FLAN-T5-XVL & 76.67\% & 60.17 & 77.34 & 72.88\% & 56.74 & 38.73 \\ FLAN-UL2 & 78.78\% & 61.08 & 69.40 & 81.89\% & 62.91 & 41.36 \\ GPT-3.5-turbo & 75.98\% & 65.86 & 75.83 & 76.08\% & 60.89 & 40.58 \\ GPT-4.17\% & 78.88\% & **69.07** & 73.59 & 28.99\% & 64.55 & 42.76 \\ GPT-Nee 2.7B & 61.22\% & 50.08 & 44.16 & 52.74\% & 42.93 & 82.26 \\ OPT 2.7B & 76.24\% & 46.11 & 53.98 & 69.33\% & 45.25 & 33.15 \\ \hline BERT & - & 41.46 & - & - & - & - \\ RoBERTa & - & 46.47 & - & - & - & - \\ AttCIM & - & 42.80 & - & - & - & - \\ WinSSC & - & 37.58 & - & - & - & - \\ EvCMM & - & 45.81 & - & - & - & - \\ MultiCTX & - & 46.08 & - & - & - \\ BERT+ctx & 84.90\% & 56.88 & - & 83.36\% & 74.01 & 66.97 \\ BERT+ctx (**\(\ast\)\(\ast\ 

we increment the number of shots provided, uncovering the significance of context in the manner of examples. It is also observed that no significant performance change occurs after k approaches 6, however there is a slight rise in F1-scores of INF and LEX from 6-shot, when 10-shot contextual examples are introduced. These results indicate that having an augmented context as shot as input to the LLM leads to a notable performance improvement, while GPT-4 only needs a few examples for classifying political bias on BASIL.

### Error Analysis

To gain a deeper insight into the zero-shot abilities of LLMs in our tasks, we performed a qualitative analysis of the predictions made by GPT-4 when subject to different prompting techniques based on context. Table 7 shows samples of correct and incorrect contexts extracted from the different sources (article/event) which result in different predictions, deviating the model from its baseline performance. We noted that in those examples where the context aligns with the target sentence, the model is generally able to make the right prediction. However, there are still cases where there are nuanced representations of context, on which the model faces difficulties in understanding the overall input, resulting in incorrect predictions.

We also compare the performance of fine-tuned BERTs Fan et al. (2019); Chen et al. (2020); Maab et al. (2023a,b) against our approach with GPT-4 on a set of a hundred randomly selected samples from both informational bias and lexical bias categories without context, denoted as "Regular (No context)." These were then contrasted with augmented context samples of the same examples using BANC, ABTA, and EBTA from Maab et al. (2023b). See Appendix B for additional details on this study.

Figure 3 shows confusion matrices summarizing our obtained results, where GPT-4 exhibits difficulties in predicting lexical bias. The reasons for the weak performance of LLMs in detecting INF/LEX bias, in comparison to pre-trained BERT models, can be summarized by various factors. Firstly, the INF/LEX bias detection task is more challenging due to the potential scarcity of lexical bias sentences even after context-augmentation, and their limited span within bias sentences. Notably, informational bias spans predominantly encapsulate entire sentences, while lexical bias spans are ob

\begin{table}
\begin{tabular}{c c} \hline \hline
**Model Input** & **Pred.** \\ \hline \multicolumn{2}{l}{At one point he suggested that those who opposed his position were} & X \\  & hearless. _Nail door r want to build a fence, Romney said. You_ \\  & _ital about aspects, you put it place a magnet. Perry has also_ \\  & _ital to feel forward personnel should extend work visas permitting_ \\  & _undocumented immigrants to move freely between the U.S. and_ \\  & _their home countries, but stressed that he will optense amnestry or_ \\  & _a path to citizenship._ \\ \hline \hline \end{tabular} 
\begin{tabular}{c c} \hline \hline On that same day, Obama had lunch with Clinton. & X \\  & _The report comes one day after President Obama insisted nothing_ \\  & _improper happened with Senata._ & On that same day, Obama had \\  & lunch with Clinton. This is _guinishing brute phone._ \\ \hline \multirow{5}{*}{_I think_} & On that same day, Obama had lunch with Clinton. _As the Hoffeston Post reported on Thursday, various political historians and_ \\  & _ethics lyners have approached the Sextat news with years, noting_ \\  & _that grid post post and brackroom job offers are fairly common in_ \\  & _administrations._ \\ \hline \hline \multicolumn{2}{l}{   [noitemsep=0pt] \(\cdot\) I’m very frustrated.} & ✓ \\  & _J think this is overdue, and I think other states should jump on board._ & X \\  & _German salf of the mountain._ | m Very frustrated. I take an outh of_ \\  & _office, as does serve your choice before office in this country._ \\ \hline \multicolumn{2}{l}{   [noitemsep=0pt] \(\cdot\) In very frustrated.} & _Colorado's attorney general, John Sudues, a Republican, said in a statement that the challenge from Nebraska_ \\  & _and Oklahoma was wikout meet. Like many elected officials in Colorado, Ms. Sulbers had opposed Amendment off, which legalized_ \\  & _marjuana. But on Thursday, he said we will vigorously defend_ \\  & _against the lowest attempting to undo it._ \\ \hline \hline \multicolumn{2}{l}{   [noitemsep=0pt] \(\cdot\) A leader only starts a fight he’s prepared to finish.} & X \\  & _But any dramatic robrecen can walk into a bar and start at fight._ & X \\  & _A leader only starts a fight he’s prepared to finish. The field of_ \\  & _confirmed and potential GOP breakfast candidates include more than a dozen people._ \\ \hline \multicolumn{2}{l}{   [noitemsep=0pt] \(\cdot\) A leader only starts a fight he’s prepared to finish. And if someone_ \\  & _can capture how the blue-collar, working-class Republic, the conservatives, many of them even union members, as well as event-geldles, there’s a real pathway to the nomination. Metabolize, an orained Southern Baptist minister is a celebrated figure among evangelcol Christians. He was the longest-serving Arkansas governor, from 1998 to 2007._ \\ \hline \end{tabular}
\end{table}
Table 7: Examples of zero-shot DETAILED prompt performance of GPT-4 on the INF/OTH bias task when using our context-augmented prompting techniques. In the table, italic portions of the model input denote context obtained using the corresponding technique.

Figure 3: Comparison of INF/LEX results on hundred randomly selected examples of each using GPT-4 + CTX and BERT +CTX



served in concise words or phrases Fan et al. (2019); Maab et al. (2023a). Similarly, according to Chen et al. (2020), distinguishing informational and lexical bias is more difficult compared to detecting any type of bias. Secondly, pre-trained BERT models Maab et al. (2023a), benefit from fine-tuning on tasks with ample training data, i.e., when combined with additional augmented context using backtranslation (BT), the detection of both informational and lexical bias becomes more apparent and fluent. Consequently, this leads to superior performance compared to language models (LLMs) that in this work are utilized without any training or fine-tuning.

### Concise vs. Detailed Prompt

Finally, we also compare the performance of zero-shot Concise versus zero-shot Detailed prompting. Although Detailed prompting enhances overall performance, OPT demonstrates superior performance when Concise prompting is employed. We examined bias samples to showcase how OPT benefits from context information with a Concise prompt. As found by Zhang et al. (2022), OPT models such as OPT-175B struggle with declarative instructions or straightforward interrogatives. Our Detailed prompt aligns with this observation when compared with the Concise prompt, which affords more flexibility. Similarly, in line with our findings, OPT benefits from the addition of context information. See Table 10 in Appendix for more details.

To explore OPT, we conducted a comparison between Zero-shot-Concise and Zero-shot-Detailed prompts specifically for the task of INF/OTH i.e., detecting informational bias only. This analysis also highlights the distinctions in contextual information over regular non-context samples. In Table 10, the examples selected include text with and without contextual information, where Incorrect predictions are attributed to the absence of a sufficient amount of context for the target sentence. We observe that OPT performs well when context makes sense, particularly with Concise prompts. For instance, in Table 10, the third biased sentence sample "He also praised the campaign that Mr. Sanders ran." is integrated with a context involving Obama endorsing Clinton and discussing the candidacy. The inclusion of this context fails to provide an explanation for the target 'Bernie Sanders,' leading to an incorrect model prediction for both Concise and Detailed prompts, contrary to the reasonable context in the second example, "He called it a very good night, and said Holder's visit had let people know their voices had been heard," where the context also describes Holder's visit as receptive to finding solutions and offering assistance to the target 'Eric Holder'. Thus the model makes the correct prediction even with only the Concise prompt. However, OPT capability is limited in detecting irregular context when only a Concise prompt is provided. This analysis of samples reveals that certain contexts do not contribute significantly to correct predictions. The primary trend suggests that the zero-shot approach with ctx tends to be more effective.

## 6 Conclusion

This paper aims at advancing our understanding of prompt-based identification of media bias by LLMs, a crucial and challenging task to the research community, news media, and social media companies. We establish a framework for examining prompt-based models in a zero-shot and few-shot configuration to classify sentences that manifest bias. Our approach demonstrates the utility of using context-augmented informational and lexical bias from prior work, an area that we think has received inadequate attention. We provide insights into the dynamics between LLMs, tasks, and prompts, discerning their individual capacities in the detection of bias. Notably, we find considerable performance improvements when adding context-augmented information to both small and large-parameter LLM models, indicating that in-context learning can approach the performance levels of traditional models in classifying media bias. While our study demonstrates that LLMs are applicable to the area of misinformation detection, they do not always represent a huge improvement. Formulating prompts that strike a balance between the simple and the (over)-elaborate remains a challenge, as is particularly evident in GPT-Neo and OPT.

We hope our work inspires further research in bias detection as a means to gain insight into enhancing LLMs. Future work includes exploring additional aspects of bias, e.g., bias related to culture, race, or age, as well as exploring the understanding and limitations of LLMs. We also think other similar bias-detection tasks could be grouped together in our prompting setting, following Lee et al. (2021).

 



## Limitations

The topic of bias detection using zero and few-shot prompts is relatively young and not addressed properly. Therefore, the amount of existing probable directions to research is immense. As there is a scarcity of bias representations and annotated media coverage in other languages, our work is exclusively founded on bias representations of English news articles, and BASIL stands alone as the sole annotated dataset containing informational bias. Also, the state-of-the-art models used for comparison are not competitors to the models used in this paper. Hence, another direction of future work is to implement similar kind of models for understanding bias and discovering knowledge limits of large language models.

## Ethical Considerations

Political entities are subjects of continuous debate in news media, and it is crucial to necessitate the development of fast and reliable methods to disseminate accurate information. This becomes especially challenging because conventional bias detection models encounter rigorous training. To address this, we must adopt a more flexible and tolerant approach, especially when utilizing large language models. However, it is essential to strike a balance between bias detection, model's adaptability, and the preservation of diverse viewpoints.

Finally, we think that one major potential risk of our work is that given the simplicity and effectiveness of our prompt-based techniques, companies may choose to deployed in online products and services that are available to the masses. However, since in this paper we have not tested nor analyzed the predictions from a qualitative point of view, it is impossible for us to foresee potential impacts that this may have in the future at this point. For example, it is plausible that the predictions of our models may contain other kinds of biases, which may be amplified when deployed at a massive scale. We urge for extreme caution in utilizing our techniques in production-level environments, and call researchers and practitioners to help by conducting studies to further understand the nature and potential repercussions of using the outputs of our models in broader contexts.

## Acknowledgements

The authors wish to express gratitude to the funding organization as this work has been supported by the Mohammed bin Salman Center for Future Science and Technology for Saudi-Japan Vision 2030 at The University of Tokyo (MbSC2030).

## References

* M. Pervez Akhter, Z. Jiangbin, I. Raza Naqvi, M. Abdelmajeed, A. Mehmoood, and M. T. Sadiq (2020)Document-level text classification using single-layer multisize filters convolutional neural network. IEEE Access8, pp. 42698-42707. Cited by: SS1.
* C. Anil, Y. Wu, A. Andreassen, A. Lewkowycz, V. Misra, V. Ramasesh, A. Slone, G. Gur-Ari, E. Dyer, and B. Neyshabur (2022)Exploring length generalization in large language models. Advances in Neural Information Processing Systems35, pp. 38546-38556. External Links: ISSN 0037-69X5, Document, Link Cited by: SS1.
* R. Baby, G. Karadzhov, A. Saleh, J. Glass, and P. Nakov (2019)Multi-task ordinal regression for jointly predicting the trustworthiness and the leading political ideology of news media. arXiv preprint arXiv:1904.00542. Cited by: SS1.
* J. Baumgartner, S. Zannettou, B. Keegan, M. Squire, and J. Blackburn (2020)The pushshift reddit dataset. In Proceedings of the international AAAI conference on web and social media, Vol. 14, pp. 830-839. Cited by: SS1.
* I. Beltagy, A. Cohan, R. Logan IV, S. Min, and S. Singh (2022)Zero-and few-shot nlp with pretrained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: Tutorial Abstracts, H. Wallerstein, W. Durrell, K. Heissel, and F. Schiele (Eds.), pp. 32-37. Cited by: SS1.
* S. Black, L. Gao, P. Wang, C. Leahy, and Stella Biderman (2021)GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow. Technical report If you use this software, please cite it using these metadata. Cited by: SS1.
* T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. (2020)Language models are few-shot learners. Advances in neural information processing systems33, pp. 1877-1901. Cited by: SS1.
* K. Chang, V. Prabhakaran, and V. Ordonez (2019)Bias and fairness in natural language processing. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP): Tutorial Abstracts, Hong Kong, China, pp. 1877-1889. Cited by: SS1.
* W. Chen, K. Al-Khatib, B. Stein, and H. Wachsmuth (2020)Detecting media bias in

news articles using gaussian bias distributions. _arXiv preprint arXiv:2010.10649_.
* Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. _arXiv preprint arXiv:2210.11416_.
* Cohan et al. (2019) Arman Cohan, Iz Beltagy, Daniel King, Bhavana Dalvi, and Daniel S Weld. 2019. Pretrained language models for sequential sentence classification. _arXiv preprint arXiv:1909.04054_.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Entman (2007) Robert M Entman. 2007. Framing bias: Media in the distribution of power. _Journal of communication_, 57(1):163-173.
* Face (2021) Hugging Face. 2021. The ai community building the future. _URL: https://huggingface. co_.
* Fan et al. (2019) Lisa Fan, Marshall White, Eva Sharma, Ruisi Su, Prafulla Kumar Choubey, Ruihong Huang, and Lu Wang. 2019. In plain sight: Media bias through the lens of factual reporting. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 6343-6349, Hong Kong, China. Association for Computational Linguistics.
* Fletcher and Park (2017) Richard Fletcher and Sora Park. 2017. The impact of trust in the news media on online news consumption and participation. _Digital journalism_, 5(10):1281-1299.
* Gao et al. (2020a) Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020a. The pile: An 800gb dataset of diverse text for language modeling. _arXiv preprint arXiv:2101.00027_.
* Gao et al. (2020b) Tianyu Gao, Adam Fisch, and Danqi Chen. 2020b. Making pre-trained language models better few-shot learners. _arXiv preprint arXiv:2012.15723_.
* Gentzkow and Shapiro (2010) Matthew Gentzkow and Jesse M Shapiro. 2010. What drives media slant? evidence from us daily newspapers. _Econometrica_, 78(1):35-71.
* Gentzkow et al. (2015) Matthew Gentzkow, Jesse M Shapiro, and Daniel F Stone. 2015. Media bias in the marketplace: Theory. In _Handbook of media economics_, volume 1, pages 623-645. Elsevier.
* Greene and Resnik (2009) Stephan Greene and Philip Resnik. 2009. More than words: Syntactic packaging and implicit sentiment. In _Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics_, pages 503-511, Boulder, Colorado. Association for Computational Linguistics.
* Guo and Zhu (2022a) Shijia Guo and Kenny Q Zhu. 2022a. Modeling multi-level context for informational bias detection by contrastive learning and sentential graph network. _arXiv preprint arXiv:2201.10376_.
* Guo and Zhu (2022b) Shijia Guo and Kenny Q. Zhu. 2022b. Modeling multilevel context for informational bias detection by contrastive learning and sentential graph network.
* Hamborg et al. (2019) Felix Hamborg, Karsten Donnay, and Bela Gipp. 2019. Automated identification of media bias in news articles: an interdisciplinary literature review. _International Journal on Digital Libraries_, 20(4):391-415.
* Hao et al. (2022) Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. 2022. Language models are general-purpose interfaces. _arXiv preprint arXiv:2206.06336_.
* Holtzman et al. (2023) Ari Holtzman, Peter West, and Luke Zettlemoyer. 2023. Generative models as a complex systems science: How can we make sense of large language model behavior? _arXiv preprint arXiv:2308.00189_.
* Hube and Fetahu (2019) Christoph Hube and Besnik Fetahu. 2019. Neural based statement classification for biased language. In _Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining_. ACM.
* Iyyer et al. (2014) Mohit Iyyer, Peter Enns, Jordan Boyd-Graber, and Philip Resnik. 2014. Political ideology detection using recursive neural networks. In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1113-1122, Baltimore, Maryland. Association for Computational Linguistics.
* Kiros et al. (2015) Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. _Advances in neural information processing systems_, 28.
* Kojima et al. (2022) Takeshi Kojima, Shixiang Shane Gu, Michel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. _Advances in neural information processing systems_, 35:22199-22213.
* Lee et al. (2021) Nayeon Lee, Belinda Z. Li, Sinong Wang, Pascale Fung, Hao Ma, Wen-tau Yih, and Madian Khabsa. 2021. On unifying misinformation detection. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5479-5485, Online. Association for Computational Linguistics.
* Lei et al. (2022) Yuanyuan Lei, Ruihong Huang, Lu Wang, and Nick Beauchamp. 2022. Sentence-level media bias analysis informed by discourse structures. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 10040-10050.
* Lin et al. (2006) Wei-Hao Lin, Theresa Wilson, Janyce Wiebe, and Alexander G Hauptmann. 2006. Which side are you on? identifying perspectives at the document and sentence levels. In _Proceedings of the Tenth Conference on Computational Natural Language Learning (CoNLL-X)_, pages 109-116.
* Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes 

good incontext examples for gpt-3. In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_, pages 100-114.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Maab et al. (2023a) Iffat Maab, Edison Marrese-Taylor, and Yutaka Matsuo. 2023a. An effective approach for informational and lexical bias detection. In _Proceedings of the Sixth Fact Extraction and VERification Workshop (FEVER)_, pages 66-77.
* Maab et al. (2023b) Iffat Maab, Edison Marrese-Taylor, and Yutaka Matsuo. 2023b. Target-aware contextual political bias detection in news. In _Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics_, pages 782-792, Bali, Indonesia.
* Mehta et al. (2022) Nikhil Mehta, Maria Leonor Pacheco, and Dan Goldwasser. 2022. Tackling fake news detection by continually improving social context representations using graph neural networks. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1363-1380.
* Mikolov et al. (2013) Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. _Advances in neural information processing systems_, 26.
* Min et al. (2021) Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2021. Metaicl: Learning to learn in context. _arXiv preprint arXiv:2110.15943_.
* Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. _Advances in Neural Information Processing Systems_, 35:27730-27744.
* Pan et al. (2018) Jeff Z Pan, Siyana Pavlova, Chenxi Li, Ningxi Li, Yangmei Li, and Jinshuo Liu. 2018. Content based fake news detection using knowledge graphs. In _The Semantic Web-ISWC 2018: 17th International Semantic Web Conference, Monterey, CA, USA, October 8-12, 2018, Proceedings, Part I 17_, pages 669-683. Springer.
* Perez-Rosas et al. (2017) Veronica Perez-Rosas, Bennett Kleinberg, Alexandra Lefevre, and Rada Mihalcea. 2017. Automatic detection of fake news. _arXiv preprint arXiv:1708.07104_.
* Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
* Prat and Stromberg (2013) Andrea Prat and David Stromberg. 2013. The political economy of mass media. _Advances in economics and econometrics_, 2:135.
* Radford and Mikolov (2018) Alec Radford and Tomas Mikolov. 2018. Improving language understanding by generative pretraining. Technical report, OpenAI.
* Radford et al. (2019) Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. _OpenAI Blog_, 1(8).
* Rae et al. (2021) Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Milican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. _arXiv preprint arXiv:2112.11446_.
* Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. _arXiv e-prints_.
* Rashkin et al. (2017) Hannah Rashkin, Eunsol Choi, Jin Yea Jang, Svitlana Volkova, and Yejin Choi. 2017. Truth of varying shades: Analyzing language in fake news and political factchecking. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2931-2937, Copenhagen, Denmark. Association for Computational Linguistics.
* Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutiawika, Zaid Ayafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. Multitask prompted training enables zero-shot task generalization. _arXiv preprint arXiv:2110.08207_.
* Schick and Schutze (2020) Timo Schick and Hinrich Schutze. 2020. Exploiting cloze questions for few shot text classification and natural language inference. _arXiv preprint arXiv:2001.07676_.
* Shapiro (2016) Jesse M Shapiro. 2016. Special interests and the media: Theory and an application to climate change. _Journal of public economics_, 144:91-108.
* Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. _arXiv preprint arXiv:2010.15980_.
* Tay et al. (2022) Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaxiiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. Unifying language learning paradigms. _arXiv preprint arXiv:2205.05131_.
* Thoppilan et al. (2022)Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. 2022. Lamda: Language models for dialog applications. _arXiv preprint arXiv:2201.08239_ 

Petter Tornberg. 2023. Chatgpt-4 outperforms experts and crowd workers in annotating political twitter messages with zero-shot learning. _arXiv preprint arXiv:2304.06588_.
* van den Berg and Markert (2020)Esther van den Berg and Katja Markert. 2020. Context in informational bias detection. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 6315-6326, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Anjana Arunkumar, Arijun Ashok, Artul Selvan Dhanasekaran, Atharva Naik, David Stap, et al. 2022. Supernaturalinstructions: Generalization via declarative instructions on 1600+ nlp tasks. _arXiv preprint arXiv:2204.07705_.
* Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. _arXiv preprint arXiv:2109.01652_.
* Yang et al. (2022) Xiaohan Yang, Eduardo Peynetti, Vasco Meerman, and Chris Tanner. 2022. What gpt knows about who is who. _arXiv preprint arXiv:2205.07407_.
* Zhang et al. (2022) Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. _arXiv preprint arXiv:2205.01068_.

## Appendix A Parsing the output of the LLMs

Table 8 shows two examples of LLM answers generated by OPT and GPT-Neo that require additional processing to map onto expected answers ("answer cleansing"). Table 9 provides additional details on the algorithm we apply.

## Appendix B Robustness study

Table 11 provides a detailed ablation analysis of the contribution of the different contextualization strategies to various base LLM models on the INF/LEX bias classification task. We implemented the BERT models of [1] using HuggingFace (Face, 2021), utilizing BERT-base [Devlin et al., 2018], with a batch size of 32, a learning rate of \(5\times 10^{-5}\), and 15 epochs. Finally, Table 12 illustrates these findings with concrete examples



* [15] M. C. Bans, A. K. S. Gatton, and R. J. E. Hinton, "A review of machine learning algorithms for machine learning," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 209-218, 2017
* [16] D. P. Kingma and Ba, "Adam: a method for stochastic optimization," in _Advances in neural information processing systems_, vol. 34, no. 3, p. 133-146, 2014
* [17] T. L. Chuang, "Learning to improve the dimensionality of deep networks," _arXiv preprint arXiv:1406.03015_, 2014.
* [18] H. Li, Y. Zhang, Z. Liu, and X. Sun, "Deep residual learning with convolutional neural network," in _NIPS conference on computer vision_, pp. 1-10, 2015.
* [19] W. Li, Y. Zhang, and X. Sun, "Deep residual learning with convolutional neural networks," _CoRR_, abs/1412.1080, 2014.
* [20] W. Li, Y. Zhang, and X. Sun, "Deep residual learning with convolutional neural networks," _IEEE Transactions on Neural Networks_, vol. 10, no. 1, pp. 1-10, 2015.
* [21] W. Li, Y. Zhang, and X. Sun, "Deep residual learning with convolutional neural networks," _Neural Information Processing Systems_, vol. 10, no. 1, pp. 1-1, 2015.
* [22] W. Li, Y. Zhang, and X. Sun, "Deep residual learning with convolutional neural networks," _Journal of Machine Learning Research_, vol. 10, no. 1, pp. 1-1, 2015.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{l l} \hline \hline
**INF/OTH** & **Model Input and Prompt Answer** \\ \hline Model Input (No context) & The NRA said it needs time to digest the findings, but it commended Hutchinson and his team for contributions that will go a long way to making America’s schools safer. _The report marks the culmination of a controversial process that began in December, when NRA CEO Wayne LaPierre held a press conference the week after the shooting in Newtown, Conn. The 225-page report also called for states to require schools to conduct a safety assessment and for a pilot program to be created assessing mental health as a pre-indicator for violence. Mark Mattioli, the father of one Newton shooting victim, also spoke after Hutchison's announcement to appland the review effort. The review comes as Congress and a number of state legislatures pursue gun control legislation. This is a recommendation for solutions, real solutions that will make our kids safer. That’s what we need. At least one state, Indiana, is considering the idea of armed officers at schools. \\ \cline{3-3} Concise Detailed & ‘bias’ \textbackslash{} revert \\  & ‘bias’ (a). ✓ It does not address the root causes of violence, nor does it suggest the specific actions needed to combat it; rather, it identifies a number of potential solutions. It does so with an airy disregard for individual rights and without a clear understanding of how many of the proposals would affect gun ownership and Second Amendment rights in practice.” \\ \hline \hline Model Input (No context) & He called it a very good night, and said Holder’s visit had let people know their voices had been heard. \\ \cline{3-3} Concise & ’No bias’. X He was the first to admit that the visit to the local community had not met with universal approval, but he said the fact it had brought people together was “extraordinary”. ” No bias’. X By the next day, the story had fallen entirely silent. Classification: ‘Bias’ Text: One of the first things Holder said Sunday, he added, was ’let me say we need a justice system that really represents the African-American community. A bias detector would give an output of true (this is neither bias nor no bias) if the text is either. A bias detector would have a false positive (false positive) if it’s output is false; \\ \cline{3-3} Model Input + ctx & _Johnson said there were six arrests Wedensday, compared to 47 the previous night._ He called it a very good night, and said Holder’s visit had let people know their voices had been heard. _In nearby Clayton, a grand jury began hearing evidence to determine whether Wilson should be charged in Brown’s death. We didn know it was Eric Holder [coming]. Rev. John Paul Hopping of Our Lady of Guadalupe, told HuffPost. They were really nice and they were really listening. They are looking for solutions and they want to help._ \\ \cline{3-3} Concise & ‘bias’. ✓ According to the article, the riot that ensued erupted over a rumor that a protester had been shot, CNN reported. The city Cleveland issued a curfew at 9 p.m., and police arrested a 30-year-old man on curfew violation. \\ Detailed & ’No bias’. X Obama will have to wait until the summer before trying to restart his push for immigration reform. In remarks at the National Press Club, Obama said its an issue that will have to be addressed legislatively. \\ \hline \hline Model Input (No context) & He also praised the campaign that Mr. Sanders ran. \\ Concise & How X he got so many votes remains a mystery. However, there is nothing unusual about the campaign. \\ Detailed & ‘No bias’. X My friend and colleague, the Vice President of the United States, said one of the reasons he thought Mr. Trump would be so easy to beat is that voters would have no interest in voting for an authoritarian. \\ \cline{3-3} Model Input +ctx & _Shortly after their meeting, Mr. Obama endorsed Mrs. Clinton in a video._ He also praised the campaign that Mr. Sanders ran. _The visit came a day after the senator huddled with his team at his headquarters in Vermont to discuss the fate of his candidacy._ \\ Concise & ‘No bias’ X \\ Detailed & ‘No bias’. X Shortly after their meeting, Mr. Obama endorsed Mrs. Clinton in a video. He also praised the campaign that Mr. Sanders ran. \\ \hline \hline \end{tabular}
\end{table}
Table 10: Contrasting contextual information vs. regular non-context samples: A comparison of zero-shot-Concise and zero-shot-Detailed with OPT, illustrated with few sample examples.





\begin{table}
\begin{tabular}{l c c c c c c} \hline \hline  & **OPT** & **FLAN-UL2** & **GPT-3.5-turbbo** & **GPT-4** & **BERT + BT** & **BERT (*)** & **BERT (**)** \\ \hline Regular (No Context) & 32.8 & 37.9 & 45.8 & 48.7 & **51.4** & - & - \\ + BANC & 33.5 & 41.0 & 52.5 & 54.6 & - & 58.6 & **62.3** \\ + ABTA + EBTA & 32.9 & 43.1 & 51.7 & 55.1 & - & 63.3 & **65.5** \\ + ctx & 35.8 & 50.5 & 56.8 & 59.5 & - & 67.6 & **71.3** \\ + k-shots & 33.4 & 46.1 & 52.9 & **53.7** & - & - & - \\ + ctx + K-shots & 38.6 & 58.2 & 62.4 & **65.5** & - & - & - \\ \hline \hline \end{tabular}
\end{table}
Table 11: Robustness study of zero-shot DETAILED prompts using Regular (No Context), ctx (combined context of BANC, ABTA, and EBTA [1]), k-shots (no context few-shots), andctx + k-shots (with context few-shots) using various LLM models and reimplementation of BERT in using BERT + BT from Maab et al. (2023a), BERT (*) = BERT + BANC + ABTA + EBTA Maab et al. (2023b), and BERT (**) = BERT + BANC + ABTA + EBTA + BT Maab et al. (2023c). We randomly picked up 100 samples from informational bias and 100 samples from lexical bias from BASIL without context named as Regular (No Context). Using the target-aware techniques, BANC, ABTA, and EBTA from [1], we contextualized the picked samples termed ctx. For Regular (No Context), BANC, ABTA+ EBTA, and ctx, LLM models, such as OPT, FLAN-UL2, GPT-3.5-turbo, and GPT-4, use DETAILED prompt. In case of fine-tuned BERT models, no context-augmented data is provided during testing, however out of the remaining picked-up samples the relevant context-augmentation technique is applied for training data with non-overlapping samples. Please note that BERT+BT and BERT(**) model requires additional data using BT over context-augmented data for training [1, 1], where BT stands for backtranslation.

\begin{table}
\begin{tabular}{p{113.8pt} p{113.8pt} p{113.8pt} p{113.8pt}} \hline \hline INF/LEX & **Model Input** & **GPT-4** & **BERT (***)** \\ \hline \multirow{8}{*}{**Benchmark**} & His advisers have complained that Democrats are slowing the process, and resisting the kind of swift confirmation the Senate gave many of Obama’s nominees in 2009. & ✓ & ✓ \\ \cline{2-4}  & AARP’s new and welcome position is a positive step towards the type of reforms I’ve championed, and I look forward to working with the organization to shape the changes in a way that makes the least detrimental impact to present and future retirees, she said in a statement. & X & ✓ \\ \cline{2-4}  & At one point he suggested that those who opposed his position were heartless. & ✓ & ✗ \\ \cline{2-4}  & Cordova is just one of 198 people that Weiner follows on Twitter, though he has nearly & ✗ & ✗ \\ \cline{2-4}  & 50,000 followers. & & & \\ \hline \hline \multicolumn{4}{p{113.8pt}}{On Saturday, the mayor of Hoboken, Dawn Zimmer, said Mr. Christie’s lieutenant governor and another senior administration official had **threatened** in May to withhold federal recovery aid for Hurricane Sandy unless she supported a development favored by the Christie administration.} & X & ✓ \\ \cline{2-4}  & Gay and civil rights groups **raised** the ruling. & X & ✓ \\ \cline{2-4}  & WASHINGTON – The U.S. Attorney’s office in New Jersey has subpoenaed documents from the reelection campaign of New Jersey Gov. Chris Chris Christie (R) and the New Jersey Republican State Committee, as part of its investigation into the ”Bridgeate” **scandal.** & X & ✓ \\ \cline{2-4}  & The Christie administration closed down two of the three George Washington Bridge access lanes in Fort Lee, N.J., in September, in what appeared to be a **political retribution scheme** aimed at the borough’s Democratic mayor. & X & X \\ \hline \hline \end{tabular}
\end{table}
Table 12: Classification results produced by GPT-4 + ctx and BERT(*) + ctx using Zero-shot-DETAILED prompt. The lexical bias spans are highlighted in bold.