Measuring Political Bias in Large Language Models: What Is Said and How It Is Said (2024)

Abstract: We propose to measure political bias in LLMs by analyzing both the content and style of their generated content regarding political issues. Existing benchmarks and measures focus on gender and racial biases. However, political bias exists in LLMs and can lead to polarization and other harms in downstream applications. In order to provide transparency to users, we advocate that there should be fine-grained and explainable measures of political biases generated by LLMs. Our proposed measure looks at different political issues such as reproductive rights and climate change, at both the content (the substance of the generation) and the style (the lexical polarity) of such bias. We measured the political bias in eleven opensourced LLMs and showed that our proposed framework is easily scalable to other topics and is explainable.

heading: 1 Introduction
text: As the pervasiveness of AI in human daily life escalates, extensive research has illuminated its limitations and potential harms such as gender and racial biases and hallucinations (Weidinger et al., 2021; Blodgett et al., 2020; Ji et al., 2023; Sheng et al., 2021; Solaiman and Dennison, 2021; Ganguli et al., 2022; Kumar et al., 2023). Among these, political bias in AI, a notably crucial yet underexplored facet, poses significant risks by potentially distorting public discourse and exacerbating societal polarization (Garrett, 2009; Stroud, 2010; DellaVigna and Kaplan, 2007).
Existing scholarly efforts have explored political bias ingrained in LMs, mainly focused on stance at the political-orientation level (i.e., left or right/liberal or conservative) (Liu et al., 2021; Feng et al., 2023; Rozado, 2023; Durmus et al., 2023). The political orientation tests-based methodology (e.g., The Political Compass test1) is often
1https://www.politicalcompass.org/test
employed, yet it may be inadequate to fully capture the complex dynamics of bias within LLMgenerated content (Rozado, 2023). Furthermore, this approach might not provide the detailed insights necessary to understand the subtleties of political biases in LLM-generated content.
This study introduces an interpretable and granular framework for measuring political bias in LLM-generated content, going beyond traditional political-orientation level analyses. Political bias, characterized by a prejudiced perspective towards political subjects, mandates a nuanced evaluation of the models’ positions on diverse political issues. This bias predominantly manifests through framing, which entails the deliberate selection and emphasis of specific informational elements, both in content and style, to shape perceptions (Entman, 1993; Chong and Druckman, 2007; Scheufele, 1999; Giles and Shaw, 2009; Saez-Trumper et al., 2013). Given the multifaceted nature of political bias, our framework employs a two-tiered approach for its assessment, encompassing both topicspecific stance and framing.
To assess the models’ stance on distinct political topics, we implement a method of extreme anchor comparison, quantifying the similarity be-
11142
tween model outputs and two opposed stances – advocacy and opposition – across various political subjects. Subsequently, to dissect the political bias of LLMs more thoroughly, we examine framing by decomposing both the content and style. This involves a detailed content analysis leveraging Boydstun’s frame dimensions and entity-based frames, coupled with an evaluation of stylistic bias, including the examination of media bias in writing styles and the presence of non-neutral sentiment towards salient entities of topic. Collectively, our framework not only discerns topic-specific stances but also explores the intricate dynamics of the content (“what” is said) and style (“how” it is said) concerning contentious topics. The ultimate aim is to provide a measurement of political biases inherent in various LLMs, thereby paving the way for the development of strategies to diminish these biases and enhance the reliability and equity of LLM applications.
Drawing upon the empirical evidence and analytical insights derived from our frameworks, this study elucidates a set of findings that furnish a guide for subsequent research endeavors within the community. The key discoveries include: (1) LLMs show different political views depending on the topic, such as being more liberal on reproductive rights and more conservative on immigration; (2) Even when LLMs agree on a topic, they focus on different details and present information differently; (3) LLMs often discuss topics related to the US; (4) topic-level analysis aligns with previ-
ous finding that LLMs usually lean towards liberal ideas; (5) Larger models aren’t necessarily more neutral in their political views; (6) Models from the same family can have different political biases; (7) the impact of multilingual capabilities (e.g., Yichat, Jais-chat) on the thematic focus of content, diverging from models primarily trained in English. By facilitating both model-specific and comparative analyses, our framework seeks to advance the development of AI systems that are safer and more aligned with ethical standards. We release the codebase to promote research in this area. 2

heading: 2 The Framework
text: We present an evaluation framework that provides fine-grained, topic-specific analysis for political bias in LLMs rather than a generalized ideological examination. Our approach recognizes the complexity of bias, where, an LLM might exhibit liberal stances on certain topics (e.g., reproductive rights) while leaning conservative on others (e.g., death penalty), reflecting a sophisticated and variable political landscape. Moreover, since we evaluate models’ actual generation, we can also understand how biases are conveyed to users in practical applications.
Specifically, we begin by preparing a set of natural language instructions X = {X1, X2, ..., Xt} covering a total of t sensitive political topics. For the LLM fLLM that we want to evaluate, we obtain
2https://github.com/HLTCHKUST/ llm-political-bias.git
reference extreme anchor distributions P (Ypro), P (Yopp) for stance analysis (e.g., proponents and opponents ). For instance, on the left-top corner about reproductive rights, the model generation distribution (grey color) overlaps with the proponent distribution, which refers to the model showing an advocacy stance on reproductive rights.
the distribution of model generated response by fLLM(Xi) → P (Ŷi) for each topic. To measure the political bias inherent in P (Ŷi), we introduce a two-tiered framework, focusing first on the political stance (Section 2.1) analysis, and a more detailed examination of framing bias (Section 2.2) analysis. Hereafter, for simplicity, we will omit the topic index i.

heading: 2.1 Political Stance Analysis
text: Stance refers to a position or perspective that an individual, group, or institution takes on a specific issue, policy, or topic. For instance, we investigate if the model has a position of “pro-choice” or “prolife” about the topic of reproductive rights. This analysis can reveal “What stance does the model have about the specific topics”.
Certain political stance can be represented by a vector s⃗ in a vector space S obtained via a “stance extractor” (P (Ŷ )) → s⃗. Different orientations of s⃗ represent different stances taken, while the norm ∥s⃗∥ represents the degree of leaning to a particular stance. An optimal, non-biased model would ideally have a very small norm, i.e. ∥s⃗∥ < ϵs.
Unfortunately, there are currently no existing off-the-shelf stance extractors that allow us to directly obtain the s⃗ vector. Instead, we tackle this problem in a surrogate manner. As shown in the top row of Figure 2, we assume the stance extractor preserves relative distance relationships, and hypothesize there exists of a pair of reference anchor distributions P (Ypro), P (Yopp), which respec-
tively elicit a proponent stance s⃗pro and an opponent stance s⃗opp in the stance space. These reference vectors satisfy the conditions s⃗pro = −s⃗opp and |s⃗pro| = |s⃗opp| = 1. We then proceed to analyze the overlap between the model generation distribution P (Ŷ ) and these anchor distributions using a similarity metric d. The degree of imbalance between d{pro/opp} = d(P (Ŷ )||P (Y{pro/opp})) serves as a representation of the model’s stance. An optimal, non-biased model would ideally exhibit minimal imbalance, i.e., ||dpro − dopp|| < ϵz .
By comparing the imbalance between dpro and dopp, we can determine which stance the fLLM has adopted and the extent to which it leans in that direction.

heading: 2.2 Framing Bias Analysis
text: Framing refers to “selecting some aspects of a perceived reality and make them more salient in a communicating text” Entman (1993), which comprises content bias and style bias. This section will focus on investigating “What is said” (content) and “How it is said” (style) to reveal the detailed mechanism underlying the formation of framing bias.
As shown in the bottom row of Figure 2, we assume that the model response generation process of fLLM consists of two steps: firstly, g generates a latent variable z from the input instruction X , then h translates Z into natural language form, i.e., fLLM : X
g−→ Z h−→ P (Ŷ ). The latent variable z encapsulates all the information about the model’s response, which can be further decomposed into
two parts: the content variable C and the style variable S, i.e., Z = [C, S]. These respectively contain information about frame selection (“what to say”) and lexical presentation (“how to say”), which are then translated by hC and hS into P (Ŷ ).
We are interested in the content and style variables (C, S). Unfortunately, we cannot directly access the function g since the two-step generation process is explicitly done by fLLM. However, obtaining C and S from the reverse direction is relatively more feasible. This involves employing a content extractor h−1C and a style extractor h −1 S , which are respectively the inverse functions of hC and hS . Employing these content and style extractors allows us to disentangle the components of the model’s response generation process, providing valuable insights into how framing bias is encoded through g.
After obtaining the content and style variables, another challenge arises: how to analyze their biases. Designing a reliable yet scalable approach to derive optimal, comprehensive C and non-biased S as references (i.e., “what should be said” and “how it should be said”) is exceedingly difficult. Therefore, in our framework, rather than measuring the deviation of C and S compared to some golden references, we compare different C and S across a diverse range of models.
This granular methodology for framing bias analysis facilitates a deep dive into the LLM-generated content, enabling a thorough exploration of both the substantive and stylistic elements of political bias.

heading: 3 Framework Implementation
text: In this section, we describe how we implement the evaluation framework proposed in Section 2.

heading: 3.1 Political Stance Analysis
text: Political Topics We selected 14 politically divisive topics in which polarized views are prevalent. The list of topics is comprised of reviewing existing studies about political bias (Card et al., 2015; Hamborg, 2020; Liu et al., 2022) as well as referring the research centers (e.g., Pew Research Center3), or media bias-related websites (e.g., Allsides.com 4). Our framework is not confined to the listed topics but is scalable. The topics consist
3https://www.pewresearch.org/topics/ 4https://www.allsides.com/
topics-issues
of 10 political topics (Reproductive Rights, Immigration, Gun Control, Same Sex Marriage, Death Penalty, Climate Change, Drug Price Regularization, Public Education, Healthcare Reform, Social Media Regulation) and four political events (Black Lives Matter, Hong Kong Protest, Liancourt Rocks dispute, Russia Ukraine war).
Task Instruction To focus the generation scope to the most suitable for political bias evaluation, we probe models by conducting news headline generation about political topics. This choice of scope is grounded in the insight that news article titles, as demonstrated by (Lee et al., 2022), serve as effective indicators of framing bias. Given that headlines encapsulate the essence of articles and set the tone for the subsequent content, we can analyze the setup of where political biases are particularly pronounced.5 The prompt template we used is as follows: Write 10 news headlines about the topic of “{topic}". Separate each with a tag ‘Title:’..
Reference Anchor Generation We obtain a pair of reference anchor distributions P (Ypro), P (Yopp) by prompting each LLM under evaluation to generate responses that reflect opposed stances (e.g., “Pro same-sex marriage” versus an “Anti same-sex marriage.”). We keep the prompt to be as same as task instruction but with adding the specific stance tags. For instance, the prompt for obtaining P (Ypro) for the topic of “Same sex marriage” is as follows:
“Write 10 Pro same-sex marriage news headlines about the topic of Same Sex Marriage. Separate each with a tag ‘Pro same-sex marriage Title:”’ . Table 1 illustrate the examples of anchor generation.
Distance Function We implement distance measurement d{pro/opp} = d(P (Ŷ )||P (Y{pro/opp})) using sentence embedding from SentenceBERT
5This can be generalized into other prompt templates. For instance, article generation or opinion piece generation.
(SBERT) (Reimers and Gurevych, 2019). 6 First, sim(·, ·) denotes the cosine similarity between two sentence embeddings and measures the semantic similarity between them. Then we sample n model generations {ŷ1, ŷ2, ..., ŷn} from the distribution of fLLM generation P (Ŷ ). For each sample, we retrieve the nearest neighbor from the reference distribution, for example, yNN−kpro denotes the nearest neighbor of ŷk in the proponent distribution P (Ypro). Then the distance function d can be implemented as:
d{pro/opp} = − 1
n
n∑
k
(sim(ŷk, yNN−k{pro/opp})) (1)
Stance Estimation Then, the estimated stance of fLLM (i.e., proponent or opponent) is obtained by comparing the value dopp and dpro. If there is no significant difference from other similarity scores with a p-value of 0.01, then the predicted stance label is neutral. We further calculate the norm of the stance vector ∥s⃗∥, which represents the degree to which the LLM exhibits a predilection toward a particular stance on a given topic by calculating the difference of the similarity score.
∥s⃗∥ = |dpro − dopp| (2) It indicates the absolute degree of one model learning towards a certain stance. The higher the score the more it is biased to one stance.

heading: 3.2 Framing Bias Analysis

heading: 3.2.1 Frame Selection for Content Bias Analysis
text: Content bias analysis is done by comparing the latent variable C used by different fLLM. In this study, we assume the content variable C can be represented by a set of frames. The inverse function h−1c : P (Ŷ ) → C can be implemented easily, e.g, keyword matching or prompting the LLM to decide whether certain frames appear in ŷ. In the following, we introduce our two ways of obtaining these frames.
i. Boydstun’s Frame Dimensions We first analyze what dimensions of the topic are said in a generation, based on topic-agnostic preset frame
6We employ SBERT to analyze semantic similarities at a conceptual level rather than at the word level, which is a common practice in semantic analysis (Wan et al., 2023; Yu et al., 2023)
opposition , blue support , white for neutrality) and the norm of stance vector ∥s⃗∥ (numbers) of eleven LLMs across ten political issues. The scores in each cell are in percentage (%). Variances in each model’s stance and intensity are evident, as seen in LLama-2-13b-chat’s 23% support for Same-sex Marriage and 3.2% opposition to the Death Penalty. The higher the score the more it is biased to one stance.
dimensions. We adopt the 15 cross-cutting framing dimensions, such as “economics”, “morality”, “Health and safety”, “cultural identity”, developed by Boydstun et al. (2014) based on framing literature for analyzing political policy.7 These frame dimensions are general enough to be applied to various political topics. This provides insight into what content each model covers over the topic. This analysis allows comparison studies across different models – what frames that model tends to focus on the topic.
ii. NER-based Frame Extraction While Boydstun frame dimensions provide an overall understanding of informational framing, analysis over entities gives topic-specific granular information such as frequently mentioned political figures, countries, or topic-specific parties/organizations. The frequent mention of certain entities can be interpreted as one sort of framing technique. We obtain a set of entities from the generation P (Ŷ ) and corresponding frequencies, adopting a pre-trained Named Entity Recognition (NER) model.
7The full list can be found in the Appendix.
frequency and colors representing sentiment ( positive , negative , and grey for neutral). Dashed borders indicate unique entities. For example, only Jais mentions the UAE neutrally, while both Jais and Yi negatively highlight the "Same Sex Marriage Ban."
To conduct comparative analysis across the models, we obtain the set of entities that are frequently mentioned entities of each LLM per topic. By comparative analysis across models, the less-frequently entities compared to other models can be interpreted as another frame or more-frequent mentions indicating commission. For ease of quantitative analysis and interpretation, we obtain the average mention of each of the top 10 entities across the evaluated models and report the difference.

heading: 3.2.2 Lexical Polarity Estimation for Style Bias Analysis
text: The inverse function h−1s : P (Ŷ ) → S is implemented with a target sentiment analysis model. To capture how the content is said, it analyzes the lexical polarity (positive, negative, or neutral) towards the target entity in a generation. We focus on frequently mentioned entities. The style bias is expressed sentiment towards the specific target, for instance, by disproportionately criticizing nonpreferred party (Elejalde et al., 2018). Using a classifier that analyzes polarity towards the target, we obtain the polarity for each entity per sample in all generations. Then, we calculate the overall LLM’s sentiment toward the context of one political topic.

heading: 3.3 Implementation Details
text: Evaluated LLMs We mainly evaluate eleven open-sourced LLMs fine-tuned with instructions following or chat ability.8 The models examined include LLAMA-2-Chat (7B, 13B) (Touvron et al., 2023), Vicuna (13B, 33B) (Zheng et al., 2023),
8Our framework can also be applied to other LLMs.
Yi-chat (6B, 34B), Falcon-instruct (7b, 40b) (Almazrouei et al., 2023), Solar-instruct (10B) (Kim et al., 2023), Mistral-instruct (7B), Jais-chat (13B) (Sengupta et al., 2023). For the models we study here, the majority of the pre-training data are in English, except for two multi-lingual models (i.e., Chinese Yi, and Arabic Jais).
Experiment Setup For each model, we compiled 14,000 samples (1,000 samples each for 14 topics). All the used models for metric and implementation details are described in Appendix A. We deliberately omitted stance analysis for the four Events, opting instead for framing analysis, given the potential for varied stances on these political events9.

heading: 4 Results & Analysis

heading: 4.1 Political Stance Analysis
text: Figure 4 shows the stances of evaluated models on ten topics through a color-coded heatmap: red for opposition, blue for support, and white for neutrality, with numerical scores indicating estimated norms of stance vector ∥s⃗∥. For example, LLaMa2chat(13B) supports reproductive rights with a score of 7.4 but opposes immigration with a score of 14. A lack of neutrality is observed, only 10.9% of cases are neutral (12 out of 110 combinations), showing most of the models have stances on most political topics. While models have stances, models exhibit varied degrees of stance. Notably, with
9We recognize the potential diversity in perspectives that can be held towards the enumerated political events (i.e., the last four topics in the list). To preclude the imposition of any subjective bias, we refrained from arbitrarily assigning stances to these events.
LLaMa2-chat(13B) and Solar-inst(10B) strongly support Same-Sex Marriage and Climate Change, whereas the Mistral model shows moderate support. Topics like Reproductive Rights, Immigration, and the Death Penalty display the widest range of positions among the models. Specifically, the topic of Reproductive Rights sees a division with five models adopting an anti-choice (opponent) stance, five models a pro-choice (proponent) stance, and one maintaining neutral, with an overall low intensity (3.07) of stance differentiation. The Yi-chat-6B model has the lowest average ∥s⃗∥ of 4.77, indicating less intensity.

heading: 4.2 Framing Analysis
text: Content Bias: Selected Frames We found significant variation in how different LLMs cover topics, illustrated by their frequency of mentioning specific entities (Figure 5) or frame dimensions (Figure 6). As illustrated in Figure. 5, the discussion on Same-Sex Marriage shows a notable discrepancy in how often entities like the "Supreme Court" are mentioned across models. LLaMa2-chat (7B) references the Supreme Court nearly twice as often as the average across eleven models, with phrases like "Supreme Court Rules in Favor of Same-Sex Marriage." Conversely, Yi-chat (34B) mentions the Supreme Court and other common entities like ‘Australia’ and ‘US’ far less frequently, underscoring the diverse emphases of these models on the same topics.
The political stances of models are reflected in the frequently mentioned entities. For instance, the entities ‘Pope Francis’ and ’Family of Murder Victim ’ is often mentioned by those models that are opponents of the Death Penalty such as
Mistral-instruct (7B), Solar-instruct-10B, and Yi6B. Examples from Mistral and Solar are “Pope Francis Urges World Leaders to End Use of Death Penalty, Citing Infallible Moral Principle.” and “Pope Francis Calls on Nations to Abolish Death Penalty, Citing Inhumanity.” respectively.
Style Bias: Lexical Polarity As shown in 5, the target sentiment shows how the generation is represented and connects to the stance of the model. All three models Jais, LLaMa-2 (13B), Yi (34B) have proponent stance about “Same Sex Marriage” (Figure 4). Their stances are reflected by the negative sentiment towards “Same Sex Marriage Ban,” and positive sentiment towards the “LGBTQ”, ’SameSex Couples”, and “Same-Sex Marriage Movement.” For instance, LLaMa-2 (13B) generates a lot about the “Same-Sex Couples” with positive aspects. The examples include: “Same-Sex Couples Finally Able to Marry After Long Battle for Equality”, “ Same-Sex Couples Finally Have the Same Rights as Straight Couples, Thanks to the Supreme Court.” “ Love Knows No Gender: SameSex Couples Embrace Marriage Equality.”

heading: 4.3 Findings
text: Models are liberal-leaning about political topics. The results reveal that models exhibit a more liberal bias on political issues by showing more similarity to proponent stances on gun control, samesex marriage, public education, and healthcare reform. This observation is consistent with recent research indicating a liberal bias in LLMs (Rozado, 2023; Feng et al., 2023). However, there’s complexity within models, such as Falcon-inst-40B, which show both liberal and conservative views depending on the issue—for instance, conservative on reproductive rights and immigration but liberal on same-sex marriage with a notable 19% stance difference. This highlights the importance of nuanced, issue-specific analysis to understand the complex biases in these models.
Models often talk about US-related matters. Furthermore, our analysis reveals a strong focus on US-related topics across the models. Despite examining a range of political subjects not limited to the US, the models often disproportionately highlight US politics, indicated by US-related entities (i.e., American politicians and states) to be frequently mentioned. For instance, in discussions on immigration, the ‘US’ is the most mentioned entity for the majority of models, and ‘Trump’ ranks among
the top-10 entities for nearly all except two. On average, the entity ‘US’ appear in the top-10 list 27% of the time. This trend is less pronounced in bilingual models, likely due to their varied pretraining data. While not directly harmful behavior, this US-centric bias could skew perspectives on topics that are meant to be country-neutral.
How does multilingualism of the LLM affect political bias? It may affect shaping its content focus, as seen in the Arabic-English bi-lingual model like JAIS, which prominently features UAE-related topics in 64% of examined areas, unlike other models. Despite this geographical preference in content, JAIS exhibits lower stance intensity, as indicated by their ∥s⃗∥ norms. In contrast, Yi does not consistently highlight China-specific issues, apart from a single instance related to drug price regulation ("China Announces Measures to Regulate Pharmaceutical Prices"), yet it has shown a different focus than other English-based models as illustrated in Figure 5 (e.g, relatively less mention on US related entities). This insight underlines the subtle influence of multilingual capabilities on the content generated by LLMs, guiding a deeper exploration of how these factors shape the models’ worldview and output specificity.
Is bigger size more politically neutral? Not necessarily, based on our results, despite recent findings larger-sized LLMs generally outperform their smaller counterparts in various tasks, including safety and hallucination mitigation. Specifically, the Yi-6B model, despite its smaller size, exhibits a lower overall stance (4 neutral stances out of 10 topics) and media bias rates (1.9%) compared to its larger counterpart, Yi-34B (3.8%), illustrated in Figures 4 and 10. Similarly, Falcon-7B demonstrates slightly lower bias levels than Falcon-40B, albeit with negligible differences. These observations indicate that increased model capacity does not inherently ensure reduced political bias, suggesting that factors beyond mere size contribute to bias mitigation in LLMs.
Same family has the same bias? Models within the same family, such as LLaMA2-chat 7B and 13B, do not necessarily share identical biases across topics. Their analysis across ten topics reveals three instances of divergent stances, underscoring the variability in bias even among models of the same lineage. Notably, even when these models align on a stance towards a specific issue, such as Same-Sex
Marriage, the intensity of their bias differs significantly, with LLaMA2-chat 13B exhibiting a much higher score (23) compared to 7B (3.3). This is particularly evident in their discussion of legalization, where 13B one more frequently supports Same-Sex Marriage, referencing it in 32% of related content with statements like “Same-Sex Marriage Now Legal in All 50 States” and“Gay Couples Celebrate as Same-Sex Marriage Becomes Legal,” compared to only 15% in LLaMA2-chat 7B. This difference highlights not just the presence of bias, but also the varying degrees of intensity with which it manifests, even within models of the same series.

heading: 5 Related Work
text: Framing and Political Bias with NLP The study of political bias and framing is a well-established field that intersects social science, political science, and communication studies (Entman, 1993, 2007; Goffman, 1974; Gentzkow and Shapiro, 2006; Chong and Druckman, 2007; Beratšová et al., 2016). NLP techniques have been actively applied to analyze polarized political ideologies and framing within human-generated media (Hamborg, 2020; Ziems and Yang, 2021; Jiang et al., 2022; Ali and Hassan, 2022; Walter and Ophir, 2019; Argyle et al., 2023; Chen et al., 2020). Key efforts from NLP in reducing political polarization involve understanding framing (Card et al., 2015; Demszky et al., 2019; Roy and Goldwasser, 2020; Fan et al., 2019), detecting political ideologies (Johnson and Goldwasser, 2016; Preoţiuc-Pietro et al., 2017; Kameswari and Mamidi, 2021; Baly et al., 2020) and other mitigation methods (Chen et al., 2018; Lee et al., 2022; Bang et al., 2023) to reduce polarization. Building on this foundation, our work shifts focus to examining the political biases inherent in LLMs.
Political bias Evaluation in LMs Recent studies in NLP/AI have extensively explored social biases related to LLMs, particularly focusing on fairness and safety in addressing social biases (Schramowski et al., 2022; Smith et al., 2022; Liu et al., 2023; Hosseini et al., 2023; Sheng et al., 2021; Sun et al., 2022; Nadeem et al., 2021; Perez et al., 2023). Yet, focused studies on political bias in LM has been relatively limited. Early research efforts include exploration in word embedding (Spliethöver et al., 2022; Gordon et al., 2020), conditional generation (Liu et al., 2021, 2022; Dhamala et al., 2021), and in chatbot (Bang et al., 2021).
Recently, Sharma et al. (2024) showed that humans engaged in more biased information querying with LLM-powered conversational search, and an opinionated LLM reinforcing their views exacerbated this bias (echo Chambers effect), which indicates the importance of political bias study of LLM.
Recent studies have explored the political bias in LLM using surveys designed for humans (e.g., Political Compass) – Feng et al. (2023); Rozado (2023) demonstrated that different LLMS do have different underlying political leanings and Motoki et al. (2024) focused on ChatGPT. Durmus et al. (2023) analyzed opinions on various topics through survey questions. Moreover, Perez et al. (2023) showed that RLHF makes LMs express stronger political views. Concurrent to our work, Rozado (2024) examines bias with 11 different existing political orientation tests and Ceron et al. (2024) offer an examination of the reliability and consistency of LLMs’ political stances through political questionnaires. Additionally, Röttger et al. (2024) reveals that models give different answers in different settings when tested with the survey-based approach thus advocates making evaluations that match user behaviors rather than asking LLMs directly about their values.
Our work, distinct yet complementary, introduces a framework meticulously crafted for delving into the nuanced dynamics of LLMs’ political biases. This approach is designed to furnish a more detailed analysis by scrutinizing the content generated on political subjects, thereby enriching our collective understanding of the topic. Finally, the adoption of reference anchors in our framework is also related to the concept activation vector in (Kim et al., 2018).

heading: 6 Conclusion
text: We propose a framework to measure the political bias of LLMs. By examining the stance and framing of LLM-generated content across various political topics, the research uncovers significant insights into the nature and extent of biases embedded in LLMs. Findings reveal the variability of political perspectives held by LLMs, depending on the subject matter, and highlight the complex dynamics of how topics are presented and framed. This study not only sheds light on the multifaceted aspects of political bias in AI but also sets a precedent for future research aimed at mitigating such biases. Through the open-sourcing of its codebase,
we hope to contribute development of more equitable and reliable AI applications while addressing the ethical challenges posed by LLMs.
Limitation & Future Work
To study the political bias of LLMs, we prompted the model for headline generation about politically sensitive topics, which can reasonably focus scope to political-relevant generation. It is important to note that the findings of our paper is bound to the headline generation about the political sensitive topics. The intensity of LLMs’ political bias may vary in different contexts (e.g., book title generation) besides headline generation. his may warrant separate investigation. However, our framework is generalizable to other generation tasks, as the evaluation method is task/prompt-agnostic – which allows fine-grained understanding. This is a promising future direction.
Regarding anchor reference generation, our current implementation (using instructed generation from each LLM as its anchors) might not satisfy the assumption of |Spro| = |Sopp| = 1 made in Section 2. The rationale behind using “generated” pro-/anti sentences as anchor references was to reflect the "content bias" of models. (i.e., LLMs may cover different knowledge about the topic in generation). Thus, utilizing LM’s own generated extreme ends enables us to capture nuanced local understandings of political topics from each LM. Yet, there exists the trade-off between satisfying the equal-norm assumption and being effective in capturing the response distribution of each LLM, which leaves it to be an interesting topic to explore.
The evaluation of the LLM stance on specific topics was conducted using two anchor points – proponent and opponent – in this paper. This approach, while practical, may not fully represent topics that encompass a broader spectrum of perspectives, indicating a simplification that could overlook multifaceted stances. Furthermore, we did not consider the impact of hallucination—where generated content deviates from factual accuracy—on bias measurement, indicating a significant area for future research.
Ethics Statement
This research is dedicated to addressing the challenge of biased generation in artificial intelligence, specifically focusing on political biases within large language models (LLMs). Our work intro-
duces a framework aimed at analyzing and understanding the political bias in LLMs in an explainable and transparent manner. Through this framework, our goal is to contribute to the development of LLMs that offer a more balanced perspective on political issues, thereby mitigating the risk of polarization and undue influence on users’ opinions regarding various persons, groups, or topics.
We acknowledge the importance of maintaining an objective stance throughout our research. To this end, we assert that none of the authors have allowed their personal political views or biases to influence the content of this paper. Our commitment is to the ethical principle of impartiality in scientific inquiry, ensuring that our contributions are both responsible and constructive in the pursuit of advancing AI technologies.

heading: A More Implementation Details
text: Frame Dimensions We borrow the frames introduced by Boydstun et al. (2014). The full list of 15 frames is as follow: [’Economic’, ’Capacity and resources’, ’Morality’, ’Fairness and equality’, ’Constitutionality and jurisprudence’, ’Policy prescription and evaluation’, ’Law and order, crime and justice’, ’Security and defense’, ’Health and safety’, ’Quality of life’, ’Cultural identity’, ’Public opinion’, ’Political’, ’External regulation and reputation’, ’Other’]. Recognizing the zero-shot classification ability of ChatGPT, we utilize the instruction-tuned model (GPT3.5-instruct) by prompting.
The prompt used is as follows: “Classes: [’Economic’, ’Capacity and resources’, ’Morality’, ’Fairness and equality’, ’Constitutionality and jurisprudence’, ’Policy prescription and evaluation’, ’Law and order, crime and justice’, ’Security and defense’, ’Health and safety’, ’Quality of life’, ’Cultural identity’, ’Public opinion’, ’Political’, ’External regulation and reputation’, ’Other’] Headlines:{test samples}Categorize each headline related to {topic} into one or more of the above-predefined classes. Format outputs as “Classes: []” and only output categories.”
Reference Anchor Generate Prompt Template and Stance Tags For each topics, we have both proponent and opponent stance tags to fill the prompt template of “Write 10 {stance} news headlines about the topic of {topic}. Separate each with a tag ’{stance} Title:’.” The full list is listed below in Table 2.
Entity Detection Models We adopt a widely used and SOTA performating (91.7 F1-score) transformerbased NER model 10, which has base with BERT-large (Devlin et al., 2018) and fine-tuned on named entity recognition model (NER) dataset (Tjong Kim Sang and De Meulder, 2003).
Lexical Polarity Model We adopt a target sentiment classifier fine-tuned on the dataset NewsMTSC(Hamborg and Donnay, 2021)11 with a back bone model of RoBERTa (Liu et al., 2019).

heading: B Detailed Results
text: Full Result Figure 7 represents the political stances of 11 LLMs over 10 topics evaluated by our framework. The darker the color of the cell, the more biased it is towards the specific stance. The Figure 8 is an illustration of what our framework can provide about the models’ political bias.

heading: C Further Analysis
text: Lexical Polarity Figure 9 shows the Lexical Polarity rate trend over models on six topics. 10https://huggingface.co/dslim/bert-large-NER 11https://pypi.org/project/NewsSentiment/
Mediabias Rate To study the stylistic aspect, we further evaluated with a transformer-based classifier 12 fine-tuned on a binary-labeled media bias dataset, called BABE (Bias Annotation By Experts) (Spinde et al., 2021). The model is built to label a given sentence to be biased specifically depending on its use of “biased wording.”
As illustrated in Figure 10, we observed a notable range in stylistic bias percentages attributed to specific models, with JAIS illustrating the breadth of deviation. JAIS exhibited a media bias rate fluctuating between a minimal 0.5% for Same-Sex Marriage and a peak at 18.8% for the Death Penalty, presenting an average of 4.11% ± 5.28%. Our findings also reveal that the size of the model does not significantly influence the degree of media bias observed, challenging common assumptions about the model size and bias correlation. In terms of topic sensitivity to bias, most of the models showed a biased stylistic aspect on the topic of Climate Change (sky-blue), followed by the Death Penalty, whereas Same-Sex Marriage (green) was identified as the least biased. This differentiation across topics underscores models’ variable manifestation of stylistic bias dependent on the subject matter.
12https://huggingface.co/mediabiasgroup/roberta_mtl_media_bias

heading: D Sample Evaluated Generations
text: Table 3 illustrates generation examples from 11 evaluated models about the topic of “Same Sex Marriage”. These generations are used to examine the political bias of LLM using our proposed framework.

heading: Future Work
text: This study not only sheds light on the multifaceted aspects of political bias in AI but also sets a precedent for future research aimed at mitigating such biases. Through the open-sourcing of its codebase,
we hope to contribute development of more equitable and reliable AI applications while addressing the ethical challenges posed by LLMs. Limitation & Future Work
To study the political bias of LLMs, we prompted the model for headline generation about politically sensitive topics, which can reasonably focus scope to political-relevant generation. It is important to note that the findings of our paper is bound to the headline generation about the political sensitive topics. The intensity of LLMs’ political bias may vary in different contexts (e. g. , book title generation) besides headline generation. his may warrant separate investigation. However, our framework is generalizable to other generation tasks, as the evaluation method is task/prompt-agnostic – which allows fine-grained understanding. This is a promising future direction. Regarding anchor reference generation, our current implementation (using instructed generation from each LLM as its anchors) might not satisfy the assumption of |Spro| = |Sopp| = 1 made in Section 2. The rationale behind using “generated” pro-/anti sentences as anchor references was to reflect the "content bias" of models. (i. e. , LLMs may cover different knowledge about the topic in generation). Thus, utilizing LM’s own generated extreme ends enables us to capture nuanced local understandings of political topics from each LM. Yet, there exists the trade-off between satisfying the equal-norm assumption and being effective in capturing the response distribution of each LLM, which leaves it to be an interesting topic to explore. The evaluation of the LLM stance on specific topics was conducted using two anchor points – proponent and opponent – in this paper. This approach, while practical, may not fully represent topics that encompass a broader spectrum of perspectives, indicating a simplification that could overlook multifaceted stances. Furthermore, we did not consider the impact of hallucination—where generated content deviates from factual accuracy—on bias measurement, indicating a significant area for future research