Profiling News Media for Factuality and Bias Using LLMs and the
  Fact-Checking Methodology of Human Experts
Authors: name: Zain Muhammad Mujahid; name: Dilshod Azizov; name: Maha Tufail Agro; name: Preslav Nakov
Published: 2025-06-14 15:49:20

Abstract: In an age characterized by the proliferation of mis- and disinformation
online, it is critical to empower readers to understand the content they are
reading. Important efforts in this direction rely on manual or automatic
fact-checking, which can be challenging for emerging claims with limited
information. Such scenarios can be handled by assessing the reliability and the
political bias of the source of the claim, i.e., characterizing entire news
outlets rather than individual claims or articles. This is an important but
understudied research direction. While prior work has looked into linguistic
and social contexts, we do not analyze individual articles or information in
social media. Instead, we propose a novel methodology that emulates the
criteria that professional fact-checkers use to assess the factuality and
political bias of an entire outlet. Specifically, we design a variety of
prompts based on these criteria and elicit responses from large language models
(LLMs), which we aggregate to make predictions. In addition to demonstrating
sizable improvements over strong baselines via extensive experiments with
multiple LLMs, we provide an in-depth error analysis of the effect of media
popularity and region on model performance. Further, we conduct an ablation
study to highlight the key components of our dataset that contribute to these
improvements. To facilitate future research, we released our dataset and code
at https://github.com/mbzuai-nlp/llm-media-profiling.