Benchmarking LLMs for Political Science: A United Nations Perspective
Authors: name: Yueqing Liang, name: Liangwei Yang, name: Chen Wang, name: Congying Xia, name: Rui Meng, name: Xiongxiao Xu, name: Haoran Wang, name: Ali Payani, name: Kai Shu

Large Language Models (LLMs) have achieved significant advances in natural
language processing, yet their potential for high-stake political
decision-making remains largely unexplored. This paper addresses the gap by
focusing on the application of LLMs to the United Nations (UN) decision-making
process, where the stakes are particularly high and political decisions can
have far-reaching consequences. We introduce a novel dataset comprising
publicly available UN Security Council (UNSC) records from 1994 to 2024,
including draft resolutions, voting records, and diplomatic speeches. Using
this dataset, we propose the United Nations Benchmark (UNBench), the first
comprehensive benchmark designed to evaluate LLMs across four interconnected
political science tasks: co-penholder judgment, representative voting
simulation, draft adoption prediction, and representative statement generation.
These tasks span the three stages of the UN decision-making process--drafting,
voting, and discussing--and aim to assess LLMs' ability to understand and
simulate political dynamics. Our experimental analysis demonstrates the
potential and challenges of applying LLMs in this domain, providing insights
into their strengths and limitations in political science. This work
contributes to the growing intersection of AI and political science, opening
new avenues for research and practical applications in global governance. The
UNBench Repository can be accessed at:
https://github.com/yueqingliang1/UNBench.