Sentient Agent as a Judge: Evaluating Higher-Order Social Cognition in
  Large Language Models
Authors: name: Bang Zhang; name: Ruotian Ma; name: Qingxuan Jiang; name: Peisong Wang; name: Jiaqi Chen; name: Zheng Xie; name: Xingyu Chen; name: Yue Wang; name: Fanghua Ye; name: Jian Li; name: Yifan Yang; name: Zhaopeng Tu; name: Xiaolong Li
Published: 2025-05-01 19:06:10

Abstract: Assessing how well a large language model (LLM) understands human, rather
than merely text, remains an open challenge. To bridge the gap, we introduce
Sentient Agent as a Judge (SAGE), an automated evaluation framework that
measures an LLM's higher-order social cognition. SAGE instantiates a Sentient
Agent that simulates human-like emotional changes and inner thoughts during
interaction, providing a more realistic evaluation of the tested model in
multi-turn conversations. At every turn, the agent reasons about (i) how its
emotion changes, (ii) how it feels, and (iii) how it should reply, yielding a
numerical emotion trajectory and interpretable inner thoughts. Experiments on
100 supportive-dialogue scenarios show that the final Sentient emotion score
correlates strongly with Barrett-Lennard Relationship Inventory (BLRI) ratings
and utterance-level empathy metrics, validating psychological fidelity. We also
build a public Sentient Leaderboard covering 18 commercial and open-source
models that uncovers substantial gaps (up to 4x) between frontier systems
(GPT-4o-Latest, Gemini2.5-Pro) and earlier baselines, gaps not reflected in
conventional leaderboards (e.g., Arena). SAGE thus provides a principled,
scalable and interpretable tool for tracking progress toward genuinely
empathetic and socially adept language agents.