Leveraging Word Guessing Games to Assess the Intelligence of Large
  Language Models
Authors: name: Tian Liang; name: Zhiwei He; name: Jen-tes Huang; name: Wenxuan Wang; name: Wenxiang Jiao; name: Rui Wang; name: Yujiu Yang; name: Zhaopeng Tu; name: Shuming Shi; name: Xing Wang
Published: 2023-10-31 14:37:42

Abstract: The automatic evaluation of LLM-based agent intelligence is critical in
developing advanced LLM-based agents. Although considerable effort has been
devoted to developing human-annotated evaluation datasets, such as AlpacaEval,
existing techniques are costly, time-consuming, and lack adaptability. In this
paper, inspired by the popular language game ``Who is Spy'', we propose to use
the word guessing game to assess the intelligence performance of LLMs. Given a
word, the LLM is asked to describe the word and determine its identity (spy or
not) based on its and other players' descriptions. Ideally, an advanced agent
should possess the ability to accurately describe a given word using an
aggressive description while concurrently maximizing confusion in the
conservative description, enhancing its participation in the game. To this end,
we first develop DEEP to evaluate LLMs' expression and disguising abilities.
DEEP requires LLM to describe a word in aggressive and conservative modes. We
then introduce SpyGame, an interactive multi-agent framework designed to assess
LLMs' intelligence through participation in a competitive language-based board
game. Incorporating multi-agent interaction, SpyGame requires the target LLM to
possess linguistic skills and strategic thinking, providing a more
comprehensive evaluation of LLMs' human-like cognitive abilities and
adaptability in complex communication situations. The proposed evaluation
framework is very easy to implement. We collected words from multiple sources,
domains, and languages and used the proposed evaluation framework to conduct
experiments. Extensive experiments demonstrate that the proposed DEEP and
SpyGame effectively evaluate the capabilities of various LLMs, capturing their
ability to adapt to novel situations and engage in strategic communication.