Probing the Multi-turn Planning Capabilities of LLMs via 20 Question Games (2024)

Abstract: Large language models (LLMs) are effective at answering questions that are clearly asked. However, when faced with ambiguous queries they can act unpredictably and produce incorrect outputs. This underscores the need for the development of intelligent agents capable of asking clarification questions to resolve ambiguities effectively. This capability requires complex understanding, state tracking, reasoning and planning over multiple conversational turns. However, directly measuring this can be challenging. In this paper, we offer a surrogate problem which assesses an LLMs’s capability to deduce an entity unknown to itself, but revealed to a judge, by asking the judge a series of queries. This entity-deducing game can serve as an evaluation framework to probe the conversational reasoning and planning capabilities of language models. We systematically evaluate various LLMs and discover significant differences in their performance on this task. We find that strong LLMs like GPT-4 outperform human players by a large margin. We further employ Behavior Cloning (BC) to examine whether a weaker model is capable of imitating a stronger model and generalizing to data or domains, using only the demonstrations from a stronger model. We finally propose to use Reinforcement Learning to enhance reasoning and planning capacity of Vicuna models through episodes of game playing, which lead to significant performance improvement. We hope that this problem offers insights into how autonomous agents could be trained to behave more intelligently in ambiguous circumstances.

heading: 1 Introduction
text: In uncertain circumstances, conversational agents may need to take the initiative to reduce their uncertainty by asking good questions proactively, thereby solving problems more effectively. This requires intricate, interactive, strategic decisionmaking and reasoning about the agent’s next move
in a multi-turn conversation. This capability is crucial in various applications like task-oriented chatbots, recommendations, and conversational search.
Traditional dialogue systems have been achieved by modularizing various aspects of such tasks into sub-tasks such as understanding, state tracking, planning, and generation. However, recent advances in LLM have made it possible to create an end-to-end pipeline for developing autonomous agents that can complete complex tasks using enhanced planning and memory capabilities. Promising autonomous agents, such as ReAct (Yao et al., 2022), AutoGPT (Significant Gravitas, 2023), and LangChain (Langchain-AI, 2023), require the underlying LLM to recall information from previous dialogues, resembling the understanding and state tracking stage. They also rely on the LLM to decompose larger tasks into more manageable components, analogous to the planning stage. Among them, some approaches (e.g., HuggingGPT (Shen et al., 2023)) use a static planning strategy by first generating the complete plan via LLM and subsequently tackling subtasks. Other approaches (e.g., AutoGPT) adopt an interactive planning strategy, where the generation of each action is conditioned on the outcome of the previous planning steps.
While LLM-powered agents are powerful, they can perform poorly when a user’s instructions are unclear. This poses a critical challenge – how to accurately elicit and capture the user’s intents, which are often nuanced and noisy, to enable dynamic rather than static human-computer interactions. For example, in Figure 1, the agent aims to: 1) accurately assess the current dialog state; 2) disambiguate user’s intent and satisfy the user demand; 3) ask as few questions as possible.
Progress in this direction is difficult because directly measuring complex understanding, reasoning and planning capabilities is challenging. In this study, we investigate this somewhat overlooked research problem – how good the LLMs are at asking
1495

heading: Do you want to know their information, or to play their album?
text: questions and deducing intent. Instead of directly evaluating disambiguation ability of LLMs, we study a proxy task in a controlled setting with clear evaluation. We propose to use entity-deducing games, specifically the 20 questions game (Q20) (Akinator, 2007), to assess the complex reasoning and strategic planning capability of LLMs, in formulating precise questions/guesses over long conversations (Figure 1). This serves as a first-step towards probing the underlying cognitive abilities required for effective disambiguation. This game requires a model to infer an unidentified entity with as few queries as possible. To achieve this, the model needs track the dialogue state over turns, and use its reasoning and planning skills to effectively partition and narrow down the search scope.
We systematically evaluated several LLMs on this task, and found significant differences in their conversational reasoning and planning capabilities. We then investigated a set of research questions to enhance open-source models using demonstrations from high-performing closed-source models. Finally, we employ PPO (Schulman et al., 2017) to improve vanilla Vicuna (Chiang et al., 2023) models using multi-turn game playing directly. We will provide the code and dataset to facilitate future research. Our findings offer insights into potential future advancements in autonomous agents that proactively lead the conversation.

heading: 2 Entity-deduction Arena (EDA)
text: During an entity-deducing game session, two players engage in a game regarding a specific entity. One player, “judge” (J), is provided with the entity
and is expected to respond to queries or guesses from the guesser using only the responses “Yes,” “No,” or “Maybe,”. The other player, “guesser” (G), is agnostic about the entity, and is supposed to pose a series of questions to deduce the entity using as few queries as possible.
The judge does not require access to the dialogue history and only needs the entity and current question to provide a response1. This task is akin to closed-book QA (Roberts et al., 2020), which current LLMs can reasonably handle. On the other hand, playing the guesser is more demanding. A proficient G necessitates several multi-turn dialogue capabilities working in synergy: 1) State Tracking and Understanding: G must comprehend multiturn context, track asked questions, and understand its position in the game and coreference resolution. 2) Strategic Planning: G needs to strategically ask questions to progress efficiently towards a better state, avoiding redundant queries and ensuring consistency with prior knowledge. 3) Inductive Reasoning: G must use conversation comprehension to generate conjectures based on acquired knowledge. G must inherently establish a taxonomy representation to efficiently and accurately identify the correct entity among numerous options.
Our assessment, referred to as the EntityDeduction Arena (EDA), focuses on evaluating the capability of various LLMs as the guesser, as a proxy to probe their overall capabilities in handling complex multi-turn conversational tasks involving
1Our experiment indicates that incorporating entire dialogue history negatively impacts accuracy as the additional information tends to confuse the judge.
proactively asking clarification questions. While EDA may not evaluate all aspects of reasoning and planning ability, it capture several key capabilities which are important for any conversational AI agent that needs to actively elicit information from users through clarifying questions. The task provides a targeted way to probe these capabilities in a controlled setting, and provides a new and complementary benchmark compared to existing tasks for probing language models’ capabilities from a different angle focused on multi-turn strategic conversation.

heading: 2.1 Experimental settings
text: Datasets We conducted the evaluation on two proposed datasets: Things and Celebrities. The Things dataset consists of 500 entities that are commonly found in the web corpus (Raffel et al., 2020). It encompasses a wide range of categories, such as common-life objects, animals, foods, plants, vehicles, clothing, professions, materials, instruments, places, birds, sports, buildings, furniture, celestial bodies, mythical creatures, events, activities. The larger category “objects” account for 20% percentage of the items. The Celebrities dataset consists of 500 celebrity names covers 32 different nationality, diverse eras of life and various occupations. Among the nationalities, the United States holds the largest proportion with 27.7%. In terms of occupations, the largest categories are actor/actress, accounting for 18.4%. Both datasets were divided into training, evaluation, and testing, with sizes of 300, 100, and 100 respectively. The composition of each dataset is provided in Appendix A.
Judge (J) We employ GPT-3.5-turbo as the judge. The judge takes the entity, questions from the guesser and the following prompt to generate a response of “Yes,” “No,” or “Maybe” for Things dataset. In guessing the celebrity name, the choices are “Yes,” “No,” or “Dunno”. Consequently, the resulting prompt is slightly different. The prompt, and rationale for using “Dunno” for the Celebrities dataset are provided in Appendix B)
Based on your knowledge about the entity: {entity}, respond to the following question or guess. Limit your respond to only “Yes.”, “No.” or “Maybe.”, with no explanation or other words. Never say the answer in your response. If the question is to solicit the answer, respond “No.”. \n Question/Guess: {question} (Yes/No/Maybe)
Whenever the correct answer is contained in the
generation from G as an exact substring match, we manually set the output of J to be “Bingo!” and G wins this game. At the penultimate step of J, an extra prompt “You must guess now, what’s it?” will be appended to J’s response to guide G in making the ultimate guess. To emulate more deterministic responses from J, we use a temperature of 0.2 for the generation. Admittedly, we observe that the judge model exhibits occasional inaccuracies in its responses, resulting in a noisy environment. We asked internal non-paid volunteers to annotate 300 responses from J for each dataset, and the error rate is 3.07% and 2.95% for Things and Celebrities, respectively. Nevertheless, this noisy environment of J is consistent with all models. Addressing judge inaccuracies and inconsistency is an interesting direction for future work.
Guesser (G) The guesser model receives the current dialogue history as input and generates the next question or final guess, guided by the instructions provided in the following (the prompt for Celebrities is provide in Appendix B):
Your task is to ask a series of questions to deduce the entity that I’m thinking of with as few queries as possible. Only ask questions that can be answered by
“Yes,” “No,” or “Maybe”. Do not ask for hint. Make your question brief with no linebreaker. Now start asking a question. \n {dialog history}
We used a sampling approach with a temperature of 0.8 for all experiments, which sufficiently yields diverse outcome. The conversation format for each LLMs is based on their official guidelines.
Evaluation metrics We assess the model’s performance by evaluating its final prediction using the Exact Match (EM) criteria 2. This evaluation considers four key metrics: 1) #Turns, which represents the average number of turns taken to complete the game. Games terminate at 20 turns if failed. Lower is better. 2) Success rate, which indicates the percentage of games won by G. Higher is better. 3) #Yes, which represents the average number of “yes” responses received from the J. 4) Score, which is a combined game score of #Turns and success rate, defined in Eq. (1).
S = (1−λ ·max(#Turns−5, 0)) · I(G wins) (1) 2A more lenient evaluation metric could be employed, e.g. embedding similarities or LLM judges. However, these metrics rely on the specific model or LLM judge employed thus are less consistent than the EM.
where I(·) represents the identity function, λ = 0.02 is a balancing hyperparameter. A higher value for λ would result in assigning inadequate rewards for successful games that take longer, while a smaller value would penalize the model too little for taking a longer turns. In our experiments, we discovered that a value of 0.02 led to models with a reasonable and stable reward. It is worth mentioning that reward design is more of a heuristic, and there may be superior choices available. The #Yes is more of a statistic than a evaluation metric, but we have observed some correlation between #Yes and the outcome. Intuitively, a losing game is often characterized by a high frequency of unproductive guesses (with “No” or “Maybe” response from J).

heading: 3 Benchmarking LLMs on EDA
text: We assess several widely-used LLMs, such as GPT4, GPT-3.5, Claude-1/2, Vicuna (7B-v1.3, 13Bv1.3) (Chiang et al., 2023) and Mistral-7B (Jiang et al., 2023) using our EDA benchmarks. Unfortunately, we are unable to evaluate Llama-2 or its derivatives due to license restrictions. Llama1 (Touvron et al., 2023a) is also excluded from the comparison as it lacks a conversation finetuned model, and as such it often does not adhere to our instructions. The corresponding results are presented in Table 1. The means and standard deviations for each dataset and LLM were calculated using 30 items from 5 separate runs.
In general, GPT-4 attained the best performance among the models evaluated. Around 1/3 of the entities were accurately predicted. The runner-ups are ChatGPT-3.5 and Claude-2. ChatGPT-3.5 is superior on the Things datasets, while Claude-2 shows better strength on the Celebrities datasets. Despite being much smaller, open-source models such as the Vicuna 13B and Mistral 7B model performed comparably well, not trailing too far behind the closed-source models. In particular, Vicuna 13B outperformed Claude-1 when it comes to Things. This suggests that open-sourced models have significant potential. Naturally, models with a higher success rate usually require fewer turns.
It is plausible that a small portion of entity deduction game sessions may exist on the web and were a part of the pretraining data. However, these sessions likely contribute only a minuscule fraction. Nevertheless, robust models such as GPT-4 exhibit consistently strong performance across numerous tested entities. We have also conducted a
large-scale human baseline in a subset of 30 tested examples for each dataset. The discussion of these results can be found in Appendix B.
We present several real game runs in Table 2 on Things. Examples on Celebrities are in Appendix H. We observe that an effective game strategy typically featured by narrowing down the scope through initial questions that divide the remaining search space into two halves of similar capacity, similar to a binary tree search process. During the late game when it becomes challenging to find good binary-partitioning questions, the model start enumerates a smaller number of possibilities. Interestingly, the more advanced models like GPT-4 seem to possess this capability to a decent extent, while weaker models exhibit different failure modes in planning and reasoning abilities (Table 2): 1) (Early Enumeration) initiating enumeration too early without first posing high-level questions to narrow down the scope; 2) (Redundancy) asking questions similar to previous ones; 3) (Inconsistency) asking questions or making guesses that are inconsistent to the answers received in previous turns.
We also compare several games between GPT-4 and human players in Appendix G, highlighting that strong LLMs share similarities with humans in their ability to backtrack when they are on the wrong path. One advantage of LLM over humans is that LLM may have a superior level of knowledge (Appendix G, Table 8), empowering them to ask questions using tacit knowledge among LLMs. One weakness of the LLMs comparing to humans is that LLMs tend to enumerate or repeat to follow the same pattern of previous turns, demonstrating self-reinforced undesirable behaviors. Weaker models tend to be attracted to this absorbing state of repetition or artifacts, and accumulate errors until they ultimately fail (Appendix H). Stronger models occasionally display the ability to escape from this repetitive patterns and rectify early errors, which presumably be attributed to their stronger adherence to the high-level plan. We then address the following Research Questions (RQs):
RQ1: How does the LLM generate strategies? We conducted an analysis to probe the internal dialog states underneath the LLMs, to understand how these states evolve throughout the game, and how do they affect the strategy of the model. To this end, at each turn before the model asks a question, we prompt the LLM to predict top 5 candidates given
the current dialogue history. This probing process is separate from (thus will not affect) the gameplay. The probing prompt is provided in Appendix C. A GPT-4 game session of guessing “guitar” with dialog state is shown in Table 3. A session on Celebrities dataset of guessing “LeBron James” is shown in Table 12 in Appendix C.
From Table 3, it seems that the model seeks a strategy to ask questions to effectively partition the space among the most promising candidates in the initial 10 turns. After the question is answered, the ranked list of top guesses is adjusted, with certain top candidates being eliminated. For instance,
during the third turn, the model asks about the item’s feasibility of being held in one hand, leading to the elimination of the larger objects upon receiving an affirmative answer. During the fourth turn, the model asks if the item is electronic, which ultimately eliminates the top-ranked guess, “smartphone”, after receiving a negative response. The remaining items are shifted one position forward to include a new item to the top guesses.
Interestingly, it is not necessary for the model to always ask questions in order to optimally partition the top-5 candidates. We observed that the strategy also depends on factors including the ease with
which a question can be asked to partition the space, and the level of uncertainty the model has about the current top predictions. We provide details on the measurement of uncertainty in Appendix D. In situations where the model is uncertain, it may occasionally backtrack and reexamine categories that were previously missed. For instance, during the 11th turn, question was asked that ruled out all of the top candidates. This could be due to the fact that the top items are similar in nature, and the model realize there is a significant proportion of other classes that have been overlooked. The successful questioning in these cases led to the recovery of these overlooked classes.
To summarize, a strong G exhibits the following behavior: 1) prioritizing high-level questions before addressing specific details and enumerations, 2) being aware of the current state and asking questions to effectively bi-partition the search space, and 3) being able to occasionally backtracking to consider previously overlooked options.
Additionally, we noticed a high level of consistency in GPT-4’s gameplay strategy across multiple repetitions (see example in Appendix L), despite some fluctuation in the order of the questions asked. This suggests that GPT-4 may rely on its own im-
plicit taxonomy representation of entities to make decisions, which remains consistent throughout.
RQ2: Which one is more important in this task, planning or reasoning? Planning and reasoning abilities affects different stages of game play. Early questions require careful planning to efficiently partition the space, while late game requires more deductive reasoning skills to make an educated guess. We consequently assume that the last turn would only require reasoning ability as no strategic move is needed. With this assumption, we designed the following experiment to investigate the model’s planning and reasoning ability in a finer granularity. Given a stronger model GPT-4 and a weaker model Vicuna 7B, and their respective game play trajectories, we only replay the last turn in each trajectory by swapping the guesser model G.
The results are presented in Table 4. Comparing the GPT-4 → Vicuna 7B with Vicuna 7B, we observe that stronger planning ability from GPT-4 results in significant improvement. Regarding different reasoners, Vicuna 7B → GPT-4 does not show much improvement over Vicuna 7B. This indicates that planning deficiency could result in an unproductive trajectory, poses significant challenges for reasoning during the final step. Moreover, GPT-4 → Vicuna 7B shows regression over GPT-4, emphasizing the importance of reasoning in addition to a strong planning capability. Therefore, it is crucial for both planning and reasoning abilities to be strong and work in synergy to achieve optimal performance. See the full table including the results using human trajectory in the Appendix E.

heading: 4 Enhancing the Open-source Models
text: We further investigate the potential of open-source models. We used Behavior Cloning (BC) to distill capabilities exhibited by the stronger models into the smaller Vicuna models. We first collected game demonstrations from the GPT-3.5 model over all 600 training examples from Things and Celebrities. We then fine-tuned the Vicuna models using these demonstrations. The experimental settings are provided in the Appendix I.
RQ3: Can smaller open-source models benefit from imitating larger close-source models? We first fine-tuned the Vicuna 7B model using a total of 600 training instances from both datasets. The results are presented in Table 1 as the V-FT 7B (All) model. This yielded a more than 70% improvement in both datasets, indicating that weaker models can follow the stronger models’ demonstration to use a more effective strategy to steer the conversation on unseen entities or names. We have also observed that BC training can effectively mitigate undesirable behavior such as repetition.
RQ4: Should models learn from successful demonstration exclusively? To answer this, we selected all the game runs of the entities or names where the teacher model finally won the game from both datasets. This subset was then used to train the Vicuna 7B model, similar to the Rejection Sampling (Touvron et al., 2023b) . The results, referred to as V-FT 7B (Suc.) in Table 1, show that imitating from successful experiences outperforms imitating from all experiences (V-FT 7B (All)). It gets close to the teacher model (GPT-3.5) on Things. However, when it comes to the Celebrities dataset, the improvement is more limited.
RQ5: Does the model size matter? We further compared the performance improvement achieved through finetuning for both Vicuna 7B and 13B models. The model finetuned on Vicuna 13B, VFT 13B (Suc.), also achieved much higher score than Vicuna 13B. However, the improvement was less substantial than the improvement on V-FT 7B (Suc.). Again, the improvement over the Celebrities dataset is marginal. We hypothesize that guessing the celebrity requires a more case-specific strategies compared to guessing things, thus likely to be more challenging to be superficially imitated.

heading: 5 Reinforcement Learning via Gameplay
text: We conducted further experiments to investigate whether the performance could be further enhanced through learning solely from the model’s own experience. We employed Proximal Policy Optimization (PPO) (Schulman et al., 2017) to train the model by playing with with the judge, J, a technique we refer to as Reinforcement Learning from Game-play RLGP. We made modifications to the trlX repository (Castricato et al., 2023) to enable multiturn setup to facilitate RLGP training. During training, we assigned the reward defined in Eq. (1), to the final turn of each rollout. Additionally, we assigned an intermediate reward to turns that received a “Yes” response from J. We trained the V-FT 7B (Suc.) and V-FT 13B (Suc.) models in Table 1 on 300 Things training dateset. Further experimental details can be found in Appendix K. Table 1 presents the performance of the RL-trained models, denoted as V-RLGP.
V-RLGP models exhibit improvement compared to the vanilla Vicuna when tested on the in-domain dataset Things. V-RLGP 13B seems to unlock the potential of the Vicuna model, outperforming the V-FT 13B model, matching the performance of the runner-up GPT-3.5. V-RLGP 7B also show substantial improvement over V-FT 7B model. On the out-domain dataset Celebrities, interestingly, VRLGP models achieves some improvement, showing its generalization potential.

heading: 6 Breakdown Analysis: Do the models agree on successful predictions?
text: We present a comprehensive breakdown analysis of each model’s performance on a subset of 30 evaluated items for each dataset in Figure 2. Within the evaluated items of Things, four items consistently defy successful predictions by all models. On the other hand, Celebrities exhibits a more scattered pattern, with each celebrity being correctly predicted by at least one model.
There are correlations between the entities or names that each model can correctly identify, but different models exhibit their own strengths on different subsets. For instance, the RLGP 13B model can accurately predict “Sunglasses”, whereas the stronger models like GPT-4 consistently fails. We also provide some case studies including why GPT4 consistently fails on “Yoga mat” in Appendix L.
We observed that RLGP models tend to
strengthen the performance on items that vanilla models occasionally succeed in, thereby improving their success rate on these specific items. However, RLGP models do not effectively facilitate learning about new items. Conversely, BC fine-tuning excel in achieving success on new items. Interestingly, the BC fine-tuned models displays different strengths compared to both the initial checkpoint and the expert whose demonstration it mimics. For example, the V-FT 13B (Suc.) model achieves high accuracy in identifying Post-it Note and VR headset, whereas neither the Vicuna 13B nor the GPT-3.5 performs as well in this regard.

heading: 7 Related Work
text: Complex Reasoning Benchmarks Extensive benchmarks have been evaluating the complex reasoning abilities of LLMs (Huang and Chang, 2022). These include HELM (Liang et al., 2022) BIGbench (Srivastava et al., 2022), SuperGLUE (Sarlin et al., 2020), LAMA (Petroni et al., 2019), and CoT-Hub (Fu et al., 2023), which have enabled researchers to assess LLMs across a spectrum of tasks involving arithmetic and math (GSM8K) (Cobbe et al., 2021), commonsense (ARC) (Clark et al., 2018), knowledge (MMLU) (Hendrycks et al., 2020), and coding (HumanEval) (Chen et al., 2021) have been developed to gauge their realworld abilities. We aim to introduce a novel benchmark focus on evaluating multi-turn state tracking and strategical planning capability to complement existing reasoning benchmarks.
Evaluation of Planning Evaluation of the planning abilities of LLMs is relatively rare. Valmeekam et al. (2022) proposed an assessment
framework to gauge the planning capabilities of LLMs to generate valid actions to achieve a special goal and provide a rationale for the plan. Their evaluation on 8 planning tasks reveals LLMs, like GPT3 (Brown et al., 2020), seem to display a dismal performance. Valmeekam et al. (2023) further evaluates on GPT-4 and suggests the autonomous learning capacity of LLMs to formulate plans is limited and dependent on properly designed heuristics. Xie et al. (2023) similarly indicate that LLMs may encounter difficulties in generating planning goals related to numerical or spatial reasoning, while being responsive to the specific prompts used. Unlike these studies, our task solely relies on textual representations of goals.
Multiturn benchmarks MT-Bench (Zheng et al., 2023) assesses the multi-turn conversation and instruction-following ability of LLMs by annotating their responses to questions involving writing, role-play, extraction, reasoning, math, coding and knowledge. Bang et al. (2023) evaluates LLMs on 23 tasks related to logical reasoning, non-textual reasoning, and commonsense reasoning. The study reveals that incorporating interactive features in LLMs can enhance their performance by employing multi-turn prompt engineering. LMRL Gym (Abdulhai et al., 2023), a concurrent work, investigated Multi-Turn Reinforcement Learning with Language Models across multiple tasks. In contrast, our work focuses on the depth by presenting a thorough analysis of the entity deduction task.
Entity-deduction game Testing the model’s ability to deduct an ambiguous entity or asking clarification questions (Aliannejadi et al., 2019; Cho et al., 2019) has been utilized as a testbed on dia-
logue systems and visual reasoning tasks. InfoBot (Dhingra et al., 2016) uses reinforcement learning to learn an optimal dialog policy for identifying movie entries from a movie database. ESP (Von Ahn and Dabbish, 2004) and Peekaboom (Von Ahn et al., 2006) demonstrated that deduction games can effectively gather labeled data. GuessWhat?! (De Vries et al., 2017) and ReferIt (Kazemzadeh et al., 2014), assess the visual reasoning capabilities of tested models by ask them to guess the referred object in the image based on the conversation. Our work instead aims to gauge on the model’s ability on generating the conversation.

heading: 8 Conclusion
text: Motivated by a need to develop agents capable of effectively addressing ambiguous user intents, we introduce a testbed for evaluating LLM’s strategic planning and deductive reasoning abilities in asking entity-deducing questions. Our findings indicate that SOTA LLMs are able to maintain an intrinsic taxonomic representation of knowledge entities to a certain extent. We further show that this capability can be enhanced through Behavior Cloning or Reinforcement Learning, revealing great potential for further advancements. In future research, we intend to investigate whether the implementation of CoT prompting can further enhance the model’s performance in related tasks.

heading: 9 Limitations
text: The Entity-Deduction Arena presents a narrow aspect of LLM multiturn planning capabilities, focusing on iterative refinement over selective taxonomies. As such, the results of this work presents a peak into LLM planning behavior, but doesn’t necessarily apply to all LLM planning problems.
Despite our best efforts to ensure the diversity of the datasets, they could still be subject to bias such as origin language, popularity and time relevance due to their web-crawled nature.
Finally, while Behavior Cloning and RLGP are shown to be useful to improve the performance of Vicuna models, future work could measure their effectiveness across a variety of open source model families.

heading: A Dataset composition
text: We provide the dataset composition on Things and Celebrities in Table 3.
B Prompt used for Celebrities and rationale for using “Dunno”
For Celebrities, the J use the following prompt to generate a response.
Based on on your knowledge about the celebrity: {entity} respond to the following question or guess. Limit your respond to only ’Yes.’, ’No.’ or ’Dunno.’, with no explanation or other words. Never say the name {entity} in your response. Do not say ’Dunno.’ if it can be answered by ’Yes.’ or ’No.’ If the question is to solicit the answer, respond ’No.’.\n Question/Guess: {question}(Yes/No/Dunno)
The G use the following prompt to generate a question or guess.
Your task is to ask a series of questions to deduce the celebrity that I’m thinking of with as few queries as possible. Only ask questions that can be answered by “Yes”, “No”, or “Dunno”. Do not ask for hint. Make your question brief with no linebreaker. Now start asking a question. \n {dialog history}
Rationale for using “Dunno” The Things dataset contains entities that has many different forms, such as if asking “is this made by metal?” for the entity “Chopstick”. The answer to this question could be “Maybe” because sometimes they are made by metal and sometimes are not. On the other hand, in the Celebrities dataset, most questions can typically be clearly answered with a yes or no. These questions often refer to gender, birth year, or other specific information about a person. However, there is a possibility that the model might not know the answer to certain questions, such as “Does this celebrity smoke?”. In these cases, the answer could be “Dunno” as it is considered very personal information. As a general observation, we have found that less than 1% of the responses in the Celebrities dataset are “Dunno”, while around 20% of the responses in the Things dataset are “Maybe”. Note that the percentage of “Dunno” or “Maybe” responses can vary depending on the specific G being used.
Human baseline Collecting static human annotation for this study is a challenging task due to the interactive nature of this research. In order to establish a baseline of human performance, we conducted a human-in-the-loop study. We subsampled Things and Celebrities into 2 smaller subsets containing 30 sample datapoints each, Things-30 and Celebrities-30. We set up a game server and recruited 108 non-paid internal volunteers to interact with the J, and collected a total of 145 and 71 human game play sessions for Things-30 and Celebrities-30, respectively. Human guessers were given the same instructions as the LLM guessers and were provided with a tutorial. Optionally, for training, they could also view the question GPT-3.5 would have asked at the previous step. Statistics, experimental details and UI are provided in Appendix F.
To ensure the quality of the data, we manually inspected human game plays to filter out sessions that do not complete the entity deduction task, contain irrelavant chit-chat, or contain nonsensical repeated questions, leaving only sensible game plays. In the end 201 sessions (93.1% of the total game play) were selected as valid game play.

heading: C Additional results and prompt used for Dialog State Probing
text: We used the following prompt to probe the model for its top guesses during each turn.
{dialog history} Based on the information provided, here are the top 5 most likely concrete entities I think you are thinking of:
We provide additional top K probing results from GPT-4 on Celebrities in Table 12. Similarly, the model seems to be able to leverage its own hierarchical representation of the celebrities’ space to ask questions that steer to narrow down the search effectively.

heading: D Measurement of uncertainty
text: We asked the Guesser model, “On a scale of 1-5, how confident are you in your top guesses based on the current conversation history?” We observed a correlation between the model’s predicted uncertainty and significant shifts in its top guesses. However, this pattern was mainly observed in more powerful models such as GPT-4. In the case of weaker models like Vicuna, the top guesses or confidence scores generated by the model were not very sensible.

heading: E Full ablation results on RQ2, planning vs reasoning
text: The full results comparing planning and reasoning are presented in Table 6. Interestingly, when GPT-4 is employed for the last step reasoning in human trajectories, there is a decrease in performance. One possible explanation for this is that while GPT-4 may excel in comprehending model-generated trajectories, it may struggle to fully interpret the planning trajectory of humans based on the observed turns. This suggests that models may be more compatible with model-generated trajectories rather than human trajectories.

heading: F Experimental details for human baseline
text: In total, we recruited 108 human volunteers to participate in the study. To ensure the quality of the data, we manually inspected human game plays to filter out low quality data. We also prioritized games that
received fewer qualified plays to present to the human players. After a 30-day trial period, we collected a total of 140 and 68 human game play sessions for Things and Celebrities, respectively. Each entity or celebrity is covered at least by two game plays.
We show the UI for the in game demo server in Figure 4. The UI contains 3 sections – game play, tutorial and leaderboard. In the leaderboard, we compute the Wilson confidence interval for each player, and include the performance of LLMs as benchmarks for the human players to refer to.
For Human game plays, entities with fewer game plays are oversampled so that all entities have the same amount of game play. Game plays are then randomized into separate runs to calculate means and standard deviations. Note that this might not be exactly comparable to the model’s numbers.

heading: G Comparison of GPT-4 vs Human player
text: We provide additional examples in Table 7 (Things) and Table 8 (Celebrities) for GPT-4’s generation and compare them with the human players, highlighting that strong LLMs share similarities with humans in their ability to backtrack when they are on the wrong path. One advantage of LLM over humans is that LLM may have a superior level of knowledge (Table 8), empowering them to ask questions using tacit knowledge among LLMs. One weakness of the LLMs comparing to humans is that LLMs tend to enumerate or repeat to follow the same pattern of previous turns, demonstrating self-reinforced undesirable behaviors. Weaker models tend to be attracted to this absorbing state of repetition or artifacts, and accumulate errors until they ultimately fail (Appendix H). Stronger models occasionally display the ability to escape from this repetitive patterns and rectify early errors, which presumably be attributed to their stronger adherence to the high-level plan. We then present a detailed analysis on each dataset:
G.1 Things dataset
In the first example “Guitar”, we observed that both GPT-4 and the human player efficiently converged on a music-related object within approximately 10 turns. However, the human player deviated from the correct path by thinking about an electronic device, whereas GPT-4 promptly identified the item as a musical instrument. Notwithstanding this, the human player’s capacity for back-tracking their steps was evident when they recognized the erroneous direction and ultimately win the game. In this game, the human player retraced the error and asking a reconfirming question “Is it some kind of music player?”.
We then ask the question if the GPT-4 model also has this back-tracking ability. The answer is yes. In Table 7 (Guitar, Human + GPT-4), we ask GPT-4 to complete the game based on partial gameplay provided
by human players until they recognize their mistake (up to “G: Can it display images? J: No.”). GPT-4 promptly comprehends the game scenario and identifies that it has taken the incorrect route. Subsequently, it formulates the appropriate question that encompasses the overlooked category, thereby salvaging the game. We have also observed this similar capability in several other game sessions from GPT-4.
In the second game “Bald eagle”, human player has a luck guess after identifying the entity is a bird. GPT-4, on the contrary, still roll out 3 more turns to gather more information before making an informative guess.
In the third example “Jigsaw Puzzle”, GPT-4 interleaves questions regarding size, purpose and location and combine multiple aspects of information to make a guess. Prior to making the final guess, GPT-4 demonstrates caution by reconfirming if the object is a physical toy, after receiving a “No” with “Is it a board game?”. On the contrary, the human player seems clueless and may ask repetitive or irrelevant questions.
G.2 Celebrities dataset We present a comparative analysis between GPT-4 and human players on Celebrities in Table 8. Our findings reveal that GPT-4 demonstrates a propensity for asking questions that necessitate domain-specific background knowledge (as highlighted in Table 8). Such knowledge could be tacit knowledge between LLMs, which human players may lack. With its extensive information repository, GPT-4’s inquiries can delve into highly specific topics, sometimes requiring professional expertise that human judges may not possess. This characteristic confers an additional strength and advantage to GPT-4 during gameplay. Conversely, human players tend to rely more on direct guesses, which may yield fortunate outcomes in certain scenarios, but limits their overall effectiveness when compared to GPT-4.
H Game runs and failure modes on Celebrities and Things
We provide additional gameplay sessions on Celebrities, as shown in Table 9. Similar findings can be observed as they are in the experiment section. We observed that stronger models display proficient gameplay by accurately identifying the relevant question that separates the current space, while weaker models encounter difficulties in planning and reasoning.
We also presents several other failure modes observed with the Vicuna 7b model in Table 10. These failure modes illustrate the repetitive behavior and generation of artifacts. When a less robust model is drawn towards an absorbing state of repetitive pattern, it becomes increasingly challenging for them to escape from it.

heading: I Experimental settings for Bahavior Cloning
text: We used 4x A100 Nvidia GPU for training. The learning rate was configured to be 2e-5, while the batch size was set to 32. We utilized the BF16 method for training and consistently employed deepspeed stage 2 across all our experiments. As part of the training process, we masked tokens from the turns of the J within the loss function. In all of our experiments, we continued training the model until the validation loss ceased to decrease any further.
J Generalization ability of Vicuna 7B finetuned on Things to Celebrities
A comparison of the gameplay between Vicuna 7B and the V-FT 7B (Things) is provided in Table 11. We observe that Vicuna 7B model may be more inclined to ask irrelevant questions, while the V-FT 7B (things) ask questions directly to the point. In terms of game strategy, the V-FT 7B (things) is more generative effective, indicating the planning ability obtained from imitation might be able to generalize to out of domain scenario.

heading: K Experimental settings for Reinforcement Learning from Game-Play (RLGP)
text: We adhered to the default settings of the TRLX library for our experiments, but made modifications to certain hyperparameters. Specifically, we set the rollout generation temperature to 0.8 and disabled the topK or topP sampling. The value function coefficient for the value hydra head was set to 0.05. For the
discounted reward, we set γ to 1 and λ to 0.97. The RL models were trained using four Nvidia A100 cards, with a total of 32 rollouts for each training entity. We trained the Vicuna 7B and 13B models on a dataset of 209 entities from the Things domain, which at least one of the off-the-shelf LLMs had won at least once. This is to ensure that model learn from experiences where it has some chance to receive some final rewards.
The intermediate reward was determined as follows. We utilized a linear decaying function to gradually decrease the intermediate reward from 2 to 0, based on the following equation:
Rintermediate = max(0.2− 0.025 · #Turns, 0) (2)
The rationale behind this choice is that achieving more “yes” responses in the earlier turns is considered a positive signal. We observed that this approach improved the performance of the RLGP model. If the initial intermediate reward is too high (e.g., 0.5), the model tends to generate trivial questions like “is it an entity” more frequently. Our choice is based the empirical results obtained from the development set. All RLGP models were trained for 600 iterations, ensuring that the model does not deviate significantly from the initial Vicuna model.

heading: L GPT-4 failure cases
text: We notice that the strongest model GPT-4 consistently fail on the entity “Yoga Mat”, while some other models can adeptly deduct it. We investigated this issue and it turns out that the GPT-4 model will always ask a misleading question “Is it made of plastic” in the 5 tested runs, leading to self-misinterpretation. See examples in Table 13. GPT-4 seems to possess a stereotype of plastic items and excludes “Yoga mat” after knowing this item is made of plastic. The effect of this question is more evident in Table 14. After confirming that the entity is made of plastic, the model proceeds to focus on plastic only items in its top guesses for all following turns. This observation suggests that even the most robust model, which may demonstrate backtracking when encountering a wrong path, can still become trapped in erroneous assumptions.

heading: Future Work
text: Finally, while Behavior Cloning and RLGP are shown to be useful to improve the performance of Vicuna models, future work could measure their effectiveness across a variety of open source model families