Venue: NAACL 2024

# Assessing Logical Puzzle Solving in Large Language Models:

Insights from a Minesweeper Case Study

Yinghao Li, Haorui Wang, Chao Zhang

Georgia Institute of Technology, Atlanta, USA {yinghaoli,hwang984,chaozhang}@gatech.edu

###### Abstract

Large Language Models (LLMs) have shown remarkable proficiency in language understanding and have been successfully applied to a variety of real-world tasks through task-specific fine-tuning or prompt engineering. Despite these advancements, it remains an open question whether LLMs are fundamentally capable of reasoning and planning, or if they primarily rely on recalling and synthesizing information from their training data. In our research, we introduce a novel task--Minesweeper--specifically designed in a format unfamiliar to LLMs and absent from their training datasets. This task challenges LLMs to identify the locations of mines based on numerical clues provided by adjacent opened cells. Successfully completing this task requires an understanding of each cell's state, discerning spatial relationships between the clues and mines, and strategizing actions based on logical deductions drawn from the arrangement of the cells. Our experiments, including trials with the advanced GPT-4 model, indicate that while LLMs possess the foundational abilities required for this task, they struggle to integrate these into a coherent, multi-step logical reasoning process needed to solve Minesweeper. These findings highlight the need for further research to understand the nature of reasoning capabilities in LLMs under similar circumstances, and to explore pathways towards more sophisticated AI reasoning and planning models.

+
Footnote †: Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 59–81

June 16-21, 2024 ©2024 Association for Computational Linguistics

## 1 Introduction

Large Language Models (LLMs) have made remarkable strides in the Natural Language Processing (NLP) arena, capturing the spotlight with their multifaceted capabilities. These models have been effectively utilized across a spectrum of NLP tasks, including information extraction Agrawal et al. (2022); Zhu et al. (2023b), answering arithmetic and common-sense questions Li et al. (2022); Yuan et al. (2023); Imani et al. (2023), as well as aiding in strategic planning Yao et al. (2023b); Wang et al. (2023c) or acting as game agents Callison-Burch et al. (2022); Wang et al. (2023a); Gupta (2023). With growing scale, LLMs begin to exhibit "emergent abilities" Wei et al. (2022a); Schaeffer et al. (2023), including marked improvements in their ability to follow instructions, perform multi-step reasoning and planning, and even extend to comprehending humor OpenAI (2023). Such flourishing capability has triggered excitement within the research community, cultivating the belief that LLMs could be instrumental in achieving Artificial General Intelligence Bubeck et al. (2023); Zhang et al. (2023); Tu et al. (2023). Nevertheless, a crucial question remains underexplored: _To what extent can the reasoning capabilities of LLMs go beyond the scope of their training distribution?_

This question is critical as it uncovers if LLMs are reliable for sophisticated hands-off planning in various scenarios such as biomedical experimental design Bran et al. (2023); O'Donoghue et al. (2023) and autonomous driving Sha et al. (2023), where the agents have to deal with off-distribution inputs occasionally. Unfortunately, the answer is not straightforward, and the debate is going on Huang et al. (2022); Helbling et al. (2023); Wu et al. (2023); Huang et al. (2023); Valmeekam et al. (2023); Stechly et al. (2023). The reason is that LLMs' substantial parameter counts enable them to store and recall vast amounts of information from their training material Gudibande et al. (2023); Rosenfeld et al. (2020); Kaplan et al. (2020); Brown et al. (2020), which may contain conventional reasoning datasets such as GSM8K Cobbe et al. (2021), MultiArith Roy and Roth (2015), or StrategyQA Geva et al. (2021). This could exaggerate LLMs' reasoning abilities on such benchmarks Qin et al. (2023); Deng et al. (2023), leading to false a promise when applying LLMs to practical scenarios.

Although attempts to develop new benchmarks that challenge LLMs suggest potential zero-shot

learning successes (Suzgun et al., 2023; Bao et al., 2023), there is an underlying concern that these benchmarks may not be distinct enough from the LLMs' training data, thereby giving LLMs an undue advantage and masking their true abilities (Valmeekam et al., 2022). For instance, Gudibande et al. (2023) suggest that smaller LLMs only learn specific tasks when fine-tuned with data from larger models and that these improvements do not generalize well. Drawing an analogy between smaller and larger LLMs to humans implies that LLMs may not exhibit genuine intelligence or intrinsic reasoning beyond their training distribution (Liu et al., 2023; Bubeck et al., 2023). To robustly claim that LLMs possess reasoning abilities, they must be tested under a wider array of conditions. A fundamental approach to overcoming these challenges is to decouple the test inputs from the LLMs' pre-training data, compelling them to rely on task descriptions for reasoning and planning rather than prior knowledge. This entails an environment with clear instructions, a rich but well-defined action space, and a distinct objective. Such a setup facilitates the verification of the reasoning process and qualitative performance assessment, while it also reduces the chances of successful random guessing.

Minesweeper, a well-known logic puzzle game illustrated in Figure 1, is proposed as an ideal testing ground under these considerations. The game presents a range of complexities: while its fundamental principles are straightforward and accessible to beginners, achieving proficiency requires advanced logical reasoning skills that go beyond mere pattern recognition on the game board. For LLMs, the challenge in Minesweeper lies in interpreting the states of individual cells, making use of numerical hints, and understanding the spatial interconnections between cells. This is essential for accurately determining the locations of mines and strategizing subsequent moves based on incomplete information----a process that is instinctive for human players. Despite LLMs' prior knowledge of the universally known rules of Minesweeper, the transformation of game boards from visual representations to machine-readable text has been limited, with no existing format aligning with our representation. Consequently, LLMs must rely on their intrinsic logic for problem-solving by understanding and following the rules, as pattern matching from training data is impractical.

Building upon our initial motivation, we conduct a comprehensive set of experiments with LLMs including GPT-3.5-16k, GPT-3.5-instruct, and GPT-4 (OpenAI, 2022, 2023), probing their inherent reasoning skills. We introduce the Minesweeper game in various formats to determine how different types of input influence LLM performance. Objective measures like the count and proportion of accurate moves, correctly flagged mines, and completely resolved boards were employed to gauge effectiveness. However, a critical aspect of our evaluation is the manual examination of the LLMs' reasoning processes, inspecting the validity and logical soundness of their intermediate deductions. Our findings indicate that these models generally struggle to maintain consistent logical reasoning chains. The GPT-3.5 versions, in particular, tended to repeat information from provided examples or previous conversations, often failing to adjust to the updated board layouts. While GPT-4 showed improvements in response diversity, relevance, and coherence, it still faced issues with logical inconsistencies and incomplete reasoning. Our research categorizes errors in a detailed simulated environment, which sheds light on the specific skills required for solving logical puzzles, underscoring the importance of these competencies in future LLM-focused studies. Moreover, we notice that LLMs tend to generate ad-hoc sequences of actions rather than following logical reasoning chains based on action history, a finding not commonly reported, providing novel insights into LLM operational behaviors. To support forthcoming research and comprehensive evaluation of LLMs, we have published our code and data at [https://github.com/Yinghao-Li/Minesweeper-forLLM](https://github.com/Yinghao-Li/Minesweeper-forLLM). We hope it contributes to LLMs' broader understanding and future development.

## 2 Related Works

Recent studies have explored the reasoning capabilities of LLMs, with several investigations underway to understand (Qin et al., 2023; Bang et al., 2023; Liu et al., 2023; Xu et al., 2023; bench authors, 2023; Bao et al., 2023), enhance (Wei et al., 2022b; Imani et al., 2023), and leverage (Yao et al., 2023; Sha et al., 2023) these abilities for various applications. Some critics argue that the datasets used for evaluation are overly simplistic and potentially compromised by inclusion in the training data of LLMs (OpenAI, 2023; Wei et al., 2023), casting doubt on the authenticity of these purported capabilities (Valmeekam et al., 2022; Bao et al., 2023). In response, new tasks and evaluation frameworks 

have been suggested (Valmeekam et al., 2022; Xu et al., 2023; Bao et al., 2023; Vervoort et al., 2023; Wang et al., 2023a). However, these tend to focus narrowly on specific aspects of arithmetic or commonsense reasoning, often presented in unstructured natural language that intersects with LLMs' pre-existing world knowledge. This overlap presents a challenge in fully assessing LLMs' inductive reasoning abilities. In contrast, the game of Minesweeper encompasses symbolic, arithmetic, and commonsense reasoning in a structured format lying outside the scope of LLMs' training data, and thus demands genuine reasoning skills for successful resolution.

Alone another track, the use of LLMs in gameplay has gained attention, with efforts to harness them for diverse games such as card games (Gupta, 2023; Guo et al., 2023), interactive narrative games (Callison-Burch et al., 2022; Tsai et al., 2023; Cui et al., 2023; Zhu et al., 2023a), chess (Noever et al., 2020; Stockl, 2021; Suzgun et al., 2023; Feng et al., 2023), and video games like Minecraft (Wang et al., 2023a, b). Nonetheless, we contend that such games are intrinsically distinct from Minesweeper in both structure and the level of challenge they pose to LLMs. Card and interactive narrative games largely draw on common sense and narrative understanding, with a constrained range of actions. This limitation can obscure whether LLM decisions stem from deep reasoning or instinctual response. Chess, while offering a more extensive range of possibilities, presents an abstract playfield that typically requires tailored pre-training for LLMs to grasp its rules. Minecraft poses a further challenge with its complex syntax and the need for meticulous, often biased, prompt engineering, making it an impressive display but not an accurate indicator of LLMs' functional application. Minesweeper, by comparison, offers a straightforward yet strategically rich gameplay experience with clear objectives and direct evaluation criteria, making it an exemplary model for assessing the practical reasoning capabilities of LLMs.

## 3 Minesweeper

Minesweeper, a classic logic game, has been a part of the Windows operating system since around 1990s. Though the rule is simple, it features a rich and discrete action space, a clear goal, and direct feedbacks and evaluation metrics. The objective is straightforward: players must unveil all cells on a board peppered with hidden mines without detonating any. Each cell on the board can exist in one of four possible states (Figure 0(b)):

* **Unopened** ("?"): The state of these cells is unknown; they could transition into any of the other three states.
* **Numbered** ("!" to "8"): These cells reveal the count of adjacent mines, including diagonals, providing critical clues for safe navigation.
* **Blank** (". "): These cells have no adjacent mines, often opening up larger areas of the board.
* **Flagged** ("F"): Cells suspected to contain mines are marked accordingly for safety. Notably, the choice of symbols is arbitrary and should not impact the model performance, as they serve as mere representations of concepts.

Figure 1: An example of Minesweeper on a \(9\times 9\) board containing \(10\) mines, along with its interaction format. Subfigure 0(a) displays the game’s GUI; Subfigure 0(b) shows a plain-text, table-formatted representation of the game board, enhanced with color for improved visualization; Subfigure 0(c) depicts the coordinate-based plain-text representation of the board; and Subfigure 0(d) provides a log of the player’s (in this case, the first author’s) actions, where “L”, “R”, and “M” denote left-click, right-click, and middle-click actions, respectively.

 

During gameplay, players may execute one of three actions per turn: left-click ("L") to reveal a cell's contents, right-click ("R") to flag a suspected mine, and middle-click ("M") to verify the correctness of flagged cells based on adjacent numbers. On a beginner's \(9\times 9\) board (Figure 0(a)), this results in \(243\) potential actions per round (Figure 0(d)), creating an extensive action landscape despite numerous actions being invalid (as listed in Table 3). The game provides immediate feedback for each action, with outcomes ranging from revealing or flagging cells to error messages, game endings or victories, allowing models to adjust their strategies in real-time. Victory is reached by correctly identifying and flagging all mines or revealing all non-mine cells, whereas failure occurs upon missteps such as erroneous left-click on mines or incorrect flag placements during middle-click verifications. With action space and game objective established, metrics such as the proportion of valid moves, the percentage of successfully solved games, and the average number of moves to resolve a board can be readily computed. In summary, Minesweeper serves as a less knowledge-intensive, more symbol-comprehension and basic math-reasoning benchmark, pivoting on spatial reasoning over pure information retrieval.

We employ two plain-text formats to present the game board when interacting with LLMs, aiming to understand LLMs' capabilities of comprehending each of them and to reduce the potential influence of input format on the reasoning process. The first format is **table representation**, depicted in Figure 0(b). Here, the game board is portrayed as a table with the states of each cell enclosed in LaTeX-style quotation marks (grave accent "" and apostrophe "") to differentiate them from the cell indices and separators. Rows and columns are delineated by line break ""n" and comma ""," respectively, with indices for the first row and column serving as the coordinates for the cells. This format is intuitive for human interpretation; LLMs equipped with capabilities to understand tables, as suggested by the literature Chen et al. (2023); Singha et al. (2023), should process it similarly. For comparison, we also employ **coordinate representation** (Figure 0(c)). It explicitly associates each cell's coordinates with its state in the form of a look-up table, potentially offering a more direct description for LLMs to comprehend the board's configuration.

The following sections first explore two critical skills required for Minesweeper agents (SS 4). Then, they evaluate the proficiency of LLMs in playing Minesweeper, detailing objective scores and case studies that highlight their reasoning and planning capabilities (SS 5). Each section begins with a brief introduction of the experimental setup, with comprehensive details available in appendix A.

## 4 Board Understanding

### Experiment Setup

A fundamental skill for LLMs in Minesweeper is the ability to comprehend the game board. To assess this capability, we conducted two types of straightforward experiments: **board navigation** and **neighbor counting**. In board navigation, LLMs are presented with a game board and a specific coordinate, and they are tasked to identify the corresponding cell state. For neighbor counting, LLMs are given an additional arbitrary state from the set {^?, _~!', _~!', _~?',} and are required to calculate the number of occurrences of this state (ranging from \(0\) to \(8\)) surrounding the provided coordinate. We annotated \(100\) randomly generated \(9\times 9\) boards, each containing \(10\) mines, with complete action steps (refer to Figure 0(d)). The boards were randomized at different stages of the game, either fully or partially revealed, and \(3\) sets of coordinates were selected randomly for the experiments.

### Results

The performance of GPT-3.5-16k and GPT-3.5-instruct was evaluated using both table and coordinate representations, as shown in Table 1. With the

\begin{table}
\begin{tabular}{l l c c} \hline \hline Task & Representation & 3.5-16k & 3.5-instr \\ \hline  & table & 66.7 & 53.3 \\  & – ids & 63.7 & 40.7 \\ Board & \(+\) example & 64.3 & 47.0 \\ Navigation & \(+\) verification & 66.3 & - \\ \cline{2-4}  & coordinate & 82.3 & 66.7 \\  & \(+\) example & **89.7** & 61.0 \\ \hline \hline Neighbor Counting & table & 33.3 & 16.7 \\  & coordinate & **37.3** & 30.3 \\ \hline \end{tabular}
\end{table}
Table 1: Comparative analysis of GPT-3.5 variants in board understanding tasks. The performance are quantified using exact-matching accuracy percentages. The notation “\(-\) ids” indicates the omission of indices in the table representation, while “\(+\) example” denotes the inclusion of additional examples in the prompts. The “\(+\) verification” symbol refers to the application of the self-verification technique as described in Weng et al. (2022). As GPT-3.5-instruct is not optimized for conversational contexts, the self-verification is not applicable.

 

table representation, GPT-3.5-16k demonstrated a notable deficiency in accurately identifying the cell state, failing approximately one-third of the time in board navigation tasks. GPT-3.5-instruct showed even lower performance. Attempts to enhance performance through various modifications to the boards and prompts, including removing indices, adding examples, and requesting models to revise their outputs, were unsuccessful. However, with the coordinate representation, both models exhibited improved results. This suggests that the challenge of interpreting tables might be a limiting factor for the models in subsequent gameplay tests. Nonetheless, GPT-3.5 displayed a failure rate of over \(10\%\), which is suboptimal given that the task essentially involves copying and pasting values in coordinate-state mappings. In neighbor counting tasks, both models achieved a maximum accuracy of about one-third, significantly better than random guessing. This indicates that GPT-3.5 models possess basic arithmetic and geographical planning skills, which are essential for Minesweeper. These findings suggest that the models have a moderate level of board understanding, which is sufficient to not severely impede the gameplay tests that follow.

## 5 Minesweeper Gameplay

### Experiment Setup

In our study, we utilize \(5\times 5\) Minesweeper boards containing \(4\) hidden mines, a setup less complex than the conventional beginner level. LLMs are instructed to start the game by left-clicking the center cell and then proceed with either left, right, or middle-click actions on cells they assess as beneficial for advancing gameplay. We craft and annotate \(100\) boards for GPT-3.5 models, ensuring the initial left-click would reveal at least \(10\) cells (\(40\%\) of the board). This strategy aims to streamline the length of the prompts and minimize the number of interactive rounds required. For GPT-4, we randomly subsample \(10\) boards from the pool to reduce the experiment time and expense.

Prompting techniques such as chain-of-thought reasoning Wei et al. (2022b) and few-shot inncontext learning appendix A.3) are employed to activate the LLMs' reasoning capabilities and provide sufficient information for decision-making. Each game session permits a maximum of \(10\) actions per board, concluding either when the model achieves victory, triggers a mine, or produces unrecognizable responses. Given that human annotators, on average, complete each board with an average of \(6.14\) actions, we deemed a \(10\)-action limit adequate for the model to effectively resolve most boards, contingent on its reasoning ability.

To balance the influence of prompt details and accommodate models with shorter prompt constraints, our study introduces two distinct prompting approaches: 1) **Natural Conversation** (NC) mode, and 2) **Compact History** (CH) mode, as illustrated in Figure 2. In the NC mode, each game commences with a detailed explanation of rules and examples. During each round, we prompt LLMs to execute actions along with their associated reasoning. The game system then provides feedback on these actions and updates the board in a new user message, while maintaining the complete history of the interaction. CH mode, on the other hand, condenses all relevant information and interaction history into a concise, singular prompt, eliminating the need for extensive conversational details. This approach significantly shortens the length of the conversation history. Given the turn-based nature of Minesweeper, the CH mode is effective in providing all vital information required for the model to make well-informed decisions.

### Metrics

The evaluation of our model's performance is anchored in objective metrics, which encompass: the count of valid actions, the accurate identification of mines, the number of successfully completed games, and the logical coherence of reasoning

Figure 2: Interaction prompting modes. The “Natural Conversation” mode encompasses the full interaction history, whereas the “Compact History” mode condenses the actions generated by the LLM and the game’s feedback into a succinct, unified prompt.

 

chains. A "valid" action is one that advances gameplay without activating a mine. Table 3 details a broad spectrum of negative examples. The metric for correctly flagged mines is based on the quantity of mines identified by the model that align with the actual mine locations at the game's conclusion. A game is deemed "solved" when all mines are accurately marked and no extraneous cells are flagged. To assess the reasoning chains, we initially select \(5\) game boards where each model executed the highest number of valid actions. We then perform a manual inspection to ensure these chains are logically sound and directly pertinent to the gameplay. Due to the variable action counts across different models and boards, the totals of actions and reasoning chains differ, as Table 2 illustrates.

### Results and Discussion

Table 2 showcases the performance comparisons between various GPT family models and a human benchmark, focusing on valid actions, correctly identified mines, completed games, and logical reasoning chains. In line with the findings on board comprehension, models utilizing coordinate representation exhibit superior performance across all metrics compared to table representation. This supports our previous assertion that LLMs struggle with tables containing symbols not found in their training data, preferring familiar in-domain terminology as evidenced in the studies by Chen (2023) and Singha et al. (2023).

Objective ScoresIn assessing specific gameplay metrics, we first focus on three main areas: Game Outcomes, Mine Identification, and Action Accuracy. Among the GPT-3.5 variants, 3.5-instruct demonstrates better overall performance compared to 3.5-16k in these metrics, albeit with a marginally increased failure rate, which diverges from the board understanding results. A detailed analysis of GPT-3.5-instruct's _initial_ move reveals a consistent anomaly: the model persistently selects the R(2,2) action, contrary to the given instruction of L(3,3) (refer to Table 3 for invalid actions and Listing 5, Line 83 for the specific prompt). After receiving feedback, it switches to the L(1,1) action. This action often uncovers fewer cells in our specially selected boards, usually just the chosen cell itself (as illustrated in Figure 3), which increases the probability of the next action being valid. This be 

havior could also explain the higher failure rate and lower repeatance rate observed in GPT-3.5-instruct compared to the 3.5-16k model. Furthermore, GPT-3.5-instruct tends to make conservative moves by flagging all cells on the board, which accounts for its inability to solve any board despite having a higher ratio of flagged mines. Overall, all GPT-3.5 models present a noticeable pattern of repetitive actions during board solving, persisting even with direct instructions to avoid repetition (see Listing 5, line 29). This repetitive behavior suggests a significant limitation in the models' capacity to fully understand and reason about the nuanced variations occurring on the board at each turn. As Transformer-based models designed primarily for next-token prediction Vaswani et al. (2017), their focus likely remains on static aspects of cell states, overlooking subtle changes. This attentional bias, compounded by the lack of pre-training on tasks resembling this one, often leads to the repeated execution of identical actions.

In contrast, GPT-4 exhibits a marked improvement in following instructions, achieving a zero rate of repetition while maintaining a high frequency of valid actions. Notably, it flags the most mines, despite not solving any board and recording a high percentage of game failures, which is partly attributed to its tendency for divergent actions. When both operated in NC mode, GPT-4 is more prone to generating unrecognizable actions compared to GPT-3.5-16k, as illustrated in Figure 5, even when the conversation history is within the token limit. A plausible interpretation of this behavior is that GPT-4 demonstrates heightened sensitivity to recent changes in board states, allocating more attention to these immediate alterations. Conversely, it appears to pay less regard to the more distant historical context, where the constraints governing action formats are established.

Figure 4: A case study of a “valid” action and its corresponding reasoning generated by GPT-3.5-16k for solving Minesweeper. Blue indicates logical reasoning; red and golden are illogical ones.

Figure 3: A detailed analysis of the example interactions performed by GPT-3.5-instruct. Arrows oriented to the left and right signify left and right mouse clicks, respectively. The arrow pointing upwards represents a middle-click. The majority actions are technically allowed but do not effectively advance the gameplay.

 

Reasoning ChainsThe foregoing discussion highlights that merely counting valid actions is not an adequate measure of the models' reasoning skills. Therefore, we shift our focus to the logical coherence of their reasoning chains. A review of the final section in Table 2 reveals that the GPT-3.5 variants struggle to generate coherent reasoning. For instance, in a typical case from GPT-3.5-16k, shown in Figure 4, the model correctly identifies cell ~4' at (\(2,4\)) and understands that there are four mines around, but fails to grasp the concept of "neighboring cells" or to accurately count the surrounding cell states, as indicated by the statement marked in red. Furthermore, the statement in golden font is not well-grounded, making it challenging to discern any logical connection between its first and second halves. This sentence appears more like an unsuccessful attempt to replicate our in-context example "_Given that there's just one adjacent mine, it's logical to deduce that the unopened cell at (2,3) contains the mine_" (refer to Listing 5, Line 51) rather than a display of genuine reasoning. While this example is specifically for GPT-3.5-16k, similar errors are prevalent across all GPT-3.5 outputs. It appears that despite possessing a certain degree of individual capabilities in elementary symbolic understanding, geographical reasoning, and arithmetic calculation, as suggested by the board understanding results, the GPT-3.5 models fall short in integrating these skills cohesively for action planning, resulting in a fragmented reasoning chain.

GPT-4 exhibits more encouraging results compared to its predecessors. While its achievements are modest, GPT-4 demonstrates a certain level of logical understanding in this unfamiliar task, suggesting an ability to grasp the game's rules and dynamics beyond mere replication. To explore the extent and origin of this reasoning, we delve deeper into GPT-4's reasoning chains, analyzing a representative case in Figure 5. It shows that GPT-4 can logically reason if the number of neighboring un

Figure 5: This figure presents a case study of the reasoning sequences formulated by GPT-4 with coordinate board representation during action planning. The GUI on the left shows the board states and actions taken by the agent during the game. Elements highlighted in blue represent accurate facts and logical inferences as assessed by human evaluation, while those marked in red indicate incorrect observations or illogical conclusions. Notably, the final generated action “F(\(3,1\))” deviates from the permissible action formats, resulting in the termination of the game.

 

opened cells equals the remaining unflagged mines near a numbered cell, specifically in simple scenarios like '1'and '2', and particularly around the board's corners (_e.g._, \((1,2)\) or \((5,4)\)). Yet, even in such basic situations, GPT-4 shows around \(50\%\) fails. The model struggles particularly with configurations where exactly three unopened cells surround a '3'cell and other complex setups, as seen in the lower rows of the example. To determine if GPT-4's ability stems from the rules or the examples, we conducted a control experiment by omitting Example 1 from the initial prompt (Listing 5, Line 43), which closely mirrors the successful scenarios and is the sole example of left-clicking. This led to a dramatic \(90\%\) reduction in coherent reasoning with the coordinate board representation, indicating the significant role of examples in GPT-4's reasoning process, likely by providing a familiar context for reference and imitation.

Another notable point from Figure 5 is GPT-4's non-linear action sequence. For a human player, following the initial action of R(1,2), the logical next step would be to mark the adjacent unopened cell as a mine, _i.e._, R(1,1). However, GPT-4 instead opts for R(5,1), an irrational choice that strays from the focus of the previous action, before returning to \((1,1)\). This behavior suggests that GPT-4 might not fully consider its historical actions and lacks the capability for long-term, multi-step planning, a crucial aspect in evaluating an agent's intrinsic reasoning abilities as opposed to mere summarization capabilities.

It is important to note that the results discussed were derived from experiments conducted on \(5\times 5\) boards, and it is reasonable to anticipate that performance may further deteriorate on larger boards with more flexible mine arrangements.

## 6 Conclusion

In this study, we assessed the capabilities of LLMs in solving logical puzzles by examining their performance in the game Minesweeper, a task for which they have received limited exposure during the pre-training phase. This approach aims to assess the intrinsic reasoning capabilities of LLMs. Our experiments reveal that GPT-3.5 models exhibit basic abilities in spatial navigation, symbol recognition, and counting. However, they struggle with multi-step planning and generating coherent reasoning chains. GPT-4 shows improvement in simpler scenarios with shorter logic chains but still encounters issues with hallucinations and inconsistent context awareness. We find that GPT-4 can learn from examples and apply this knowledge to similar situations but is less adept at deducing underlying rules and applying them to novel scenarios, unlike GPT-3.5 models which tend to replicate examples. Our conclusion seems to be in correlation with Yadlowsky et al. (2023): LLMs do not produce new knowledge (a situation where logical reasoning is necessary); they are just (good) handlers of existing experience.

We do not intend to diminish the significance of LLMs or their transformative impact on natural language processing. Rather, we believe it is premature to consider LLMs as an intelligent object and a threat to human society. They resemble a comprehensive dictionary, useful for reference but lacking in comprehension. We hope that our work can inspire future research on this topic, and we will continue exploring the capabilities of LLMs.

### Limitations

In our study, we employed two board representation methods: table and coordinate representations. Our results indicate that coordinate representation is more effective for LLM comprehension and offers a wider range for evaluating the model's reasoning abilities. However, we did not explore other board representation methods, such as different table formats (_e.g._, HTML, Markdown) or image-based approaches. According to Singha et al. (2023), alternative table formats may not significantly differ in effectiveness from our chosen method. Still, it's possible that other representations could be more suitable for our specific context. Moreover, our initial attempts to use an imagened Web interface with board GUI were not successful. Therefore, we focus on text-based representations in this study. While advanced prompting methods like Program of Thoughts (PoT, Chen et al., 2022) or Tree of Thoughts (ToT, Yao et al., 2023) may boost performance, their use could diverrt from assessing the true reasoning abilities of LLMs and their applicability in simplifying complex issues without extensive external input. Hence, we deferred the application of these sophisticated techniques to future studies.

Another area for exploration is fine-tuning LLMs like Llame-2-70b (Touvron et al., 2023) with specific reasoning capabilities on our dataset labeled with human actions. The idea is that if a fine-tuned 

model outperforms the original or GPT-4 when the fine-tuning and testing data match in terms of symbol usage, board size, and representation, and does not improve with variations in these factors, we can infer that the "reasoning capability" is tied to the training data and not broadly generalizable. However, due to constraints in computational resources, we haven't conducted these experiments, leaving them for future research.

At last, we would like to emphasize that we can easily extend the Minesweeper dataset to a scale of \({n_{\mathrm{row}}\times n_{\mathrm{column}}\choose n_{\mathrm{mine}}}\) with zero effort. Nonetheless, we keep a relatively small scale of test cases as it is sufficient to support our conclusion. With the future development of LLMs, a larger evaluation set might be desired for a more robust and unbiased objective metric-based evaluation.

## Acknowledgments

This work was supported in part by NSF IIS-2008334, IIS-2144338, and ONR MURI N00014-17-1-2656.

## References

* Agerwal et al. [2022] Monica Agrawal, Stefan Hegselmann, Hunter Lang, Yoon Kim, and David Sontag. 2022. Large language models are few-shot clinical information extractors. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 1998-2022, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Bang et al. [2023] Yejin Bang, Samuel Cahyawijaya, Nayeon Lee, Wenliang Dai, Dan Su, Bryan Wilie, Holy Lovenia, Ziwei Ji, Tiezheng Yu, Willy Chung, Quyet V. Do, Yan Xu, and Pascale Fung. 2023. A multitask, multilingual, multimodal evaluation of chatgpt on reasoning, hallucination, and interactivity.
* Bao et al. [2023] Qiming Bao, Gael Gendron, Alex Yuxuan Peng, Wanjun Zhong, Neset Tan, Yang Chen, Michael Witbrock, and Jiamou Liu. 2023. A systematic evaluation of large language models on out-of-distribution logical reasoning tasks. _CoRR_, abs/2310.09430.
* bench authors [2023] BIG bench authors. 2023. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. _Transactions on Machine Learning Research_.
* Bran et al. [2023] Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. 2023. Chemcrow: Augmenting large-language models with chemistry tools.
* Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Grectchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesch, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_.
* Bubeck et al. [2023] Sebastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott M. Lundberg, Harshu Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4. _CoRR_, abs/2303.12712.
* Callison-Burch et al. [2022] Chris Callison-Burch, Gaurav Singh Tomar, Lara Jean Martin, Daphne Ippolito, Suma Balis, and David Reitter. 2022. Dungeons and dragons as a dialog challenge for artificial intelligence. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022_, pages 9379-9393. Association for Computational Linguistics.
* Chen [2023]Wenhu Chen. 2023. Large language models are few(1)-shot table reasoners. In _Findings of the Association for Computational Linguistics: EACL 2023_, pages 1120-1130, Dubrovnik, Croatia. Association for Computational Linguistics.
* Chen et al. [2022] Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. _CoRR_, abs/2211.12588.
* Cobbe et al. [2021] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reitichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. _CoRR_, abs/2110.14168.
* Cui et al. [2023] Christopher Cui, Xiangyu Peng, and Mark O. Riedl. 2023. Thespian: Multi-character text role-playing game agents. _CoRR_, abs/2308.01872.
* Deng et al. [2023] Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Gerstein, and Arman Cohan. 2023. Investigating data contamination in modern benchmarks for large language models. _CoRR_, abs/2311.09783.
* Feng et al. [2023] Xidong Feng, Yicheng Luo, Ziyan Wang, Hongrui Tang, Mengyue Yang, Kun Shao, David Mguni, Yali Du, and Jun Wang. 2023. Chessgpt: Bridging policy learning and language modeling. _CoRR_, abs/2306.09200 

* Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant. 2021. Did aristothe use a laptop? A question answering benchmark with implicit reasoning strategies. _Trans. Assoc. Comput. Linguistics_, 9:346-361.
* Guibande et al. (2023) Arnav Guibande, Eric Wallace, Charlie Snell, Xinyang Geng, Hao Liu, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023. The false promise of imitating proprietary llms. _CoRR_, abs/2305.15717.
* Guo et al. (2023) Jiaxian Guo, Bo Yang, Paul Yoo, Bill Yuchen Lin, Yusuke Iwasawa, and Yutaka Matsuo. 2023. Suspicion-agent: Playing imperfect information games with theory of mind aware gpt-4.
* A pre-flop analysis. _CoRR_, abs/2308.12466.
* Helbling et al. (2023) Alec Helbling, Mansi Phute, Matthew Hull, and Duen Horng Chau. 2023. LLM self defense: By self examination, llms know they are being tricked. _CoRR_, abs/2308.07308.
* Huang et al. (2022) Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. 2022. Large language models can self-improve. _CoRR_, abs/2210.11610.
* Huang et al. (2023) Jie Huang, Xinyun Chen, Swaroop Mishra, Huaxiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. 2023. Large language models cannot self-correct reasoning yet.
* Imani et al. (2023) Shima Imani, Liang Du, and Harsh Shrivastava. 2023. Mathpromter: Mathematical reasoning using large language models. In _Proceedings of the The 61st Annual Meeting of the Association for Computational Linguistics: Industry Track, ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 37-42. Association for Computational Linguistics.
* Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. _CoRR_, abs/2001.08361.
* Li et al. (2022) Xiang Lorraine Li, Adhiguna Kuncoro, Jordan Hoffmann, Cyprian de Masson d'Autume, Phil Blunsom, and Aida Nematzadeh. 2022. A systematic investigation of commonsense knowledge in large language models. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhadi, United Arab Emirates, December 7-11, 2022_, pages 11838-11855. Association for Computational Linguistics.
* Liu et al. (2023) Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, and Yue Zhang. 2023. Evaluating the logical reasoning ability of chatgpt and GPT-4. _CoRR_, abs/2304.03439.
* Noever et al. (2020) David Noever, Matthew Ciolino, and Josh Kalin. 2020. The chess transformer: Mastering play using generative language models. _CoRR_, abs/2008.04057.
* O'Donoghue et al. (2023) Odhran O'Donoghue, Aleksandar Shatefritski, John Ginser, Ralph Abboud, Ali Essa Ghareeb, Justin Booth, and Samuel G. Rodrigues. 2023. Bioplanner: Automatic evaluation of llvms on protocol planning in biology. _CoRR_, abs/2310.10632.
* OpenAI (2022) OpenAI. 2022. Introducing ChatGPT. (Accessed on Jun 18, 2023).
* OpenAI (2023) OpenAI. 2023. GPT-4 technical report. _CoRR_, abs/2303.08774.
* Qin et al. (2023) Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? _CoRR_, abs/2302.06476.
* Rosenfeld et al. (2020) Jonathan S. Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. 2020. A constructive prediction of the generalization error across scales. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Roy and Roth (2015) Subbro Roy and Dan Roth. 2015. Solving general arithmetic word problems. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 1743-1752, Lisbon, Portugal. Association for Computational Linguistics.
* Schaeffer et al. (2023) Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023. Are emergent abilities of large language models a mirage? _CoRR_, abs/2304.15004.
* Sha et al. (2023) Hao Sha, Yao Mu, Yuxuan Jiang, Li Chen, Chenfeng Xu, Ping Luo, Shengbo Eben Li, Masayoshi Tomizuka, Wei Zhan, and Mingyu Ding. 2023. Langaugempc: Large language models as decision makers for autonomous driving. _CoRR_, abs/2310.03026.
* Singha et al. (2023) Ananya Singha, Jose Cambronero, Sumit Gulwani, Vu Le, and Chris Parmin. 2023. Tabular representation, noisy operators, and impacts on table structure understanding tasks in llms. _CoRR_, abs/2310.10358.
* Stechly et al. (2023) Kaya Stechly, Matthew Marquez, and Subbarao Kambhampati. 2023. GPT-4 doesn't know it's wrong: An analysis of iterative prompting for reasoning problems. _CoRR_, abs/2310.12397.
* Stockl (2021) Andreas Stockl. 2021. Watching a language model learning chess. In _Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), Held Online, 1-3 September, 2021_, pages 1369-1379. INCOMA Ltd.
* Suzgun et al. (2023) Mirac Suzgun, Nathan Scales, Nathanaael Scharli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoco V. Le, Ed Chi, Denny Zhou, and Jason Wei. 2023. Challenging big-bench tasks and whether chain-of-thought can solve them. In _Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023_, pages 13003-13051. Association for Computational Linguistics 

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiyal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Klounmann, Artem Korenev, Punit Singh Koruz, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, DiamisLiskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Mobbyog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binth Tang, Ross Taylor, Adina Williams, Jian Xiang Kaun, Pavin Xu, Zheng Yan, Hiyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharn Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. _CoRR_, abs/2307.09288.
* Tsai et al. (2023) Chen-Feng Tsai, Xiaochen Zhou, Sierra S. Liu, Jing Li, Mo Yu, and Hongyuan Mei. 2023. Can large language models play text games well? current state-of-the-art and open questions. _CoRR_, abs/2304.02868.
* Tu et al. (2023) Tao Tu, Shekoofeh Azizi, Danny Driess, Mike Schaekermann, Mohamed Amin, Pi-Chuan Chang, Andrew Carroll, Chuck Lau, Ryutaro Tanno, Ira Ketena, Basil Mustafa, Akankssha Chowdhery, Yun Liu, Simon Kornblith, David J. Fleet, Philip Andrew Mansfield, Susant Parkash, Rene Wong, Sunny Virmani, Christopher Semturs, Sara Mahdavi, Bradley Green, Ewa Dominowska, Blaise Aguera y Arcas, Joelle K. Barral, Dale R. Webster, Gregory S. Corrado, Yossi Matias, Karan Singhal, Pete Florence, Alan Karthikesalingam, and Vivek Natarajan. 2023. Towards generalist biomedical AI. _CoRR_, abs/2307.14334.
* Valmeekam et al. (2022) Karthik Valmeekam, Alberto Olmo Hernandez, Sarath Sreedharan, and Subbarao Kambhampati. 2022. Large language models still can't plan (A benchmark for llms on planning and reasoning about change). _CoRR_, abs/2206.10498.
* Valmeekam et al. (2023) Karthik Valmeekam, Matthew Marquez, and Subbarao Kambhampati. 2023. Can large language models really improve by self-critiguing their own plans? _CoRR_, abs/2310.08118.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 5998-6008.
* Vervoort et al. (2023) Louis Vervoort, Vitaliy Mizyakov, and Anastasia Ugleva. 2023. A criterion for artificial general intelligence: hypothetic-deductive reasoning, tested on chatgpt.
* Wang et al. (2023a) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlefx, Chaowei Xiao, Yukue Zhu, Linxi Fan, and Anima Anandkumar. 2023a. Voyager: An open-ended embodied agent with large language models. _CoRR_, abs/2305.16291.
* Wang et al. (2023b) Zihao Wang, Shaofei Cai, Guanzhou Chen, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023b. Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents.
* Wang et al. (2023c) Zihao Wang, Shaofei Cai, Anji Liu, Xiaojian Ma, and Yitao Liang. 2023c. Describe, explain, plan and select: Interactive planning with leg enables open-world multi-task agents. _CoRR_, abs/2302.01560.
* Wei et al. (2022a) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022a. Emergent abilities of large language models. _Trans. Mach. Learn. Res._, 2022.
* Wei et al. (2022b) Jason Wei, XueZhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2022b. Chain-of-thought prompting elicits reasoning in large language models. In _NeurlPS_.
* Wei et al. (2023) Tianwen Wei, Liang Zhao, Lichang Zhang, Bo Zhu, Lijie Wang, Haihua Yang, Biye Li, Cheng Cheng, Weiwei Lu, Rui Hu, Chenxia Li, Liu Yang, Xilin Luo, Xuejie Wu, Lunan Liu, Wenjun Cheng, Peng Cheng, Jianhao Zhang, Xiaoyu Zhang, Lei Lin, Xiaokun Wang, Yutuan Ma, Chuahai Dong, Yanqi Sun, Yifu Chen, Yongyi Peng, Xiaojuan Liang, Shuicheng Yan, Han Fang, and Yahui Zhou. 2023. Skywork: A more open bilingual foundation model. _CoRR_, abs/2310.19341.
* Weng et al. (2022)Yixuan Weng, Minjun Zhu, Shizhu He, Kang Liu, and Jun Zhao. 2022. Large language models are reasoners with self-verification. _CoRR_, abs/2212.09561.
* Wu et al. (2023) Yue Wu, Shrimai Prabhumoye, So Yeon Min, Yonatan Bisk, Ruslan Salakhutdinov, Amos Azaria, Tom M. Mitchell, and Yuanzhi Li. 2023. SPRING: GPT-4 out-performs RL algorithms by studying papers and reasoning. _CoRR_, abs/2305.15486.
* Xu et al. (2023) Fangzhi Xu, Qika Lin, Jiawei Han, Tianzhhe Zhao, Jun Liu, and Erik Cambria. 2023. Are large language models really good logical reasoners? a comprehensive evaluation and beyond.
* Yadlowsky et al. (2023) Steve Yadlowsky, Lyric Doshi, and Nilesh Tripuraneni. 2023. Pretraining data mixtures enable narrow model selection capabilities in transformer models. _CoRR_, abs/2311.00871.
* Yao et al. (2023a) Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narayanhan. 2023a. Tree of thoughts: Deliberate problem solving 

with large language models. In _Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023_.
* Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023b. React: Synergistic reasoning and acting in language models. In _The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023_. OpenReview.net.
* Yuan et al. (2023) Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023. How well do large language models perform in arithmetic tasks? _CoRR_, abs/2304.02015.
* Zhang et al. (2023) Chaoning Zhang, Chenshuang Zhang, Chenghao Li, Yu Qiao, Sheng Zheng, Sumit Kumar Dam, Mengchun Zhang, Jung Uk Kim, Seong Tae Kim, Jinwoo Choi, et al. 2023. One small step for generative ai, one giant leap for agi: A complete survey on chatgpt in aige era. _arXiv preprint arXiv:2304.06488_.
* Zhu et al. (2023a) Andrew Zhu, Lara J. Martin, Andrew Head, and Chris Callison-Burch. 2023a. Calypso: Llms as dungeon masters' assistants.
* Zhu et al. (2023b) Yutao Zhu, Huaying Yuan, Shuting Wang, Jongnan Liu, Wenhan Liu, Chenlong Deng, Zhicheng Dou, and JiRong Wen. 2023b. Large language models for information retrieval: A survey. _CoRR_, abs/2308.07107.

 

Experiment Setup

### GPT Versions

In this paper, we discuss three distinct versions of the GPT model: GPT-3.5-16k, GPT-3.5-instruct, and GPT-4. The term GPT-3.5-16k is used to denote the "gpt-3.5-turbo-16k" model as listed on the OpenAI API model webpage, specifically the checkpointed 0613.1 The GPT-3.5-instruct model refers to the "gpt-3.5-turbo-instruct" variant, with the checkpoint version dated 2023-09-15. Lastly, GPT-4 represents the "gpt-4" model, utilizing also the 0613 checkpoint version. All mentioned checkpoints are hosted on Microsoft Azure,2. The model temperature is set to \(0\) in all cases.

Footnote 1: [https://platform.openai.com/docs/models](https://platform.openai.com/docs/models)

Footnote 2: *.openai.azure.com

### Board Understanding

In our board understanding experiments, we initially generate \(1{,}000\) randomly arranged boards, each featuring a \(9\times 9\) grid with \(10\) hidden mines. Notably, the central cells, positioned at (5,5), are always mine-free. From this pool, we select \(100\) boards, ensuring that the initial click at the center \(\texttt{L}(5,5)\) reveals at least \(10\) cells. Subsequently, we conduct gameplay using the graphical user interface depicted in Figure 0(a). The actions of the annotators, who are average players in Minesweeper and unaware of the board's layout beyond the initial instruction to click the center, are recorded as shown in Figure 0(d). Although not all trials are successful---with about \(90\%\) boards being solved and the remainder failing due to human errors or 50-50 guesses---the entire action history, including unsuccessful attempts, is preserved for analysis.

For each board, we randomly choose an action (excluding the first and last from the history) and associate board state as the focus. We then select \(3\) random coordinates per board, setting two challenges for LLMs: to determine the status of each selected cell (board navigation), and to count the number of specific symbols within {^?,^,_',^,_'1', '2'} in their neighbors, including diagonally adjacent ones (neighbor counting). This process yields \(300\) test cases for each task.

As outlined in SS 4, we employ two distinct board representations: table and coordinate. Each required a slightly different approach in prompting. Specifically, the table representation uses LaSXstyle quotation marks, while the coordinate representation does not use quotations for board cell states and employs regular double quotes in the necessary positions in the game description. Examples illustrating these differences are provided below.

```
1Youwillbepresentedwith=9by$boardfortheMinesweepeargame.Theboardswrappedina1bly1table,wherethefirstrowandthefirstcolumnwindwnbersinddoublequotationmarksaretherowandcolumnindices.Acoordinate(x,y)representsthecellatex-xthrowandytchulum,wherexandystartingfrom1,aretherowandcolumnindices,respectively.Thestateofeachcellisrepresentedbythefollowingsymbols;
2...
3..
4..
5..
7..
8..
9..
10..
11..
12..
13..
14..
15__Perepresentsnumberedcellswiththatnumberofminesintheadjacentcells.
16....
17..
18..
19..
20..
21..
22..
23..
24..
25..
26..
27..
28..
29..
30..
31..
32..
33..
34..
35..
36..
37..
38..
39..
40..
41..
42..
43..
44..
45..
46..
47..
48..
49..
50..
51..
52..
53..
54..
55..
56..
57..
58..
59..
60..
61..
62..
63..
64..
65..
66..
67..
68..
69..
70..
71..
72..
73..
74..
75..
76..
77..
78..
79..
80..
81..
82..
83..
84..
85..
86..
87..
88..
89..
90..
91..
92..
93..
94..
95..
96..
97..
97..
98..
99..
99..
90..
91..
91..
92..
93..
94..
95..
96..
97..
97..
98..
 

12(4,4):7
13(2,1):7
14(2,2):7
15(2,3):7
16(2,4):7
17(3,1):7
18(3,2):7
19(3,3):7
20(3,4):1
21(4,1):7
22(4,2):7
23(4,3):7
24(4,4):1
25(4,4):1
26QUESTION: What is the cell at coordinate (3,4)?
27ANSWER: "1""
28
29--- END OF EXAMPLES --
30
*11--- CURRENT BOARD --
*12(1,1):7
31(2,2):7
4(1,3):7
*15(1,4):7
46(1,5):1
47(1,6):-
48(1,7):.
49(1,8):-
41(2,1):7
40(2,3):7
41(2,4):7
42(5,5):1
46(2,6):-
47(2,7):1
48(2,8):1
49(2,9):1
40(3,1):7
41(3,2):1
42(3,3):1
43(3,4):1
44(3,5):1
45(3,6):-
46(3,7):1
47(3,8):f

[MISSING_PAGE_POST] 

* 43QUESTION: How many cells '!" are neighbors (including diagonal) of the cell with coordinate (2,1)?
* 44Let's thinkstepbystep.
* 45ANSWER:

```
1Youwillbepresentedwith$9by$9boardfortheMinesweepergame,whichisdevidedintocells.Thecellsarepresentedascoordinatestate'ampapings.Acoordinate(x,y)representstheelementatsthexthrowandys-thcolumn,wherexandydy,startingfrom1,arewandandcolumontrollesides,respectively.Thestateofeachcellisrepresentedbyfollowingsymbols:2=""persepetsabankcall.3="1tto"8rrepresentsnumberedcellswiththatnumberofminesinadjacentcells.4+"Fergementsaflaggedell.5+"7"representsanunopenedcell.6{}%--EXAMPLES---PartialBOARD---9{1,1}:F=0{1,2}:7|11[1,3]:7|12[1,4]:f|13[2,1]:7|14[2,2]:7|15[2,3]:7|16[2,4]:7|17[3,3]:7|18[3,2]:f|19[3,3]:7|20[3,4]:1|21[4,1]:7|22[4,2]:7|23[4,2]:7|24[4,4]:1|25[5]
26QUESTION:Howmanycells''F\areneighboring(includingagonally)thecellwithcoordinates(2,1)?
27Let'sthinkstepbystep2ANSWER:
28
29Tofindouthowmanycellsinthevalue"1"areneighborsofthereshellwithcoordinate(2,1),weneedtolookatthe8neighboringcellsof(2,1).Thesecoordinatesare:31{1,1},{1,2},{3,1},{3,2},{2,2}.32Now,weillcheckthevaluesofthesecellsonthegivenMinesweeperboard:31
33[1,1]=F3[6,2]=?3[7,3]=?3[8,2]=Frf3[9,2]=?40
41Fromoftheseneighboringcells(1,1)and(3,2)are"F\({}^{*}\)So,thenumberofcellswith"1"thatareneighborsofthemcell(2,1)is:42ANSWER:2.44
45....ENDOFEXAMPLES---46
47==CURRENTBOARD---8
48{1,1}:7|39{1,2}:7|50{1,3}:7|51[1,4]:7|52{1,5}:1|53{1,6}:.54|55{1,7}:.55|56{1,9}:.57|57{2,1}:7|58{2,2}:7|59{2,3}:7|60{2,4}:7|61{2,5}:1|62{2,6}:.5|63{2,7}:1|64{2,8}:1|65{2,9}:1|66{3,1}:2|66{3,2}:1|66{3,3}:1|66{3,3}:1|68{3,3}:1|69{3,4}:1|70{4,5}:.5|60{4,6}:.5|61{4,7}:1|71{4,8}:1|72{4,8}:1|83{4,9}:1|84{5,1}:2|55{5,2}:2|66{5,3}:1|73{5,4}:1|85{5,5}:.5|95{5,6}:.5|95{5,7}:.5|95{5,7}:.5|91{5,8}:.5|92{5,5}:.5|93{5,8}:.5|93{5,8}:.5|94{5,9}:1|94{5,9}:2|67{6,3}:2|95{6,3}:2|96{6,4}:1|97{6,5}:1|98{6,6}:2|99{6,7}:3|100{6,8}:2|101{6,8}:2|102{6,9}:1|103{6,8}:2|104{6,9}:1|104{6,9}:2|105{7,1}:2|106{7,1}:2|106{7,1}:2|107{7,1}:2|107{7,1}:2|109{7,1}:2|109{7,1}:2|110{7,1}:2|111{8,1}:2|12{8,2}:2|13{8,3}:2|14{8,4}:2|15{8,5}:7|16{8,6}:2|177{8,7}:2|118{8,8}:2|191{91,8}:2|119{92,8}:2|120{9,1}:2|121{93,8}:2|122{9,9}:2|13{9,4}:7|124{9,5}:2|125{9,6}:2|126{9,7}:2|127{9,8}:2|128{9,9}:2|130QUESTION:Howmanycells~1~areneighbors(includingdiagonal)ofthecellwithcoordinate(2,1)?
131Let'sthinkstepbystep.
132ANSWER:
133
134
135
136
137
138
139
140
141 Fromoftheseneighboringcells(1,1)and(3,2)are"F\({}^{*}\)So,thenumberofcellswith"1"thatareneighborsofthemcell(2,1)is:42ANSWER:2.44
142
143ANSWER:
144
145....ENDOFEXAMPLES---46
147....CURRENTBOARD---8
151{1,1}:7|39{1,2}:7|50{1,3}:7|51{1,4}:7|52{1,5}:1|53{1,6}:.5|54{1,7}:.5|55{1,8}:.5|55{1,8}:.5|56{1,9}:.5|57{2,10}:.5|57{2,11}:2|11{8,1}:2|123{8,1}:2|13{8,1}:2|14{9,1}:2|15{1,1}:2|16{1,1}:2|17{1,1}:2|18{1,1}:2|19{1,1}:2|20{1,1}:2|21{1,1}:2|22{1,1}:2|23{8,1}:2|24{9,1}:2|25{1,1}:2|26{1,1}:2|27{1,1}:2|28{1,1}:2|29{1,1}:2 

In contrast, the board navigation task is straightforward, involving a direct query about a specified target cell. We have observed that incorporating COT into the neighbor counting process notably enhances its performance.

Furthermore, we have made the inclusion of in-context examples optional for the board navigation task, but mandatory for neighbor counting. The need for examples in neighbor counting arises from the models' propensity to generate unclear outputs in this task. By providing an example, we aim to guide the models towards producing more interpretable responses.

### Minesweeper

The detailed setup of the Minesweeper experiment is presented in SS 5. We hereby provide the actual initial prompts with table-formatted board representation used in our experiments. The coordinate representation closely mirrors the variations found in Listing 3 and Listing 4 and will not be omitted.

```
1>>SYSTEM:
2Youareahelpulassistantwhoisgoodatplayingsuotationmarkstoindicaterowandcolumnindices.Cellshavemultiplepossiblestates?
37Unopenedcells(representedly'!',whichcovertheboardattoftsthefangemeanalshapeprovenvingflags)
88Numberclts(representedly'lto's,whichindicateshumberofminestheeightneighborecells,includingthosediagonallyadjacent)
99Blankcells(representedly'.',whichhavenoneighbororingmines)
10L>Flaggedcells(representedly'f',whicharcmarkedbytheplayertoindicatepotentialiminclocation)
11
12Al playerselectsacelltopopenit.Ifaplayeropensocallcontaininganime,thengamendsinalossOtherwise,themopedcelldisplayedeithernumber,indicatingthunotheroffamesdignonallyand/oradjacenttotintorablantetile(sometimesshownasab),andalladjacentcellswillautomaticallydepend.TowinagameofMinesweeper,allnon-ninecellsmustbecomedwithoutopeningane.insue 

* ENDOFEXAMPLES
* >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >> >>  >> >> >> >> >> >> >> >> >>  >> >> >> >> >> >>  >> >> >> >> >> >>  >> >> >> >> >>  >> >> >> >> >>  >> >> >> >> >>  >> >> >> >>  >> >> >>  >> >> >> >>  >> >> >>  >> >> >> >>  >> >> >>  >> >> >>  >> >> >>  >> >> >>  >> >> >>  >> >>  >> >> >>  >> >>  >> >> >>  >> >>  >> >>  >> >> >>  >> >>  >> >>  >> >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >>  >> >>  >> >>  >> >>  >>  >> >>  >> >>  >> >>  >>  >> >>  >>  >> >>  >> >>  >> >>  >> >>  >>  >> >>  >> >>  >>  >> >>  >> >>  >>  >> >>  >>  >> >>  >> >>  >>  >> >>  >> >>  >>  >> >>  >>  >> >>  >> >>  >>  >> >>  >>  >> >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >>  >> >>  >> >>  >>  >> >>  >>  >> >>  >> >>  >>  >> >>  >>  >> >>  >> >>  >>  >> >>  >> >>  >> >>  >>  >> >>  >> >>  >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >>  >> >> >>  >> >>  >> >>  >> >> >>  >> >>  >> >> >>  >> >>  >> >> >>  >> >> >>  >> >> >>  >> >> >>  >> >> >>  >> >> >> >>  >> >> >>  >> >> >> >>  >> >> >> >> >>  >> >> >> >>  >> >> >> >> >> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>028573496188888888888888888888888888888888888888888888888888888888888888888888888888888882882828285828282828238238238232323232323232323 

actions within game description and action feedbacks with abstract symbols (_e.g.,_ "L/R/M"). In the other set of experiments, Arabic numerals that indicate the numbers of surrounding mines (_e.g.,_1,2,...,8) were converted to their Roman counterparts (_e.g.,_I, II, \(\cdots\), VIII), with other elements remaining unchanged. The outcomes are summarized in Table 4. They show that these variations have minimal effect on model performance, indicating that these factors do not significantly hinder the model's reasoning abilities. Our findings suggest that the general exposure to Minesweeper game rules across the internet does not provide a distinct advantage for GPT models in solving these puzzles, and our assumption of "Minesweeper being a new challenge for GPTs" holds.

## Appendix C Interaction Examples

In this section, we provide more case studies on the reasoning chains generated by LLMs as support of our discussion in SS 5. **For better readability, we convert all coordinate representations to table representations.** Please refer to our GitHub repository for a complete set of experiment results.

```
1//InitialInstrutionsareOmitted
2>>>ASSISTANT:
3#REASONING:Sinetheboardiscompletelycoveredwithunopenedcells,thereisnospecificinformationavailabletomakeaninformeddecisionaboutwhichcelltoopenfirst.Therefore,Iwillstartbyrandomlyselectingcalltoconopen.
6ACTION:L{3,3}
7
8>>USE:
9---CURRENTBOARD....
10*!","1-"2-"3","4-",5"
12"","2-","2-","2-","?" 

62 >>USER:
63Your previous action "L(5,5)" is invalid. Error Message:
64Invalid action: cannot left-click a blank cell. Left -click is only for unopened cells ('?').
65Please follow the instructions and try again.
66
67... <URRENT BOARD ---
68...
69"=","1","2","3","4","5"  70","1","2","2","1","2","1","2","1","2","2","2","2","2","2","3","1","1","1","1","1","1","1","a","4","5",".","1 

* You do not duplicate actions.
* You submit only one action at a time.
* ReasoningING:
* Action:
* >> ASSSISTANT:
* BEASONING: Apologies for the repeated mistakes. Since the cell at (5,1) is already opened and blank, we need to look for another cell to open . Based on the current information, the cell at (4,4) is surrounded by one mine. However, the cell at C=[] at (5,8) is surrounded by two mines. Therefore, it is safer toopen the cell at (5,8) next.
* Action: L(5,0)
* Action: L(5,0)
* >> USER:
* Your previous action "L(5,0)" is invalid. Error Message:
* Invalid Coordinates! Please make sure your coordinate are within [1, 5] for rows and [1, 5] for columns.
* Please follow the instructions and try again.
* 21
* 22
* CURRENT BOARD --
* 33
* 34
* 35
* 36
* 37
* 38
* 39
* 31
* 32
* 33
* 34
* 35
* 36
* 37
* 38
* 39
* 37
* 39
* 38
* 39
* 31
* 39
* 30
* 31
* 32
* 33
* 34
* 35
* 36
* 37
* 38
* 39
* 39
* 31
* 32
* 33
* 34
* 35
* 36
* 37
* 38
* 39
* 39
* 31
* 33
* 34
* 35
* 36
* 37
* 38
* 39
* 39
* 31
* 32
* 33
* 34
* 35
* 36
* 37
* 38
* 39
* 39
* 39
* 31
* 32
* 33
* 34
* 35
* 36
* 37
* 38
 

* 6B","1".2".3".4".5";
* 71".7".1".7".1".7';
* 82".7".2".2".1".7".7";
* 93".7".1".1".1".1".7";
* 10".4".3".1".1".1".1".7';
* 11".5".".7".7".7".7".7";
* 12".
* 13
* 14please ensure:
* You do not duplicate actions.
* 16- You submit only one action at a time.
* 17
* REASONING:
* ACTION:
* 20
* 21>> ASSISTANT:
* 22REASONG:
* The cell at (2,2) indicates there are 2 mines in its neighboring cells. Since (1,1), (2,1), and (3,1) are the only unopened cells around it, we can deduce that two of these cells contain mines. Similarly, the cell at (3,2) also indicates there are 2 mines in its neighboring cells. Considering that (3,1), (4,1), and (5,1) are the only unopened cells around it, and we know that two of (1,1), (2,1), and (3,1) contain mines, we can conclude that (4,1) and (5,1) are safe to open.
* 24ACTION: L{4,1}
* 25
* 26
* 27>> UERR:
* 28... CURRENT BOARD --
* 29...".1".2".3".4".5";
* 30...".1".7".1".7";
* 31".7".2".2".1".7";
* 32".7".2".1".7";
* 33".7".2".1".7";
* 34".1".1".1".7";
* 35".7".7".7".7".7";
* 36
* 37
* 38plesensure:
* You do not duplicate actions.
* 39<- You submit only one action at a time.
* 31
* 31
* 32
* 33
* 34
* 35
* 36
* 37
* 38
* 39<- Yous submit only one action at a time.
* 39<- Yous submit only one action at a time.
* 31
* 39<- Yous submit only one action at a time.
* 31
* 31
* 32
* 33
* 34
* 35
* 36
* 37
* 38
* 39<- Yous submit only one action at a time.
* 32
* 39<- Yous submit only one action at a time.
* 33
* 34
* 35
* 36
* 37
* 38
* 39<- Yous do not duplicate actions.
* 39<- Yous submit only one action at a time.
* 39<- Yous submit only one action at a time.
* 39<- Yous submit only one action at a time.
* 39<- Yous submit only one action at a time.
* 39<- Yous submit only one action at a time.
* 40
* 41
* 42
* 43
* 44
* 45
* 46
* 47
* 48
* 49
* 49
* 41
* 42
* 43
* 44
* 45
* 46
* 47
* 48
* 49
* 49
* 41
* 42
* 43
* 44
* 45
* 46
* 47
* 48
* 49
* 49
* 49
* 41
* 43
* 44
* 45
* 46
* 47
* 48
* 49
* 49
* 49
* 41
* 42
* 43
* 44
* 45
* 46
* 47
* 48
 

* 41 ++= CURRENT BOARD ---
* 42 += "0","1","2","3","4","5"