Monte Carlo Planning with Large Language Model for Text-Based Game Agents
Authors: Zijing Shi
Meng Fang
Ling Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Large language model
Monte Carlo tree search
Text-based games
Primary area: foundation or frontier models, including LLMs

Abstract: Text-based games provide valuable environments for language-based autonomous agents. However, planning-then-learning paradigms, such as those combining Monte Carlo Tree Search (MCTS) and reinforcement learning (RL), are notably time-consuming due to extensive iterations. Additionally, these algorithms perform uncertainty-driven exploration but lack language understanding and reasoning abilities.
In this paper, we introduce the Monte Carlo planning with Dynamic Memory-guided Large language model (MC-DML) algorithm. MC-DML leverages the language understanding and reasoning capabilities of Large Language Models (LLMs) alongside the exploratory advantages of tree search algorithms. Specifically, we enhance LLMs with in-trial and cross-trial memory mechanisms, enabling them to learn from past experiences and dynamically adjust action evaluations during planning. We conduct experiments on a series of text-based games from the Jericho benchmark. Our results demonstrate that the MC-DML algorithm significantly enhances performance across various games at the initial planning phase, outperforming strong contemporary methods that require multiple iterations. This demonstrates the effectiveness of our algorithm, paving the way for more efficient language-grounded planning in complex environments.

Published as a conference paper at ICLR 2025
MONTE CARLO PLANNING WITH LARGE LANGUAGE MODEL FOR TEXT-BASED GAME AGENTS
Zijing Shi
Meng Fang
AAII, University of Techholoog Sydney zi jijing- - shi @stuueet.... .ut. edu.au au
University of Liverpool
Meng. Fang@ vivrrooolll.. acc .k
Ling Chen
AAlI University of Technology Sydney ling. cwwn@uts........................ edw.uutteeeuu.a........ eu
ABSTRACT
Text-based games provide valuable environments for language-based autonomous agents. However, soaniing-thenleanee Monte Carlo Tree Search (MCTS) and reinforcement learning (RL), are notably time-consuming due to extensive iterations. Additionally, these algorithms per- form uncertainty-driven exploration but lack language understanding and reason- ing abilities. In this paper, we introduce the Monte Carlo planning with Dynamic Memory-guides Large language model (MC-DML) algorithm. MC-DML lever- ages the language understanding and reasoning capabilities of Large Language Models (LLMs) alongside the exploratory advantages of tree search algorithms. Specifically, we enhance LLMs with in-trial and cross-trial memory mechanisms, enabling them to learn from past experiences and dynamically adjust action eval- uations during planning. We conduct experiments on series text-based games from the Jericho benchmark. Our results demonstrate that the MC-DML algorithm significantly enhances performance across various games at the initial planning phase, outperforming strong contemporary methods that require multiple itera- tions. This demonstrates the effectiveness of our algorithm, paving the way for more efficient language-grounded planning in complex environments.
paradigms, such those combining
INTRODUCTION
Text-based games serve valuable environments for studying various natural language processing (NLP) and sequential decision-making problems (Narasimhan et al., 2015) Xu et al. 2020). In these games, agents navigate environments using textual commands and deal with limited observability. Unlike simple synthetic games (Côté et al.) 2019). human-designed adventure games feature dy- namic state spaces and sparse rewards, presenting significant challenges (Hausknecht et al., 2020) Existing game agents are typically based on reinforcement learning (RL) and employ c-greedy or softmax policies for action exploration. However, they lack long-term planning abilities (Osborne) al. 2022).
Planning for text-based games presents unique challenges, as each language action treated as discrete token, making uncertainty-drrven exploration without understanding the potential future impacts or the natural language described game state. Previous works that integrate Monte Carlo Tree Search (MCTS) with learning models have shown remarkable proficiency in classical games such as Go and Atari (Browne et al.) 2012 (Sweechoskki et al. (2023). These methods are based on the architectures pioneered by AlphaGo and AlphaGo Zero, which utilize policy and value net- works evaluate and prioritize potential moves, thereby significantly enhancing gameplay (Silver et al. 2016 2017). However, MCTS still faces challenges in real-world scenarios. Learning models typically need substantial warm-up period to learn effectively. Their performance relies data obtained from MCTS planning, which is confined by the practical limitations of tree size and depth.
Our code is available at http:://eengaammgggguuu..ieeeeee

---

Published as a conference paper at ICLR 2025
Specifically, in text-based games. MCTS lacks the necessary language understanding and reasoning abilities. In response, Jang et al. (2020) suggested guiding exploration in MCTS planning through the evaluation of action's 's semantic similarity. This approach assumes that action may possess value if its similar actions taken previously have high Q values. While effective in certain games, this assumption may be less reliable in more dynamic settings.
The recent emergence of Large Language Models (LLMs) have shown remarkable capabilities in quickly generating viable initial plans for decision-making tasks, even with minimal or no prior examples. Some research explores various prompting techniques, such as Reflection (Shinn et al. 2024) and Tree-of- Tree-of-Thought (Yao al. 2024), which further enhance LLM reasoning for interactive tasks. Despite achieving near saturated performance in simpler environments such as ALFworld (Shridhar et al. 2020), they continue to face challenges in more complex settings. A primary chal- lenge lies in translating the plans generated by LLMs into executable actions. Furthermore, LLMS often struggle to balance exploration and exploitation, which hinders their ability to navigate exten- sive state spaces efficiently.
In this study, we explore the potential of LLMs to enhance MCTS planning in complex interactive tasks. We aim to answer two questions: (1) Can LLMs, with their encoded knowledge, enhance action exploration within MCTS planning, thereby improving sample efficiency and task perfor- mance? (2) Can LLMs, with their few-shot learning capabilities, dynamically adapt action guidance based on past experiences during planning? To address these questions, we introduce the Monte Carlo planning with Dynamic Memory-guider Large language model (MC-DML). This algorithm leverages the language understanding and commonsense reasoning capabilities of LLMs and the ex- ploration benefits of tree-search approaches. By integrating both in-trial and cross-trial memory into LLMs, MC-DML enables dynamic adjustments of action evaluation during the MCTS planning.
We conduct experiments using a series of text-based games from the Jericho benchmark (Hausknecht iet al. 2020). These games are characterized by numerous branching paths and sparse rewards. The agent, operating under limited observability, must extensively explore the environment solve com- plex puzzles. Our results demonstrate that the MC-DML algorithm enhances performance across various games the initial planning phase, outperforming strong contemporary methods that require multiple iterations for policy optimization. Additionally, perform ablation studies to highlight the role of the memory mechanism LLM policy.
Our main contributions are summarized as follows: First, we propose an MCTS-based algorithm that integrates an LLM to enhance action exploration in complex textual interactive tasks. Second, develop an LLM agent equipped with both in-trial and cross-trial memory, enabling dynamic language action value estimation in tree-search planning phrase. Third, our experiments on series of text-based games demonstrate that the proposed algorithm significantly improves performance across multiple games at the initial planning phase, outperforming strong contemporary methods that require multiple iterations.
2 PRELIMINARY
2.1 MONTE-CARLO TREE SEARCH
Upper Confidence bound for Trees (UCT) MCTS (Kocsis & Szepesvári 2006 Coulom 2006 operates by iteratively developing a decision tree through four key phases: selection, expansion, simulation, and backpropagation. Within the search tree, the standard MCTS method uses UCT to choose action a" at each node, balancing exploitation based on the Q-value with exploration driven by uncertainty. The formula for UCT is as follows:
In (s)) ((s,a)
Q(s,a) + Cust
a" = arg max
acA(s)
where Q(s, a) is the average reward for action in state 8, (s, a) is the number of times action is chosen in state s, (ss) is the visit count to state s, Cuot is constant that balances exploration and exploitation.
Predictor UCT (PUCT) One limitation of UCT is its dependence on Monte Carlo averages to estimate state values, which can result in significant search inefficiencies, especially in text-based
2

---

Published as a conference paper at ICLR 2025
games with high branching factors. PUCT partially overcomes these challenges by incorporating PUCB, which utilizes a prior action distribution <--[s) to estimate action values under state s and prioritize exploration (Silver et al. 2017 2018). The formula for PUCT is as follows:
((())
a" arg max aEA((s)
Q(s, + Opuct [[ass)- 1 ((s
(2)
Here, r(-[s) is usually neural network that trained via behavioral cloning using (s, a²) samples from previous tree search results. However, PUCT still faces several challenges. Firstly, the training data for policy network sourced from MCTS planning. In practice, due limitations tree size and depth, the training data may not be optimal. To achieve effective learning, it often requires multiple iterations using a planning-then then-learning paradigm, which is time-consuming. Addition- ally, this training data is specific to certain environments, limiting the ability of the policy network to generalize across various games. Furthermore, the nature of its supervised learning approach restricts its strategic depth and impairs its long-term planning capabilities.
2.2 TEXT-BASED GAMES AS POMDPS
The text-based game can be modeled as a Partially Observable Markov Decision Process (POMDP) Narasimhan al. 2015), represented by (S, A, O, R, y). At each time t, the agent cannot directly observe the environmental state 8t from S. Instead, it infers the state through a textual observation o from O. When the agent performs action a, from A, the environment transitions the next state according to the hidden transition function T. Meanwhile, the agent receives a reward r; R( (z) and the subsequent observation ot+1 from the game environment. The agent's goal is to optimize actions to maximize the expected total discounted rewards Rs = E 7're]. where
y ranges from 0 to indicating the discount factor.
3 METHOD
In this study, we focus on husan-designed text adventure games, which present two significant challenges. First, these games feature a vast combinatorial action space. To manage this complexity, benchmarks such as Jericho provide a predefined set of valid actions each step by filtering out inadmissible commands. However, this still results in dynamic action space that varies with the game state, leading to numerous game branches. Moreover, these games are characterized by sparse rewards and multiple bottleneck states. Figure [ the game 2orkl. Based on the current observation, the agent selects from the available actions, leading to different game branches. An agent optimized for cumulative rewards might choose open t crapdoor, resulting in a significant immediate reward but also leading to subsequent death. To progress in the game, the agent must explore necessary actions without receiving any immediate reward signals. This requires the agent to combine semantic reasoning with long-term planning capabilities.
illustrates
example of a bottleneck state in
Rewart 25 open traddoor
dowwn
It
black,
obs: are
you likely by You have died!
living room...
closed deor feet
You dark and damp cellar with narrow leading northi th....
open trapdoor
down
take lartern
turn lantem
Reward +55
Figure 1: An example bottleneck state from the game Zorkl.
To address these challenges and the limitations of current MCTS -based algorithms, we introduce the MC-DML algorithm. We provide a comprehensive introduction to MC-DML in SectionB.11 outline the algorithm during single planning process Section 3.2 and discuss the innovative aspects of MC-DML. in Section 3.3

---

Published as a conference paper at ICLR 2025
PUCT in MCTS
MC with Dynamic Memory Guided LLM (MC-DML)
Prior Policy m/lly/
Update Cross-Tiral Nemory
Action Selection
Uiinng
letar
=..05
tako labers LLM (1 1
id
grgmax.
U of visit countr- LLM(6)5)
Thia unsucurssfel pewsslay
are playing qamn current ubservaciio mesurty Your wenery about tris
Please adencaty
rescons for fallure and
suppest the noxt remn).
deaed on your osservelion,
#rgmax.(0
Froe (Ual nol (86s) to
oc vizit cownts ztels)
Noinng doer into the dark
Simulation (s,
source or mny other itema
CAKe Jatere
ardb
GAME PAUSI
V Uiilizes miir information
x Time--sonuuring with multiple neeaations Polley performance depends on simulated data
v Easy imploment
Good Intiaal pollcy and ability improve during planning
Figure 2: A comparison of the PUCT and MC-DMI. algorithms. PUCT trains its policy through imitation learning from self-play data. In contrast, MC-DMI. uses a L.L.M as the initial policy. During planning, the LLM learns from past failure trajectories and adjusts the action value estimates. This approach more closely aligns with the human thought process.
3.1 MONTE CARLO PLANNING WITH DYNAMIC MEMORY- GUUDDD LLMS (MC-DML)
MC-DML consists of four stages: selection, expansion, simulation, and backpropagation, and finally predicts an action based on the simulations. During the expansion phase, MC-DML employs an LLM as the prior policy within the PUCT algorithm. Based on the current tree state, the LLM assigns non-uniform search priorities to each optional action. After an action is selected, the expansion phase adds a new node to the search tree. In the simulation phase, MC-DMI. conducts multiple rollouts from this new node to evaluate the potential outcomes of the chosen action. The simulation results are then backpropagated to update the Q-value estimates and visit counts the relevant nodes. In text-based games, the current state is not fully observable. Therefore, we equip the LLM with a dynamic memory mechanism, utilizing in-trial memory M; and cross-trial memory Mc. M; contains the current trajectory history, representing the game state, while Me includes experiences from previous failure trajectories, used to dynamically adjust the action value estimation.
Learning from In- Tn-rrial Memory The in-trial memory M, includes sequence of past observa- tions and actions. Using this memory, we prompt the LLM generate probability distribution of valid actions T--Is) grounded in commonsense. The probability of an action a is calculated by accumulating the conditional probabilities of its tokens. We use the GPT-3.5 model, which provides log probabilities for the top potential answers. These probabilities are then normalized using the softmax function, For APIs where token probabilities are unavailable, this can be achieved through
self-consistency (Wang et al. 2022) and verbalized methods (Lin et al. 2022
Reflection on Cross-Trial Memory In-trial memory is form of short-term memory that relies on the LLM's commonsense but lacks flexibility. Inspired by (Shinn et al. (2024), we develop cross- trial memory Mes interpretable and enduring form of episodic memory that allows agents to learn from past failures. In MCTS, the agent repeatedly simulates from the root node to explore various paths. This restart mechanism allows agent reflection on the segment of trajectory and resume play from that save point. Figure illlsttraees the process of updating and utilizing cross- trial memory in MC-DML. During the tree search, when the agent encounters a terminal state due to game failure, the LLM analyzes this trajectory and generates a reflection. In subsequent simulations
selff-consistency methods estimate the probability of an answer by sampling multiple responses from the I.M. Verbalized methods leverage well-designed prompt to instruct the I.I.M to output the most likely answers along with their corresponding prohahilities.
4

---

Published as a conference paper at ICLR 2025
under the same root node, combine in-trial memory and cross-trial memory to adjust the LLM's action estimations. The formula for action selection in MC-DML is as follows:
(s)
a" arg max aEA
(s,,a) Cpuct (LM Me, 1+ (s,
where (a p) is the action probability distribution generated by the LLM policy, is the given prompt consisting of an instruction and optional actions A, Q a) is the average reward for action in tree state s.
3.2
ALGORITHM
Algorithm Monte Carlo Planning with Dynamic Memory-Guided LLM (MC-DML)
1:
procedure SEARCH(So)
32: procedure SELECTACTIONCH
2: $g
+ (50)
33:
M; <- LASTPART(H)
3: ho 6000 On
34:
Talss) LLM(M;; MesPPactioonpniii
4: repeat
a" +- arg max (Q(+,a) aEA
5: SIMULATE(SO, ho.
6: until MaxDepthReachee()
(p N(+.0+1
N(e)
7:
return arg max Q(ho, a) aeAA
35:
if (s, a) then
8: end procedure
36:
rollout <- true
9:
procedure SIMULATE(S. h,
37: else
10:
GAMEFAIL( then reflection <- LLM(h, Preftection)
38:
rollout +- false
11:
39: end if
12:
Me + MM + reflection
40: return [a", rollout]
13:
return 0
41: end procedure
14:
end
42: procedure ROLLOUT(h, s,t)
15:
i
planning horizon H then return 0
43:
GAMEFAIL(s) then
16:
44:
reflection L.MM(h, Preflecioo)))))
end if
45:
Mr. + M + reflection
18:
[a, rollout < SELECTACTION())
46:
return
[r, + Tss.a)
47:
end if
20:
h' + + +a+
48:
if planning horizon // then
21:
if rollout then
49:
return 0
22:
R' + ROLLOUT(S' M, (111)
50:
end if
23:
else
51:
O(s)
24:
R' +- SIMULATE(S N
52:
a Uniform(AA)
25:
end
53:
[r.s',o'] ((s
26:
R R r N......R R'
54:
return r + y- ROLLOUT(S", t+
27:
N(h) t N(h)+ +1
55: end procedure
28:
N(h,a) -Nhhaa +1
R-Q(haa) Q(h,a) + Q(h, + N(h.))
29:
30:
return R
end procedure
We now describe the operation of MC-DML in a single planning episode, as outlined in Algorithm I For each simulation, MC-DML initializes the root node sp and new trajectory ho to construct the state (lines 2-3). An action a* is chosen based on the Q value, visit counts, and the LLM policy (lines 34-35). The L.M policy determines the action probability distribution using the in-trial memory M; and the cross-trial memory M The in-trial memory M, is portion of the trajectory h (line 33), while the cross-trial memory M, consists of reflections generated from previous failed trajectories (lines 10-13, 44-47). MC-DML iteratively selects actions execute and updates the visit counts and estimated Q values (line 26-29). When encountering leaf nodes, it expands the tree and performs rollout, using uniform policy to sample actions and returning the discounted reward. Upon completion of the search process, the agent will execute an final action based on the estimated Q value and receive observation.
5

---

Published as a conference paper at ICLR 2025
3.3 NOVELTY IN COMPARISON TO PRIOR ALGORIHHM
We now more explicitly discuss comparisons to a few other approaches. The approach most related ours LLM-MCTS, which uses an LLM as a prior policy in MCTS (Zhao et al., 2024). While the LLM can serve as a good imitial policy, it cannot improve the policy based on past experience and external feedback. This makes LLM-MMTS well-suited for commonsense planning tasks, such as object rearrangement in household environments, but less effective in uncertain environments like text-based games.
For text-based games, MC-LAVE-RL is one of the SOTA methods that combines MCTS with RL while considering the semantic sharing between actions (Lee et al. (021). During MCTS planning. an exploration reward is added to each action a estimated through the Q-values of its semantically similar actions. This approach addressed the bottleneck state in the game 2orkl (see Figure D Actions such collecting items typically have high value in games, resulting in the action take lantern r being assigned a higher exploration bonus than the action open trapdoor. However, its performance beyond games remains be validated. In this study, rather than relying item col- lection within LLM prompts, we focus on developing a more general solution. MC-DML simulates human gameplay by combining in-trial and cross-trial memory, mimicking how humans retain both recent detailed information and significant past experiences. We avoid introducing any prior game knowledge or human -esiined hints in the LLM prompts.
4
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Environments We conduct experiments on text-based games provided by the Jericho bench- mark (Hausknecht et al.) 2020). These games provide a variety of challenges such as darkness, nonstandard actions, inventory management, and dialog Among them, Zorkl, Deephome, and Ludi corp are categorized as difficult games, their optimal completion paths require over 300 steps, with an average of more than 14 available actions per step. The remaining games are catego- rized as possible games (Hausknecht et al., 2020). At each step the observation from the Jericho game engine includes a description of the state. This is augmented with location and inventory in- formation by issuing the "look" and "inventory" commands, forming or Additionally, we utilize the valid action handicap provided by Jericho. For further analysis of these games, refer to Appendix A
Implementation Details For the LLM policy, we use gpt- 5.3.--turbbooo 0125 as the backend model with sampling temperature set to O. We query the LLM for the index of the optimal action and retrieve the log probabílities for the top 20 tokens at that index. For absent actions, we assign a log probability of These log probabilities of 5. The in-trial memory is defined as (o,-1, 4;-1, ot),, and the size of the cross-trial memory k is to 3. We allow each root node to store up reflections. If the number of reflections exceeds k. cross-memory collection is terminated early due to the sufficient experiences. When necessary, we compress the input information using a truncation function to ensure that fits within the LLM's input window constraints.
then normalized using softmax with temperature
For the tree search component, we adopt dynamic pruning strategy; the search depth dynami- cally adjusted between a minimum depth dmin and maximum depth dmux- The algorithm begins with dmin- If the highest Q-value of the selected action node is O, the depth increased by Ad and the search is repeated, up to dmax- This setting takes into account the uneven distribution of steps with rewards in the game. We provide a statistical analysis the game's 's step distribution in Ap- pendix A and an ablation study of this setting in Section 4.3 Further details about the experimental implementation can be found in Appendix
Evaluation We first evaluate the performance of MC-DML in comparison with baseline methods on a series of text-based games. Next, we compare MC-DML with the intermediate scores of MCTS- based baselines during multiple iterations in the game Zork1. Then, we conduct ablation studies on subset of games to evaluate the importance of different memory mechanisms in MC-DML. Finally, we provide further qualitative analysis, including how MC-DML addresses bottleneck states in the game Zork1.

---

Published as a conference paper at ICLR 2025
Baseline We consider 10 baselines, including RL-based agents, LLM-basee agents, and MCTS- based agents. All these baselines except MC!Q*BERT assume access to valid actions from the Jericho benchmark. Among these, PUCT-RL and MC-LAVE-RL algorithms serve as direct com- parators to MC-DML. (1) DRRN (He et al. 2015) The Deep Reinforcement Relevance Network (DRRN) is RI.-based agent that utilizes Q-based softmax policy. This policy is parameter- ized with GRU encoders and decoders and is trained using the Temporal-Differrece (TD) loss. (2) KG-AZC-Hard (Ammanabrolu & Hausknecht 2020): An actor-critic method using knowledge graph for state representation, with hard valid action constraint. (3) MERTT (Ammanabrolu al. 2020): An extension of KG-A2C that leverages BERT for knowledge graph construction and includes wwwellegg-grahh-aaeee
intrinsic rewards. (4) MPRC-DQN (Guo et al.) 2020): A DQN- based method that formulates action selection as a multi-paragraph reading comprehension task, re- trieving relevant historical observations and jointly encoding them with the current state for Q-value prediction. (5) RC-DQN (Guo et al. 2020); A DQN-based agent that applies a reading compre- hension model to compute Q-values from the current observation alone, without multi-paragrapn saelledggggrphh-.e based A2C agent augmented with case-based reasoning framework that explicitly stores and reuses successful experiences to improve out-of-distribution generalization. (7) LLM agent: We employ the LLM directly as the agent interact with the environment, aiming assess potential data contamination within the LLM. In this setting, the LLM selects actions from the valid action set based on the current tra- jectory history complete the game. We set the temperature of the LLM 0 and select the action with the highest output probability. (8) Reflection agent (Shinn et al. (024); We allow the LLM to perform reflection, which is then used to guide interactions with the environment in the next round. (9) PUCT-RL (Jang et al. (0000): PUCT-RL uses PUCT as a policy improvement operator for DRRN, alternating between PUCT planning and supervised learning of self-generated actions. (10) MC-LAVE-RL (Jang et al., 2020); MC-LAVE is one of the SOTA models on the Jericho benchmark that combines MCTS with RL while considering the semantic sharing between actions.
retrieval. (6) BiKE+CBR (Atzeni et al.) 202T): A
4.2 MAIN RESULTS
Tables I and 2 present the performance of MC-DML compared with RL-based baselines and LLM/MCTS- based baselines, respectively, across set of 9 games. We observe the following key findings. First, when compared to RL-based baselines, MC-DML achieves better comparable results in 6 out of 9 games. In comparison to LLM/MCTS- based baselines, MC-DML outperforms matches their performance in 8 out of games. Secondly, in challenging games like Deephome, MC-DML overcomes multiple bottlenecks, achieving nearly double the performance of MC-LAVE- RL. In possible games like Pentari and Detect ive, MC-DML
completes the games fully. In other possible games, such as Library and Temple, it also approaches the best possible score within the given number of steps. Finally, the LLM policy performs poorly, likely due to its inability to balance exploration and exploitation. This also indicates that LLM does not have knowledge of the game's walkthrough under the current prompting setting.
Algorithms
DRRN KG-A2C-Hard MCIQ"BERT MPRC-DON RC-DON BiKE CBR
MC-DML
Games
Zork1
32.6
40.2
41.6
38.3
38.8
44.3
48.66 + 1.89
Deephome
2.1
1
1.41
Ludicorp
13.8
19.8
22.8
19.7
17
23.8
19.67 + 1.7
Pentari
27.2
4,
58
44.4
43.8
52.1
70 + 0.0
Detective
197.8
338+3..4
330
317.7
291.3
326.1
346.67 9.43
Library
17
0.0
17.7
18.1
22.3
21 0.0
Balances
10
10
10
10
11.9
+ 0.0
Temple
7.4
8 0.0
8
8
7.8
8 + 0.0
Zuu
21.6
5 0.0
11.8
23.67 1.9
Table 1: Comparison of MC-DML. with RL-based baselines on text-based games from Jericho benchmark. MC-DML indicate averages over independent runs. We omit the results of some baselines on the ztuu game because they revealed unbounded reward loops (Guo et al.) 2020).

---

Published as a conference paper at ICLR 2025
Algorithms
I..M-based
MCTS-based
Ours
Games
LLM
Reflection
PUCT-RL
MC-LAVE-RL
MC-DML
Zork1
0
5
38.2 =
45.2 1.2
48.66 1.89
Deephome
28.6
0.6
67 1.41
Ludicorp
4
4
18 + 0.0
22.8 + 0.2
19.67 = 1.7
Pentari
64
70
Detective
30
30
2.0
3.0
346.67 9,43
Library
1.0
1.0
21 0.0
Balances
10
10
10 0.0
100
10 0.0
Temple
0.0
8.0
0.0
Ztuu
5
0.0
7.2.7
23.67 1.9
Table 2: Comparison of MC-DML with LLM-based and MCTS-based baselines on Jericho bench- mark text-based games.
Table 3 shows the results of MC-DML. alongside the intermediate scores of the MCTS- base- lines during multiple iterations the game Zork1. 1. We would like emphasize that for the PUCT- RL and MC-LAVE-RL. algorithms, the final result is computed based on the policy and Q-function obtained after iterations, which is when convergence reached. In each iteration, these algorithms conducted 25 independent planning sessions to collect trajectories and experience replay for policy learning. Unlike these approaches, MC-DML does not require planning-then. teannnng paradigm. It can adjust the initial policy and estimated action values guided by dynamic memory.
PUCT-RL
MIC-LAVE-RL
MC-DML
Tree Search
RL
Tree Search
RL
Tree Search
Iteration
31.9 1.4
36.6
3.0
36.6 1.0
48.66 + 1.89
Iteration 2
35.8
1.0
3.1
38.2 0.8
N/A
Iteration 3
0.2
39.0
41.2
43.0 1.0
N/A
Iteration 4
35.2
0.8
4.1
45.2
N/A
Table 3: Experimental results of MC-DML with the intermediate scores of the MCTS-based baseline during multiple iterations on the game Zorkl. Our MC-DMI. achieves superior results with its initial planning.
4.3 ABLATION STUDIES
To evaluate the importance of the memory mechanisms and dynamic pruning strategy in MC-DML,
we conduct several ablation studies on a subset games. We compare the performance of MC- DML without dynamic pruning (DP), without Me, without DP, and without MM and DP, MM, and M, When disregarding the DP, we follow the experimental setup of Jang et al. 2020), using a fixed search depth for each game. Without Mc, the LLM's 's's action estimates are based on the current historical trajectory. Without both M, and M,, the LLM's action estimates at time t rely only the current state node og. The results show that using DP significantly improves performance in the game Zutt, but has little effect on other games. Removing M. reduces game scores, and removing both Me and M, results in even larger drop in scores, highlighting the importance of these memory mechanisms.
MC-DMI.
w.o. Me
w.o. DP
wo. Mo, DP
w.O. Me, DP
Zork1
48.66 1.89
38.33 + 2.89
48 2.45
38 5.2
31.67 4.7
Deephome
67 = 1,41
62.66 0.94
67.4 0.8
64.33 # 0.94
11.
Detective
346.67 9.43
326.67 4.71
334 4.9
323.33
320 0.0
Ztuu
23.67
20.66 0.47
0.56
0.88
6.33 + 0.94
Table 4: Ablation results subset of games. For the ablation models, we report the average score over independent runs. Overall, both the Me and M; are crucial our MC-DML..
8

---

Published as a conference paper at ICLR 2025
4.4 ANALYSIS
Table [ presents an illustrative example of search results for MC-DML and MC-DML w.o. Me on bottleneck state in the game 2orkl. Without the reflection module Me the LLM assigns a high value to the action open trap due to its semantic alignment with the current state, which also provides an immediate reward. Although this action eventually leads to failure, the agent, lacking the ability to reflect on its mistakes, continues to explore it, resulting in both a high Q-value and N (s, a). Consequently, the agent ends up selecting this action during the final execution step, which explains why it gets stuck in a bottleneck state. However, in MC-DML, the LLM generates reflection based on failed trajectories and store it in the memory. The reflection might be, "Ensure you have a light source before entering dark areas," altering the action probability distribution in subsequent simulations. After sufficient exploration, the agent obtains an accurate value estimation and ultimately selects the take lantern action at the current state. Ultimately, the agent selects the optimal action t take lant even though it does not provide any immediate reward. Similar bottleneck states are also addressed in the game Deephome. Additional trajectory examples of MC- DML playing Deephome are provided in the Appendix D
MC-DML
open trap
open
take sword
take lantern
take all
east
turn on lantern
Q(s,a)
4.41
11.41
9.31
14.26
0.00
-8.12
-1.42
LLM(alMM M..p)
0.16
0.13
0.10
0.22
0.10
0.08
0.17
a(s,a)
21
39
27
252
6
wo. Me
open trap
open case
take sword
take lantern
take all
east
turn on lantern
Q(s,a)
13.02
9.92
8.38
12.66
3.17
-1.85
4.73
LLM(alM...D)
0.24
0.20
0.21
0.10
0.10
0.05
0.06
a(s.a)
176
72
17
10
Table 5: An illustrative example of search results for MC-DML and MC-DML w.o. MMc on a bottleneck state in the game Zorkl. Regarding the differing scales between LLM value and Q(s,a). during simulations, the LLM value is multiplied by a scale factor CPUCT.
RELATED WORK
Action Selection in MCTS MCTS-based algorithms thrives in large search spaces by selectively sampling promising actions (Osborne et al. 2022). The prevalent PUCT algorithm enhances this process by predicting action values using prior knowledge, typically obtained from historical data through imitation learning (Silver et al. 2017 2018). Current research on MCTS is directed to- wards developing contextual action value estimators to further refine action exploration (Lee et al.) 2021 (Styglic et al. (02T). Specifically, in aanguage-gruum settings, |Branavan et al. (2.12) utilizes multi-layer neural network to extract relevant text segments from game documents. This text is then integrated into the Monte-Carlo search framework to facilitate linguistically-inte innfoorrm decision- making. Jang et al. (2020) introduced MC-LAVE-RL RL for solving text-based games, method that incorporates value sharing among actions during the search process. Specifically, an action is en- couraged for exploration if similar actions taken previously have high Q-values. While effective in certain games, this assumption may be less reliable in more dynamic settings.
Interactive Planning with LLM It is important to underscore our research focus. While recent studies have introduced search-based prompting approaches to enhance LLMs' reasoning capabili- ties by exploring generated thoughts (Yao et al. 2024 Ding et al. (023). our study takes distinct direction, emphasizing the large-scale planning under limited observability. In this domain, some studies utilize large LLMss as direct policies for interactive tasks, which yield interesting results but also exhibit certain limitations (Huang et al. 2022 Zhu et al. 2023 Fang et al., 2024). One such limitation is the difficulty in translating the plans created by LLMs into executable actions. Another is the inability of LLMS to balance exploration with exploitation. To address these issues, some research efforts use LLMs formulate high-level plans and guide RL agents in performing spe- cific actions (Shukla et al., 2023 Liu et al.) 2024 Dalal et al. 2024] Zhang et al. (2024). However, these RL agents often struggle to conduct long-term planning. The study most closely aligned with

---

Published as a conference paper at ICLR 2025
ours is Zhao et al. (2024), which employs LLM fixed prior policy within MCTS address common sense tasks. Whereas these tasks are more intuitive and can be effectively addressed by leveraging the world prior knowledge of LLMs, text-based games present greater uncertainty, thus posing significant challenges.
Text-based Game Playing Agent Recent research has explored RL agents with varying archi-
tectures for solving text- based games (He et al. 2015 Narasimhan et al. 2015 Ammanabrolu & Hausknecht 2020; Xu et al.] 202T Ryu al. [022]] Tuyls et al., [022] SShi et a.. 2023b a). Inno-
vations in this field address the problem of combinatorial action spaces (Zahavy et al. 2018 Yao
al. 2020), modeling state space utilising knowledge graphs (Ammanahrolu & Hausknecht 2020
and reading comprehension modules (Ammanabrolu et al. 2020; Xu et al. [022 Dambekodi et al. 2020). These agents rely on e-greedy softmax policies, which restrict their capacity for long-term term planning. To overcome this limitation, Jang et al. (2020) proposed MC-LAVE-RL, which integrated MCTS with RL to solve text-based games, while also considering semantic sharing of actions between nodes. Following this line, our study aims to extend the capabilities of these agents by combining MCTS with LLM, enhancing their strategic depth and adaptability in complex scenarios.
Adhikari et al. 2020 Xu et al. 2020). integrating iwestion-answernig
CONCLUSION
In this study, propose the MC-DML algorithm. MC-DML leverages the prior knowledge em- bedded in LLM to guide action exploration during MCTS planning. The LLM is equipped with dynamic memory mechanism adjust action value estimation based on historical experience. MC- DML simulates human gameplay by mimicking how humans retain both recent detailed information and significant past experiences. Our results demonstrated that the MC-DMeeenhances across various games.
enhances performance
Limitation We utilise an LLM for value estimation during MCTS planning that combines in-trial memory and cross-memory. However, for simplicity in our current setup, we define in- memory as the trajectory within shorter time window. In these games, some puzzles may relate to clues encountered much earlier, such as a spell or an item seen long ago. This places demands on the L.MM's "Needle In Haystack" ability. Future work could explore more efficient in-trial memory storage and retrieval mechanisms.
ACKNOWLEDGMENTS
This project is partially supported by ARC DP240101349. We are grateful the anonymous re- viewers for their insightful feedback, which significantly enhanced our work.
REFERENCES
Ashutosh Adhikari, Xingdi Yuan, Marc-Alexande Côté, Mikuláš Zelinka, Marc-Antoine Rondeau, Romain Laroche, Pascal Poupart, Jian Tang, Adam Trischler, and Will Hamilton. Learning dy- namic belief graphs to generalize text-based games. Advances in Neural Information Process- ing Systems, 33:3045-300 3057, 2020.
Prithviraj Ammanabrolu and Matthew Hausknecht. Graph constrained reinforcement learning for natural language action spaces. arXiv preprint arXiv:2001.08837, 2020.
Prithviraj Ammanabrolu, Ethan Tien, Matthew Hausknecht, and Mark O Riedl. How to avoid being eaten by a grue: Structured exploration strategies for textual worlds. arXiv preprint #rXiv:2006.07409 2020.
Mattia Atzeni, Shehzaad Dhuliawala, Keerthiram Murugesan, and Mrinmaya Sachan. Case- based reasoning for better generalization in textual reinforcement learning. arXiv preprint arXiv:2110.08470, 2021.
SRK Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in a monte- carlo framework. Journal of Artificial Intelligence Research, 43:661-704, 2012.
10

---

Published as a conference paper at ICLR 2025
Cameron B Browne, Edward Powley, Daniel Whitehouse, Simon M Lucas, Peter Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A survey of monte carlo tree search methods. IEEE Transactions Computational Intelligence and Al in games, 4(1): 2012.
Marc-Alexandre Côté, Akos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla El Asri, Mahmoud Adada, al. Textworld: A learning environment for text-based games. In Computer Games: 7th Workshop, CGW 2018, Held in Con- junction with the 27th International Conference on Artificial Intelligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Papers 7, pp. 41-75. Springer, 2019.
Rémi Coulom. Efficient selectivity and backup operators in monte-carlo tree search. In International conference on computers and games, pp. 72-83. Springer, 2006.
Murtaza Dalal, Tarun Chiruvolu, Devendra Chaplot, and Ruslan Salakhutdinov. Plan-seq- learn: Language model guided rl for solving long horizon robotics tasks. arXiv:2405.01534, 2024.
arXiv preprint
Sahith Dambekodi, Spencer Frazier, Prithviraj Ammanabrolu, and Mark O Riedl. Playing text-based games with common sense. arXiv preprint arXiv:2012.02757, 2020.
Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Ra- jmohan, Qingwei Lin, and Dongmei Zhang. Everything of thoughts: Defying the law of penrose triangle for thought generation. arXiv preprint urXiv:2311.042555 2023.
Meng Fang, Shilong Deng, Yudi Zhang, Zijing Shi, Ling Chen, Mykola Pechenizkiy, and Jun Wang. Large language models are neurosymbolic reasoners. arXiv preprint arXiv:2401.0333 2024.
Xiaoxiao Guo, Mo Yu, Yupeng Gao, Chuang Gan, Murray Campbell, and Shiyu Chang, Interac- tive fiction game playing as multi paragraph reading comprehension with reinforcement learning. arXiv preprint arXiv:2010.023:0 2020.
Matthew Hausknecht, Prithviraj Ammanabrolu, Marc-Alexandre Côté, and Xingdi Yuan. Interac- tive fiction games: colossal adventure. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 7903-7910, 2020.
Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. Deep reinforcement learning with a natural language action space. arXiv preprint arXiv:1511.0460 2015.
Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pp. 91447 PMLR, 2022.
Youngsoo Jang, Seokin Seo, Jongmin Lee, and Kee-Eung Kim. Monte-carlo planning and learning with language action value estimates. In International Conference on Learning Representations 2020.
Levente Koesis and Csaba Szepesvári. Bandit based monte-carlo planning. In European conference on machine learning, pp. 282-293 Springer, 2006.
Jongmin Lee, Wonseok Jeon, Geon- Hyeong Kim, and Kee-Eung Kim. Monte-carlo tree search in continuous action spaces with value gradients. In Proceedings the AAAI conference on artificial intelligence, volume 34, pp. 4561-4568, 2021.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. arXiv preprint arXiv: 14334, 2022.
Shaoteng Liu, Haoqi Yuan, Minda Hu, Yanwei Li, Yukang Chen, Shu Liu, Zongqing Lu, and Jiaya Jia. Rl-gpt: Integrating reinforcement learning and code-as-policy. arXiv preprint arXiv:2402. /9299, 2024.
Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. Language understanding for text-based games using deep reinforcement learning. arXiv preprint / 115550006444 2015.
11

---

Published as a conference paper at ICLR 2025
Philipp Osborne, Heido Nömm, and André Freitas. A survey of text games for reinforcement learning informed by natural language. Transactions of the Association for Computational Linguistics, 10: 873-887, 2022.
Dongwon Ryu, Ehsan Shareghi, Meng Fang, Yunqiu Xu, Shirui Pan, and Reza Haf. Fire burns, sword cuts: Commonsense inductive bias for exploration in text-based games. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 515-522, Dublin, Ireland, May 2022. Association for Computational Lin- guistics. doi: (00186655/1120222acllshotrtt
uttpttss: httptttppss//aaaaattttttceoeeeee htt.:////////olooooooooooo//////
1e1lssoor..5
Zijing Shi, Meng Fang. Yunqiu Xu, Ling Chen, and Yali Du. Stay moral and explore: Learn to behave morally in text-based games. The Eleventh International Conference on Learning Representations, 2023a.
Zijing Shi, Yunqiu Xu, Meng Fang, and Ling Chen. Self-imitation learning for action generation in text-based games. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 703-726, 2023b.
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint arXiv:2010.03766 2020.
Yash Shukla, Wenchang Gao, Vasanth Sarathy, Alvaro Velasquez, Robert Wright, and Jivko Sinapov. Lgts: Dynamic task sampling using llm-generated sub-goals for reinforcement learning agents. arXiv preprint arXi::2310.09454, 2023.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schritwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, al. Mastering the game of go with deep neural networks and tree search. nature, 529(7587):484448) 2016.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-35,, 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science, 362(6419): 140- 1144, 2018.
Maciej Swiechowski, Konrad Godlewski, Bartosz Sawicki, and Jacek Mańdziuk. Monte carlo tree search: A review of recent modifications and applications. Artificial Intelligence Review, 56(3): 2497-2562, 2023.
Ori Sztyglic, Andrey Zhitnikov, and Vadim Indelman. Simplified belief-dependent reward mets planning with guaranteed tree consistency. arXiv preprint arXiv:2105. /4239, 2021.
Jens Tuyls, Shunyu Yao, Sham Kakade, and Karthik Narasimhan. Multi-stage episodic control for strategic exploration in text games. arXiv preprint arXiv:2201.01255 2022.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery. and Denny Zhou Self-consistency improves chain of thought reasoning in language models. arXiv preprint ar0iv:2203.11 / 2022.
Yunqiu Xu, Meng Fang. Ling Chen, Yali Du, Joey Tianyi Zhou, and Chenggi Zhang. Deep rein- forcement learning with stacked hierarchical attention for text-based games. Advances Neural Information Processing Systems, 33:16495-16507, 2020.
12

---

Published as a conference paper at ICLR 2025
Yunqiu Xu, Meng Fang, Ling Chen, Yali Du, and Chengqi Zhang. Generalization in text-based games via hierarchical reinforcement learning. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1343-1353, 1353, Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10.18653/v 1/2011 htt.://.............eelee .indings-emmlp.ll. URL
httts::/acclantiin wwwpmthhh.......g./210111111
ndi ngs-iinnnnns-mnnnn 116
Yunqiu Xu, Meng Fang, Ling Chen, Yali Du, Joey Zhou, and Chengqi Zhang. Perceiving the world:
Quedtion-guided reinforcement learning for text-based games. In Proceedings of the 60th Annual
Meeting of the Association for Computational Linguistics (ACL), pp. 538-560) 2022. doi: 10.
06653/1/2022.aclllog
ogy- org/ /eargg20000 2022 acl -ll-lloggg long. 41
Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. Keep CALM and plore: Language models for action generation in text-based games. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 8736-8754, 2020. doi: 10. 12..665//122222..mmmmmmaaai
URL piteps://aclantiie org/
2020. ewnlp-main 704
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Ad- vances Neural Information Processing Systems, 36, 2024.
Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor. Learn what not to learn: Action elimination with deep reinforcement learning. Advances in neural information processing systems, 31, 2018.
Shenao Zhang, Sirui Zheng, Shuqi Ke, Zhihan Liu, Wanxin Jin, Jianbo Yuan, Yingxiang Yang. Hongxia Yang, and Zhaoran Wang. How can Ilm guide rI? value-based approach. arXiv preprint 4rXiv:2402.1618/ 2024.
Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models commonsense knowledge for large-scale task planning. Advances in Neural Information Processing Systems, 36, 2024.
Xizhou Zhu, Yuntao Chen, Hao Tian, Chenxin Tao, Weijie Su, Chenyu Yang, Gao Huang, Bin Li, Lewei Lu, Xiaogang Wang, al. Ghost in the minecraft: Generally capable agents for open- world enviroments via large language models with text-based knowledge and memory. arXiv preprint 2:X.::2305.17144 2023.
13

---

Published as a conference paper at ICLR 2025
A GAME STATISTICS
We conduct experiments upon games provided by the Jericho Game Suite (Hauskmecht et al, 2020). Different from those generated through pre-defined simple rules (Côté et al. 2019), the games we use are more complex, making them even challenging for the human players. These games have diverse themes and genres. For example, in the game "Ludicorp". the player appears to be a modern citizen being located in an office building. In another game "Zork1", the player is put into a fantasy world that she/he has find the treasure in the mazes while escaping from the troll. Some games contain nonstandard actions (e.g.. the spells), which are unlikely to be understood by the language model pre-trained with commonsense knowledge.
Aosless the game statistics calculated from the walkthrough of each game. The Avg Actions per Step refers to the average number of valid actions available at each step of the game. The Walk- through Length represents the minimum number of steps required to complete the game optimally, showing the shortest possible solution. The Avg Steps per Reward measures the average distance between two reward-trige triggerring steps, reflecting how frequently rewards distributed throughout the game. The Max Steps per Reward indicates the maximum number of steps a player might take between rewards, highlighting the sparsest distribution of rewards. Finally, the Max Score rep- resents the highest possible score an agent can achieve in the game.
Avg Actions Step
Walkthrough Length
Avg Step Reward
Max Step Reward
Max Game Score
Zorkl
15.96
396
9.12
51
350
Deephome
19.47
327
5.90
53
300
Ludicorp
14.52
364
3.69
45
150
Pentari
49
6.43
Detective
7.16
51
1.96
360
Library
7.73
52
3.67
6
30
Balances
23.18
122
13.44
54
50
Temple
15.25
182
21.38
35
Zhuu
33.96
84
4.53
Table 6: Game statistics on text-based games from Jericho benchmark.
B IMPLEMENTATION DETAILS
MC-DML. We set the discount factor to 0.95 and the number of simulations to 50 multiplied by (A)).). We set Cpuct 50. Specifically, it set 20 for the games Deephome and Library. and to 200 for the game Detect ive. The above configuration follows the work of Jann et al. (2020). We set dmin 10, dmax 30, and the step increment Ad to 20. This configuration allows the algorithm to start with a conservative search depth and expand progressively when necessary. The LLM policy uses gpt-3 -3. 5 turbo-O0 01255 as the backend model with sampling temperature set to O. We query the LLM for the index of the optimal action and retrieve the log probabilities for the top 20 tokens at that index. For absent actions, we assign a log probability -10. These log probabilities are then normalized using softmax with temperature of 5. The in-trial memory is set to (o--1, 4,-1, The size of the cross-trial memory K set to 3.
LLM agent. For the LLM agent baseline, we use gpt 1.tt-3.5 turboooooo--o 0125 with sampling temperature of 0.1. The agent selects actions based on the highest probability from the model's output. However, we observe that the agent often enters loops due to ineffective exploration. To address this, if the agent repeats the same action in the same state five consecutive times, it switches to random action.
Reflection agent. For the Reflection agent baseline, use the same settings the LLM agent baseline but equip it with a cross- cross- memory module, allowing the agent to learn through reflection and tndal-and-error. Specifically, the LLM plays multiple rounds of the game, generating reflections the end of each round, which are then used as input for the next round. In our experiments, we limit the number of reflections to 3.
14

---

Published as a conference paper at ICLR 2025
C LLM PROMPTS
In this section, we provide the prompts used for action value estimation by the LLM, as well as the prompts used for reflection.
C.I PROMPTS FOR ACTION VALUE ESTIMATES
You are player in text-based adventure game. Your task is to evaluate and select actions that promising based the given context.
Your memory of playing this game previously is: 1ANGS__RRRAI_MMOOO You are now facing the follewing state: (EN_TRY)__MMMORT
Considering the current state and previous memories, please select the action most worth exploring from the following list: WVLID_ACTIONS
Respond by providing
index of the action only. Your response should be a single integer, without any extra formatting, spaces, punctuation, or text.
C.2 PROMPTS FOR REFLECTION
You will receive a log of unsuccesstul gameplay:
(TRAJECTORY]
Please briefly identify the reascns for failure and provide suggestion for improvement the next round. not summarize the gameplay trajectory; respond with your suggestion. in single sentence.
D TRAJECTORY EXAMPLE
[OBS] mining center This the main center all of the mining that takes place in beephone. To the east are coal mines, rarely used because of the nature of the generator, and to the south are the ore mines, used for simultaneously mining all kinds of metal ores from the deeps of the mountains. To the north is the main street. [LOOK] This the main center of all the mining that takes place Deephome. To the east are coal mines, rarely used because of the nature of the generator, and to the south are the ore mines, used for simultancously mining all kinds of metal mountains, To the north the main street. [INV] You are carrying: king's s order warning note a lantern (providing light)
from the deeps of
[ACTION] east [REWRRD] 1
GAME SCORE| 15
[OBS]
coal mines The coal mines are rather stark, for by the time the dwarves moved into Deephome, they had already developed the technelogy to use the world's core
energy, what they called geothermal power. The rock on the wall looks like it has been dug in some, however. The mine opens up to the mining center to the west. [LOOK] coal mines The coal mines are rather stark, for by the time the dwarves moved into Deephome, they had already developed the technelogy to use the world's core for energy, what they called geothermal power. The rock on the wall looks like it has been dug in some, however. The mine opens up to the mining center to the west. [INV] You are carrying: king's order warning note lantern (providing light)
15

---

Published as a conference paper at ICLR 2025
[ACTION] push coal to warning [REWARD] 10
[GAME SCORE] 25
[OBS] You open the generator furnace. [LOOK] city generator Here is large wonder of technology and ingenuity. The generator
the
natural heat of the world, having pipes that lead all the way to near the core of the world. However, it requires a small fire to start the process of activating the power. The generator itself is large and black, having small furnace in the very front. doorway leads to the west, out into the street. [INV] are carrying: king's s order a warning note - a lantern (providing light) [ACTION] turn lever on [REWARD] 30
[GAME SCORE] 56
[OBS]
extreme south main street [LOCK] remr south main street Only people working for the city actually ever come down here, unless tour. few mining operations 1ic to the south of this the
they
road, and to the east is the main generator for the city. southwest, there is doorway leading to the city water works. The street ends here but continues to the north. [TNV] You are carrying: king's s order
warning note lantern (providing light)
|ACTION] get in southwest
[REARRD]
[GAME SCORE] 57
water works This room is filled with scores and scores of pipes, leading this way and that, all connecting with the walls point or another. à large wheel sticks out of a rather significant pipe and there is a sign above it. To the northeast is the main road. (LOCK) water works This room is filled with scores and scores of pipes, leading this way and that, all connecting with the walls one point or another. A large wheel sticks out a rather signifi ficant pipe and there is a sign above it. To the northeast is the main road. [INV] You are carrying: king's order - warning note lantern sproviding light) [ACTION] get in northeast
[OBS]
[REWARD] 0
(AME SCORE] 57
[OBS]
extreme south main street [LOOK] extreme south main street Only people working
the city actually ever come down here, unless they are on a tour. A few mining operations 1ic to the south of this road, and to the east is the main generator for the city. To the southwest, there is doorway leading to the city water works. The street ends here, but continues the north. [INV] You carrying: king's s order warning note lantern (providing light)
[ACTION] north [REWARD] 0
[GAME SCORE]
intersection [LOOK] intersection Here meet the two major roads of the city proper: Main Street, which rurs north and south, and the Royal Road which runs from here. To the is guard post with sign next to it. [INV] You are carrying: king's order warning note [ACTION] west [REWARD] 1
[OBS]
lantern (providing light)
[GAME SCORE] 58
[OBS] east royal road This road leads east to an intersection and west toward the castle of the regional lord. It is lined with high walls and gates that open to Lesser nobles' palaces. There is gate open
16

---

Published as a conference paper at ICLR 2025
the south. [IOOK] east royal road This road leads intersection and west toward the castle of the regional lord. lined with high walls and gates that open to lesser nobles' palaces.
There is gate open to the south. [INV] You are carrying: king's order warning note - lantern (providing light) [ACTION]
[REWARD]
GAME SCORE] 59
This road is quite beautifui, decorated on its sides with fluorescent mosses that feed on the minerals in the stones that line the sides of the roads. Somehow, the mosses do not leave their designated stones. High walls on both sides make the street feel more like a hall than an open passageway, and gates leading palaces break up the monotony of the stone. single gate open to the north. The road continues east and to the west the outer court of the lord's palace. [IOOK] west royal road This road is quite beautiful. decorated on its sides with fluorescont mosses that feed on the minerals in the stones that 1ine the sides of the roads. Somehow, the nosses do High walls on both sides make the street feel more like a hall than an open passageway, and gates leading to palaces break up the monotony of the stone. A single gate is open to the north. The road continues east and to the west is the outer court the Lord's palace. [INV] You are carrying: king's order a warning note lantern (providing light) [ACTION] west
[OBS]
west royal
leave their designated stones.
[REWARD] 1
[GAME SCORE] 60
[OBS]
outer court High walls surround this court, and a long pathway leads fron. the gates to the cast to the throne room to the west. Most of the palace is locked securely, but the courtyard is still open, for the very reason of allowing dwarven reclaimer to have access to the throne room. [LOOK] outer court High walls surround this court, and long pathway leads from the gates to the east to the throne room to the west. Most of the palace is locked securely, but courtyard is still open, for the very reason of allowing dwarven reclaimer to have access to the throne room. [INV] You are carzying: king's order a warning note a lantern (providing light)
[ACTION] west
[REWARD]
[GAME SCORE] 61
[OBS] throne room The throne room is much more simple and quaint than you might expect. A large tapestry hangs behind the throne itself and Long red carpet Leads from the great doorway to the east, and except for the throne itself, there is little here. You can see a luminous spirit here. [LOO]] throne room The throne roon is much more simple and quaint than you might expect. A large tapestry hangs behind the throne itself and long red carpet leads from the great doorway to the east, and except for the throne itself, there is little here. You can see a luminous spirit here. [INV] You are carrying: king's order warning note lantern (providing light)
[ACTION] cast
[REWARD] 0
(GAME SCORE] 61
[OBS] outer court [LOOK] outer court High walls surround this court, and long pathway leads from the gates to the east to the throne room to the west. Most of the palace is locked securely, but the courtyard is still open, for the very reason of allowing dwarven
17

---

Published as a conference paper at ICLR 2025
reclaimer to have to the throne room. [INV] You are carrying: king's & order warning note lantern (providing light)
[ACTION] east [REWARD] 0
[GAME SCORE] 61
[OBS] west royal road [LOOK] west royal road This road is quite beautiful, decorated on its sides with fluorescent mosses that feed the minerals in the stones that 1ine the sides of the roads. Somenow, the both sides make passageway, and gates leading to palaces break up the monotony of the stone. A single gate is open to the north. The road continues east and to the west is the outer court of the lord's s palace. [INV] lantern
not Leave their designated stones. High walls street feel more like a hall than an open
You carrying: king's order warning (providing light) [ACTION] north
[REWARD] 1
[GAME SCORE] 62
[OBS] greater nable's palace This palace rises high, probably consisting of ten or twelve floors, all carved out of the mountain. The walls are covered with beautiful murais and tapestries, the floors are mosaies with fountains and baths interrupting them. Many gold and silver objects sit here gleaming in the light. It clearly a wealthy abode. To the south is the gate that leads to the main street. [LOCK] greater noble's palace This palace rises high, probably consisting of ten twelve floors, all carved out of the mountain. The walls tapestries, the floors are mosaics with fountains and baths interrupting them. Many gold and silver objects sit here gleaming in the light. It is clearly wealthy abode. To the south is the gate that leads to the main street. [INV] You carrying: king's order warning note lantern (providing light)
covered with beautiful murals and
[ACTION] south
[REWARD] 0
[GAMK SCORE] 62
[OBS] west royal road [LOOK] west royal road This road is quite beautiful, decorated on its sides with fluorescent mosses that feed the minerals in the stones that line the sides of the roads. Somehow, the mosses do not leave their designated atones. High walls on both sides make passageway, and gates leading to palaces break up the monotony of the stone. A single gate is open to the north. The road continues east and to the west is the outer court of the lord's palace. [INV] You carrying: king's order warning (providing light)
street feel more like a hall than an open
lantern
[ACTION] east [REWARD] 0
[GAMK SCORE] 62
[OBS] east royal road [LOOK] east royal road This road leads east to an intersection and west toward the castle of the regional lord. It is lined with high walls and gates that open to lesser nobles' palaces. There is gate open to the south. [INV] You carrying: king's order warning note lantern (providing light)
[ACTION] south [REWARD] 1
[GAME SCORE] 63
[OBS] lesser noble's palace This palace is quite beautiful, with many mosaics on the floors and several tapestries hanging on the walls. The columns that support the building are etched with pictures from dwarven lore and exquisitely done. However, despite the
18

---

[OBS] dwarven library The library quite large: floor after floor books rises before you, clinging to the walls of this circular room. Stairs lead the middle, spiraling along and pausing at landings at each flcor. At the very center of the building you can look straight up and west. You can see Fresto's Exorcisms and Tasty Hors d'uuvre and Leshosh' s Encyclopedia here. [LOCK] dwarven library The library is quite Large: floor after floor of books rises before you, clinging to the walls of this rcular room. Stairs lead up the middle, the very Published as a conference paper at ICLR 2025
attention to beauty, there is little of anything of extreme here, for dwarven cranhip always ornate and admirable. The gates to the wall lead north to the royal road. [LOOK] lesser noble's s palace This palace is quite boautiful, with many mosaics on the flcors and several tapestries hanging the walls. The columns that support the building and are exquisitely done. However, despite the attention to beauty, there is little of anything of extreme cost here, for dwarven craft is always ornate and admirable. The gates to the wall Lead north to the roya- road. [INV] You are carrying: king's order warning note lantern (providing light) [ACTION] north
etched with pictures from dwarven lore
[REWARD] 0
[GAME SCORE] 63
[OBS] east royal road [LOOK] east royal road This road leads east to an intersection and west toward the castle of the regional lord. It is lined with high walls and gates that open to lesser nobles' palaccs.
There is gate open to the south. [INV] You are carrying: king's
order warning note lantern (providing light)
[ACTION] east
[REWARD] 0
GAME SCORE| 63
intersection [IOOK] intersection Here meet the two major roads of the city proper: Main Street, which runs north and south, and the Royal Road which runs west from here. To the east is a guard post with sign next to it. [INV] You are carrying: king's order warning note [ACTION] north
[OBS]
lantern (providing light)
[REWARD] 0
[GAME SCORE] 63
[OBS] s. main street [LOOK] s, main street This section the main street is detidated to quieter pursuits. To the east is the library, and to the west is the great temple. Main Street runs north and south from here. [iNv] You are carrying: king's order warning note lantern (provid light) |ACTION] east [REWARD] 1
[GAME SCORE] 64
the roof, and it mural. The exit is the
spiraling along and pausing at landings at each floor. center of the building you can look straight up and see the roof, and on it a mural. The exit is to the west. You can see Fresto's Exorcisms and Tasty Hors d' deuvves and Leshosh's Encyclopedia here. carrying: king's order warning note lantern
[INV] You (providing light)
[ACTION] west [REWARD] 0
[GAME SCORE] 64
[OBS] s. main streat [1OOK] s. main street. This section the main street is dedicated to quieter pursuits. To the east is the library, and to the west is the great temple. Main Street runs north and
19

---

Published as a conference paper at ICLR 2025
south from here. [INV] You carrying: king's order warning note a lantern (providing light)
[ACTION] west
[REWARD] 1
[GAME SCORE] 65
[OBS] great temple of Kraxis The ence great and shining temple of the one god, Kraxis, has been desecrated. Animal dung has been smeared the beautiful wall murals, many of the tiles of the floor have been ripped up and thrown across the room, and the altar, single large rock, unhewn by hands, has the statue of pig-headed god on top of it. The large main doors lie to the east. You can see a dark spirit here. [LOOK] great temple of Kraxis The once great and shining temple of the one god, Kraxis, has been desecrated. Animal dung has been smeared tiles of the floor have been ripped up and thrown across the room, and the altar, & singic large rock, unhown by hands, has the statue of a pig-headed god on top of it. The large main doors 1ic to the east. You can see dark spirit here. [INV] You are carrying: king's order [ACTION] east
the beautiful wall murals, many of the
warning note lantern (providing light)
[REWARD] 0
GAME SCORE] 65
[OBS] s. main street [IOOK] s. main street This section the main street is dedicated to quieter pursuits. To the east is the library, and to the west is the great temple. Main Street runs north and south from here. [INV] You are carrying: king's order warning note lantern (providing light) [ACTION] north [REWARD] 0
[GAME SCORE] 65
[OBS]
n. main street [LOOK] n. main street This was once bustling trading center with shops lining the sides of the streets, and smaller, less affluent merchants' booths sitting outside of the doorways, effering itens of lower quality rate. At this particulas point the street you see clothier to northeast, a bakery to the east, and a scrivener's to the west. To the north is the main hall, and the street zuns further south. [INV] You are carrying: king's order a warning note lantern (providing light)
substantially lower
[ACTION] get in northeast [REWARD] 1
[GAME SCORE] 66
[OBS] clothier this room you the tools of clothier's trade, generally dwarven-shaped wooden mannequins, table for cutting, and other tools scattered about. The exit is to the southwest. [LOOK] cletaier In this room you see the tools of clothier's trade, generally dwa rven-shapedd wooden mannequins, other tools scattered about. The exit is to the southwest. [INV] You are carrying: king's order a warning note - lantern (providing light)
table for cutting, and
[ACTION] get in southwest
[REWARD] 0
[GAME SCORE] 66
[OBS]
main street [LOOK] main strect This was once bustling trading center with shops lining the sides of the strests, and smaller, less affluent merchants' booths sitting outside of the doorways, offering items of lower quality at a substantially lower rate. At this particular point in the street you see a clothier to the northeast, bakery to the east, and a scrivener's to the west. To the north is the main hall, and the street runs further south.
20

---

Published as a conference paper at ICLR 2025
[INV] You carrying: king' order warning lantern (providing light)
[ACTION] west
[REWARD] 1
[GAME SCORE] 67
[OBS] scrivener's s This room is very plain and has half a dozen desks and chairs where the scriveners would sit and write documents for those who could not read. The exit is to the east. [LOOK] scrivener's This is very plain and has half dozer desks and chairs where the scriveners would sit and write documents for those who could not read. The exit is to the east. [INV] You are carrying: king's order warning note - lantern (providing light)
[ACTION] east
[REMARD] 0
[GAME SCORE] 67
[OBS]
main street [LOOK] main street This was once bustling trading center with shops 1ining the sides of the streets, and smaller, less affluent merchants' booths sitting outside of the doorways, offering itens of lower quality at a substantially lower rate. At this particular point in the street you see a clothier to the northeast, a bakery to the east, and a scrivener's s to the west. To the north is the main hall, and the street runs further south. IINV] You are carrying: king's order a warning note lantern (providing light)
[ACTION] east
[REWARD] 1
[GAME SCORE] 68
21


Reviews:
[{"review_id": "LqsZ6giFpZ", "reviewer": "Reviewer_ZBKR", "summary": "This paper introduces Monte Carlo planning with Dynamic Memory-guided Large language model (MC-DML) algorithm, which targets at bringing the language model understanding and reasoning capabilities into Monte Carlo Tree Search and RL. Specifically, the authors propose to add in-trial (which includes exploration and results in the current game), and cross-trial memory (which is reflections from previous experiences) into predictor UCT. Experimenting on 9 text-based games in Jericho benchmark, results show that the proposed MC-DML approach improve the performance over other RL-based, LLM-based, and MCTS-based approaches.", "questions": "1. Have you done ablation studies to study the impact of choosing k in the cross-trial memory? Why would 3 reflection be sufficient?\n2. How would the proposed method work in other RL setups beyond text-based games?\n3. How would the quality of the LLM (for both action prediction, and reflection) impact the overall performance?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper shows that adding LLM reasoning and planning capabilities, especially when augmenting with cross-trial reflections, can improve exploration performance in MCTS. This can be an interesting direction to study and may suggest good research results to people interested in how to make use of LLMs in RL exploration.\n2. This paper compares to previous RL-based, LLM-based, and MCTS-based methods, and shows that when combining LLM with MCTS RL methods, we would benefit from stronger reasoning and planning capabilities.", "weaknesses": "1. The main contribution in the proposed algorithm actually lies in introducing cross-trial reflection, as LLMs have been studies in the MCTS setup which is also mentioned by the authors. However, although the paper compares to LLM-based and MCTS-based baselines, there is no comparison to LLM + MCTS baselines, which seems to be the most relevant. \n2. Given that combining LLMs and MCTS has been well studies in both text-based games and other RL setups, the contribution of this paper might be relatively limited.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z5TOLdsS8Z", "reviewer": "Reviewer_jbdx", "summary": "This is a neat paper combining well known MCTS with LLM memory. More specifically, MCTS will be guided by past failure trajectories as learned by the LLM. The idea is demonstrated in a number of text games.", "questions": "- can you apply this idea for other MCTS-RL frameworks?\n- can you apply this idea for other tasks which require planning, such as robot task completion, complex math problem solving, etc.?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This is a neat and effective idea. It may also apply to other domains such as robot planning or math problem solving. It is still not clear how the formalism is incorporated into POMDP.", "weaknesses": "The method requires the LLM to remember these failure trajectories. Long term memory can be one limitation, or for complex tasks it may be needed to do some sort of RAG to retrieve the relevant memory, and this can introduce noise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BKa4vfZUGo", "reviewer": "Reviewer_djd9", "summary": "This paper introduces a new planning and learning algorithm called Monte Carlo with Dynamic Memory-guided Large language model (MC-DML). This method leverages the language understanding capabilities of LLMs and the exploratory advantages of tree search algorithms. The approach is similar to Predictor UCT: actions are selected based on the Q-value function plus a probability distribution over valid actions p(a|s). In this work, p(a|s) is not trained but produced by a prompted LLM. Applied to Jericho text-based games, GPT3.5 is used to generate p(a|s) at each game state based on both an in-trial memory (M_i = sequence of past observations & actions), and a cross-trial memory (M_c = a collection of LLM generated reflections based on previous failures). In addition, the Monte Carlo search used in this work applies dynamic pruning to save search time and favor exploration.\nExperimental results on 9 Jericho games show that MC-DML yields better performance when compared to RL-based, LLM-based, and other MCTS-based methods. Ablation studies show the importance of the cross-trial memory M_c.", "questions": "Q1. Do you take into account the next step rewards (when present) during the Monte Carlo Tree Search? If so, how? If not, why not?\n\nQ2. What is the average input token length of your two prompts? Given that they include trajectories, they may be longer than the 4191 context length limit of gpt3.5. Do you compress the trajectories in some way? Since states are made of the game state transition message + the description of the room + the description of inventory, sequences of (s, a) could become lengthy. For instance, consecutive room and inventory descriptions can be the same over multiple time steps if actions do not move the player or change its inventory.\n\nQ3. Do you include the initial description of the game as part of the state? Often the opening message of the game will give valuable information about the goal/objective of the game.\n\nQ4. When doing MCTS rollouts/simulations, how do you sample actions? Is it random or do you use an informed process during simulation as well?\n\nQ5. Why did you select these 9 Jericho games? Could you report performance across all 33 games in the appendix?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper proposes a novel method to combine the advantages of LLMs and tree-search algorithms. The idea of including both in-trial and cross-trial elements inside the decision-making process of choosing the next action is an interesting and effective technique that can inspire future work.\nThe paper is well written, the experiments correspond to the research questions, and are well motivated.", "weaknesses": "**W1.** Some technical aspects are not always clear, see below and the list of questions for things to clarify.\n\nThe paper could benefit from a more detailed explanation of the way each LLM prompt is constructed. In particular what is inside the {TRAJECTORY}, {IN-TRIAL MEMORY}, and {CROSS-TRIAL MEMORY} variables. An example in the appendix could help the reader understand what information the LLM has access to when making its decision.\nA discussion about how the lengthy game trajectories are handled could also clarify some technical aspects.\n\n**W2.** The Jericho benchmark has been used in many previous works, additional baselines could be included in the experimental results to make them more significant.\nIn particular MPRC-DQN & RC-DQN from Guo et al. [1], Bike+CBR from Atzeni et al. [2], and LDTs from Gontier et al. [3]\n\n[1] Guo, X., Yu, M., Gao, Y., Gan, C., Campbell, M., & Chang, S. (2020). Interactive fiction game playing as multi-paragraph reading comprehension with reinforcement learning. arXiv preprint arXiv:2010.02386\n\n[2] Atzeni, M., Dhuliawala, S., Murugesan, K., & Sachan, M. (2021). Case-based reasoning for better generalization in textual reinforcement learning. arXiv preprint arXiv:2110.08470.\n\n[3] Gontier, N., Rodriguez, P., Laradji, I., Vazquez, D., & Pal, C. (2023). Language Decision Transformers with Exponential Tilt for Interactive Text Environments. arXiv preprint arXiv:2302.05507.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)