Policy Learning from Tutorial Books via Understanding, Rehearsing and Introspecting
Authors: Xiong-Hui Chen
Ziyan Wang
Yali Du
Shengyi Jiang
Meng Fang
Yang Yu
Jun Wang
Venue: NeurIPS 2024 oral (2024)
Keywords: Reinforcement Learning
Large Language Model
Agent
Retrieval Augmented Generation
Primary area: reinforcement_learning
TL;DR: We present Policy Learning from Tutorial Books (PLfB), a new topic to train policy networks using text resources. Our implementation combines advanced LLM and RL techniques, achieving strong results in Tic-Tac-Toe and Football game.

Abstract: When humans need to learn a new skill, we can acquire knowledge through written books, including textbooks, tutorials, etc. However, current research for decision-making, like reinforcement learning (RL), has primarily required numerous real interactions with the target environment to learn a skill, while failing to utilize the existing knowledge already summarized in the text. The success of Large Language Models (LLMs) sheds light on utilizing such knowledge behind the books. In this paper, we discuss a new policy learning problem called Policy Learning from tutorial Books (PLfB) upon the shoulders of LLMs’ systems, which aims to leverage rich resources such as tutorial books to derive a policy network. Inspired by how humans learn from books, we solve the problem via a three-stage framework: Understanding, Rehearsing, and Introspecting (URI). In particular, it first rehearses decision-making trajectories based on the derived knowledge after understanding the books, then introspects in the imaginary dataset to distill a policy network. 
 We build two benchmarks for PLfB~based on Tic-Tac-Toe and Football games. In experiment, URI's policy achieves at least 44% net win rate against GPT-based agents without any real data; In Football game, which is a complex scenario, URI's policy beat the built-in AIs with a 37% while using GPT-based agent can only achieve a 6\% winning rate. The project page: https://plfb-football.github.io.

Policy Learning from Tutorial Books via Understanding, Rehearsing and Introspecting
Xiong-Hui Chen -henl-*,ZZyyn Wang"*, Yali Du o Shengyi Jiang",
Meng Fang". Yang Yu Jun Wang National Key Laboratory for Novel Software Technology, Nanjing University, China School of Artificial Intelligence, Nanjing University, China 2 Cooperative AI Lab, Department of Informatics, King's s College London Al Centre, Department of Computer Science University College London 4 University of Liverpool 'The University of Hong Kong
Abstract
When humans need to learn new skill, we can acquire knowledge through written books, including textbooks, tutorials, etc. However, current research for decision- making, like reinforcement learning (RL), has primarily required numerous real interactions with the target environment to learn a skill, while failing to utilize the existing knowledge already summarized in the text. The success of Large Language Models (LLMs) sheds light utilizing such knowledge behind the books. In this paper, we discuss a new policy learning problem called Policy Learning from tutorial Books (PLfB) upon the shoulders of LLMs' systems, which aims to leverage rich resources such as tutorial books to derive a policy network. Inspired by how humans learn from books, solve the problem via three-stage framework: Understanding. Rehearsing, and Introspecting (URD. In particular, it first rehearses eeciisinnmaking trajectories based on the derived knowledge after understanding the books, then introspects about the imaginary dataset to distill a policy network. We build two benchmarks for PLfB based Tic-Tac- Tac-Toe and Football games. In the experiment, URI's policy achieves a minimum of 44% net winning rate against GPT-based agents without any real data. In the much more complex football game, URI's policy beat the built-in Als with a 37% winning rate while GPT-based agents can only achieve 6% winning rate. The project page: plfb-football, github..o....
Introduction
Humans can acquire new skills through various written materials that provide condensed knowledge, without the need for direct interactions with the target environment. In contrast, traditional policy learning paradigms. such as reinforcement learning (RL) [1] primarily rely on trial and error [2-4]. Despite recent advances in offline RL [5, 6] showing that policy improvements can be achieved simply by using pre-collected data, in fields such embodied AI for robotics [7], the process of collecting large amounts of decision- impossible. Therefore, a question arises: similar to how humans learn, can policy learn from non-real data, e.g., from tutorial books?
data remains costly and, at times, impractical or even
We argue that the recent successes of Large Language Models (LLMs), such GPT-4 [8], and LLaMA [9] already demonstrated the potential to learn from textual content. However, current studies focus on using LLM directly for decision making [10, 11] or integrating them as auxiliary modules in other machine learning workflows [12-14]. In this study, we introduce a novel topic built upon the shoulders of LLMs' systems: Policy Learning from Books (PLiB). PLIB aims derive a policy
Equal Contribution. + Work done during Xiong-Hur Hui Chen's visit King's College London. 9 Corresponding: yali.du kcl.ac.k, yuy@nju.edu.coo @nj..edu.cn.juu. wang uck.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(A) Policy Learning from Real Data
(B) Policy Learning from Tutorial Books
collect
RL
write
Real World
Dataset
Policy Network
Human
Tutorial Book
Policy Network
(C) The three-stage learning methodology for Policy Learning from Tutorial Books
Stage Understanding
Stage 2
Stage 3
Rehearsing
Introspecting
extract
genterate
disti
Knowledge
marinerry Duteset
Database
Simulator Information
Figure 1: (A-B) A comparison between the problem of policy learning from tutorial books (PL(B) and from data; (C) An illustration of the understanding, rehcarsal, and introspection for PLfB.
network directly from natural language texts, bypassing the need for numerous real- woold interaction data, as shown in Fig. (A--B). This can be viewed as a further step towards enabling more resources for policy learning and also more generalized form of offline RI. problem, which uses textbooks to learn a policy offline. The essential challenge of PL.fB comes from the inevitable large modality gaps between the text space, which includes the knowledge related to recision-making. and the policy network space, which formulates the parameters of the policy function.
To realize PLIB, inspired by human learning processes, we propose a three-stage learning method- ology: understanding, rehearsal, and introspection (URI), which is shown in Fig. 1(C). For under- standing, it first extracts knowledge from books to form knowledge database; then it rehearses imaginary decision -aaiing trajectories with the help of the knowledge retrieved from the database; finally, it introspects on the imaginary dataset to distill policy network for decision-making. We present the first practical implementation of URI, employing L.LMs to convert book paragraphs into pseudocode for policy, dynamics, and reward functions. This pseudocode forms a code-based knowledge database, which is used to generate an imaginary dataset based on retrieval augmented generation techniques [15]. A policy learning technique inspired by offline RL [6] is then applied to distill a policy network that addresses inaccuracies the imagined actions, rewards, and states.
In the experiments, we build two benchmarks from Tic-Tac-To Toe and Football games for PLfB. We first validate URI on Tic-Tac-Te Toe game, where the tutorials are manually constructed to cover the complete knowledge for gpcision-makings The results show that our method achieves at least 44% net win rate against GPT-based agents without any real game interaction. In addition, we build testhed based on football tasks, which is popular in recent studies [16] as difficult decision- task, and focus on policy learning from football tutorial books, which naturally contain condensed knowledge, especially information closely related to football skill acquisition, the environment and dynamics of football games, and ways to evaluate football behaviors. We collect football tutorial books from RedPajama [17]. The policy is deployed in the Google Football simulator [18] directly. Our agent controlled by the policy network could beat the built-in with a 37% winning rate on average while using GPT as the football agent only achieve 6% winning rate. The experiments demonstrate the scalability of our approach from simple board games to complex scenarios.
2 Related Work
LLMS have demonstrated remarkable potential in achieving human-level ability various tasks, sparking a surge in studies investigating LLMM-based autonomous agents. Based on the different roles of LLM in the agents, there are two types of work. Firstly, LLM can be used as an actionable agent directly [10]. Early works [19-21] prefer open-loop plans and hard-code search algorithms to improve the long-term planning ability of LLM. Closed-loop planning [22-24] leverages environmental feedback, thereby facilitating more adaptive decision matiiiinnmmking.... Secondly, LLMs can be used assist or accelerate the learning of agents. The form of assistance can be quite diverse. It can generate adapter to convert human instructions into structured inputs [36]. In our work, LLMs play a different role compared all the above-mentioned types. I.LMs are to understand the knowledge in the books, rehearsing the decision-makerr process to derive an imaginary dataset based on the knowledge. The policy to control the agent is simple neural network distilled from the imaginary data.
high-level plans [25, 26), (surrogate) rewards [27-32], transitions [33-35],

---

3 Preliminaries
Markov Decision Process (MDP) is defined as a tuple of M == (S, R. 7, po) [1] of the target environment, where S is state space, A is action space, T : x - A(S) is the transition function (A(-) is the probability simplex), R : S x A + [0, Rmax] is the reward function, Y E [o, 1) is the discount factor, and Po is the initial distribution over states. A policy π 8 ((A) describes the distribution of actions for each state.
Retrieval Augmented Generation (RAG) [15] has shown great ability to improve LLM's generation ability for seowledge-intensiine tasks. RAG enables LLMs to query external data sources to obtaín relevant information before proceeding to answer questions generate text. Formally, given an LLM M, the retrieval module R(x, (v:) }) takes a textual query x as input, finds the most relevant textual segment y² by finding the most similar E(y) from the knowledge database E {v:} compared to E(z). Augmentation transforms the content into an additional input for the LLM's generation. In this paper, RAG is used to implement the URI algorithm. We use standard RAG techniques as the retrieval module, i.e., cosine similarity matching [37, 38], within the implementation of URI. In particular, we use GPT embedding to index the texts in the database and also the current query, then use cosine similarity to find the top-n matching data for downstream tasks, ie., M(x, [v ..., V.l).
Offiie RL [39] addresses the problem of learning policies from pre-collected dataset D. Existing studies can be classified into two categories: model-free and model-based methods. Model-free [5, 40- 45] methods learn a policy directly from the dataset 'D through specially designed conservative policy learning loss which usually aims to avoid policy taking actions unseen in D. Model-based offline algorithms [6, 46-52] first estimate the dynamics and reward model 7 and it from the dataset D. The policy is learned by iterating with I and R. In the process, specially designed penalties R adopted to discourage the policy from visiting states where the model predictions are of high uncertainty.
4 Problem Formulation of Policy Learning from Tutorial Books
The goal of Policy Learning from Tutorial Books (PLiB) is to use an algorithm ALG to learn a policy A ALG((B (Ml) from textual books B, which can maximize the cumulative discounted reward ()() = Σ, Eu R(s,, 0t), where the book is defined as N₂ segments b, divided by paragraphs, i.e., B = (b), (b,...,bb; ..., ... bN. ), and |M] denotes the brief textual descriptions of the structures of MDP, including the descriptions of state space, action space, the task we faced, and the initial state distribution. [M] is inevitable to be used since B only gives general knowledge of the target environment while |M| defines the specific information, e.g., the exact format to interact with the simulator. Unlike standard RL, during the learning process of PLíB, the algorithm has no access to the environment. To ensure the fcasibility of extracting non non-trivial policy, we make a mild assumption that B contains the descriptions of transition functions (7', and reward functions (ft that can be regarded as the approximations of T and R, respectively. These descriptions could range from simple game rules to complex natural language paragraphs, depending on the books. It also contains descriptions of relatively high-qualin behavior policies.
5 Understanding, Rehearsing, and Introspecting
In this section, we first present our motivation for the proposed three-stage framework in Sec. 5.1. After that, we explain how we implement those stages in Sec. 5.2, 5.3, and 5.4 respectively.
5.1 Motivation of the Three-Stage Framework
As shown in Fig. 1, the methodology to solve PLIB consists of three major stages: understanding, rehearsing, and introspecting (URI). The motivation of these three modules can be easily understood by reviewing how humans learn from books: as humans, we first extract knowledge from books to extend our knowledge database. Then, to acquire skills, we often rehearse the possible consequences of applying the skill in our mind, integrating knowledge from the books with our prior life knowledge. Finally, we will re- re-examine the steps we took in our minds and think how we could have done better until we confirm that we know how to execute the real world. In this paper, we mimic the above three steps via the following modules: Understanding module takes paragraphs of books input and forms a knowledge database organized in podudo-code. Rehearsing module iteratively takes the current imagined state as input and outputs the action, next state, and reward with guidance from the
3

---

Knowledge Plece
(A) Staget: Understanding
(C) Stage3; Introspecting
teettal selicoianee
extracted knowledge
Knowledge Aggregator
Offine Policy Learning from Imaginary Datasct
K" [K]
Mage
in the bua inrrustion
policy
be trmmoout cornig
aggregated knowledge
#*
she suats
Khowlodge Extractor
X = {K t.... xxp)
Imaginary Dataset
Bi de Resl
or aaals actinnl
Mes
Dine
Tutorial Books
Stage2: Rehearsing
Player in possession of the ball. Carrying ball avaiiable...
for dynamics Kr
instantiated knowledoge
as
R
Fri Roos
Knowlodge Rerrieval Module
My, M.
by: Together with the general principles, the operational ones should also
(3)
for rewarrds
query
de
(2)
Maups
update
Simulator Information MM|
instantittdd knowledege
LLMs as M,,
About the State Space for state, firtt score
for policios
Rusqe
Kit
Mavi
About the Action Space we have discrete action, for idle,
query
(1)
The knowledge extractor and aggregator modules process paragraphs from books to form structured knowledge database organized as pseudo- code. (B) Rehearsing: Using the knowledge database, the simulator generates and iterates through imagined states, actions, and rewards to create an extensive imaginary dataset. (C) Introspecting: The introspection module refines the policy network by evaluating and correcting errors in the generated states, actions, and rewards to ensure accurate and effective policy implementation. The pscudocode of the pipeline is in Appendix F.8.
Figure 2: The URI pipeline consists of three major stages: (A) Understanding:
database. After gathering this imagined content to form a dataset, Introspecting module distills a policy network, which should consider errors of generations of state, action, and rewards.
We give the overall architecture of our implementation of URI in Fig. A detailed description of how they realize these functionalities will be discussed in the following.
5.2 Book Content Understanding
The understanding module is responsible for extracting knowledge := {K..... K......... KNx } from the books B, where K; denotes one piece of knowledge. For we humans, the knowledge is naturally stored in our brains. For machines, the first question is what should be the appropriate format to represent the knowledge. Since LLM has shown superior performance in code generation and codes themselves as interpretable, compact, and expressive languages [53], we also choose it as the basic format of knowledge representation. Different from previous works [10] which directly asked for runnable code for the downstream tasks using, we do not execute this code. Instead, we just use it as more flexible and abstract description of knowledge which still maintains a rigorous control flow. As shown in Fig. 2(A), a knowledge extractor module is used as the first step. The knowledge extractor an LLM-injected model. We iteratively ask K2- = Mext (b:) to examine whether paragraph is related to the decision decision-making elements, i.c., policy functions, reward functions, dynamics functions, and how to represent them as pscudocode.
Moreover, humans often learn by repeatedly reading the texts across pages and even books, updating existing knowledge with newly learned ones, and summarizing them into more general and abstract forms. A similar procedure is achieved by the knowledge aggregator module, shown in Fig. 2(A), as the second step of understanding. The knowledge aggregator also an LL.--ijjecte. model. We iteratively ask Mags ([K,, Kj, Ki....) to aggregate relevant knowledge among different segments based on a similarity estimation of the embedding model E. Formally, let X := (KP K
be the paragraph-wise knowledge extracted by Maxt- For the j-the iteration, [Kg+1, _Kt+1 [/,+,,..]-=
Magg ([K], KY,KK,,... ..),.. where [K], K/. K......] is from KC' by selecting the most similar Nags pieces of knowledge by comparing their cosine similarity under the embedding model E(K) of GPT and o, p, i, 1 n here denote the indexes. [K Kit!" ....] are then added Ki+1 The iteration will stop || >= [K'), where IC is the pieces of knowledge at j-th iteration. After iteration stops, the remaining knowledge pieces constitute the knowledge databases by splitting them into cenamics-relatedd knowledge Kr, reward-relate knowledge KR., and pelicy-related knowledge IC, for later modules to use. More details are provided in Appendix F.1.

---

5.3 Knowledge-based Rehearsing of Decision-Making
When humans develop rough understanding of new skill from knowledge K. they will usually imagine what choice they will make given certain situations and what are the possible consequences in their minds. This rehearsing procedure would help humans correct apparent mistakes and consider better actions for long-term benefits [54, 55].
We implement closed-loop generation process involving the LLM-injected dynamics function Mr. reward function MR. and policy M,. as depicted in Fig. 2(B). This approach resembles the model rollout in traditional model-based RL, where the policy, reward function, and dynamics function are represented by LLMs. Given the current imagined state &t. the LLMs M2 first predict the most plausible action âz Subsequently, Mr and MR generate the next state $141 and the associated reward fr based on this action and the current state. In the process, a knowledge retrieval module is involved in selecting relevant knowledge pieces enhancing the LLM input with this information for LLM'' predictions, e.g., å = M2 (&,, R(s Kx)), fr = Mn (8, 0e R(s,, Kn)) and 8441 M7 âz, Rsst, )) The knowledge retrieval module includes the following two steps:
The fundamental problem of standard RAG techniques, i.e. embedding-vectoo similarity matching, in this scenario, is the modality gap between the query 8, and the knowledge K. Standard RAG approaches aim to identify information closely related to the query, while here we need to ask the most suitable knowledge to be applied ward/policy function that has the best predictions for the queried state and action. Standard RAG techniques tend to retrieve knowledge pieces that include similar text patterns the queried states and fail to find the best knowledge for predictions. To address this, we propose a knowledge scope retrieval method that includes simple yet effective preprocessing step to bridge the modality gap. In particular, we traverse all knowledge pieces K; E K. by iteratively sampling Rscope knowledge pieces In from the database K. combining them with simulator information [M] and using LLM {K²) Macope Macope ( {K:} nicope: M) to determine the preferred scopes K² for each piece of knowledge. K. is defined by the preferred subspace of state to use the knowledge. Then a standard RAG technique is applied to identify the most relevant knowledge scope Kÿ and its relevant knowl- edge Ky. Formally. ({K,), }. {K}])
Matee-ased Knowledge Scope Retrieval:
dynamics/re-
Recope (8, XC5), where KC is the scopes of the knowledge database. This method is effective for embedding models in retrieving the correct knowledge as the texts of keys and queries both are about the descriptions of states.
Post-Retrieval: Knowledge Instantiation: Predicting based on code knowledge requires the LLM's robust understanding of code. We refine this process employing LLM to instantiate the code K² <== Minet (8, [M] {K+}) based on the current state 8, the simulator information description [M and the knowledge {Ký} retrieved by Recope- Knowledge instantiation involves generating domain-specific pseudocode based on the knowledge coded in the retrieved information, tailored to the target environment's 's current state and action requirements.
Finally. three LLMS are involved to generate the imaginary dataset Ding- including à, = M2 (8,, Mins( (Bc, [M], Rscope (())),, ft = MR(s,, â,, Minst (st, [M], Recope (5, x)))) and 8 = MT (8t, ât, Minat [M], Recope (8, K)))), where KCS. K.S. and K.G. are the scope of the knowledge database Kz,, KR, and Ky respectively. To start the rollout of a trajectory, we need state as the initial state. It has several choices to achieve, c.g., sampled from S, generated by another LLM, or pre-collected a small number of real states from the target environments as part of simulator information, i.c., initial state distribution pa. Since the first choice might introduce unrealistic initial states in complex scenarios, in Football tasks, we opt for more practical approach by pre-collecting a small set of real states from the target environments for our experimental validation. The detailed setup is in the experiment section. Besides, during the rollout over H steps, we reuse instantiated knowledge to reduce the overload of LLM's callings. More details are provided in Appendix F.2.
5.4 Introspecting based on the Imaginary Dataset
The direct output LLM in the rehearsing stage can be sub-optimal or incorrect. We would like to re-examine the collected data and try to distill policy that can avoid the side effects. We can regard the data Ding, collected during the rehearsing stage as an offline dataset and apply offline RL algorithms to train an improved policy from the dataset. However, directly applying existing offline RI. algorithms over-simplifies the problem. Compared with the standard offline setting where only the behavior policy is sub-optimal, there is an additional misalignment in the data generated
5

---

during the rehearsal: the transition and the reward function estimated by the LLM are also inaccurate. Overlooking such inaceuracy would result in a policy exploiting the sub-optimal transition and reward function and cause performance degradation or even risky behaviors in the final deployment.
To solve the problem, in this paper, we adopt the Conservative Q-learning [5] as the base offline RL. algorithm, whose learning objective is as follows:
min max (Bg~Dimee a (a (a (8) (Q a)- Bg,a- [Q(8,)) (). R(x))
+ Bg (Q a) Bd (33,
where B. Q is the bellman update operator update the Q-value function [1], and the first term to learn a policy with conservative Q value [5]. As a solution introspecting from imaginary data, as shown in Fig. 2(C), we add the uncertainty of the reward and transition estimation as the regularization terms, RR and Ry over the original reward f output by the LLM MR. In practice, we adopt these regularization terms by applying them when we backup the Q%:
QG, = (RRR(S, - RT, à) + -lEg''DDam (mma (0 (Q(("("
where 8' D ing is to sample the next state given 8, â, DR and yrr are two hypeer-aramettrs to control the weighting of the uncertainty terms. Inspired by Model-based Offline Policy Optimization (MOPO) [6], the uncertainty is estimated by an ensemble of Nens Gaussian models and R, which is learned by maximizing logarithmic likelihood from the imaginary dataset. Then the uncertainty is estimated by Rx(s,a) = mI.,,el of (s, a) and Rr a) = max.g(I,..,.N...... of (s, a), where of and of are the i-th reward and dynamics model's predicted standard deviation for s, respectively. We call the solution of CQL with BJ the bellman update operator Conservative Imaginary Q-Learning (CIQL). This regularization is easy to implement and can force the policy to generalize better over the regions where the LLM outputs inconsistent next states and rewards.
6 Experiments
We build two benchmarks with increasing complexity to evaluate PLfB: a classical Tic-Tac- game with discrete states and deterministic rules, and the Google Research Football environment (GRF) [56] featuring continuous states and multi-agent interactions. In Sec. 6.1, we introduce the experimental settings for both environments. In Sec. 6.2, we analyze the performance of URI against various baselines. In Sec. 6.3 and 6.4, we examine the effectiveness of our knowledge extraction and retrieval mechanisms. Finally, in Sec. 6.5-6.8, we provide detailed analysis of the training process, including data generation, ablation studies, inference efficiency, and dataset visualization.
6.1 Experiment Setups
Tic- Vie-Tac-Te Game To build a proof-of-concept benchmark for PLIB, we first construct Tic- Tac- Toe (TTT) environment where players take turns placing their marks (X or O) on a 3x3 grid, aiming to form a line of three marks. This classical game serves as an ideal testbed for two reasons: (1) it has known optimal solution (minimax algorithm), allowing us to evaluate how close our learned policy to optimality: (2) due to its discrete and deterministic nature, we can collect complete set of optimal trajectories covering all possible game states. Based on these trajectories, we can use GPT to generate comprehensive tutorial texts that contain complete knowledge of game mechanics and winning strategies, thereby controlling for the impact knowledge incompleteness on algorithm performance. In our experiments, each policy plays as 'X' and moves first, with performance measured by win rate, draw rate, loss rate, and net win rate (win rate minus loss rate) across different opponents. For detailed game rules and setup, please refer to Appendix A.
Google Research Football [56] is physics-based 3D football simulator that supports the main football rules such as goals, fouls, corners, penalty kicks, and offside. Google Research Football (GRF) includes built-in Al bot for the opposing team, whose difficulty can be adjusted between 0 and 1. An illustration of the game can be seen in Fig. 7. We define three custom difficulty levels for our experiments on the l scenario: easy, medium, and hard. The difficulty levels differ in the bot's reaction time and decision-making defined in GRF, with higher difficulty corresponding to a stronger opponent. The major metric our experiment Goal Difference per Match (GDM), calculated as the average number of goals scored in all league matches minus the average number of

---

Table 1: Performance of different policies in Tic-Tac- Tac-Toe. Each policy plays as 'X' and moves first, tested
across 100 matches (50 for 1LM--assi methods).
aggen
LLM-RAG
Random Policy
Minimax-
Player X
D
L
W-L
D
L
W-L
W
D
L
W-L
W
L
W-L
LLM-av-agent
58%
2256
206
+38%
52%
68
42%
1.0%%
16%
4%
80%
-64%
LLM-RAG
72%
18%
10%
162%
52%
14%
34%
+18%
10%
18%
72%
-62%
(Oues)
80%
6%
14%
466%
62%
108
18%
444%
70%
12%
18%
452%
36%
54%
10%
+26%
Minimax (Optimal)
163
0%
4843
328
0%
+088
0%
+78%
0%
goals conceded per match. For URI, for each seed, we selected the average performance of the top checkpoints among all recorded checkpoints as the final performance. For more details about the GRF implementation, please refer to Appendix B.
Datasets We utilize three datasets in our approach. For Tic-Tac- we generate tutorial texts by having GPT analyze all possible game trajectories collected from optimal minimax policy (see Appendix A.2). For football, we collect the textbook dataset from the open-source book dataset RedPajama- IT [57], focusing on titles and abstracts related to football or soccer. After filtering, we obtain a curated set of ninety books closely aligned with the domain. For both environments, we need initial states to start our imaginary data generation process. In Tic-Tac-Toe, since we play as 'X' and move first, we always start from an empty board. In GRF, we sample 7,500 states from the rollout of a rule-based policy competing with the hard built-in AI. Due to limited resources, distill the policy, we imagine 75,000 transitions via URI, which is 10 times compared to the initial states.
Baselines We compare URI against several baselines across both environments. LLM-as-agent directly uses large language model (GPT 3.5) to output actions conditioned on the current state description. LLM-RAG enhances LLM-as-agent by retrieving relevant knowledge from the database extracted from the tutorial books, similar to the retrieval step in our rehearsing stage, but directly outputs the action without policy learning. For Tic-Tac- Toe, we additionally include Minimax-noise. which follows the optimal minimax strategy but with 30% random action selection to serve as a near- optimal baseline. For GRF, we include Rule-based-Al from the Kaggle Football Competition [58], which is hand-designed and serves as reference for the performance of hand-coded policies. Random Policy randomly chooses actions in both environments. For the implementation details about the baselines, please refer to Appendix D.
6.2 Policy Performance Evaluation
TTT Results We first evaluate URI's performance in the Tic-Tac-Toe environment. Tab. 1 shows the head-to-head match results between different policies, where each policy plays 'X' and goes first. URI demonstrates superior performance across all opponents, achieving net win rates (win rate minus loss rate) of +66%, +44%, and +52% against LLM-as-agent LLM-RAG, and Random Policy respectively. Morcover, when playing against Minimax-now to the optimal minimax strategy, URI maintains a positive net win rate of +26% while all other baselines suffer negative net win rates. This indicates URI's ability to learn effective strategies from tutorial texts even in this classical game setting.
which introduces 30% randomness
The results in Tab. demonstrate the superiority of the proposed URI approach compared to the baselines in the 11 vs 11 full-game scenarios of the GRF environment. The LLM-based agents, including LLM-as-agent and LLM-RAG, exhibit zero-shot task completion capabilities, outperforming the Random Policy. However, even with the use of RAG techniques, the best-performing LLM agent can only barely match the performance of the Medium-lewe built-in Al and fails to achieve any wins against the Hard-level built-in AI. In contrast, URI surpasses the performance of the baseline methods on all difficulty levels. Surprisingly, in the Hard task, URI achieves higher win rate than the Rule-based Policy. We believe this is due to URI's ability to leverage knowledge from domain textbooks and generate high-quality imaginary data for policy learning, enabling it to learn more adaptable and robust policies compared to the hand-crafted rule-based AI.
GRF Results
These results across both simple and complex environments highlight the scalability of URI in learning strong policies from well-defined board games to challenging scenarios.
6.3 Effectiveness of Code-based Knowledge Extraction and Aggregation
7

---

Table 2: Performance Comparison of Different Policies Against Built-in AI Levels in a GRF 11 11 settings. where the performance URI is averaged among three different seeds, LLg--s- LLM-RAG tested with 10 matches, and URI policy and random policy is tested with 40 matches.
Level
I.I. Li-M-as-agent
L.M--AG
Random Policy
URI (Ours)
Rule-hased-AI
Win
20%
30%
2%
37% 4%
70%
Easy
Draw
60%
60%
55%
57% 1: 4%
30%
Lose
20%
10%
43%
6% 1 4%
0%
GDM
0.2
-0.58
0.14
0.7
Medium
Win
0%
20%
2%
42% 1: 12%
70%
Draw
60%
60%
43%
50% 8%
30%
Lose
40%
20%
55%
8% + 4%
0%
GDM
-0.4
0.0
-0.76
0.43 + 0.24
0.7
Win
0%
0%
3%
32% = 14%
30%
Hard
Draw
50%
40%
43%
58%
70%
Luse
50%
60%
53%
10% 1. 7%
0%
GDM
-0.5
-0.6
-0.73
0.32 1 0.14
0.3
Win
6.7%
9.4%
16.7%
12.5%
2.3%
0.5%
40.3%
56%
Average
GDM
-0.30 + 0.22
-0.13
0.34
-0.69
0.08
0.38 + 0.05
0,56
According to Sec. 5.2, we perform an iterative process of code extraction and aggregation to
Number of Code Segments
465
understand the tutorial books and obtain ex- ecutable knowledge. As shown in Fig. 3(a),
205
480
350
220
for Tic-Tac- Toe, the initial extraction yields ap-
150
107
54
proximately 600 code segments which are ef-
fectively consolidated through the aggregation process. Similarly, in the more complex foot- ball domain (Fig. 3(b)), the number of code seg- ments for the dynamics, policy, and reward fune- tions decreases significantly over the aggrega- tion rounds. Through iterative aggregation, the number of code segments, decreases significantly in both environments, which helps condense the extracted knowledge into a more compact and coherent form, This consistent pattern across domains of varying complexity demonstrates the robustness of our knowledge extraction and aggregation approach.
Jd0 &
Aggregation Round
Figure 3: Knowledge Segment Aggregation.
6.4 Correlation between Knowledge Embedding and Current State Embedding
To validate the effectiveness of our code embedding method mentioned in Sec. 5.3, we conducted experiments in GRF task comparing it with two baselines: Vector Embedding, which directly compares the state and code embeddings, and Code Summary, which first summarizes the code segments before comparing the embeddings. We evaluate the top- 15 hit rate in a hand-crafted test dataset. The observations and actions in the dataset we used are collected by the tule-based policy interacting with the real environment, while the ground-truth codes are labeled by GPT and aggregated using similar way as in Sec. 5.2. As shown in Fig. 4(a), our method significantly outperforms the other two baselines on both pre-trained language models. The hit rate, which measures the proportion of relevant code segments retrieved, is consistently higher for URI across all three random seeds, demonstrating its robustness and superiority in improving the correlation between the state embedding and the code embedding. These results highlight the importance of learning dedicated matcher for effective code retrieval in our framework.
6.5 Tracing the Data Generation Process
In URI, it is crucial to guarantee that the imaginary data include states, actions, and rewards. The task is non-trivial since it has involve several transformations to align the gap between textual contents in books and decision-making trajectories in MDP. To demonstrate this process in detail, we use the football environment as a case study. We trace how M, outputs an imaginary action on an imaginary state š, which is actually collected at the start of a football game in GRF. We record all the intermediate outputs to trace the action generation process. The result is shown in Fig. 5. It is clearly shown that the output action "dribble" is fully supported by the logic in"has in"has_space" space" branch in K7, "assess_ forward decisions" function in K, and the paragraph of "Player in possession of the ball" in the raw content of tutorial books. However, we would like to point out that there also some
8

---

0.35
0.6
0.30
0.4
Hit Rate
0.25
Vector Em Embodding
0.2
Code Summary
0.0
0.15
URI (Ours)
0.10
0.2
URI rethaariinn
URR-wwo-pennllt
0.05
0.4
CQU-real-date
0.00
baalbbge-larr large-en text- enteedddbddgg ada-000
GDM
Win Rate
Draw Rate
Lose Rabe
(a)
(b)
Figure 4: (a). Comparison of different code retrieval methods on two pre- pre-trained language models. (b). Perfor-
mance comparison of different variants of the URT framework in the GRF This figure illustrates the average GDM, win, draw, and lose rates among the three levels of built-in Als. The error bars in the figure indicate the standard deviation from the mean performance for each configuration in three random seeds.
Carrossndiar Buce Cantaalin Bedaak sccetiics peicpa tenobrttion
1al asrass, payer aow sigpppt dtenndirr rean parnor
Carnepeadoor.c Arprerated Knontedger
'onalyer" ooyy'"" Instaniatedd
shong
player,
Sayar =1 (1
daliciie pr dindiy bour fin in tis deeen eat
R Line ortea and oppooeet poal ira.
1slerrce beerder
nittor difitee poes to tharente in better paciii Plgee 2 poeatio Antiivuilyy player to promass in saro 111, which riigtt vistan
posenoiinn on the bell and ooparents gool
rash www..oeeeee.e.. per sapo wett apa coneniiinn
tanu wos recovee's
ot regin rinton
stack opparaition Great adue sageois stackin ahaaticss is
calf rould phoolgio gold Unftl
ralch contoc domraatoo.ge goa). hal steeretiond
corance poal player midence benne
of loll.
Camping thoough oualiakle opain
ream eaporane:
(sheads
ees00s rerwad seeeeneerant stserwttess rame corted
Pertonning detanss ce oureerio
geel. shoolre
tropen
cangtgaall
oopreer torw)) oppowen
enants ser aeseenee" the play towerds the
14, cleayyyy...ooooooooooo raatund)
by ebservetirs
mam Thar
Corninng the hall loverts the soponentss poel
rstu lake shot"
oppanwrte
Purtoming detaoica toveards the opporents
rstum pasa bettrr powtiooedd jurmmrr ewer
rouu 15 a action shot.
concitiona for pusslunittance to
cauen aokken p4600)
cpes Inserates Deoeeeate temmenaee fearemutes cower)
calouitte devteretchhettssee teamrnitee
pam:
sicka leehe "Doee "ormant
reporet's
autton ids;
Tiae
Faahho,tkoy...e Paaayeer playeds refere
(1 retrieved by (RR) (X (RR)
ball positio, bharenatee, agg apaaennyy.
commggedding sonsarioe fiessiond by rindlaser)
(1,1%)
refare actioon Jtbr".
Suitnn paaye's Satte skyne Innnedt pudr Ntraal
Imagitary
generated M, (i)
"Mere Pland" Tall" Tmar hoper" sea'll "Rremten" "aar).
Imaaginary Actioa:
instantiard E₁ Mas (A, (X))
An example of the URI data generation process in the football game. The imaginary state 8 is collected from the real environment, which is 12 timesteps of football game, while the rendered image the corresponding sccnario generated by the simulator. The imaginary action is "dribble", where the logic supported by the "has_space" branch in Ky, Based on this, we bold the relevant information in the predecessor nodes and skip irrelevant information with the ellipsis
Figure 5:
examples that demonstrate LLMs having hallucinations in the generation, and the retrieved module might also miss the ground-truth piece of knowledge. More results are provided in Appendix H. These results indicate the necessity of introspecting in URI. The relevant ablation studies are in Sec. 6.6.
6.6 Importance of the components in URI
We validate the effectiveness of the rehearsing technique presented Sec. 5.3 and the CIQL method introduced in Sec. 5.4 through ablation studies. Specifically, we constructed the following variants of the URI framework in GRF task: (I) URI-Wo-rehearring which solves the policy without using the rehearsing dataset and relics a pre-collected dataset of 7,500 samples for offline RL policy training; (2) UURI-W/-penalty where penalties 7r and 7) are zero, same as the standard CQL for policy learning; (3) CQL-real-data, where we collect real data of equivalent scale to the rehearsing- generated chherrsing-genertted data using rule- based Al and apply standard CQL for offline RL policy training. The results shown in Fig. 4(b).
Firstly, URI-W/o-rehearsing demonstrates that without generating a substantial amount imaginary data through rehearsing, solely relying on offline RI. algorithms to train a policy with our pre-collected 7,500 samples is ineffective. The results indicate that cannot even beat the on Easy difficulty, though it still performs better than random strategy. URI-w/o-penalty underscores the importance

---

of penalizing the uncertain aspects of the outcomes generated from the imaginary data. Neglecting this penalty leads results worse than those of URehwii-re Finally, our method slightly outperforms -QLareal-ddaa. We attribute this improvement to the strategies inferred from prior knowledge by L.LMs, which are partially superior to built-in Al behaviors, or possibly because L.MSs generate more diverse data. The successful ablation results in such a challenging domain illustrate the considerable potential of PLIB.
6.7 Efficiency in inference
Lower cost of inference is one of the benefits for agents controlled by URI policies. We compare the inference time per action for different methods in the GRF task in Tab. 3. Our approach takes only 0.009 seconds on average to choose an action, which is significantly faster at least 300 times than using LLMs directly as agents (2.84 seconds) with retrieval-augmented generation (RAG) (4.12 seconds) in complex environments. This makes URI more suitable for real-time decision-making in the football simulator. The efficiency gain comes from the fact that URI distills the knowledge from the LLM into a compact policy network, which can be executed quickly without the need for expensive LLM inference at each step.
Table 3: Comparison of inference time per
action for different methods in GRF.
Time Cost(s)
LLMM-a-Agen LLM-RAG URI
2.84 = 0.71
4.12 = 1.46
0.009 + 0.0004
6.8 Imaginary Dataset Visualization
We visualize the imaginary datasets to analyze the quality of generation and uncertainty estimation. Here we present the results from the GRF, while the TTT results can be found in Ap- pendix G.I. We choose t-SNE [59] as the visualization method and project the imaginary dataset and a real dataset collected by the rule-based policy into 2-d space for comparison. The results are in Fig. 6. The "real data" marks the data collected by the rule-based policy, while "low-unc. data" and "high-unc. data" represent segments of the imaginary dataset categorized by their uncertainty scores Ry and R falling within the lower and upper 50% percentiles, respectively. The real data and the imaginary data follow a similar data distribution, which indicates the effectiveness of the rehearsing process in URI. Besides, as highlighted by the yellow dashed circles, the uncer- tainty score also identifies parts of the clusters that are out of the real data distribution, which will be penalized when introspecting via CIQI..
low-unc data
high-urc data
Figure 6: Visualization of the pro- jected distributions for real and imag- inary datasets in the football environ- ment.
7 Conclusion and Discussion
Inspired by the learning behavior of humans when they try to acquire a new skill, we propose PLfB that trains agent from books, instead of numerous interaction data with the real environment. We also implement a practical algorithm of understanding, rehearsing, and introspecting modules to realize such learning paradigm. The result of deploying our method in Tic-Tac- Toe and football game environments demonstrates a huge improvement in the winning rate over the baseline methods. This success proves the feasibility of utilizing knowledge stored in various written texts for decision- making agents, which was neglected by the community for a long time.
One major limitation of URI is its implicit dependence on the quality of the "tutorial books": they should cover sufficiently the dynamics, policy, and rewards of the targeted environment so that relevant knowledge and imaginary data can be extracted to train the policy. It is also important to develop metrics evaluating the textual data quality to decide whether to use URI.
Besides, we hope that the promising result in current experiments will initiate more research on PLfB. We leave more discussion of several interesting open problems in PLIBthat URI does not address in Appendix I.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the Jiangsu Science Foundation (BK20243039). The authors thank anonymous reviewers for their helpful discussions and suggestions for improving the article.
References
[1] Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press. 2018.
[2] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learning, In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016.
[3] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford. and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.0633 2017.
[4] Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Tracey, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602(7897): 414-419, 2022.
[5] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33: 1179-1191, 2020.
[6] Tianhe Yu, Garrett Thomas. Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. MOPO: Model-based offline policy optimization. Advances Neural Information Processing Systems, 33:14129-14142, 2020.
[7] Soroush Nasiriany, Abhiram Maddukuri, Lance Zhang, Adect Parikh, Aaron Lo, Abhishek Joshi, Ajay Mandlckar, and Yuke Zhu. RoboCasa: Large-scale simulation of everyday tasks for generalist robots. arXiv preprint arXiv:2406.02525 2024.
[8] Josh Achiam, Steven Adler, Sandhini Agarwal. Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08777, 2023.
[9] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. 2023.
[10] Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.1622
2023.
[11] Andrew Szot, Max Schwarzer, Harsh Agrawal, Bogdan Mazoure, Rin Metcalf, Walter Talbott, Natalie Mackraz, R Devon Hjelm, and Alexander T Toshev. Large language models general- izable policies for embodied tasks. In The Proceedings of the 12th International Conference on Learning Representattinss 2023.
[12] Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang. Senjie Jin, Enyu Zhou, al. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.0780 2023.
[13] Sihao Hu, Tiansheng Huang, Fatih Ilhan, Selim Tekin, Gaowen Liu, Ramana Kompella, and Ling Liu. A survey on large language bodel-based game agents. arXiv preprint ArX4vv2404.000 02039, 2024.
[14] Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey progress and challenges. arXiv preprint arXiv:2402.01680. 2024.

---

[15] Patrick Lewis, Ethan Perez, Aleksandra Piktus. Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih. Tim Rocktäschel, et al. Retrieval- auggeeeed generation for inowledge-intensive nlp tasks. volume 33, pages 9474, 2020.
[16]
Zhe Wang, Petar Veličković, Daniel Hennes, Nenad Tomašev, Laurel Prince, Michael Kaisers, Yoram Bachrach, Romuald Elie, Li Kevin Wenliang, Federico Piccinini, William Spearman, Ian Graham, Jerome Connor, Yi Yang, Adrià Recasens, Mina Khan, Nathalie Beauguerlange, Pablo Sprechmann, Pol Moreno, Nicolas Heess, Michael Bowling, Demis Hassabis, and Karl Tuyls. Tacticai: an ai assistant for football tactics. Nature Communications, 15(1):1906) Mar 2024.
ISSN 2041-1723.
[17] Together Computer. Redpajama: An open source recipe to reproduce llama training dataset, 2023. URL https:///ithh/. wwt.cooo//oooo wogethercomputer/RRee Reddaaj ama- -ata.
[18] Karol Kurach, Anton Raichuk, Piotr Stanczyk, Michal Zajac, Olivier Bachem, Lasse Espe- holt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, and Sylvain Gelly. Google research football: A novel reinforcement learning environment. arXiv preprint 19X1v1190..11180
[19] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824 24837, 2022.
[20] Zirui Zhao, Wee Sun Lee, and David Hsu. Large language models as commonsense knowledge for large-scale task planning. arXiv preprint arXiv:2305. 14078, 2023.
[21] Shuqi Liu, Bowei He, and Linqi Song. Bi-chainer: Automated large language models reasoning with bidirectional chaining. arXiv preprini arXiv:2406.06558, 2024.
[22] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022.
[23] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.0560 2022.
[24] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik R Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In Proceedings of the 37th Conference on Neural Information Processing Systems, 2023.
[25] Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Lim+ p: Empowering large language models with optimal planning proficiency, arXiv preprint arXiv:2304.1144 2023.
[26] Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Lim-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2998-3009, 2023.
[27] Meng Cao, Lei Shu, Lei Yu, Yun Zhu, Nevan Wichers, Yinxiao Liu, and Lei Meng. DRLC: Reinforeement learning with dense rewards from Ilm critic. arXiv preprint arXiv:2401.07382. 2024.
[28] Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint S.ti::233300000. 2023.
[29] Yue Wu, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, and Tom M Mitchell. Read and reap the rewards: Learning to play atari with the help of instruction manuals. volume 36, 2024.
[30] Jing-Cheng Pang, Pengyuan Wang, Kaiyuan Li, Xiong-Hur Chen, Jiacheng Xu, Zongzhang Zhang, and Yang Yu. Language model self-imppoveemet by reinforcement learning contempla- tion. Proceedings of the 12th International Conference on Learning Representations, Vienna, Austria, 2024.
12

---

[31] Ziyan Wang, Meng Fang, Tristan Tomilin, Fei Fang. and Yali Du. Safe multi agent reinforcement learning with natural language constraints. arXiv preprint arXiv:2405.20018, 2024.
[32] Martin Klissarov, Pierluca D'Oro, Shagun Sodhani, Roberta Raileanu, Pierre-Luc Bacon, Pascal Vincent, Amy Zhang, and Mikael Henaff. Motif: Intrinsic motivation from artificial intelligence feedback. In Proceedings of the 12th International Conference on Learning Representations, 2024.
[33] Abhay Zala, Jaemin Cho, Han Lin, Jachong Yoon, and Mohit Bansal. Envgen: Generating and adapting environments via Ilms for training embodied agents. arXiv preprint 120104 2024.
[34] Ruiyang Ren, Peng Qiu, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Hua Wu, Ji-Rong Wen, and Haifeng Wang. Bases: Large-scale web search user simulation with large language model based agents, arXiv preprint wrXiv:2402.17505 2024.
[35] Jessy Lin, Yuqing Du, Olivia Watkins, Danijar Hafner, Pieter Abbeel, Dan Klein, and Anca D. Dragan. Learning model the world with language. In Proceedings of the 41st International Conference on Machine Learning, 2024.
[36] Hengyuan Hu and Dorsa Sadigh. Language instructed reinforcement learning for human- ai coordination. In Proceedings of the 40th International Conference Machine Learning, 2023.
[37] Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextual- ized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39 48, 2020.
[38] Ehsan Kamalloo, Xinyu Zhang, Odunayo Ogundepo, Nandan Thakur. David Alfonso-Hermen Hermelo, Mehdi Rezagholizadch, and Jimmy Lin. Evaluating embedding apis for information retrieval. arXiv preprint arXiv:2305.06333 2023.
[39] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint vrXiv:2005.01645 2020.
[40] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1911.
2019.
[41] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning, Advances in neural information processing systems, 34:20132-20145, 2021.
[42] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[43] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, pages 104-114. PMLR, 2020.
[44] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-leamiig. arXiv preprint 10.0661 2021.
[45] Yuhang Ran, Yi-Chen Li, Fuxiang Zhang, Zongzhang Zhang, and Yang Yu. Policy regular- ization with dataset constraint for offline reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, 2023.
[46] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. volume 33, pages
2020.
[47] Xiong-Hui Chen, Fan-Ming Luo, Yang Yu, Qingyang Li, Zhiwei Qin, Wenjie Shang, and Jieping Ye. Offline model-based adaptable policy learning for decision-making in out-of-support regions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[48] Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-bellme inconsistency for model-based offline reinforcement learning. In Proceedings of the 40th International Conference on Machine Learning, 2023.
13

---

[49] Fan-Ming Luo, Tian Xu, Xingchen Cao, and Yang Yu. Reward-consistent dynamics models are strongly generalizable for offline reinforcement learning. In Proceedings of the 12th International Conference on Learning Representations, 2024.
[50] Ruifeng Chen, Chengxing Jia, Zefang Huang, Tian-Shue Liu, Xu-Hui Liu, and Yang Yu. Offline transition modeling via contrastive energy learning. In Proceedings of the 4 41th International Conference on Machine Learning, 2024.
[51] Ruifeng Chen, Xiong-Hui Chen, Yihao Sun, Siyuan Xiao, Minhui Li, and Yang Yu. Policy- conditioned environment models are more generalizable. Proceedings of the 41 Interna- tional Conference on Machine Learning, 2024.
[52] Xiong-Hui Chen, Yang Yu, Zhengmao Zhu, Zhihua Yu, Zhenjun Chen, Chenghe Wang, Yinan Wu, Rong-Jun Qin, Hongqiu Wu, Ruijin Ding, and Fangsheng Huang. Adversarial counterfac- tual environment model learning. In Advances in Neural Information Processing Systems 36, 2023.
[53] Ke Yang, Jiateng Liu, John Wu, Chaoqi Yang, Yi Fung. Sha Li, Zixuan Huang, Xu Cao, Xingyao Wang, Yiquan Wang, et al. If Ilm is the wizard, then code is the wand: A survey on how code empowers large language models to serve as intelligent agents. arXiv preprint arXiv:2401.0087 2024.
[54] Chengxing Jia, Chenxiao Gao, Hao Yin, Fuxiang Zhang, Xiong- Chen, Tian Xu, Lei Yuan, Zongzhang Zhang, Yang Yu, and Zhi-Hua Zhou. Policy rehearsing: Training generalizable policies for reinforcement learning. In Proceedings of the 12th International Conference on Learning Representations, 2024.
[55] Jing- King-Cheng Pang, Si-Hang Yang, Kaiyuan Li, Jiaji Zhang, Xiong-Hui Chen, Nan Tang, and Yang Yu. Knowled Knowledgeable agents by offline reinforcement learning from large language model rollouts. The 38th Annual Conference on Neural Information Processing Systems, 2024.
[56] Karol Kurach, Anton Raichuk, Piotr Stańczyk, Michal Zajac, Olivier Bachem, Lasse Espeholt, Carlos Riquelme, Damien Vincent, Marcin Michalski, Olivier Bousquet, et al. Google research football: A novel reinforcement learning environment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 4501-4510, 2020.
[57] TogetherComputer Redpajama: open dataset for training large language models, 2023. URL https://gi thub. wwt http://oetheecooptttooo /Redaaj ama- Data.
[58] Sarvar Anvarov. Solution ranked 35th in kaggle football competition. https://github. com/ Sarvar- Anvararoo--orrrroog-e-ee--- Research- Fohttbootl 2020.
[59] Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 986):2559-2265, 2008
[60] Tian-Zuo Wang, Tian Qin, and Zhi-Hua Zhou. Estimating possible causal effects with latent variables via adjustment. Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pages 36308-36333 PMLR, 23-29 Jul 2023.
[61] Jerry Liu. Llamalndex, 11 2022. URL hwR.antpee//gtaaab.cc////rr////p 1adellmmm_indexea___ee
[62] Takuma Seno and Michita Imai. d3ripy: An offline deep reinforcement learning library. Journal of Machine Learning Research, 23(315): 2022.
[63] Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Ri-2: Vision-language- action models transfer web knowledge to robotic control. arXiv preprint 2...:23377.18111 2023.
[64] Leo Gao. An empirical exploration in quality filtering of text data. arXiv preprint 21X 2021.
14

---

[65] Alexander Wettig, Aatmik Gupta, Saumya Malik, and Danqi Chen. QuRating: Selecting high- quality data for training language models. In Proceedings of the 41st International Conference on Machine Learning, Vienna, Austria, 2024.
[66] Simin Fan, Matteo Pagliardini, and Martin Jaggi. DOGE: domain reweighting with general- ization estimation. In Proceedings of the 41st International Conference on Machine Learning, 2024.
[67] Lei Yuan, Ziqian Zhang, Lihe Li, Cong Guan, and Yang Yu. A survey of progress cooperative multi-agent reinforcement learning in open environment. arXiv preprint arXiv:2312.011 2023.
15

---

Appendix
Table of Contents
A Tic- Tac-Toe (TTT) Task Setup
17
A.1 Environment
17
A.2 Tutorial Text Generation
17
B Google Research Football (GRF) Tasks Setup
B.I State Space
18
B.2 Action Space
22
B.3 Transition
22
C Additional Related Work
23
Baseline Implementations for TTT
23
LLM-as-agent
23
D.2 LLM-RAG
24
D.3
Additional Baselines
25
E
Baseline Implementations for GRF
25
E.1 ILM-as-agent
25
E.2
LLM-RAG
27
F
URI Implementation
28
F.I
Book Content Understanding
................................
28
F2
Knowledge-based Rehearsing of Decision Decision-Making ................................
32
F33
Tic-Tac-Tow Toe Prompt Configurations
37
F4
Football Prompt Configuration
39
F5
Introspecting based on the Imaginary Dataset
43
F6
General Recipe for Prompt Design
43
Hyper-parameters
43
F88
URI Algorithm
44
G
Additional Ablations
45
G.1 Data Aggregation Visualization for Toe
45
G.2
Performance Comparison with Additional Baselines in GRF
45
Examples Data Generation and Policy Execution Videos
45
Open Problems and Future Directions
45
Compute Resources
46
K Broader Impact Statement
46
16

---

A Tic-Tace Fre-Tac-Toe (TTT) Task Setup
A.1 Environment
Tic-Tac-Toe is a classical two-player game played on a 3x3 grid. Players take turns placing their symbols or O) in empty cells, with the goal of forming a line of three marks horizontally, vertically, or diagonally. The game state is represented by a 9-dimensional vector, where each dimension corresponds to a cell that can be emply (0), marked by X (I), or marked by O (2). The action space consists of integers from 0 to 8, representing the cell indices for placement.
A.2
Tutorial Text Generation
A.2.1
Data Collection
We first collect comprehensive game trajectories by having minimax player (optimal policy) play against all possible opponent moves. The data collection process:
Start from empty board
2. For X's moves (minimax player):
Use optimal minimax strategy
First move randomized for diversity
3. For O's moves (opponent):
Explore all possible valid move
Record complete trajectory
4. Store trajectories with:
Complete state sequence
Actions taken
Player turns
Game outcomes
Winning/losing patterns
This systematic exploration generates dataset covering all possible game scenarios under optimal play.
A.2.2 Knowledge Extraction
We GPT to analyze these trajectories and extract strategic knowledge. The prompt are used as follow:
Prompt 1: Knowledge Extraction Prompt
Given a Tic-Tac- game trajectory where X known be the oracle (optimal player), analyze the game and provide extracted knowledge about various aspects:
1. Observe the entire game trajectory, including moves by both X and O.
2. Formulate concise statements of knowledge that capture insights about:
Provided Trajectory Analysis: Overall assessment of game flow and key decisions
Learned Game Mechanics: Turn-taking, move validity, board state changes
Learned Winning Conditions: Victory achievement and prevention
Learned Strategic Principles: General strategies demonstrated by optimal player
Your output should consist of 3-4 concise statements, each focusing on one aspect. These statements should be generalizable and applicable broadly.
17

---

Example output:
Trajectory Analysis: "This game showcased defensive strategy by X, consistently blocking O's potential winning moves while building towards a win."
operates alternating turns, with each player placing their symbol in empty cells."
Game Mechanics: "Tic-Tac-
Winning Conditions: "Achieving three symbols in a line secures victory, highlighting both offensive and defensive needs." "
Strategic Principles: "Controlling the center provides the most potential winning lines."
Please provide 3-4 such knowledge statements based on the specific game trajectory presented.
A.2.3 Generated Knowledge
The complete knowledge base is available at: https://githhi Neur NeurIPS/ SaauuIII///oaaaaaaa www.oo/mmannttii tac_toe knooe_dnooeedee... json1
Representative examples include:
Strategic Principles:
"Prioritize creating two potential winning lines simultaneously to force the oppo- nent into a defensive position and increase chances of victory."
2. Position Control:
"Control the center and occupy adjacent positions to create multiple threats while forcing opponent into defensive posture."
3. Opening Strategy:
"Claim the center early when playing X to maximize board control and create multiple winning opportunities."
4. Tactical Patterns:
"When establishing dominance, prioritize claiming center and a row/column to establish multiple winning opportunities while blocking opponent threats."
This knowledge base provides complete coverage of optimal strategies, serving ground truth for evaluating PLtB's s ability to learn from textual instructions.
B Google Research Football (GRF) Tasks Setup
this section, we mainly focus on the GRF implementation" and the modifications we made under this environment.
B.1
State Space
The raw state of each player contains information about the game state, the ball and all players on the pitch. The game state includes scores, game modes indicating free kicks, corner kicks other game stages, and game time represented as steps. The ball information includes its spatial position, direction, and an indicator for ball ownership (identifying the player and team that possess it). The player information comprises position, direction, roles, yellow card records, and more. For players on the opposite team, positions and directions are mirrored. The details of the raw state follows:
18

---

STEP: 311 1 19m G G R
SCORE: BALL OWNEDD:
PLAYER SPRINT ORIBBLE DIRECTION ACTION
Figure Illustration of the game football simulator.
Ball Information:
ball y, position of the ball.
ball_direction - [z, y, ball movement vector.
ball_rotation [z,9. y. 2] rotation angles in radians.
ball_owned, team - [-1, O, 1), where indicates the ball not owned, denotes the left team, and the right team.
ball_owmed_player (O.NN1) integer denoting the index of the player owning the ball.
Left Team:
left_team N-elements vector with [z; y] positions of players.
left_ ceft__teamdiircction N-elements vector with [x, y) movement vectors of players.
left_ ttt___emm_tiree__tee factor N-elements vector of floats in the range {O..1}. means the player not tired at all.
beff__temm_yello._cmm card N-elements vector of integers denoting the number of yellow cards a given player has (0 or 1).
N-elements vector of booleans denoting whether a given player is playing the game (False means the player got red card). left_ Terf__tam_roleee N-elements vector denoting roles of players, where:
teftt_temmacctee
e Stl___aayeRool_____ goakkeeper, e weeeellaaeeoleeeeeeeee centre back,
2 e e___aeeerollee LB left back.
Right Team: Same attributes as for the left team.
Controlled Player Information:
active (O.NN-) integer denoting the index of the controlled players.
designated - (O.N-1) integer denoting the index of the designated player.
sticky_actions 10-elements vectors of Os or Is denoting whether corresponding action is active.
Match State:
score Pair of integers denoting the number of goals for the left and right teams, respectively, steps steps_leftt left- How many steps are left till the end of the match.
game_mos mode Current game mode.
For the imaginary dataset generation, we created our own Imaginary state, which includes the following information:
Game Information:
19

---

Sticky actions: A list of currently active sticky actions.
Game mode: The current game mode.
Score: The current of the game.
Time: The current game time in minutes and seconds.
Active player: The index of the currently active player.
Active player role: The role of the currently active player.
Ball ownership: The team currently possession of the ball (none, left team, or right team).
Ball ownership player: The index of the player currently in possession of the ball.
Ball zone: The zone where the ball is located.
Ball direction: The direction in which the ball is moving.
Player Information:
Team: The team the player belongs to (left team or right team).
Role: The role of the player.
Zone: The zone where the player is located.
-Direction: The direction the player is facing.
The imaginary state is designed provide compact and informative representation of the game state compared to the raw state. It extracts and processes the relevant information from the raw state, making it easier understand and use for generating imaginary data.
The sticky actions, game mode, score, time, active player, and active player role provide a high-level overview of the current game state. The ball ownership and ball ownership player indicate which team and player are currently control of the ball. The ball zone and direction give spatial information about the ball's location and movement.
For each player, the Imaginary state includes their team, role, zone, and direction. This information helps in understanding the positioning and orientation of the players on the pitch.
By including these key pieces of information in the Imaginary state, aim to capture the essential aspects of the game state that are relevant for generating realistic and diverse imaginary data. The imaginary state serves as a preprocessed and structured representation of the state, making more suitable for the subsequent steps in the imaginary data generation process. Here, show example of imaginary state,
20

---

Imaginary State
('active_player': 5,
'actiiee_player_roles _role': 'Defender',
?ball_direction': 'west', 'ball_owmership's 2,
DballLownershippplayerr" 18,
'ball_zone': [15, 9],
'game_mode': 'Normal', 'player_0': ('direction'; 'east', 'role': 'Goalkeeper', 'team': 'Left', 'zone': [2, 7]}, ('direction': 'east', 'role': 'Forward', 'team': 'Left', 'zone': [12, 11]}, ('direction': *east', 'role': 'Forward', 'team': 'Lett' 'zone': [16, 4]}, 'player_111 11:: ('direction': 'southwest', 'role': 'Goakkeeper,, 'team': 'Right', 'zone': [19, 7}}, ('direction': 'west', 'role': 'Forward', 'team': 'Right', 'zone': [14, 4]}, ('direction': 'north', 'role': 'Forward', 'team": 'Right', 'zone': [14, 8]},
'player_1;
'player_10',
'player_12'
'player_13 13':
'score': [0, 0],
'step': 295,
"sticky_actions" ['BottomRight', 'Sprint'],
'time' '8 minutes 51 seconds'}
To enhance the understanding of player and ball positions beyond mere coordinates, our methodology involves dividing the football field into grid of 20 x 12 rectangular zones. This granular zoning approach serves dual purpose: firstly, abstracts the complex spatial dynamics the game into more manageable form, allowing our L.LM to process and generate data with increased effectiveness. Secondly, it provides a framework for more accurately simulating the spatial strategics and movements employed in real-world football scenarios. Each zone, defined by specific dimensions, acts unique spatial identifier, offering precise reference points for player positioning and ball location. This system not only simplifies the representation of the field but also enriches the strategic depth of the game model, as actions and decisions can be tailored to the distinct characteristics of each zone. By adopting this zoning strategy, aim to bridge the gap between the high-level strategic understanding required for football and the detailed, positional awareness needed to implement those strategies effectively within the simulated environment.
The simulator information |M] includes the description of the state space What we use this paper is as follows.
First, provides information such as the time and score of the match.
Second, which side has control of the ball, and the active player in the left team, you need to propose corresponding policies to him.
21

---

Next, we present the position and role information of each player: In this text description, the football grass field divided into 240 zones.
We the (x, y) express the position of the player. "x" is the distance from the left team's penalty area to the right team's 's penalty area, ranging from to 20, and y is the distance from the lower corner to the upper corner flag, ranging from to 12. This means that the center circle position of the field is zone (10, 6), where the game start.
The lower left corner position of the left team is (1, 1), and the upper right comner position of the right team is (20, 12).
The venues never interchange or change. The direction of the position information is the direction the player is currently facing and the direction of future actions.
B.2
Action Space
The default action set comprises 19 actions, including directional movements, three various ball passing, ball shooting, sliding, sprinting and others. Throughout our experiments, we utilized the default action set. In details,
Idle actions
retioo_idee-a action__, idle 0 a no-op action, sticky actions are not affected (player maintains his directional movement etc.).
Movement actions
action_left=1. run the left, sticky action.
action_top_left 2: run the top-left, sticky action.
action_top=3: run to the top, sticky action.
action_top_right run to the top-right, sticky action.
action_right = run to the right, sticky action.
action_bottom_rrige 6: run to the bottom- right, sticky action.
action_bottom =:: to the bottom, sticky action.
action_bottom_lett 8: run the bottom-left, sticky action.
Passing/ Passing/Shooting Shooting
action__ongg_pate = 9: perform a long pass to the player on your team. Player to pass the ball to is auto-determined based the movement direction.
Decion_high_pase 10: perform high pass, similar action_ con_loon--pas_
aetion_short_pee 11: perform a short pass, similar to pertion_long_pae
action_shot 12 perform a shot, always in the direction of the opponent's 's goal.
Other actions
action_spring 13: start sprinting, sticky action. Player moves faster, but has worse ball handling.
action_releaee diriaatiodreeeioonet 14: reset current movement direction.
action_release_sse = 15: stop sprinting.
action_sliding 16: perform a slide (effective when not having ball).
action_dribble 17 start dribbling (effective when having ball), sticky action. Player moves slower, but is harder to take over the ball from him.
18: stop dribbling.
B.3 Transition
The game dynamics in GRF closely resemble realistic football games. Players can move sprint with or without the ball. They can also execute various passes and shots. Besides different actions, complex interactions such collisions and trips between players simulated well. Additionally, the game engine introduces stochasticity in the dynamics, which can influence the passes and shots randomly.
In our experiment, to address the challenge of executing fine-grained maneuvers within game environment, we employ a strategy where actions generated by rule-based policies [58] are utilized
22

---

to replace those produced by the policies. including URI and all of the baselines. This substitution is particularly enacted when considering the spatial segmentation of the play area into zones. Using zones as pivotal input, the approach facilitates more nuanced control over in-zone activities. The rationale behind this decision rooted in the state that precise actions within these designated zones can significantly impact the outcome of play, necessitating method that allows for detailed operational control. This method ensures that the agents can perform more sophisticated strategies, particularly in scenarios that demand high levels of precision and situational awareness within the confined spaces of each zone.
C Additional Related Work
Offline RL addresses the problem of learning policies from a pre-collected dataset. Most methods can be classified into two categories: model-free and model-based methods. Model-free [40-43, 5, 44, 45] methods learn conservative policy directly from the dataset. Model-based offline algorithms [6, 46- 49] first estimate a generalizable dynamics model [60, 52] from the dataset and perform policy learning or planning based this learned model. our work. to achieve introspection from the data generated by the LLMs, we build our policy distillation algorithm based on several existing techniques offline RI., including the uncertainty penalty MOPÓ [6] which constructs a pessimistic model that discourages the policy from visiting states where the model is inaccurate; and conservative Q-learning loss [5] to obtain robust value function that does not overestimate unseen state-action pairs too much.
D Baseline Implementations for TTT
In this section, we detail the implementation of baselines for both Tic-Tac-Ti Tac-Toe.
D.1 LLM-as-agent
The the following prompts:
baseline uses GPT-3.5 to directly generate actions for both environments.
Prompt 2: LLM M-a--saaeent Global Prompt for Tic- Tac-TTT--oo Toe
# Tic-Tac-To
## Introduction
Tic- Tac-TTa-Teee is classic two-player game, unfolds a 3-by-3 grid where the objective is to align three of one's symbols, Xs for the first player and Os for the second, either horizontally, vertically, or diagonally. Strategic placement is crucial: besides aiming for three in a row, players must also block their opponents' potential alignments to avoid defeat. Players can place their next move in an empty cell the 3-by-3 grid.
## Rules
1. The game is played on 3x3 grid.
2. Players take turns placing their symbol (X or O) in empty cells.
3. The first player get 3 of their symbols in row (horizontally, vertically, or diagonally) wins.
If all cells are filled and no player has won, the game is a draw.
## Game State Representation
The game board is represented as string of characters, where:
'X' represents the first player's moves
'O' represents the second player's moves (space) represents an empty cell
23

---

## Your Task:
Respond with a JSON object containing your move and reasoning.
The move should be single integer from to 8, representing the cell index.
Provide brief explanation for your choice in the reasoning field.
Prompt 3: LLM-as ALL--s-ageet Query Prompt for Tic-Tac- Tac-Toe
## Choose the next move
You are playing as "(player'..
The current state of the board is follows:
state.board[0]
state.board[1] state [tard]bbard[2]
state.board[3] state.board[4] state.boardyy
state. ]6] state.board[7] state. [0ard[)
The next move should be placed in an empty cell the 3-by-3 grid.
The available positions for the next move are:
Estate.availate _mools())
Positions are represented by the numbers 0-8, corresponding to the cell's index:
2
| |
Analyze the game state and choose your next move. Respond with a JSON object in the following format:
""move": <integer 0-8>, "reasoning": "<brief explanation for your choice>")
D.2 LLM-RAG
The LLM-RAG baseline enhances LLM-as-agent by retrieving relevant knowledge from tutorial books.
Prompt 4: LLM-RAG Global Prompt for Vie-Tac-Tee Toe
[Same as LLM-as-agent Global Prompt with additional instruction:]
The policy knowledge from tutorials will be provided to guide your decision.
24

---

Prompt 5: L.MM-AG Query Prompt for Tic- Tie-Tac-Toe
The relevant policy knowledge for the current state is:
iiolicy_knowlegge
## Choose the next move based on the policy knowledge
[Rest same as LLM-as-agent Query Prompt]
You should follow the provided policy knowledge to make your decision.
Base your reasoning on how your chosen move aligns with the strategic principles described in the policy knowledge.
For the RAG part, use the following knowledge retrieval approach:
Use GPT embeddings to encode both state descriptions and tutorial knowledge
Retrieve top-3 most relevant pieces of strategic knowledge using cosine similarity
Combine retrieved knowledge with state information for move generation
D.3 Additional Baselines
Minimax (Optimal):
Uses standard minimax algorithm with perfect play
Evaluates states with +1 for win, for loss, 0 draw
Uses random first move selection for diversity
Minimax-noise:
Based on optimal minimax algorithm
Makes random moves with 30% probability
Uses optimal moves with 70% probability
Serves as a strong but imperfect opponent
Random:
Selects moves uniformly from available positions Serves as a baseline for minimal strategic play
E Baseline Implementations for GRF
In this section, we will focus on implementing of two relevant baselines.
E.1
LLM-as-agent
As mentioned the paper, the LLM ageentttent method uses LLM (GPT3.5) directly to generate the action. We provide the prompt as follows,
Prompt 6: LLM ass-a--sp agentt Query Prompt 1/2
The texted state for the current state: frext_stater
(Simulator Information [M]. Refer to Appendix F.4.)
For example, if the active player is Player 2 and you want him to be close to the ball and control it in the texted state,
25

---

Forward Player 2 is at (9,9( The ball is at Zone(1 11,)..
Given these coordinates, the ball is diagonally one zone to the right (east) and one zone down (south) from the player's current position. The most direct route the ball would indeed be diagonally towards the bottom right.
Therefore, the most appropriate action for Forward Player 2 in this situation would be:
6 action action_bottom right: This sticky action will allow the player to move diagonally in the bottom-right direction (southeast), which is the direct path to where the ball is currently located in Zone(11,8).
By choosing action_bottom bottomm_righ,, Forward Player 2 can close the distance to the ball more effectively, aligning their movement directly with the ball's current location. Once the player reaches the ball, the subsequent action can be decided based on the situation at that moment (e.g., dribbling, passing, shooting).
Question: What next action do you want this active player to take? Answer:
The global prompt shows as follows,
Prompt 7: LLM-as-agent Global Prompt
want you to act like a football manager and also an expert in Python coding and Reinforcement Learning, which is about learning a football manager policy in football simulator.
will give you the pseudocode snippets to define the specific policy for the active player in the football game.
These codes represent absolute correctness and do not add other common logic. Your task is to select the action that best fits and executes the code based on the logic of the given code.
Requirements:
About the action choosing:
Please choose the action that best fits the code logic.
About the format:
you should answer in pure JSON format with the key: 'action': an int number from 0 to 18,
'thought': why did you choose this action? without any other information or code.
For example, you should not add the "JSON" tag the answer.
Response example (you should respond in the following order):
f(
"action": 0,
"thought": "Based on your thought, tell me the optimal action you would like to select in the action set."
26

---

}}
E.2 LLM-RAG
The LLM-RAG enhances aLM--as-aeent by retrieving relevant knowledge from the database extracted from the tutorial books, similar to the retrieval step in our rehearsing stage, but directly outputs the action without policy learning. The dataset which the LLM-RAG using is the text-booked dataset. Here we show the prompts of LLM-RAG.
Prompt 8: LLM-RAG Query Prompt 1/2
The relative policy from the book you want this active player to implement as follows:
(Policy_ Str)
The texted state for the current state:
[text_statee
[Simulator Information [.M]. Refer to Appendix F.4.)
For example, if the active player is Player 2.
The most relative policy from books:
When you become back defender, you aim to get close to the ball and stop the attacker's progression towards the goal.
The texted state:
Forward Player 2 is at Zone(9,9). The ball is at Zone(1 1,8).
Given these coordinates, the ball diagonally one zone the right (east) and one zone down (south) from the player's current position. The most direct route to the ball would indeed be diagonally towards the bottom right.
Therefore, the most appropriate action for Forward Player 2 in this situation would be:
6 action _iction_btttom___m firtoom_rihht:: This sticky action will allow the player to move diagonally in the bottom-right direction (southeast), which is the direct path to where the ball is currently located in Zone(11,8),
By choosing _ction_ bottom poottmmrright, Forward Player 2 can close the distance to the ball more effectively, aligning their movement directly with the ball's current location. Once the player reaches the ball, the subsequent action can be decided based on the situation at that moment (e.g., dribbling, passing, or shooting).
Question: What next action do you want this active player to take? Answer:
The global prompt shows as follows,
27

---

Prompt 9: L.M--AG Global Prompt
want you to act like a football manager and also an expert in Python coding and Reinforcement Learning, which is about learning a football manager policy in football simulator.
I will give you the pseudocode snippets to define the specific policy for the active player in the football game.
These codes represent absolute correctness and do not add other common logic. Your task is select the action that best fits and executes the code based on the logic of the given code.
Requirements:
About the action choosing:
Please choose the action that best fits the code logic.
About the format:
you should answer in pure JSON format with the key: 'action': an int number from O to 18,
'thought': why did you choose this action? without any other information or code.
For example, you should not add the "JSON"" tag in the answer.
Response example (you should respond in the following order):
{(
"action": O,
"thought": "Based on your thought, tell me the optimal action you would like to select in the action set."
}>
For the RAG part, use the llama index [61]* for embedding the textbook paragraphs and the state and find the most relevant one as input to the query prompt.
F
URI Implementation
In this section, we will focus on implementing URI.
F1
Book Content Understanding
Book content understanding is achieved via code-based knowledge extractor and code-based knowledge aggregator. The implementation is designed to be task-agnostic, with task-specific parameters configured through prompts.
E.1.1 Knowledge Extractor
The knowledge extractor processes book paragraphs to identify and extract relevant knowledge. Algorithm shows the extraction process:
"ntts://github.commmmn aaaaallaa index
28

---

14: return I. Algorithm Code Extractor
Require: Book B consisting of segments bi, b2...., bNg
I: Initialize language model Mext
2: Define KnowledgeContexl + (Rewards, Policies, Dynamics]
3: K ++0
4: I + 0
5: for b, E do
Iterate over each paragraph in the books
6:
I.add(b;))
7:
Output +- Mext (I, Knoweedgeoontextt if d'WithKnowledge' in Output then
8:
9:
K.add(( Outpull.coee
10:
else if "Wihhouknnwwedge" in Oulpul then - end if
11:
12:
13:
end for
Task-specific configuration can be found at Appendix. F.3 and E.4
F.1.2 Knowledge Extraction Prompt
Prompt 10: Book Information Extraction Prompt
I want you to act (lask_info),
You need to analyze the given paragraph step-by-ste from related context to derive the specific theorem, principle, rule, and law of the related elements or concepts:
Bettment_tyye_prommee
hilter_stringeney_prmppt
Requirements:
About the answer:
If you think the paragraph contains the above elements, please answer The answer is 1 only when you can write the specific theorem, principle, rule, and law of the related elements into pseudocode snippet.
If you think the paragraph does not contain the above elements, please answer 0.
If you think the given paragraph is not clear enough to answer, please answer 2. Then will give you the following paragraph to help you answer.
If you think the paragraph contains the above elements but the content is not clear enough to derive the specific theorem, principle, rule, and law of the related elements, please answer 2. Then will give you the following paragraph to help you answer.
About the analysis:
If the answer is 1, you should give the specific theorem, principle, rule, and law of the related elements.
You should write the specific theorem, principle, rule, and law of the related elements via pseudo code.
Please provide the PYTHON-style pseudo code as detailed as you can to cover the most information of the original content.
29

---

About the pseudo code snippet:
soaneent_ypp_ccdee type httpr/teeeoooeeeeeeeeeeeeeieelle evamppe_eicttknowlleegegeetet _type]] {element
About the format:
You should answer in pure JSON format, without any other information or code.
Response example:
"answer": 1,
"pseudocode of/about XX": ["PYTHON pseudocode snippet1", ...]
psseudcodd of/about XX": ["PYTHON pseudocode snippet2", :..]
F.1. Knowledge Aggregation
The aggregation process combines similar pieces of knowledge into more concise representations. Algorithm 2 shows the process:
Algorithm Code-based Knowledge Aggregation (one round)
Require: Knowledge X., text inclusion flag T, aggregation number N, similarity threshold
1: Initialize language model Magg and embedding model E
2: I'' <-
Iterate each knowledge piece 4: Define KnowledgeContext + (Rewards, Policies, Dynamics}
3: for K in K. do
5: K.sim <- {K' X|caalellate-ssiilariiees////// E E(K')) 7}
6: if IICsim 2 Nagg then
7:
K +- Magg(((smm " .KnowlodgeConeett)
8:
IC' .'.add( (add(kak))
9: else
11: end (C'.ad(()
12:
end for
13:
return K"
Key parameters for aggregation:
Naggi Number of similar pieces to aggregate (default: 4)
Similarity threshold (default: 0.95)
embedding_model Model for computing text similarities
F.1.4
Knowledge Aggregation Prompt
Prompt 11:
Code Aggregation Prompt
want you to act like
I will give you several pseudocode snippets to define the specific theorem, principle, rule, and law of the related elements or concepts:
(element hype_prompttt dictlknowleddee_typ)) biiie__strnnency_rrompe
30

---

These codes might share some common logic. Your task is to aggregate the common logic of the given codes into a single code snippet, while still covering all the given codes.
Requirements:
About the aggregated code:
Please provide the PYTHON-style pseudo code detailed as you can
Using the least number of pseudocode items
Covering all of the code. However, please feel free to add more pseudocode items if needed
Since will delete the original code after getting your aggregated code, you cannot call the pseudocodes that provided in the prompt
About the pseudo code snippet:
{element tetll_yyp__ooo_o example_ hiapapleeeicckknowlllgeeeeeee
About the format:
You should answer in pure JSON format
Response example:
"pseudocode of/about XX": ["PYTHON pseudocode snippet1", ".aeet""..... ...]. "pseudocode of/about XX": ["PYTHON pseudocode snippet2", ...]
Prompt 12: Text-based Aggregation Prompt
want you to act like [task_infor
will give you several paragraphs and the corresponding summary written by code from several related books. You need to analyze the given paragraph step-by-step from related context to aggregate the specific theorem, principle, rule, and law of the related elements or concepts:
taatteessrrnnencyprmmmee
Requirements:
About the analysis:
You should write the specific theorem, principle, rule, and law via pseudo code
Please provide the PYTHON-style pseudo code as detailed as you can
You should aggregate the given information as much as you can:
1. Using the least number of pseudocode items
2. Covering all of the code and most of the original texts
3. Since will delete the original code after aggregation, you cannot call the provided pseudocodes
31

---

About the pseudo code snippet:
{element (element_type_conne httpr/eeoooeeeeeeeeeeeeeeeeeeeie evammple_cicttkoowleeeggggeee _type]
About the format:
You should answer in pure JSON format
Response example:
"aggregated pseudocode of/about XX": ["PYTHON pseudocode snippet1", ". ...].
aagregated_pseeuueegee pseudocode of/about XX": ["PYTHON pseudocode snippet2", ...]
F.2 Knowledge-based Rehearsing of Decision-Making
Knowledge-based rehearsing of decision making is achieved via state-space knowledge scope retrieval module and retrieved knowledge instantiation.
For state-space knowledge scope retrieval, we use GPT-3.5 to identify the knowledge scope repre- sented by state space, then adopt standard RAG module to retrieve the correct pieces of knowledge.
Prompt 13: Football Knowledge Code Generation Prompt
want you to act like football manager and also an expert in python coding and Reinforcement Learning that wants learn a football manager policy in a football simulator.
I will provide observation space of the football simulator and some policy functions written by others. Your task is to summarize an observation scope, which is a subspace of the observation space and most suitable to use this policy function to win the game.
The observation space: (observation_spaaee ceatrrioonspcee_eeeeeeeeee
The action space: (action_sppal saa___saaaaaaapee
We have N codes for you to analyze: {code]list
About the definition of the observation scope of each code: For each code, you should return a description of the best scope of the observation variables be used the policy function to win the game, including:
A text summary of the preferred/better scope
score is preferred/better to be in [?]
active
Laaeree_laye_rrreee role is preferred/better to be in [?]
ball owwershhpp preferred/better to be in [?]
ball www__oowes__ppaaaeeeree is preferred/better be in [?]
ball_zone
zone is preferred/better to be in [?]
ball directior is preferred/better to be in [?]
{the preferred zone of all players from 0 to 21)
NOTE:
1. You should define your observation scope as detailed and tight as possible
2. You should define the observation scope for each code by comparing the differences between the codes
32

---

About the format:
You should answer in pure JSON format, without any other information or code
Response example:
"ode_ids idx 1': {
"preferred scope description': 'a text summary of the suitable situations,
'score': 'preferred scope of the score', 'active__layer_rooee 'preferred scope of the active_player_role
,,
}
Prompt 14: Tic-Tac Tac-Toe Knowledge Code Generation Prompt
I want you to act like tic-tac-toe pro player and also expert in python coding and Reinforcement Learning that wants to learn a tic-tac-toe policy in a tic-tac-toe simulator.
I will provide observation space of the tic-tac-toe simulator and some policy functions written by others. Your task is to summarize an observation scope, which is subspace of the observation space and most suitable to use this policy function to win the game.
The observation space: jobservation_spacee spacc__desc]
The action space: faction_space
We have N codes for you to analyze: (rode_list)
About the definition of the observation scope of each code:
For each code, you should return description of the best scope of the observation variables to be used as the policy function to win the game, including:
A text summary of the preferred/better scope
Preferred board states: [(?), (?), (?), (?), (?), (?), (?), (?), (?)], where you should replace "?" with all possible values or "any" all values are suitable
Preferred/better player: [?], where you should replace "?" with possible values
NOTE:
1. You should define your observation scope as detailed and tight as possible
2. You should define the observation scope for each code by comparing the differences between the codes
About the format:
You should answer in pure JSON format, without any other information or code
Response example:
{
code_ide 1 {
33

---

'preferred scope description': 'a text summary of the suitable situations, "current_ple currentt_paayer: 'preferred scopes', 'preferred board states': 'preferred scopes'
,,
}
Then a standard RAG technique is applied to identify the most relevant knowledge scope K2 and its relevant knowledge Kj. Formally, ([Kj), [K)]) Rerope (&))) where K is the scope of the knowledge database.
F2211
Code Instantiation
After that, we employ an LLM to instantiate the code K' = Misst (8, [MI, {K₃}) based on the current state 8, the simulator information description [M] and the knowledge {Kj} retrieved by Recope- The prompt is listed in Prompt 15. Since this step requires strong understanding of the code, we use GPT-4 instead of GPT-3.5 the LLM implementation.
The code instantiation process converts general knowledge into task-specific code through a dedicated module. Here's the detailed implementation:
Prompt 15: Code Instantiation Prompt
want you to act beask_info]
will give you an observation which you are facing in the simulator, your task is to instantiate code that serves as (knowledge_ function, which is used for
#knowweede_funcction)
For example,
Formally, the format of knowledge_tye typee function is srrowledge_format/
To help you complete the task, will provide you:
1. Pieces of relevant knowledge from the tutorial, written in Python-style pseu- docode
2. The observation space and action space of the target simulator
3. The current observation you are facing
Python-style relevant knowledge from tutorial: (code (code_string]
The observation space: (observation cettrvvattonsppcceeeeeee
space_pesse _eace_ The action space: {ac- desc) Current observation: [obs}
About your code-instantiation task:
Please provide the PYTHON-style pseudo code as detailed as you can
Your task is to rewrite code that describes (knowledge_tyyee __teee____e_______ suitable to current observation
You should make the optimal decision based on analyzing current observa- tion
After analysis, rewrite the pseudocode make it most suitable for down- stream tasks
Keep a main function named "[knowledge_typen and add inner functions necessary
34

---

Important Notes:
You cannot call the pseudocodes provided in the prompt
Variable assignment can be simplified, but implement logic in detail
Do not use placeholder implementations
Additional Requirements:
For one-step use: Keep code short, direct, and focused on main logic
For multi-step use: Code should generalize to next minute of game, consid- ering:
Changes in active player
Ball position changes
Opponent position changes
Response format:
"analyze": "Analysis of current observation, focusing on code instantiation approach",
"code": "Rewritten code based current observation analysis" }
F.2.2 Rollout Process
Finally, three LLMs are involved generate the imaginary dataset Ding. including à; M2 (&,, Mins(
(5, (z))),, MR(s, å,, Minst (St, [M], Recope (5, K(i))
and 8 (St, âz, Minst (Se, [M] Recope (8, Kc))),, where K K and KiT the scope of the knowledge database Kr,, KR. and Kr respectively.
Prompt 16: Policy Inference Prompt
want you to act (lask_info)
will give you observation which you facing in simulator, your task is to response a correct results serving as (knowledge_tyypl function, which is used for (knowledge f
For example, Berplleddeefuutttoo_eeeeeee
Formally, the format of (knowledge_tyyen function is sobowweedg_formatte
To help you complete the task, will provide you:
1. Pieces of relevant knowledge from the tutorial, written in Python-style pseu- docode
2. The observation space and action space of the simulator
3. The current observation you are facing
Python-style relevant knowledge from tutorial books:
The observation space:
(observation_spee cictiiio_spae__deesee
35

---

The action space:
(action Baapinnsspppe___e _escc_
Current observation: {obs)
Requirements:
Choose the action that best fits the code logic
Answer in pure JSON format with keys:
'action": integer from 0 to 18
'thought': reasoning for the choice
Hints for inference: {hint)
linference_formatt far__lfoomate_aae _eammlee
Prompt 17: Dynamics Inference Prompt
want you to act like [task_info).
I will give you an observation which you are facing in the target simulator, your task is to response a correct results serving as dynamics function code from the tutorial.
Formally, the format of dynamics function is dynamics format. The function describes the mechanism for updating the position of the ball and players.
To help you complete the task, will provide you:
1. The dynamics function from tutorial books: [dynamics_comm _ctes_scrdddde_ee______e_e_
2. Current observation: (obs}
3. Chosen action: (action)
Hints for inference: (hint}
Response format:
(inference Baterrence_fomma__eeee format_examplee example]
Prompt 18: Reward Inference Prompt
want you to act (tas__iffo),
will give you an observation which you are facing in simulator, your task is to response correct results serving as a (reward) function code from the tutorial.
Formally, the format of (reward) function is freward_formaty, The function describes the mechanism for calculating rewards.
Inputs provided:
1. The reward function: lewward_codestteee
2. Current observation: (obs)
3. Action taken: (action)
4. Next observation: (next_obe
36

---

Hints for inference: {hint]
Response format:
Lonference_format_exaann
The rollout process follows Algorithm 3, which integrates the three inference components:
Algorithm 3 Knowledge-based Rollout
1: function ROLLOUTOONESTIEPrreenn obs, policy _ode,, neeard__code transition_coder
2: rext_obs=< < ConveriObsSoTTexttreeeee pocceeeeettbbbs)))eeeeeeee
3: action, action titionn_hhhuhhtttt + Polieylnferencee(eet obs, policy_codde
4: next_obs, trans thans_thhugtttt < eeyeencccnfeeeeeeeeeeeeee,,,,
action, transition _cdee_coee
5: next text_eee___el___eeeeee ooss + PostProcesssObsneext obs, recent obs, action)
6: rewards, reward thought t- RewardInferencee(et_..o action, next texx__eext_ooo text_obs)
7: return next_obs, action, rewards, thoughts
8: end function
F33 Tic-Tac-Tor Toe Prompt Configurations
The following configurations detail the task-specific parameters used in the previously described prompts for the Tic-Tac-Toe environment. These include the task description, knowledge type defini- tions, observation space descriptions, and other parameters that are substituted into the corresponding placeholders in the book content understanding prompts (Appendix F.1), knowledge rehearsing prompts (Appendix F.2).
[task_info)
"An expert Tic- Ti--Tac-Toe player and also expert in python coding and Reinforce- ment Learning."
Senowledge_type
Policy Function:
"The policy in Tic Tac Toe involves strategic placement of X's or O's on the game board to block the opponent's 's moves and create opportunities to complete line. For example, "Always place your mark in the center if it is your first move, as it maximizes potential winning combinations."
Environment Dynamies Function:
"This function describes the rules and changes in the game environment following player actions. For example, "If a player completes row, column, diagonal with the same symbol, they win the game. Otherwise, the game continues or ends in draw if all spaces are filled without a winner."
Rewards Function:
"This function outlines the incentives or penalties based on the game outcomes Tic Tac Toe. For example, "Completing three consecutive symbols horizontally, vertically, or diagonally results in a win and positive reward, while failing to block an opponent's winning move results in loss and negative reward."
(knowledge_ wwwctionnexamppleeee
Policy Examples:
common opening strategy to place the first mark in the center of empty board, which provides the most opportunities for setting up wins in subsequent
37

---

moves. Another strategy involves placing the second mark in corner if the center already occupied by the opponent."
Dynamies Examples:
"The board state changes with each player's move, leading to potential lines for winning. A player wins by placing three their marks horizontal, vertical, or diagonal row. If all nine squares are filled and no player has three in row, the game is a draw."
Reward Examples:
"Yo classify the outcomes follows: 2: Winning move; 1: Blocking
opponent's 's winning move; O: Neutral move; -1: Missing an opportunity to block an opponent's winning move; -2: Making move that leads directly to loss."
(observation_spee) seape_spaeee_ee_ee ]
"The observation space consists of 3x3 grid representing the game board. Each cell can be empty or contain an X or "O" symbol. We use a Ix9 array with number 0, and to represent empty, 'X', and 'O' respectively. The board state is updated after each player's ss move, reflecting the current game configuratinn. We will use another line to tell you whose turn it is, 1 for X and for O."
(action space_desc)
"The action space is discrete from 0 to 8, including 9 possible moves corresponding to the 9 cells on the game board. Players can place their mark ('X (1)' or 'O (2)) in any empty cell to make their move."
finference format example)
Policy Inference:
"For example, it is X's turn and you observe the following board state: [1, 2, 0, 0, 0, O, 0]
The strategic thought might be "Placing X in the center (cell 4) could block O's potential diagonal completion and creates multiple opportunities for X to win on subsequent turns. Therefore, the most appropriate action for in this situation would be to place in the center." Expected output format:
"though"": "Given the current board state [o, 1,
and that it's X's turn...",
"action": 4
}"
Dynamics Inference:
"Important keys in the observation:
board: Current board configuration as list of nine integers (0 for empty.
for X, 2 for O). Example: 0,000,0000000000000
ppppyaratuuulaeealeeeppppppaaaaa turn: Indicates current player (1 for X, 2 for O)
Expected output format:
"thought": "Given the board state [0, 10,1,2,0,,,000,000 and the action to place in the center, the next state of the board should be [0, 10,1,2,0,1,0,0,0..0 reflecting X's placement." "board": "player_turn": 0's turn next
}
38

---

Reward Inference:
"Based on current state, action, and resulting state, compute rewards as:
+2: Winning move
- +1: Blocking opponent's winning move
0: Neutral move
- --1: Missing opportunity to block
- -2: Move leading to direct loss
Expected output format:
"thought": "Given the board state [0,1,2,0,0,0,00000 the action to place X in the center, and the observed next state
With X's placement
resulting in no immediate win or loss and the game still in play, the reward for this action should be 0.", "eense_reward";
}"
F.4
Football Prompt Configuration
Here detail the configuration for the Football:
(task (task_info) info]
"A football manager and also an expert in python coding and Reinforcement Learning"
(knowledge }
Policy Function:
The football manager policy is to give the tactics and strategies for all players in the team, such as how players should be used in the frontcourt during match, or when forwards should take shots. For example: "When watching defenders you have to assess how they respond to their opponents as well as the ball."
Environment Dynamies Function:
Dynamics to give the dynamics function or related rules of the football game under the foothall manager policy's action, such as after shotting, the ball will be in the goal or not. For example: "When the direction of shotting vertical to the goal, the ball will be easy to the goal."
Rewards Function:
Reward is to give the reward or punishment of the football manager policy. For example: "When the forwards are restricted, the midfielder can support and take away the defenders, which is very encouraging behavior."
(knowledge format)
Policy Function:
Policy function: observation action
Environment Dynamics Function:
Dynamics function: observation, action next_observation
Rewards Function:
Reward function: observation, action, next_ oore_vvbevvtionn -> reward
39

---

[knowledge function
function_example)
Policy Function:
"4-4-2 is good formation for a team with a strong midfield, because it allows the team to control the ball and keep possession. To play this formation, the team should have two central midfielders, two wingers, and two strikers. The central midfielders should be able to pass the ball well and control the game. The wingers should be able to run up and down the wing, and cross the ball into the box. The strikers should be able score goals"
Environment Dynamics Function:
"The behavior that is encouraged is when the forwards are restricted, the midfielder can support and take away the defenders. This is a very encouraging behavior because it allows the team to keep possession of the ball and control the game. You should only identify 5 types of rewards: Optimal behavior; 1: Encouraging behavior; O: Borderline behavior; -I: Punishing behavior; -2: Worest behavior."
Rewards Function:
"When the direction of shotting is vertical to the goal, the ball will be easy to the goal."
(observation_spanl saapa__ppeee______ _escc_
"First, it provides information such as the time and score the match. Second, which side has control of the ball, and the active player in the Left team you need to propose corresponding policies to him. Next is the position, direction and role information of each player:
In this text description, the football grass field, which is 120m 90m in the real world, is divided into 240 zones in the simulator, where "x" ranges from 1 to 20 (left to right from the left team's perspective) and "y" ranges from to 12 (bottom top).
Based on this transformation, the concepts in real- wal-world football field can be mapped to the simulator, For instance:
the center circle is (10, 6), and the lower left corner from the left team's view is at (1, 1), and the upper right corner position of the Right team is (20, 12)
the distance from (10, 5) (11, 5) in the real-world is 12m, and the distance from (10, 5) to (10, 6) is 7.5m
the left team's penalty area is zone (1, 4)-(1, 8)-(2, 8)-(2, 4), and the right team's 's penalty area is zone (20, 4)-(20, 8)-(18, 8)-(18, 4)
the center circle position of the field is zone (10, 6), where the game start The position of the player is defined by the player's moving in the correspond- ing direction, including north, northeast, east, southeast, south, southwest, west, northwest
Role information: for each team, player roles are fixed, including Goalkeeper, Forward, Forward, Defender, Defender, Defender, Defender, Midfielder, Mid- fielder, Midfielders, Forward
NOTE: Venues never interchange or change. Following the position information is direction, meaning the direction the player is currently facing and the direction of future actions." "
(action space_saceeeeeee desc)
"The action set includes:
1. action_idle: A no-op action, sticky actions not affected (player maintains his direc- tional movement etc.)
2. action ll Sticky action and will change the player's direction, player will continue to move left until another action is taken. Such from zone(1 11,44 to zone(10.4)
40

---

3. aetion_top_lse Sticky action and will change the player's direction, player will con- tinue move top left until another action is taken. Such as, from zone 1,4) xone(10.5)
4. action_topy Sticky action and will change the player's direction, player will continue to move top until another action is taken. Such as, from zone(11,4) to zone(11,5)
5. ot__oootttrrrtttt___ Sticky action and will change the player's direction, player will continue to move top right until another action is taken. Such as, from zone 1,4) to zone(12.5)
6. action_right: Sticky action and will change the player's direction, player will continue to move right until another action is taken. Such as, from zone(1 to 20ne(12,4)
7. action_ action_bottom bottom_right Sticky action and will change the player's direction, player will continue to move bottom right until another action is taken. Such as, from zone( 1 1,4) to zone(12.3)
8. action, bottom: oction_bottom: Sticky action and will change the player's direction, player will continue to move bottom until another action is taken. Such as, from zone(1 1,4) to zone(1 1,3)
9. action_ action_bottom_____ continue to move bottom left until another action is taken. Such as, from zone(1 1,44 to zone(10.3)
Sticky action and will change the player's direction, player will
10. action_long_pass: The player will long pass to their teammate in his current direction. Before you pass, you should release the sprint if the agent is doing sprint. A long pass covers a large distance on the field
action_ action_highhhnn hat___ihh_p_sss___ The player will high pass to their teammate in his current direction. Before you pass, you should release the sprint if the agent is doing sprint. A high pass sends the ball into the air, often over obstacles, to reach a teammate
12. cwwooonssst__ppee pass: The player will short pass to their teammate in his current direction. Before you pass, you should tune make sure the direction is fine, using action 0-8 to change the direction. A short pass is a quick and close-range exchange between teammates, commonly used maintain possession and build an attack
13. action_shot: Players will try to shoot. When near to the opponent penalty area, such as zone(x, y), x>18, 4<y<8, try to shoot
14. action, action_sprin:: When the player will choose this action, the agent will sprint with sticky action's direction, it will make agent run faster
15. aetion_release _iiasedddreciion Player will stop moving in the current direction. Choose it when you want this agent to change the direction, after this action, you should choose another action from 0-8 to change it
16. action_release__peeeeee sprint
Player will stop sprinting, only when the agent's during the
17. action_sil ol_ddiig: The player will try to slide the tackle. If your position is on the oppo- nent's path with the ball. you can intercept it. However, if the sliding tackle fails, you will be separated by large distance, allowing the opponent to ignore the defense
18. action, dribble: Players will try to dribble. When they have the ball, dribbling will greatly improve the success rate of dribbling, especially in multi-person double-teams and difficult-to-hnndee situations
19. action_releasee driibbee: Player will stop dribbling"
(hint)
Policy hint:
For example, if the active player is Player 2 and you want him to close to the ball and control it, in the texted observation:
Forward Player 2 is Zone(9.9)
The ball is at (one(11,8)
41

---

Your thought should be: "Given these coordinates, the ball is diagonally one zone the right (east) and one zone down (south) from the player's current position. The most direct route to the ball would indeed be diagonally towards the bottom right. Therefore, the most appropriate action for Forward Player 2 this situation would be: oation_bottons hat__ttttttttrhhtttttte
Dynamics hint:
Important keys in the observation:
score: The current score the match, should be a list two integers, such [0,0]
step:
The current step of the match
ball_ownership for left team, 2 for right team, 0 for no team/during passing
The zone of the active player [x.y] ball_zone: Ball position [x.y]
NOTE: if the active player still owner the ball, the player zone and the ball zone should be the same.
Based the current texted observation and the action set, generate the next step text observation.
For instance, the current observation shows the Left team player 3 in zone (12,4) and the chosen action is "Top," " the next observation should show the player in zone (12,5).
Reward hint:
Based on the current texted observation and the action set, generate the dense reward of the current step's 's action and state, based on the reward code above. The rewards should be one of -2, -1, 0, or
finference format_ )
Policy Inference:
"thought"; "give your thought to generate the action the current step's state, based on the policy code above.". "action":
}
Dynamics Inference:
"thought": "based on the current texted observation and the action set, the ball and player are in the zone [9,4], and the active player is controlling the ball, and choose the action go top, thus consider the right have very low chance intercept the ball, the ball will be in the zone [9,5] in the next time step.", "score": [0,0],
"step": 2,
"ball_ "ball_oomership": 1,
"ball_zone". [9,5],
chett_active_playe_zooce zone": [9,5]
}
Reward Inference:
"thought": "give your thought to generate the dense reward of the current step's action and state, based the reward code above.", "dense_ :eensee_ewwrd";
42

---

F,5
Introspecting based on the Imaginary Dataset
We implement CIQL based on the open source codes CQL in d3rlpy [62], where the uncertainty of rewards and transitions are estimated by an ensemble of 20 Gaussian models based on 4-layer full- connected neural networks [6, 47J, which are learned through standard log-likelihood maximization from the imaginary dataset.
F.6 General Recipe for Prompt Design
The specific prompt design would significantly affect the quality of the LLM outputs and the algorithm built on them. Though URI does not focus on proposing new prompting techniques, we briefly summarize the general design principle as follows in our practice. These principles are adopted in implementations for both URI and all baseline methods in our experiments.
Let LLMs output the "thoughts'analyze" first before outputting the results. The concrete prompts following this principle can be found in the "Response example" part in our prompts in Ap- pendix E;
2. Make the requirements as explicit as possible. The concrete prompts following this principle can be found in the "Requirements" part in our prompts in Appendix E;
Use pseudo-code instead using natural language to represent the knowledge.
(I) and (2) are well-known principles and have been verified in other studies many times. For (3), we would like to highlight its effectiveness based on the following ablation study. We conduct the same experiment setting as Fig. 4(a) by using natural language to represent the knowledge. As shown in the table below, if we switch to the natural language representation, whatever the embedding models and retrieval techniques we used, the hit rate the retrieval will drop a lot, as demonstrated by Tab. 4. We hope that this could provide insights for people interested in designing similar LLM-based frameworks for other decision-makinn tasks.
Table Comparison different knowledge representation
code
natural language
perf drop rate (%)
embedding baai
0.076
0.035
53.9
sabbedding-opentt
0.085
0.075
11.8
summary-bart
0.079
0.055
29.7
summary-openai
0.077
0.056
27.3
URI-baai
0.342
0.051
85.1
URI-openai
0.338
0.056
83.4
avg
48.5
F.7 Hyper-parameters
Here, we list the important hyperparameters for URI here.
Table URI Hyperparameters.
Parameter
Value
pieces of knowledge for each time of knowledge aggregation Nagg retrieved segments Neat
15
learning rate (A)
0.0001
weight of transition W/T
0.5
weight of reward penalties 7J
0.5
horizon of rollout I!
10
weight of conservative loss number of ensemble Nens
60
20
43

---

F.8 URI Algorithm
This algorithm implements the URI method for policy learning from books. It consists of three main stages: 1) Understanding (L1-L2): knowledge is extracted and aggregated from the input book using CodeExtractor and CodeAggregator. 2) Rehearsing (L3-L14): An imaginary dataset is generated by iteratively applying knowledge retrieval, instantiation, and language model-based simulation. 3) Introspecting (1.15-L16): policy learned from the imaginary dataset using Conservative Imaginary Q-Learning (CIQL). The algorithm leverages language models M,, M and Mr for action selection, reward estimation, and state transition, respectively, bridging the gap between textual knowledge and reinforcement learning.
Algorithm 4 URI (Understanding, Rehearsing, and Introspecting)
Require: Book B, simulator information [M], LLM-injected dynamics function Mr, reward func- tion MR. and policy M., imaginary dataset Dang + 0.
1: KC +- CODE EXTRACTOR(B)
> Sec. 5,2 (Algorithm 1)
2: K'' < CODE AGGREGATOR (KC)
> Sec. 5.2 (Algorithm 2)
3: for = 1 to num_ ton__rrjcctories do
4:
8 <- SAMPLE INITIAL. STATE((MM)
5:
for -= 1 to H do
b Sec. 5.3
6:
KCS KNOWLEDGE SCOPE RETRIEVAL (s; IC")
7:
K2, K K4 < KNOWLEDGE INST(?,, [M], Recope (St, Kc
8:
0r + M, K4)
D. Imaginary Action Imaginary Reward Imaginary Next State
9:
ft MR(Be, des K()
10:
81+1 (T de; K()
11:
D Ding add(8e. ae, Pe, 8141)
8c Se+1
13:
end for end for
15: π + CIQL((Ing)
> Sec. 5.4
return
44

---

G Additional Ablations
G.1 Data Aggregation Visualization for Tic-Tac-Toe
Similar to the visualization analysis in the main text, we also conduct t-SNE [59] visualization for the Tic-Tac-Toe envi- ronment to examine the quality of imaginary data generation and uncertainty estimation. As shown in Fig. despite the discrete nature of the state space in Tic-Tac- Toe, we observe similar patterns in the football environment. The imaginary data (separated into low and high uncertainty regions) largely follows the distribution of real data collected from optimal minimax gameplay. The uncertainty estimator successfully identifies deviant clusters (marked by yellow dashed circles) that lie outside the distribution of optimal play data. This con- sistency in behavior across both simple discrete environments and complex continuous ones supports the robustness of URI's data generation and uncertainty estimation mechanisms.
unc
real data
Figure 8: Visualization of the pro- jected distributions for real and imagi- nary datasets in the To--Tac-Te envi- ronment.
G.2 Performance Comparison with Additional Baselines in GRF
To further validate the effectiveness of our approach, we implement three additional baselines: (1) RT-Policy: a policy trained using LLAMA following the RT-style architecture [63], using the same 7,500 initial states as URI and rule-based actions for training: (2) URI w/o BK: variant of URI that generates imaginary data without using book knowledge; and (3) CQL: pure offline RL approach using the Conservative Q-Learning algorithm with real interaction data.
Table 6: Performance Comparison of RT-policy, URI w/o book knowledge, CQL,, Against Built-in AT Levels in a GRF 11 vs 11 settings, where the performance of URI is averaged among three different seeds, all methods tested with 20 matches except RT- RT-Policy which tested with 10 matches due to computation resource limitation.
Easy
Medium
Hard
Averge
Malhed
Wi-Phiccy
055
SUS
50%
-2.00
7094
S0F
-1.40
0%
905
-3.30
-1.79
URI we BX
30%
55%
15%
0.16
22%
58%
299
0.04
22%
50%
29%
-0.06
0.04
COL
49%
181
0.20
16%
499
196
0.32
17%
5196
MIS
41.30
0.07
URI (Ours)
37%
5799445
09318
0.30
42%
30%
558445
33%
585
109
00.1
0.38 IUI
H Examples of Data Generation and Policy Execution Videos
We list more visualization results for each stage of URI in the links: attps://plfo-football gi thub. io/, including examples of knowledge extraction, code aggregation, imaginary data, and videos of policy execution in GRF
I Open Problems and Future Directions
We hope that this promising result will initiate more research on PLfB. We will first elaborate on the points mentioned in Sec. on applying URI when textual data are insufficient of low-quality.
Multimodal Data Integration: Beyond purely textual data, additional modalities such as tutorial voices, demonstration videos, and replays can be incorporated. By employing advanced multimodal large language models, these diverse forms of data can be processed in manner akin to the current handling within the URI, thereby augmenting the knowledge base and enhancing the robustness of policy learning.
Utilization of Real Interaction Data: During the introspection phase, incorporating real inter- action data with the target environment, rather than relying solely on sinulated data generated by large language models, can enhance policy learning. This mixed-data approach can be used to further fine-tune various modules within the URI framework, potentially boosting overall per- formance. It also mimics human learning behavior better humans Iearn from both reading and
45

---

actual experience. However, one might develop new techniques to utilize both sources of data for better policy learning.
Injection of Prior Knowledge:
The URI framework allows for the integration of human expert knowledge at different stages of the pipeline. Experts can provide specific code knowledge popnseenations/oommlaatonnnnnntt
when generating code-based knowledge. We also provide constraints during the rehearsal process to enhance stability and realism.
Evaluating the Quality of Textual Data: Most current studies in this fields [64-66] are still in primitives. These studies suggest frameworks and empirical strategies that can be adapted to evaluate the relevance and quality of textual and multimodal data for training purposes. However, given the significant modality gap between textual data and the neural network parameters in PLfB topics-and the largely unsupervised nature of our learning process- this is not a trivial problem. This complexity requires innovative approaches for assessment.
We are also compelled to investigate other potential applications of the framework beyond the scope of the current experiment. Domains such as embodied AI for robotics, which require a large amount of data for foundation policy learning and the robotics dataset currently, could benefit significantly from the integration of the paradigm for data augmentation. On the other hand, currently we just formulate the problem that multi-agent systems can be built for this problem in the future work [67]. This exploration opens exciting avenues for future research and application of the method.
MDP and build single-agent system for the football tasks, believe
Compute Resources
All experiments were conducted on begh-performance computing (HPC) system featuring 128 Intel Xeon processors running at 2.2 GHz, TB of memory, an Nvidia A100 PCIE-40G GPU, and two Nvidia A30 GPUs. This computational setup ensures efficient processing and reliable performance throughout the experiments.
K
Broader Impact Statement
Integrating Large Language Models (LLMs) with Reinforcement Learning (RL) in PLfB offers groundbreaking approach to Al skill acquisition. This innovation has the potential to revolutionize industries reliant automation, such manufacturing and robotics, by enabling Al agents to learn complex tasks from textual sources, reducing the need for extensive real-world data. However, ethical considerations regarding bias, privacy, and transparency paramount. Overall, PL.B promises enhanced efficiency and accessibility but requires careful navigation of its social implications.
46

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: introduce the contribution in the abstract and the introduction section.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The limitation is discussed in Appendix I.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Answer: [Yss]
Justification: We provided all details including prompt, LLM version, and hyper- parameters in Appendix F guarantee the reproducibility.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We commit to open-source the code that can reproduce all the experiment results after the paper published.
Guidelines:
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provided the experiment setup in Appendix B and the start of the experiment
section.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The results reported are averaged over three random runs except for these with high cost of running, like L-a---s-geent and LLM-RAG
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
47

---

Answer: [Yes]
Justification: We provide the compute resources in Appendix J.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: http//eerrpp..p//
Answer: [Yes]
Justification: The research conforms with the NeurIPS Code of Ethics.
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We provide the broader impacts statement in Appendix K.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks, as it does not release new pre-trained models datasets.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The assets/enviromment used in this paper are properly credited: Google Research Football [56], d3rlpy [62], and RedPajama- [17].
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
48


Reviews:
[{"review_id": "1m4v6itL5i", "reviewer": "Reviewer_uFxN", "summary": "This paper introduces Policy Learning from Books (PLfB). This framework leverages the knowledge encoded in textual books and tutorials to train decision-making policies, specifically for playing football, without requiring direct interaction with the environment. This method is a three-stage framework Understanding, Rehearsing, and Introspecting -- **Unstanding**: extracts knowledge from books, uses it to **rehearse** decision-making trajectories in an imaginary dataset, and then **introspects** on the imagined data to distill a refined policy network.\n\nThey found that the URI approach significantly outperforms baseline methods in the Google Research Football (GRF) 11 vs 11 scenarios, and that the iterative process of code extraction and aggregation significantly reduces the number of code segments for dynamics, policy, and reward functions.", "questions": "Can the URI framework be iterated -- collecting data online and summarize new knowledge. And the knowledge can be passed on to the next iteration of URI.", "limitations": "Yes.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. This paper is novel in terms of the method proposed. The URI framework is very intuitive and could be applied to other domains of agent learning. \n2. Using imaginary dataset generated based on extracted knowledge is an interesting idea for synthetic data generation. \n3. The efficiency of the method is also impressive for real world applications.", "weaknesses": "1. The main weakness could the narrow scope of application in this paper. It is not sure how data quality from the collected textbook data could affect the performance the model. If there are more domains where the authors could experiment with different data sources, the readers would have better expectation of the model\n2. The second weakness is the complex framework has many components. This on one hand is the novelty of this paper, however, it also adds to the difficulty of replicating this method, especially on other domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: If there are more domains where the authors could experiment with different data sources, the readers would have better expectation of the model.**\n\nWe agree that applying URI to more domains would strengthen readers' understanding, expectation, and belief in it. Thus, we build a new proof-of-concept benchmark based on the classic Tie Tac Toe game (TTT) and verify URI. We kindly recommend the review to check the details of the results in the first section of the global response letter.\n\n**Q2: Concern about the complex framework, which has many components. This on one hand is the novelty of this paper, however, it also adds to the difficulty of replicating this method, especially in other domains.**\n\nWe acknowledge the reviewer's concern about the complexity of the implementation. In response to the concern, we would like to use high-quality open-source code to keep this study easy to follow and reproduce. We kindly recommend the review to refer to the second section of the global response letter for our open-source plan. We will also continue to improve the text to make the whole paper more accessible.\n\n**Q3: Can the URI framework be iterated -- collecting data online and summarize new knowledge. And the knowledge can be passed on to the next iteration of URI.**\n\nIt is an open question but we truly believe the future of URI should be iterative. It is intuitive since even when humans learn a new skill from books, it is not enough to be competent in tasks by just learning from books. We need real-world practices to bridge the knowledge-utilizing gaps in books. However, it is non-trivial to design a practical method to fully utilize the **online data to improve the URI pipeline.** We have made a brief discussion about such an extension as a future research direction in Appendix F. We are excited to dive deeper into this as future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZTrEuJXWtl", "reviewer": "Reviewer_HEXR", "summary": "The paper introduces an intriguing approach to reinforcement learning (RL) through the concept of Policy Learning from Books (PLfB), which leverages textual resources like books and tutorials to derive policy networks. This methodology represents a interesting departure from traditional RL techniques that rely heavily on real interactions with the environment as recall by the authors.\n\nThe proposed URI framework outlines how the system first comprehends the textual content, then rehearses decision-making trajectories, and finally introspects to refine the policy network using an imaginary dataset. \n\nThe practical validation of this method is demonstrated by training a football-playing policy and testing it in the Google Football simulation environment. The results are promising, with the agent achieving a 37% winning rate against the built-in agent without any interaction with the environment during training. This is a substantial improvement over the 6% winning rate achieved using an LLM.\n\nIn addition, the paper addresses the question of extracting policies without direct environment interaction by incorporating descriptions of MDP structures, transition functions, and reward functions within the textual data. This ensures the feasibility of the PLfB approach and adds depth to the methodology.\n\nHowever, one aspect that could have been elaborated on is the influence of the prompting strategy used for generating the imaginary dataset. Detailing how different prompting techniques impact the quality and effectiveness of the dataset could provide valuable insights and enhance the robustness of the proposed approach.\n\nOverall, the paper is a interesting contribution to the field, proposing an original perspective on utilizing textual knowledge for policy learning wel align with this current epoch of LLMs development. The results are encouraging, and the methodology is articulated and validated through practical experiments.", "questions": "1. Prompting Strategy Exploration: Could you elaborate on the specific prompting strategies used to generate the imaginary dataset? How do you believe different strategies might influence the quality and effectiveness of the policy learned?\n\n2. Generalizability Across Domains: What steps do you envision for testing the generalizability of the Policy Learning from Books (PLfB) approach in different domains or environments? Have you considered any preliminary experiments in varied contexts?\n\n3. Quality of Textual Resources: How do you plan to address potential limitations related to the quality and comprehensiveness of the textual resources used? Are there specific criteria or methods you would recommend for selecting or evaluating these resources?", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* Innovative Concept: Introduces the novel idea of Policy Learning from Books (PLfB), leveraging textual resources for policy network derivation, which is a significant departure from traditional RL methods.\n* Human-Like Learning Process: The URI framework—understanding, rehearsing, and introspecting—mimics how humans learn from books, making the approach intuitive and biologically inspired.\n* Promising Results: Demonstrates a 37% winning rate in the Google Football simulation environment, significantly outperforming a Large Language Model (LLM) which achieved only a 6% winning rate.\n* Feasibility and Depth: Incorporates detailed descriptions of MDP structures, transition functions, and reward functions within the textual data, ensuring the feasibility of extracting useful policies without direct environment interaction.\n* Practical Validation: The methodology is well-validated through practical experiments, strengthening the credibility and significance of the research.\n* Alignment with Current Trends: The approach aligns well with the current advancements in LLMs, making it relevant and timely.", "weaknesses": "* Prompting Strategy Details: The paper lacks detailed discussion on the influence of the prompting strategy used for generating the imaginary dataset. Exploring different prompting techniques could provide valuable insights and improve the approach's robustness.\n* Textual Resource Dependence: The success of the approach heavily relies on the quality and comprehensiveness of the textual resources, which might limit its applicability in domains with sparse or low-quality textual data.\n* Generalizability: The generalizability of the method across different domains remains uncertain and needs further exploration to ensure its broad applicability.\n* Complexity of Implementation: The methodology, while innovative, might be complex to implement and require significant computational resources, which could be a barrier for some researchers or practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: Prompting Strategy Details & Exploration: Could you elaborate on the specific prompting strategies used to generate the imaginary dataset? How do you believe different strategies might influence the quality and effectiveness of the policy learned?**\n\nWe agree that the design of prompts matters in LLM-related methods. Though we do not focus on propose new prompting techniques,  in our practice, the following principles of prompt designs are important in this projects:\n\n1. Before output the results, let LLMs output the “thoughts/analyze” firstly, and we will give an example of thoughts in the prompts. Reference to the “Response example“ parts in our prompts in Appendix C and D for more details.\n2. Make our requirements as explicit as possible. Reference to the “Requirements“ parts in our prompts in Appendix D.\n3. As mentioned in Section 5.2, instead of using natural language to represent the knowledge, we use pseudo-code. \n\nSince (1) and (2) are well-known principles and have been verified in other studies many times, in the following, we ablate the effects of knowledge representation. We conduct the same experiment setting as Figure 5(a) in the main body by using natural language to represent the knowledge. As shown in the table below, if we switch to the natural-language-representation, whatever the embedding models and retrieval techniques we used,  the hit-rate of the retrieval will drop a lot. \n\n|  | code | natural language | perf drop rate (%) |\n| --- | --- | --- | --- |\n| embedding-baai | 0.076 | 0.035 | 53.9% |\n| embedding-openai | 0.085 | 0.075 | 11.8% |\n| summary-baai | 0.079 | 0.055 | 29.7% |\n| summary-openai | 0.077 | 0.056 | 27.3% |\n| URI-baai | 0.342 | 0.051 | 85.1% |\n| URI-openai | 0.338 | 0.056 | 83.4% |\n| avg | / | / | 48.5% |\n\nIn response to this concern, we commit to: (1) describe the key prompting strategies we used to implement URI to the revised paper; (2) add the above results to the revised paper.\n\n**Q2: Generalizability Across Domains.**\n\nThanks for the valuable question. We acknowledge applying URI in just single domains might raise concerns about the generalizability of the methodology. In response to this concern, we build a new proof-of-concept benchmark based on the classic Tie Tac Toe game (TTT) and verify URI. We kindly recommend the review to check the details of the results in the first section of the global response letter.\n\n**Q3: Complexity of Implementation**\n\nWe acknowledge the reviewer's concern on the complexity of the implementation. In response to the concern, we would like to use a high-quality open-source code to keep this study easy to follow and reproduce. We kindly recommend the review to check  our open-source plan in the second section of the global response letter.\n\n**Q4: Quality of Textual Resources**\n\n*Q4.1: How do you plan to address potential limitations*\n\nThe effectiveness of the policy derived by the URI framework fundamentally hinges on the quality of the textual data. It is indeed important that the textual resources should cover sufficiently the dynamics, policy, and rewards of the targeted environment so that relevant knowledge and imaginary data can be extracted to train the policy, otherwise it is impossible to derive a good enough policy.  There are several potential approaches to address this limitation:\n\n1. **Multimodal Data Integration**: Beyond purely textual data, additional modalities such as tutorial voices, demonstration videos, and replays can be incorporated. By employing advanced multimodal large language models, these diverse forms of data can be processed in a manner akin to the current handling within the URI, thereby augmenting the knowledge base and enhancing the robustness of policy learning.\n2. **Utilization of Real Interaction Data**: During the introspection phase, incorporating real interaction data with the target environment, rather than relying solely on simulated data generated by large language models, can enhance policy learning. This mixed-data approach can be used to further fine-tune various modules within the URI framework, potentially boosting overall performance. However, one might develop new techniques to utilize both sources of data for better policy learning.\n3. **Injection of Prior Knowledge**: The URI framework allows for the integration of human expert knowledge at different stages of the pipeline. Experts can provide specific code knowledge representations/formulations/templates when generate code-based knowledge.  We can also provide constraints during the rehearsal process to enhance stability and realism.\n\n*Q4.2: Criteria for Selecting and Evaluating Resources*\n\nEvaluating the quality of training data remains a challenging issue across machine learning community. There are possible methods [1-3] that might guide our evaluation. These studies suggest frameworks and empirical strategies that can be adapted to evaluate the relevance and quality of textual and multimodal data for training purposes. However, given the significant modality gap between textual data and the neural network parameters in PLfB topics—and the largely unsupervised nature of our learning process—this is not a trivial problem. This complexity requires innovative approaches for assessment. \n\n[1] QuRating: Selecting High-Quality Data for Training Language Models\n\n[2] An Empirical Exploration in Quality Filtering of Text Data\n\n[3] DoGE: Domain Reweighting with Generalization Estimation\n\nWe believe these discussions are valuable for the research community. In response to this question, we commit to include them in Appendix F of the revised paper. Additionally, we would like to highlight that many real-world decision-making scenarios have an abundance of textual tutorial resources, such as medical diagnosis, financial trading, software development, and educational tutoring. Thus, even in domains with rich textual resources, the topic remains broad and highly valuable.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wS1bStlbaD", "reviewer": "Reviewer_mSmU", "summary": "The paper introduces a novel approach to policy learning, termed \"Policy Learning from Books,\" which leverages existing textual knowledge, such as books and tutorials, to develop policy networks without the need for extensive real-world interactions. This method is inspired by how humans learn new skills from written resources. The authors propose a three-stage framework called URI (Understanding, Rehearsing, and Introspecting) to implement this approach. In the URI framework, the process begins with understanding the content of the books, followed by rehearsing decision-making scenarios based on the understood knowledge, and finally, introspecting on these rehearsed scenarios to refine a policy network. To demonstrate the effectiveness of this method, the researchers applied it to train a football-playing policy using the Google Football game. The trained agent, which did not interact with the environment during training, achieved a 37% winning rate against the built-in AI, significantly outperforming a GPT-based agent that only managed a 6% winning rate. This study highlights the potential of utilizing textual knowledge for enhancing decision-making processes in reinforcement learning.", "questions": "1. I am curious to the performance of distilled policy from the imaginary dataset generated directly by the GPT without the information from the books?\n2. Why not provide the pseucode for this work in the anonymous repository? Maybe the reproducibility of this work can be further enhanced.\n3. The caption and illustration of Figure 1 are a little bit confusing. Personally, I think book tutorial is also a kind of data, though it is not the real interaction data between the agent and the interaction. I suggest the authors revise this point in the future version.\n4. The Figure 2 is somewhat redundant, considering the detailed 3-stage framework: understanding, rehearsing, and introspecting, has been clearly provided in the Figure 3.", "limitations": "See above weaknesses and questions.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I think the studied topic: policy learning from book is interesting and meaningful to the community. Maybe it can be adopted as a kind of new policy learning paradigm from the novel data sources, beyond the traditional reinforcement learning and imitation learning. It somehow provides the potential that does not require the extensive agent-enviroment interaction data anymore.\n2. From the paper, the proposed method framework including the understanding, rehearsing, and introspecting is reasonable and intuitive.\n3. The writing of this paper is easy to follow and the paper structure has been carefully organized. \n4. The empirical results present in the experiment part are generally persuasive.", "weaknesses": "1. More datasets can be considered in the experiments.\n2. I think the potential application of this work is beyond the football game. Why not validate its effectiveness on learning some other policies and validating their performance in some different challenging environments?\n3. As for the baselines, the authors only compare their proposed framework with the LLM-based and rule-based policies. Why not compare it with some other policies learned with conventional RL algorithms, like PPO, DDPG, SAC, and so on?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: add baselines: (1) distilled policy from the imaginary dataset generated directly by the GPT without the information from the books; (2) compare it with some other policies learned with conventional RL algorithms**\n\nThanks for the valuable suggestions.  In the rebuttal period, we implement the two baselines named “URI w/o BK” and “CQL”, which can be seen in Table 2 in the attached file. Since the PLfB is more like an offline setting, we use a popular offline RL baseline CQL to demonstrate the results of conventional RL algorithms. We found that pure URI without book knowledge and pure offline RL algorithm can reach competitive performance compared with the baselines but URI still achieves significantly better results (0.38 vs 0.04 in average GDM for “URI w/o BK” and 0.38 vs 0.07 for “CQL”). The results demonstrate that purely using the dataset or the inner knowledge of LLMs cannot should the problems. We kindly recommend the reviewer check the detailed results in the attached PDF. \n\nIn response to the suggestion, we commit to adding these two baselines to our revised paper. We believe it will make the effectiveness of URI more convincing.\n\n**Q2: Why not provide the pseucode for this work in the anonymous repository? Maybe the reproducibility of this work can be further enhanced.**\n\nWe plan to add a pseudo code to the revised paper (see Algorithm 3 in the attached PDF) to make the interactions among these components unambiguous. Besides, we commit to open-source high-quality code to help users understand the full details of this project. The global response contains the complete plan and scope of the open-sourcing.\n\n**Q3: The caption and illustration of Figure 1 are a little bit confusing. Personally, I think book tutorial is also a kind of data & The Figure 2 is somewhat redundant, considering the detailed 3-stage framework: understanding, rehearsing, and introspecting, has been clearly provided in the Figure 3.**\n\nThanks to the reviewer for pointing out these two presentation problems. We agree that the book tutorial can also be regarded as a source of data. We will change it to “interaction trajectories” in the revised version. Figure 2 is the proposed general URI methodology for the problem of PLfB and we expect the readers to quickly develop an idea about our methodology. Figure 3 is more about solution implementation: it provides an overview of the exact implementation for PLfB in this study. We will improve both figures and captions to make such a distinction more clear.\n\n**Q4: More datasets & applications can be test.**\n\nWe acknowledge that applying URI in just single domains is not enough to demonstrate the generalizability of the methodology. To alleviate such a concern, we build a new proof-of-concept benchmark based on the classic Tie Tac Toe game (TTT) and verify URI's performance on it. In short, URI continues to behave well. The benchmark setting and more results are elaborated in the global response", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uDiUpOZX7D", "reviewer": "Reviewer_b6EF", "summary": "The paper presents a method to distill knowledge about a given task or domain from text based knowledge into a form that can be used train a RL policy. The method extracts knowledge from text with a LLM, which is represented in a pseudocode-like textual form, and uses the knowledge to turn the LLM into a dynamics function, a reward function, and policy. Then the LLM is used to generate example trajectories by conditioning on the different types of extracted knowledge to turn it into a dynamics function, a reward function, and a policy. Finally conservative Q-learning is applied to the generated trajectories to learn a policy robust to the noise in the trajectory dataset. The method is evaluated on the Google Research Football environment against several baselines. The results demonstrate that the method improves performance relative to the baselines.", "questions": "- How does this method relate to the use of background textual knowledge in \"Motif: Intrinsic Motivation from Artificial Intelligence Feedback\" and \"Learning to Model the World With Language\"?\n- How beneficial is learning with a LLM versus learning a policy with the task reward and no LLM?", "limitations": "The limitations are not discussed in the main body of the paper", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Directly distilling knowledge from textual sources into control tasks is an important topic, and this paper takes a strong step in that direction.\n- The paper's experiments are decently extensive and dig into the details about how/why URI has the observed improvement gains.", "weaknesses": "- It was difficult for me to follow what exactly what does in the method. The amount of notation is maybe obfuscating what the specifics of the method is.\n- A class of methods that use LLMs as the backbone is missing, e.g. RT-2 or \"Large Language Models as Generalizable Policies for Embodied Tasks\"\n- It would have been helpful to see how well URI does in a head-to-head match with the baselines (e.g. LLM-as-agent, LLM-RAG, and Random Policy). Given that the knowledge comes from books, which likely talk about semi-skilled strategies, it is unclear how important it is for the opponent to be of a similar quality as is discussed in the textual material. Head-to-head match ups would help understand how general and robust the policy is.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: It was difficult for me to follow what exactly what does in the method.** \n\nWe are sorry that the current presentation of URI is not straightforward enough. As URI’s implementation involves interactions among several components from different research domains,  it indeed might look complex, especially for readers who are not familiar with one of these domains.  In response to this question, we plan to add a pseudo code to the revised paper (see Algorithm 3 in the attached PDF) to make the interactions unambiguous. Besides, we commit to open-source our full-procedure code to help users understand the details of this project. We kindly recommend the reviewer check our open-source plan in the global response.  We hope these efforts will make this study more easy to follow.\n\n**Q2: A class of methods that use LLMs as the backbone is missing.**\n\nThanks for the valuable suggestion. RT is a classic baseline missing before. We implement an RT-style policy for football based on LLAMA and report the results in Table 2 and Figure 1 in the attached PDF. We use the same 7,500 initial states as URI and the rule-based actions to train the RT policy. After training, the loss can be reduced to normal as shown in Figure 1. However, though we tried our best to improve the performance, RT-policy turned out to fail in reaching the goal (See Table 2). We assume it is due to the size of training data is too small to be used for generalizable policy imitation. We will keep on trying but we cannot guarantee that we can get a better result. \n\nIn response to this suggestion, we plan to (1) add RT-branch studies to the related work and (2) add the best performance we can reach to the revised paper.\n\n**Q3: It would have been helpful to see how well URI does in a head-to-head match with the baselines.**\n\nWe agree that the head-to-head evaluation is valuable and will make the effectiveness of URI more convincing. Due to the limitation of the current simulator implementation, it will require considerable engineering work to modify the current Football environment to support the simultaneous evaluation of two policies. We are trying our best to finish such a job and commit to reporting the results in the revised version. \n\nOn the other hand, the extended experiment in Tic-Tac-Toe does include head-to-head evaluations in the attached PDF. The results show that URI significantly outperforms the two major baselines LLM-as-agent and LLM-RAG with **+66%** and **+44%** net win rate. We hope this result can also solve the reviewer's concern.\n\n**Q4: How does this method relate to the use of background textual knowledge in \"Motif: Intrinsic Motivation from Artificial Intelligence Feedback\" and \"Learning to Model the World With Language\"?**\n\nThanks the reviewer for pointing out these two related works. We will add a brief introduction and comparison with them in the related work section.  Specifically,\n\n1. Motif: Intrinsic Motivation from Artificial Intelligence Feedback: the language is used as a caption of the observation. LLM is used as a surrogate reward model to output an evaluation (preference) of the observation (state) based on its caption. Such a preference model is then used to distill a reward model. Given this reward model, the policy is still trained in an online fashion. This could be classified into the LLM as a reward model category in our related work.\n2. Learning to Model the World With Language: the observation contains both image and language. LLM is used as a part of the world model to predict future observations. The training paradigm is still the traditional Model-based RL, but the model here is multi-modal. This could be classified into the \"LLM as dynamics model category\" in our related work. \n\n**Q5: How beneficial is learning with an LLM versus learning a policy with the task reward and no LLM?**\n\n Since the PLfB is more like in a offlineRL setting, we tested the performance of CQL as the result of policy learning with the task reward and no LLM. The results are in Table 2 in the attached PDF. In summary, CQL also achieved a competitive performance compared with other baselines, i.e., LLM-as-agent and LLM-RAG, but still significantly underperforms URI (0.07 vs 0.38 in average GDM). This result demonstrates the benefits of URI in improving the policy's performance compared with standard policy learning methods. \n\n**Q6: The limitations are not discussed in the main body of the paper.**\n\nWe would like to kindly point out that the limitation of this paper is mentioned in Section 7. Since the space is limited in the main body, we leave a link to Appendix F in Section 7, where give our full discussions and limitations of this study.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their constructive and thoughtful feedback. We appreciate all the recognition and kind comments on our work, including **conceptual novelty and enlightenment** (R1, R2, R3, R4); **realistic, extensive, and well-motivated experiments** (R1, R2, R3, R4); **promising results** (R1, R2, R3);  **intuitive motivation** behind the method design (R1,R2,R3). Beyond the work itself, reviewers also recognize that ideas in our work could be further utilized under a **broader context** including synthetic data generation and learning from richer resources (R1, R3).

In the following, we report our responses to the common concerns and suggestions proposed by the reviewers.

### **1 Generalizability of the URI methodology to other domains**

We acknowledge applying URI in just single domains might raise concerns about the generalizability of the methodology. In response to this concern, we build a new proof-of-concept benchmark based on the classic Tie Tac Toe game (TTT). In particular, we use a minimax policy, which is the optimal policy in this game, to collect all possible trajectories in this game. Then, for each trajectory, we use GPT to derive textual books by summarizing the trajectories, and analyzing the game mechanics, winning conditions, and strategic principles we can learn from the trajectory. Since we have all optimal trajectories, the process guarantees the textual book is high-quality to cover complete knowledge in this game (ignore the information loss of GPT generations) so it is an ideal testbed for PLfB.

We then apply URI in the TTT textual book. *Note that we use the same prompt template and just modify the task-specific contents. Our results can then be achieved by tuning the weight of transition, reward penalty, and conservative loss in CIQL, which are 0.05, 0.05, and 0.1 respectively*. The results are in Table 1, and Figure 2 in the attached PDF file. Our key findings are as follows:

1. As shown in Table 1, URI demonstrates superior performance across all opponents in head-to-head matches, where it achieves the highest net win rate (win - loss) of **+66%, +44%, +52%** against LLM-as-agent, LLM-RAG, and Random Policy, respectively.
2. Besides, against Minimax-noise, URI **still maintains a positive win-loss percentage**, indicating competitive ability even when facing a near-to-optimal strategy, while all baselines can only get negative net win rates.
3. We also apply several important experiments to show the effectiveness of URI components in the main body to the TTT environment. In particular, knowledge segment aggregation (Figure 4 in the main body) is in Figure 2 and visualization of the projected distributions for real and imaginary datasets (Figure 7 in the main body) is in Figure 3. The results are similar among these two environments, which further demonstrate the effectiveness of URI components. In particular, for knowledge segment aggregation, **the number of knowledge pieces will be reduced and converge after 4 iterations**; for visualization, **high-uncertainty regions can be identified** by the uncertainty predictor (marked with yellow ovals), while low-uncertainty regions (marked with blue ovals) are generated surrounded by the real data. We also found that the generated data cannot cover the real optimal trajectories, which indicates that there is still room to improve the quality of the trajectory imaginary in the rehearsing stage.


We again sincerely thank the reviewers for this valuable question. We believe that these results strongly demonstrate the generalizability of the URI methodology to other domains and further increase the quality and potential impacts of this study. We also commit to open-source the code and data of this additional experiment, such that researchers can refer to the modifications made in this experiment and adapt URI to other domains that they are interested in.

### **2 Complex to implement URI, especially in other domains**

We acknowledge the reviewer's concern about the complexity of the implementation. In response to the concern, we would like to use high-quality open-source code to keep this study easy to follow and reproduce. In particular, we commit to:

1. *Open-source the two environments*: there are several designs based on the original simulator to run our experiments, especially the transformations between the text-based and vector-based state/action space (as shown in Appendix A). We will open-source our Football environment and Tie Tac Toc environments with full details in our experiments to help researchers with further development.
2. *Open-source the full URI procedure*:  all the data collecting and training scripts of the full URI procedure, with the generated data in the process, including the results of Football tasks and TTT tasks will be open source.
3. *Configurable implementation*: To meet the requirements of users who want to quickly check the details of URI or test URI on other domains, we will refactor our code and use a single configuration file to all set/get the domain-specific config/information.

Meta-review: The paper introduces a novel and timely approach to solve reinforcement learning problems by incorporating textual knowledge about the problems. The experiments convincingly show that the approach can be effective, and significantly better than RL baselines and ablations of the approach.

All of the reviewers agree that the paper is technically sound, clearly written, with high significance and novelty. The authors in their feedback provided additional experiments and discussion that will make the revised paper stronger.
Decision: Accept (oral)