Policy Learning from Tutorial Books via Understanding, Rehearsing and Introspecting

Authors: xiong-hui  chen
ziyan  wang
yali  du
shengyi  jiang
meng  fang
yang  yu
jun  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 19:23:34

Abstract: When humans need to learn a new skill, we can acquire knowledge through written books, including textbooks, tutorials, etc. However, current research for decision-making, like reinforcement learning (RL), has primarily required numerous real interactions with the target environment to learn a skill, while failing to utilize the existing knowledge already summarized in the text. The success of Large Language Models (LLMs) sheds light on utilizing such knowledge behind the books. In this paper, we discuss a new policy learning problem called Policy Learning from tutorial Books (PLfB) upon the shoulders of LLMs’ systems, which aims to leverage rich resources such as tutorial books to derive a policy network. Inspired by how humans learn from books, we solve the problem via a three-stage framework: Understanding, Rehearsing, and Introspecting (URI). In particular, it first rehearses decision-making trajectories based on the derived knowledge after understanding the books, then introspects in the imaginary dataset to distill a policy network. 
 We build two benchmarks for PLfB~based on Tic-Tac-Toe and Football games. In experiment, URI's policy achieves at least 44% net win rate against GPT-based agents without any real data; In Football game, which is a complex scenario, URI's policy beat the built-in AIs with a 37% while using GPT-based agent can only achieve a 6\% winning rate. The project page: https://plfb-football.github.io.

Main paper: 
 introduction: 
 Humans can acquire new skills through various written materials that provide condensed knowledge, without the need for direct interactions with the target environment. In contrast, traditional policy learning paradigms, such as reinforcement learning (RL) [1] primarily rely on trial and error [2] [3] [4] . Despite recent advances in offline RL [5, 6] showing that policy improvements can be achieved simply by using pre-collected data, in fields such as embodied AI for robotics [7] , the process of collecting large amounts of decision-making data remains costly and, at times, impractical or even impossible. Therefore, a question arises: similar to how humans learn, can a policy learn from non-real data, e.g., from tutorial books?We argue that the recent successes of Large Language Models (LLMs), such as GPT-4 [8] , and LLaMA [9] already demonstrated the potential to learn from textual content. However, current studies focus on using LLM directly for decision making [10, 11] or integrating them as auxiliary modules in other machine learning workflows [12] [13] [14] . In this study, we introduce a novel topic built upon the shoulders of LLMs' systems: Policy Learning from Books (PLfB). PLfB aims to derive a policy network directly from natural language texts, bypassing the need for numerous real-world interaction data, as shown in Fig. 1(A-B ). This can be viewed as a further step towards enabling more resources for policy learning and also a more generalized form of offline RL problem, which uses textbooks to learn a policy offline. The essential challenge of PLfB comes from the inevitable large modality gaps between the text space, which includes the knowledge related to decision-making, and the policy network space, which formulates the parameters of the policy function.To realize PLfB, inspired by human learning processes, we propose a three-stage learning methodology: understanding, rehearsal, and introspection (URI), which is shown in Fig. 1(C ). For understanding, it first extracts knowledge from books to form a knowledge database; then it rehearses imaginary decision-making trajectories with the help of the knowledge retrieved from the database; finally, it introspects on the imaginary dataset to distill a policy network for decision-making. We present the first practical implementation of URI, employing LLMs to convert book paragraphs into pseudocode for policy, dynamics, and reward functions. This pseudocode forms a code-based knowledge database, which is used to generate an imaginary dataset based on retrieval augmented generation techniques [15] . A policy learning technique inspired by offline RL [6] is then applied to distill a policy network that addresses inaccuracies in the imagined actions, rewards, and states.In the experiments, we build two benchmarks from Tic-Tac-Toe and Football games for PLfB. We first validate URI on Tic-Tac-Toe game, where the tutorials are manually constructed to cover the complete knowledge for decision-makings. The results show that our method achieves at least 44% net win rate against GPT-based agents without any real game interaction. In addition, we build a testbed based on football tasks, which is popular in recent studies [16] as a difficult decision-making task, and focus on policy learning from football tutorial books, which naturally contain condensed knowledge, especially information closely related to football skill acquisition, the environment and dynamics of football games, and ways to evaluate football behaviors. We collect football tutorial books from RedPajama [17] . The policy is deployed in the Google Football simulator [18] directly.Our agent controlled by the policy network could beat the built-in AI with a 37% winning rate on average while using GPT as the football agent can only achieve a 6% winning rate. The experiments demonstrate the scalability of our approach from simple board games to complex scenarios. 
related work: 
 LLMs have demonstrated remarkable potential in achieving human-level ability in various tasks, sparking a surge in studies investigating LLM-based autonomous agents. Based on the different roles of LLM in the agents, there are two types of work. Firstly, LLM can be used as an actionable agent directly [10] . Early works [19, 20] prefer open-loop plans and hard-code search algorithms to improve the long-term planning ability of LLM. Closed-loop planning [21] [22] [23] leverages environmental feedback, thereby facilitating more adaptive decision-making. Secondly, LLMs can be used to assist or accelerate the learning of agents. The form of assistance can be quite diverse. It can generate high-level plans [24, 25] , (surrogate) rewards [26] [27] [28] [29] [30] [31] , transitions [32] [33] [34] , or as an adapter to convert human instructions into structured inputs [35] . In our work, LLMs play a different role compared to all the above-mentioned types. LLMs are to understand the knowledge in the books, rehearsing the decision-making process to derive an imaginary dataset based on the knowledge. The policy to control the agent is a simple neural network distilled from the imaginary data. database. After gathering this imagined content to form a dataset, Introspecting module distills a policy network, which should consider errors of generations of state, action, and rewards.We give the overall architecture of our implementation of URI in Fig. 2 . A detailed description of how they realize these functionalities will be discussed in the following. 
book content understanding: 
 The understanding module is responsible for extracting knowledge K := {K 1 , ..., K i , ..., K N K } from the books B, where K i denotes one piece of knowledge. For we humans, the knowledge is naturally stored in our brains. For machines, the first question is what should be the appropriate format to represent the knowledge. Since LLM has shown superior performance in code generation and codes themselves as interpretable, compact, and expressive languages [52] , we also choose it as the basic format of knowledge representation. Different from previous works [10] which directly asked for runnable code for the downstream tasks using, we do not execute this code. Instead, we just use it as a more flexible and abstract description of knowledge which still maintains a rigorous control flow. As shown in Fig. 2 (A), a knowledge extractor module is used as the first step. The knowledge extractor is an LLM-injected model. We iteratively ask K j = M ext (b i ) to examine whether a paragraph is related to the decision-making elements, i.e., policy functions, reward functions, dynamics functions, and how to represent them as pseudocode.Moreover, humans often learn by repeatedly reading the texts across pages and even books, updating existing knowledge with newly learned ones, and summarizing them into more general and abstract forms. A similar procedure is achieved by the knowledge aggregator module, shown in Fig. 2 (A), as the second step of understanding. The knowledge aggregator is also an LLM-injected model. We iteratively ask M agg ([K i , K j , K l , ...]) to aggregate relevant knowledge among different segments based on a similarity estimation of the embedding model E. Formally, let K 0 := {K 0 1 , ..., K 0 N 0 K } be the paragraph-wise knowledge extracted by M ext . For the j-the iteration,[K j+1 o , K j+1 p , ...] = M agg ([K j i , K j l , K j n , ...]), where [K j i , K j l , K j m , ...] is from K j by selecting the most similar N agg pieces of knowledge by comparing their cosine similarity under the embedding model E(K) of GPT and o, p, i, l, n here denote the indexes. [K j+1 o , K j+1 p , ...] are then added to K j+1 . The iteration will stop if |K j+1 | >= |K j |, where |K j | is the pieces of knowledge at j-th iteration. After iteration stops, the remaining knowledge pieces constitute the knowledge databases by splitting them into dynamics-related knowledge K T , reward-related knowledge K R , and policy-related knowledge K π for later modules to use. More details are provided in Appendix F.1.When humans develop a rough understanding of a new skill π from knowledge K, they will usually imagine what choice they will make given certain situations and what are the possible consequences in their minds. This rehearsing procedure would help humans correct apparent mistakes and consider better actions for long-term benefits [53, 54] .We implement a closed-loop generation process involving the LLM-injected dynamics function M T , reward function M R , and policy M π , as depicted in Fig. 2(B) . This approach resembles the model rollout in traditional model-based RL, where the policy, reward function, and dynamics function are represented by LLMs. Given the current imagined state ŝt , the LLMs M π first predict the most plausible action ât . Subsequently, M T and M R generate the next state ŝt+1 and the associated reward rt based on this action and the current state. In the process, a knowledge retrieval module is involved in selecting relevant knowledge pieces enhancing the LLM input with this information for LLM's predictions, e.g.,ât = M π (ŝ t , R(s t , K π )), rt = M R (ŝ t , ât , R(s t , K R )) and ŝt+1 = M T (ŝ t , ât , R(s t , K T )). The knowledge retrieval module includes the following two steps:State-based Knowledge Scope Retrieval: The fundamental problem of standard RAG techniques, i.e., embedding-vector similarity matching, in this scenario, is the modality gap between the query s t and the knowledge K. Standard RAG approaches aim to identify information closely related to the query, while here we need to ask the most suitable knowledge to be applied as a dynamics/reward/policy function that has the best predictions for the queried state and action. Standard RAG techniques tend to retrieve knowledge pieces that include similar text patterns as the queried states and fail to find the best knowledge for predictions. To address this, we propose a knowledge scope retrieval method that includes a simple yet effective preprocessing step to bridge the modality gap. In particular, we traverse all knowledge pieces K i ∈ K by iteratively sampling n scope knowledge pieces {K i } nscope from the database K, combining them with simulator information |M| and using LLM {K S i } nscope = M scope ({K i } nscope , |M|) to determine the preferred scopes K S i for each piece of knowledge. K S i is defined by the preferred subspace of state to use the knowledge. Then a standard RAG technique is applied to identify the most relevant knowledge scope K s j and its relevant knowledge K j . Formally, ({K j }, {K S j }) = R scope (ŝ, K S ), where K S is the scopes of the knowledge database. This method is effective for embedding models in retrieving the correct knowledge as the texts of keys and queries both are about the descriptions of states.Post-Retrieval: Knowledge Instantiation: Predicting based on code knowledge requires the LLM's robust understanding of code. We refine this process employing an LLM to instantiate the code K I = M Inst (ŝ, |M|, {K j }) based on the current state ŝ, the simulator information description |M| and the knowledge {K j } retrieved by R scope . Knowledge instantiation involves generating a domain-specific pseudocode based on the knowledge coded in the retrieved information, tailored to the target environment's current state and action requirements.Finally, three LLMs are involved to generate the imaginary dataset D img , includingât = M π (ŝ t , M Inst (s t , |M|, R scope (ŝ, K S π ))), rt = M R (ŝ t , ât , M Inst (s t , |M|, R scope (ŝ, K S R ))) and ŝt+1 = M T (ŝ t , ât , M Inst (s t , |M|, R scope (ŝ, K S T ))), where K S π , K S R , and K S T are the scope of the knowledge database K π , K R , and K T respectively. To start the rollout of a trajectory, we need a state as the initial state. It has several choices to achieve, e.g., sampled from S, generated by another LLM, or pre-collected a small number of real states from the target environments as part of simulator information, i.e., initial state distribution ρ 0 . Since the first choice might introduce unrealistic initial states in complex scenarios, in Football tasks, we opt for a more practical approach by pre-collecting a small set of real states from the target environments for our experimental validation. The detailed setup is in the experiment section. Besides, during the rollout over H steps, we reuse instantiated knowledge to reduce the overload of LLM's callings. More details are provided in Appendix F.2.The direct output LLM in the rehearsing stage can be sub-optimal or incorrect. We would like to re-examine the collected data and try to distill a policy that can avoid the side effects. We can regard the data D img collected during the rehearsing stage as an offline dataset and apply offline RL algorithms to train an improved policy from the dataset. However, directly applying existing offline RL algorithms over-simplifies the problem. Compared with the standard offline setting where only the behavior policy is sub-optimal, there is an additional misalignment in the data generated during the rehearsal: the transition and the reward function estimated by the LLM are also inaccurate. Overlooking such inaccuracy would result in a policy exploiting the sub-optimal transition and reward function and cause performance degradation or even risky behaviors in the final deployment.To solve the problem, in this paper, we adopt the Conservative Q-learning [5] as the base offline RL algorithm, whose learning objective is as follows:min Q max π α E ŝ∼D img ,a∼π(a|ŝ) [Q(ŝ, a)] -E ŝ,â∼D img [Q(ŝ, â)] + R(π) + E ŝ,â∼D img [(Q(ŝ, â) -Bπ Q(ŝ, â)) 2 ],where Bπ Q is the bellman update operator to update the Q-value function [1] , and the first term is to learn a policy with conservative Q value [5] . As a solution of introspecting from imaginary data, as shown in Fig. 2 (C), we add the uncertainty of the reward and transition estimation as the regularization terms, R R and R T over the original reward r output by the LLM M R . In practice, we adopt these regularization terms by applying them when we backup the Qk :Bπ I Q(ŝ, â) := r -η R R R (ŝ, â) -η T R T (ŝ, â) + γE ŝ′ ∼Dimg,a ′ ∼π k (a ′ |ŝ ′ ) [Q(ŝ ′ , a ′ )],where ŝ′ ∼ D img is to sample the next state given ŝ, â, η R and η T are two hyper-parameters to control the weighting of the uncertainty terms. Inspired by Model-based Offline Policy Optimization (MOPO) [6] , the uncertainty is estimated by an ensemble of N ens Gaussian models of T and R, which is learned by maximizing logarithmic likelihood from the imaginary dataset. Then the uncertainty is estimated by R R (s, a) = max i∈{1,...,Nens} σ r i (s, a) and R T (s, a) = max i∈{1,...,Nens} σ T i (s, a), where σ r i and σ T i are the i-th reward and dynamics model's predicted standard deviation for s, a respectively. We call the solution of CQL with Bπ I as the bellman update operator Conservative Imaginary Q-Learning (CIQL). This regularization is easy to implement and can force the policy to generalize better over the regions where the LLM outputs inconsistent next states and rewards. 
experiments: 
 We build two benchmarks with increasing complexity to evaluate PLfB: a classical Tic-Tac-Toe game with discrete states and deterministic rules, and the Google Research Football environment (GRF) [55] featuring continuous states and multi-agent interactions. In Sec. 6.1, we introduce the experimental settings for both environments. In Sec. 6.2, we analyze the performance of URI against various baselines. In Sec. 6.3 and 6.4, we examine the effectiveness of our knowledge extraction and retrieval mechanisms. Finally, in Sec. 6.5-6.8, we provide a detailed analysis of the training process, including data generation, ablation studies, inference efficiency, and dataset visualization.Tic-Tac-Toe Game To build a proof-of-concept benchmark for PLfB, we first construct a Tic-Tac-Toe (TTT) environment where players take turns placing their marks (X or O) on a 3×3 grid, aiming to form a line of three marks. This classical game serves as an ideal testbed for two reasons: (1) it has a known optimal solution (minimax algorithm), allowing us to evaluate how close our learned policy is to optimality; (2) due to its discrete and deterministic nature, we can collect a complete set of optimal trajectories covering all possible game states. Based on these trajectories, we can use GPT to generate comprehensive tutorial texts that contain complete knowledge of game mechanics and winning strategies, thereby controlling for the impact of knowledge incompleteness on algorithm performance. In our experiments, each policy plays as 'X' and moves first, with performance measured by win rate, draw rate, loss rate, and net win rate (win rate minus loss rate) across different opponents. For detailed game rules and setup, please refer to Appendix A. [55] is a physics-based 3D football simulator that supports the main football rules such as goals, fouls, corners, penalty kicks, and offside. Google Research Football (GRF) includes a built-in AI bot for the opposing team, whose difficulty can be adjusted between 0 and 1. An illustration of the game can be seen in Fig. 7 . We define three custom difficulty levels for our experiments on the 11vs11 scenario: easy, medium, and hard. The difficulty levels differ in the bot's reaction time and decision-making defined in GRF, with higher difficulty corresponding to a stronger opponent. The major metric in our experiment is Goal Difference per Match (GDM), calculated as the average number of goals scored in all league matches minus the average number of goals conceded per match. For URI, for each seed, we selected the average performance of the top 3 checkpoints among all recorded checkpoints as the final performance. For more details about the GRF implementation, please refer to Appendix B.Datasets We utilize three datasets in our approach. For Tic-Tac-Toe, we generate tutorial texts by having GPT analyze all possible game trajectories collected from an optimal minimax policy (see Appendix A.2). For football, we collect the textbook dataset from the open-source book dataset RedPajama-1T [56] , focusing on titles and abstracts related to football or soccer. After filtering, we obtain a curated set of ninety books closely aligned with the domain. For both environments, we need initial states to start our imaginary data generation process. In Tic-Tac-Toe, since we play as 'X' and move first, we always start from an empty board. In GRF, we sample 7,500 states from the rollout of a rule-based policy competing with the hard built-in AI. Due to limited resources, to distill the policy, we imagine 75,000 transitions via URI, which is 10 times compared to the initial states.Baselines We compare URI against several baselines across both environments. LLM-as-agent directly uses a large language model (GPT 3.5) to output actions conditioned on the current state description. LLM-RAG enhances LLM-as-agent by retrieving relevant knowledge from the database extracted from the tutorial books, similar to the retrieval step in our rehearsing stage, but directly outputs the action without policy learning. For Tic-Tac-Toe, we additionally include Minimax-noise, which follows the optimal minimax strategy but with 30% random action selection to serve as a nearoptimal baseline. For GRF, we include Rule-based-AI from the Kaggle Football Competition [57] , which is hand-designed and serves as a reference for the performance of hand-coded policies.Random Policy randomly chooses actions in both environments. For the implementation details about the baselines, please refer to Appendix D.We first evaluate URI's performance in the Tic-Tac-Toe environment. Tab. 1 shows the head-to-head match results between different policies, where each policy plays as 'X' and goes first. URI demonstrates superior performance across all opponents, achieving net win rates (win rate minus loss rate) of +66%, +44%, and +52% against LLM-as-agent, LLM-RAG, and Random Policy respectively. Moreover, when playing against Minimax-noise, which introduces 30% randomness to the optimal minimax strategy, URI maintains a positive net win rate of +26%, while all other baselines suffer negative net win rates. This indicates URI's ability to learn effective strategies from tutorial texts even in this classical game setting.The results in Tab. 2 demonstrate the superiority of the proposed URI approach compared to the baselines in the 11 vs 11 full-game scenarios of the GRF environment. The LLM-based agents, including LLM-as-agent and LLM-RAG, exhibit zero-shot task completion capabilities, outperforming the Random Policy. However, even with the use of RAG techniques, the best-performing LLM agent can only barely match the performance of the Medium-level built-in AI and fails to achieve any wins against the Hard-level built-in AI. In contrast, URI surpasses the performance of the baseline methods on all difficulty levels. Surprisingly, in the Hard task, URI achieves a higher win rate than the Rule-based Policy. We believe this is due to URI's ability to leverage knowledge from domain textbooks and generate high-quality imaginary data for policy learning, enabling it to learn more adaptable and robust policies compared to the hand-crafted rule-based AI.These results across both simple and complex environments highlight the scalability of URI in learning strong policies from well-defined board games to challenging scenarios. According to Sec. 5.2, we perform an iterative process of code extraction and aggregation to understand the tutorial books and obtain executable knowledge. As shown in Fig. 3(a) , for Tic-Tac-Toe, the initial extraction yields approximately 600 code segments which are effectively consolidated through the aggregation process. Similarly, in the more complex football domain (Fig. 3(b) ), the number of code segments for the dynamics, policy, and reward functions decreases significantly over the aggregation rounds. Through iterative aggregation, the number of code segments, decreases significantly in both environments, which helps condense the extracted knowledge into a more compact and coherent form. This consistent pattern across domains of varying complexity demonstrates the robustness of our knowledge extraction and aggregation approach.To validate the effectiveness of our code embedding method mentioned in Sec. 5.3, we conducted experiments in GRF task comparing it with two baselines: Vector Embedding, which directly compares the state and code embeddings, and Code Summary, which first summarizes the code segments before comparing the embeddings. We evaluate the top-15 hit rate in a hand-crafted test dataset. The observations and actions in the dataset we used are collected by the rule-based policy interacting with the real environment, while the ground-truth codes are labeled by GPT and aggregated using a similar way as in Sec. 5.2. As shown in Fig. 4 (a), our method significantly outperforms the other two baselines on both pre-trained language models. The hit rate, which measures the proportion of relevant code segments retrieved, is consistently higher for URI across all three random seeds, demonstrating its robustness and superiority in improving the correlation between the state embedding and the code embedding. These results highlight the importance of learning a dedicated matcher for effective code retrieval in our framework.In URI, it is crucial to guarantee that the imaginary data include states, actions, and rewards. The task is non-trivial since it has to involve several transformations to align the gap between textual contents in books and decision-making trajectories in MDP. To demonstrate this process in detail, we use the football environment as a case study. We trace how M π outputs an imaginary action â on an imaginary state ŝ, which is actually collected at the start of a football game in GRF. We record all the intermediate outputs to trace the action generation process. The result is shown in Fig. 5 . It is clearly shown that the output action "dribble" is fully supported by the logic in"has_space" branch in K I , baai/bge-large-en text-embedding-ada-002 • Reduce the distance between the player in 'Player in possession of the ball.• Carrying the ball through the available space (with or without defenders ' 'ahead).• Performing dribbles in search of numerical advantage in attacking situations or that enable the sequence ' 'of the play towards the opponent's goal line or goal.• Carrying the ball towards the opponent's goal line or goal.• Performing dribbles towards the opponent's goal line or goal searching for favourable conditions for a pass/assistance to a teammate to resume the play.'possession of the ball and the opponent's goal or goal line;• Unbalance opponent's defensive organization;• Directly attack the opponent or the goal;• Create advantageous attacking situations in numerical and spatial ' 'terms.... ...7,500 samples is ineffective. The results indicate that it cannot even beat the AI on Easy difficulty, though it still performs better than a random strategy. URI-w/o-penalty underscores the importance of penalizing the uncertain aspects of the outcomes generated from the imaginary data. Neglecting this penalty leads to results worse than those of URI-w/o-rehearsing. Finally, our method slightly outperforms CQL-real-data. We attribute this improvement to the strategies inferred from prior knowledge by LLMs, which are partially superior to built-in AI behaviors, or possibly because LLMs generate more diverse data. The successful ablation results in such a challenging domain illustrate the considerable potential of PLfB. Lower cost of inference is one of the benefits for agents controlled by URI policies. We compare the inference time per action for different methods in the GRF task in Tab. 3. Our approach takes only 0.009 seconds on average to choose an action, which is significantly faster at least 300 times than using LLMs directly as agents (2.84 seconds) or with retrieval-augmented generation (RAG) (4.12 seconds) in complex environments. This makes URI more suitable for real-time decision-making in the football simulator. The efficiency gain comes from the fact that URI distills the knowledge from the LLM into a compact policy network, which can be executed quickly without the need for expensive LLM inference at each step. We visualize the imaginary datasets to analyze the quality of generation and uncertainty estimation. Here we present the results from the GRF, while the TTT results can be found in Appendix G.1. We choose t-SNE [58] as the visualization method and project the imaginary dataset and a real dataset collected by the rule-based policy into 2-d space for comparison. The results are in Fig. 6 . The "real data" marks the data collected by the rule-based policy, while "low-unc. data" and "high-unc. data" represent segments of the imaginary dataset categorized by their uncertainty scores R T and R R falling within the lower and upper 50% percentiles, respectively. The real data and the imaginary data follow a similar data distribution, which indicates the effectiveness of the rehearsing process in URI. Besides, as highlighted by the yellow dashed circles, the uncertainty score also identifies parts of the clusters that are out of the real data distribution, which will be penalized when introspecting via CIQL. 
conclusion and discussion: 
 Inspired by the learning behavior of humans when they try to acquire a new skill, we propose PLfB that trains an agent from books, instead of numerous interaction data with the real environment. We also implement a practical algorithm of understanding, rehearsing, and introspecting modules to realize such a learning paradigm. The result of deploying our method in Tic-Tac-Toe and football game environments demonstrates a huge improvement in the winning rate over the baseline methods. This success proves the feasibility of utilizing knowledge stored in various written texts for decisionmaking agents, which was neglected by the community for a long time.One major limitation of URI is its implicit dependence on the quality of the "tutorial books": they should cover sufficiently the dynamics, policy, and rewards of the targeted environment so that relevant knowledge and imaginary data can be extracted to train the policy. It is also important to develop metrics evaluating the textual data quality to decide whether to use URI.Besides, we hope that the promising result in current experiments will initiate more research on PLfB. We leave more discussion of several interesting open problems in PLfBthat URI does not address in Appendix I.A Tic-Tac-Toe (TTT) Task SetupA.1 Environment Tic-Tac-Toe is a classical two-player game played on a 3×3 grid. Players take turns placing their symbols (X or O) in empty cells, with the goal of forming a line of three marks horizontally, vertically, or diagonally. The game state is represented by a 9-dimensional vector, where each dimension corresponds to a cell that can be empty (0), marked by X (1), or marked by O (2). The action space consists of integers from 0 to 8, representing the cell indices for placement. 
 
 Appendix: We first collect comprehensive game trajectories by having a minimax player (optimal policy) play against all possible opponent moves. The data collection process:1. Start from empty board 2. For X's moves (minimax player):• Use optimal minimax strategy • First move randomized for diversity 3. For O's moves (opponent):• Explore all possible valid move We use GPT to analyze these trajectories and extract strategic knowledge. The prompt are used as follow:Prompt 1: Knowledge Extraction PromptGiven a Tic-Tac-Toe game trajectory where X is known to be the oracle (optimal player), analyze the game and provide extracted knowledge about various aspects:1. Observe the entire game trajectory, including moves by both X and O. "When establishing dominance, prioritize claiming center and a row/column to establish multiple winning opportunities while blocking opponent threats."This knowledge base provides complete coverage of optimal strategies, serving as ground truth for evaluating PLfB's ability to learn from textual instructions.In this section, we mainly focus on the GRF implementation 2 and the modifications we made under this environment.The raw state of each player contains information about the game state, the ball and all players on the pitch. The game state includes scores, game modes indicating free kicks, corner kicks or other game stages, and game time represented as steps. The ball information includes its spatial position, direction, and an indicator for ball ownership (identifying the player and team that possess it). The player information comprises position, direction, roles, yellow card records, and more. For players on the opposite team, positions and directions are mirrored. The details of the raw state are as follows: • Ball Information:ball -[x, y, z] position of the ball.-ball_direction -[x, y, z] ball movement vector.-ball_rotation -[x, y, z] rotation angles in radians.-ball_owned_team -{-1, 0, 1}, where -1 indicates the ball is not owned, 0 denotes the left team, and 1 the right team. -ball_owned_player -{0..N-1} integer denoting the index of the player owning the ball.• Left Team:-left_team -N-elements vector with [x, y] positions of players.-left_team_direction -N-elements vector with [x, y] movement vectors of players.-left_team_tired_factor -N-elements vector of floats in the range {0..1}. 0 means the player is not tired at all. -left_team_yellow_card -N-elements vector of integers denoting the number of yellow cards a given player has (0 or 1). -left_team_active -N-elements vector of booleans denoting whether a given player is playing the game (False means the player got a red card). -left_team_roles -N-elements vector denoting roles of players, where: * 0 = e_PlayerRole_GK -goalkeeper, * 1 = e_PlayerRole_CB -centre back, * 2 = e_PlayerRole_LB -left back, ... • Right Team: Same attributes as for the left team.• Controlled Player Information:active -{0..N-1} integer denoting the index of the controlled players.designated -{0..N-1} integer denoting the index of the designated player.-sticky_actions -10-elements vectors of 0s or 1s denoting whether a corresponding action is active. • Match State:score -Pair of integers denoting the number of goals for the left and right teams, respectively.-steps_left -How many steps are left till the end of the match.-game_mode -Current game mode.For the imaginary dataset generation, we created our own Imaginary state, which includes the following information:• Game Information:-Sticky actions: A list of currently active sticky actions.-Game mode: The current game mode.-Score: The current score of the game.-Time: The current game time in minutes and seconds.-Active player: The index of the currently active player.-Active player role: The role of the currently active player.-Ball ownership: The team currently in possession of the ball (none, left team, or right team).-Ball ownership player: The index of the player currently in possession of the ball.-Ball zone: The zone where the ball is located.-Ball direction: The direction in which the ball is moving.• Player Information:-Team: The team the player belongs to (left team or right team).-Role: The role of the player.-Zone: The zone where the player is located.-Direction: The direction the player is facing.The imaginary state is designed to provide a more compact and informative representation of the game state compared to the raw state. It extracts and processes the relevant information from the raw state, making it easier to understand and use for generating imaginary data.The sticky actions, game mode, score, time, active player, and active player role provide a high-level overview of the current game state. The ball ownership and ball ownership player indicate which team and player are currently in control of the ball. The ball zone and direction give spatial information about the ball's location and movement.For each player, the Imaginary state includes their team, role, zone, and direction. This information helps in understanding the positioning and orientation of the players on the pitch.By including these key pieces of information in the Imaginary state, we aim to capture the essential aspects of the game state that are relevant for generating realistic and diverse imaginary data. The imaginary state serves as a preprocessed and structured representation of the raw state, making it more suitable for the subsequent steps in the imaginary data generation process.Here, we show an example of imaginary state,To enhance the understanding of player and ball positions beyond mere coordinates, our methodology involves dividing the football field into a grid of 20 x 12 rectangular zones. This granular zoning approach serves a dual purpose: firstly, it abstracts the complex spatial dynamics of the game into a more manageable form, allowing our LLM to process and generate data with increased effectiveness.Secondly, it provides a framework for more accurately simulating the spatial strategies and movements employed in real-world football scenarios. Each zone, defined by specific dimensions, acts as a unique spatial identifier, offering precise reference points for player positioning and ball location. This system not only simplifies the representation of the field but also enriches the strategic depth of the game model, as actions and decisions can be tailored to the distinct characteristics of each zone. By adopting this zoning strategy, we aim to bridge the gap between the high-level strategic understanding required for football and the detailed, positional awareness needed to implement those strategies effectively within the simulated environment.The simulator information |M| includes the description of the state space S. What we use in this paper is as follows.• First, it provides information such as the time and score of the match.• Second, which side has control of the ball, and the active player in the left team, you need to propose corresponding policies to him.• 3, # action_top, sticky action and will change the player's direction, player will continue to move top until another action is taken. Such as, from zone (11, 4) to zone(11,5)• 4, # action_top_right, sticky action and will change the player's direction, player will continue to move top right until another action is taken. Such as, from zone (11, 4) to zone(12,5)• 5, # action_right, sticky action and will change the player's direction, player will continue to move right until another action is taken. Such as, from zone (11, 4) to zone(12,4)• 6, # action_bottom_right, sticky action and will change the player's direction, player will continue to move bottom right until another action is taken. Such as, from zone (11, 4) to zone(12,3)• 7, # action_bottom, sticky action and will change the player's direction, player will continue to move bottom until another action is taken.Such as, from zone (11, 4) to zone(11,3)• 8, # action_bottom_left, sticky action and will change the player's direction, player will continue to move bottom left until another action is taken. Such as, from zone (11, 4) to zone(10,3)• 9, # action_long_pass, the player will long pass to their teammate in his current direction. Before you pass, you should release the sprint if the agent is doing sprint. A long pass covers a large distance on the field.• 10, # action_high_pass, the player will high pass to their teammate in his current direction. Before you pass, you should release the sprint if the agent is doing sprint. A high pass sends the ball into the air, often over obstacles, to reach a teammate.• 11, # action_short_pass, the player will short pass to their teammate in his current direction. Before you pass, you should tune make sure the diresction is fine, using action 0-8 to chenge the direction. A short pass is a quick and close-range exchange between teammates, commonly used to maintain possession and build an attack.• 12, # action_shot, players will try to shoot.• 13, # action_sprint, when the player will chose this action, the agent will sprint with sticky action's direction, it will make agent run faster.• 14, # action_release_direction, player will stop moving in the current direction. Choose it when you want this agent to change the direction, after this action, you should choose another action from 0-8 to change it.• 15, # action_release_sprint, player will stop sprinting, only when the agent's during the sprint.• 16, # action_sliding, the player will try to slide the tackle. If your position is on the opponent's path with the ball, you can intercept it. However, if the sliding tackle fails, you will be separated by a large distance, allowing the opponent to ignore the defense.• 17, # action_dribble, players will try to dribble. When they have the ball, dribbling will greatly improve the success rate of dribbling, especially in multi-person double-teams and difficult-to-handle situations.• 18, # action_release_dribble, player will stop dribbling.The game dynamics in GRF closely resemble realistic football games. Players can move or sprint with or without the ball. They can also execute various passes and shots. Besides different actions, complex interactions such as collisions and trips between players are simulated as well. Additionally, the game engine introduces stochasticity in the dynamics, which can influence the passes and shots randomly.In our experiment, to address the challenge of executing fine-grained maneuvers within a game environment, we employ a strategy where actions generated by rule-based policies [57] are utilized to replace those produced by the policies, including URI and all of the baselines. This substitution is particularly enacted when considering the spatial segmentation of the play area into zones. Using zones as a pivotal input, the approach facilitates more nuanced control over in-zone activities. The rationale behind this decision is rooted in the state that precise actions within these designated zones can significantly impact the outcome of play, necessitating a method that allows for detailed operational control. This method ensures that the agents can perform more sophisticated strategies, particularly in scenarios that demand high levels of precision and situational awareness within the confined spaces of each zone.The simulator information |M| includes the description of the transition information and the task information T and R. What we use in this paper is as follows.• Dynamics is to give the dynamics function or related rules of the football game under the football manager policyś action, such as after shooting, the ball will be in the goal or not. For example: "When the direction of shooting is vertical to the goal, the ball will be easy to the goal. • for each step of transition prediction, you should simulate 1.5 second based on current state and action of players. • Reward is to give the reward or punishment of the football manager policy. The behavior that is encouraged is when the forwards are restricted, the midfielder can support and take away the defenders. This is a very encouraging behavior because it allows the team to keep possession of the ball and control the game. • You should only identify 5 types of rewards: A. 2 for optimal behavior; B. 1 for encouraging behavior; C. 0 for borderline behavior; D. 1 for punishing behavior; E: -2 for worst behavior.Offline RL addresses the problem of learning policies from a pre-collected dataset. Most methods can be classified into two categories: model-free and model-based methods. Model-free [39-42, 5, 43, 44] methods learn a conservative policy directly from the dataset. Model-based offline algorithms [6, [45] [46] [47] [48] first estimate a model from the dataset and perform policy learning or planning based on this learned model. In our work, to achieve introspection from the data generated by the LLMs, we build our policy distillation algorithm based on several existing techniques in offline RL, including the uncertainty penalty in MOPO [6] , which constructs a pessimistic model that discourages the policy from visiting states where the model is inaccurate; and conservative Q-learning loss [5] to obtain a robust value function that does not overestimate unseen state-action pairs too much.In this section, we detail the implementation of baselines for both Tic-Tac-Toe.The LLM-as-agent baseline uses GPT-3.5 to directly generate actions for both environments. we use the following prompts:Prompt 2: LLM-as-agent Global Prompt for Tic-Tac-Toe # Tic-Tac-Toe ## Introduction Tic-Tac-Toe is a classic two-player game, unfolds on a 3-by-3 grid where the objective is to align three of one's symbols, Xs for the first player and Os for the second, either horizontally, vertically, or diagonally. Strategic placement is crucial: besides aiming for three in a row, players must also block their opponents' potential alignments to avoid defeat. Players can place their next move in an empty cell on the 3-by-3 grid.## Rules 1. The game is played on a 3x3 grid. 2. Players take turns placing their symbol (X or O) in empty cells.3. The first player to get 3 of their symbols in a row (horizontally, vertically, or diagonally) wins.4. If all cells are filled and no player has won, the game is a draw.The game board is represented as a string of 9 characters, where: -'X' represents the first player's moves -'O' represents the second player's moves -' ' (space) represents an empty cell ## Your Task:-Respond with a JSON object containing your move and reasoning.-The move should be a single integer from 0 to 8, representing the cell index.-Provide a brief explanation for your choice in the reasoning field. ----------------------state.board [6] | state.board [7] | state.board [8] The next move should be placed in an empty cell on the 3-by-3 grid.The available positions for the next move are:Positions are represented by the numbers 0-8, corresponding to the cell's index:0 | 1 | 2 --- 3 | 4 | 5 --- 6 | 7 | 8Analyze the game state and choose your next move. Respond with a JSON object in the following format: { "move": <integer 0-8>, "reasoning": "<brief explanation for your choice>" }The LLM-RAG baseline enhances LLM-as-agent by retrieving relevant knowledge from tutorial books.Prompt 4: LLM-RAG Global Prompt for Tic-Tac-ToeThe policy knowledge from tutorials will be provided to guide your decision.Prompt 5: LLM-RAG Query Prompt for Tic-Tac-ToeThe relevant policy knowledge for the current state is: {policy_knowledge} ## Choose the next move based on the policy knowledgeYou should follow the provided policy knowledge to make your decision.Base your reasoning on how your chosen move aligns with the strategic principles described in the policy knowledge.For the RAG part, we use the following knowledge retrieval approach:• Use GPT embeddings to encode both state descriptions and tutorial knowledge• Retrieve top-3 most relevant pieces of strategic knowledge using cosine similarity• Combine retrieved knowledge with state information for move generation• Minimax (Optimal):-Uses standard minimax algorithm with perfect play -Evaluates states with +1 for win, -1 for loss, 0 for draw -Uses random first move selection for diversity• Minimax-noise:-Based on optimal minimax algorithm -Makes random moves with 30% probability -Uses optimal moves with 70% probability -Serves as a strong but imperfect opponent• Random:-Selects moves uniformly from available positions -Serves as a baseline for minimal strategic playIn this section, we will focus on implementing of two relevant baselines.As mentioned in the paper, the LLM-as-agent method uses LLM (GPT3.5) directly to generate the action. We provide the prompt as follows, Prompt 6: LLM-as-agent Query Prompt 1/2The texted state for the current state: {text_state} -------{Simulator Information |M|. Refer to Appendix F.4.} -------For example, if the active player is Player 2 and you want him to be close to the ball and control it in the texted state, -Forward Player 2 is at Zone (9, 9) .-The ball is at Zone (11, 8) .Given these coordinates, the ball is diagonally one zone to the right (east) and one zone down (south) from the player's current position. The most direct route to the ball would indeed be diagonally towards the bottom right.Therefore, the most appropriate action for Forward Player 2 in this situation would be: 6 = action_bottom_right: This sticky action will allow the player to move diagonally in the bottom-right direction (southeast), which is the direct path to where the ball is currently located in Zone (11, 8) . By choosing action_bottom_right, Forward Player 2 can close the distance to the ball more effectively, aligning their movement directly with the ball's current location.Once the player reaches the ball, the subsequent action can be decided based on the situation at that moment (e.g., dribbling, passing, or shooting).-------Question: What next action do you want this active player to take? Answer:The global prompt shows as follows,Prompt 7: LLM-as-agent Global Prompt I want you to act like a football manager and also an expert in Python coding and Reinforcement Learning, which is about learning a football manager policy in a football simulator.I will give you the pseudocode snippets to define the specific policy for the active player in the football game.These codes represent absolute correctness and do not add other common logic.Your task is to select the action that best fits and executes the code based on the logic of the given code.About the action choosing:• Please choose the action that best fits the code logic.• you should answer in pure JSON format with the key: 'action': an int number from 0 to 18, • 'thought': why did you choose this action? without any other information or code.• For example, you should not add the "'JSON"' tag in the answer.Response example (you should respond in the following order): {{ "action": 0, "thought": "Based on your thought, tell me the optimal action you would like to select in the action set."The LLM-RAG enhances LLM-as-agent by retrieving relevant knowledge from the database extracted from the tutorial books, similar to the retrieval step in our rehearsing stage, but directly outputs the action without policy learning. The dataset which the LLM-RAG using is the text-booked dataset.Here we show the prompts of LLM-RAG. ------------- For example, if the active player is Player 2.-When you become a back defender, you aim to get close to the ball and stop the attacker's progression towards the goal.-Forward Player 2 is at Zone (9, 9) . -The ball is at Zone (11, 8) .Given these coordinates, the ball is diagonally one zone to the right (east) and one zone down (south) from the player's current position. The most direct route to the ball would indeed be diagonally towards the bottom right.Therefore, the most appropriate action for Forward Player 2 in this situation would be: 6 = action_bottom_right: This sticky action will allow the player to move diagonally in the bottom-right direction (southeast), which is the direct path to where the ball is currently located in Zone (11, 8) . By choosing action_bottom_right, Forward Player 2 can close the distance to the ball more effectively, aligning their movement directly with the ball's current location. Once the player reaches the ball, the subsequent action can be decided based on the situation at that moment (e.g., dribbling, passing, or shooting).-------Question: What next action do you want this active player to take? Answer:The global prompt shows as follows, Prompt 9: LLM-RAG Global Prompt I want you to act like a football manager and also an expert in Python coding and Reinforcement Learning, which is about learning a football manager policy in a football simulator. I will give you the pseudocode snippets to define the specific policy for the active player in the football game.These codes represent absolute correctness and do not add other common logic. Your task is to select the action that best fits and executes the code based on the logic of the given code.About the action choosing:• Please choose the action that best fits the code logic.• you should answer in pure JSON format with the key: 'action': an int number from 0 to 18,• 'thought': why did you choose this action? without any other information or code.• For example, you should not add the "'JSON"' tag in the answer.Response example (you should respond in the following order):{{ "action": 0, "thought": "Based on your thought, tell me the optimal action you would like to select in the action set."For the RAG part, we use the llama index [59] 3 for embedding the textbook paragraphs and the state and find the most relevant one as input to the query prompt.In this section, we will focus on implementing URI.Book content understanding is achieved via a code-based knowledge extractor and a code-based knowledge aggregator. The implementation is designed to be task-agnostic, with task-specific parameters configured through prompts.The knowledge extractor processes book paragraphs to identify and extract relevant knowledge. Algorithm 1 shows the extraction process: Output ← M ext (I, KnowledgeContext)8:if ′ WithKnowledge ′ in Output then 9:K.add(Output.code) I ← ∅ 12:end if 13: end for 14: return K Task-specific configuration can be found at Appendix. F.3 and F.4 About the answer:• If you think the paragraph contains the above elements, please answer 1.The answer is 1 only when you can write the specific theorem, principle, rule, and law of the related elements into pseudocode snippet.• If you think the paragraph does not contain the above elements, please answer 0.• If you think the given paragraph is not clear enough to answer, please answer 2. Then I will give you the following paragraph to help you answer.• If you think the paragraph contains the above elements but the content is not clear enough to derive the specific theorem, principle, rule, and law of the related elements, please answer 2. Then I will give you the following paragraph to help you answer.• If the answer is 1, you should give the specific theorem, principle, rule, and law of the related elements.• You should write the specific theorem, principle, rule, and law of the related elements via pseudo code.• Please provide the PYTHON-style pseudo code as detailed as you can to cover the most information of the original content.About the pseudo code snippet:• {element_type_code_example_dict[knowledge_type]} About the format:• You should answer in pure JSON format, without any other information or code.Response example:{ "answer": 1, "pseudocode of/about XX": ["PYTHON pseudocode snippet1", ...], "pseudocode of/about XX": ["PYTHON pseudocode snippet2", ...] }The aggregation process combines similar pieces of knowledge into more concise representations. Algorithm 2 shows the process: I will give you several pseudocode snippets to define the specific theorem, principle, rule, and law of the related elements or concepts:K sim ← {K ′ ∈ K|calculate-similarities(E(K), E(K ′ )) > τ } 6: if |K sim | ≥ N agg then 7: K agg ← M agg (K sim , KnowledgeContext) 8: K ′ .add(K agg ) 9:These codes might share some common logic. Your task is to aggregate the common logic of the given codes into a single code snippet, while still covering all of the given codes.About the aggregated code:• Please provide the PYTHON-style pseudo code as detailed as you can • Using the least number of pseudocode items • Covering all of the code. However, please feel free to add more pseudocode items if needed• Since I will delete the original code after getting your aggregated code, you cannot call the pseudocodes that I provided in the prompt About the pseudo code snippet:• {element_type_code_example_dict[knowledge_type]} About the format:• You should answer in pure JSON format • active_player_role is preferred/better to be in [?]• ball_ownership is preferred/better to be in [?]• ball_ownership_player is preferred/better to be in [?]• ball_zone is preferred/better to be in [?]• ball_direction is preferred/better to be in [?]• {the preferred zone of all players from 0 to 21} NOTE:1. You should define your observation scope as detailed and tight as possible 2. You should define the observation scope for each code by comparing the differences between the codes About the format:• You should answer in pure JSON format, without any other information or code Response example:'code_idx 1': { 'preferred scope description': 'a text summary of the suitable situations, 'score': 'preferred scope of the score', 'active_player_role': 'preferred scope of the active_player_role', ... }, ... } Prompt 14: Tic-Tac-Toe Knowledge Code Generation Prompt I want you to act like a tic-tac-toe pro player and also an expert in python coding and Reinforcement Learning that wants to learn a tic-tac-toe policy in a tic-tac-toe simulator.I will provide an observation space of the tic-tac-toe simulator and some policy functions written by others. Your task is to summarize an observation scope, which is a subspace of the observation space and most suitable to use this policy function to win the game.The action space: {action_space_desc}About the definition of the observation scope of each code:For each code, you should return a description of the best scope of the observation variables to be used as the policy function to win the game, including:• A text summary of the preferred/better scope • Preferred board states: [(?), (?), (?), (?), (?), (?), (?), (?), (?)], where you should replace "?" with all possible values or "any" if all values are suitable• Preferred/better player: [?], where you should replace "?" with possible values NOTE:1. You should define your observation scope as detailed and tight as possible 2. You should define the observation scope for each code by comparing the differences between the codes About the format:• You should answer in pure JSON format, without any other information or code Response example:'code_idx 1': { 'preferred scope description': 'a text summary of the suitable situations, 'current_player': 'preferred scopes', 'preferred board states': 'preferred scopes' }, ... } Then a standard RAG technique is applied to identify the most relevant knowledge scope K s j and its relevant knowledge K j . Formally, ({K j }, {K S j }) = R scope (ŝ, K S ), where K S is the scope of the knowledge database.After that, we employ an LLM to instantiate the code K I = M Inst (ŝ, |M|, {K j }) based on the current state ŝ, the simulator information description |M| and the knowledge {K j } retrieved by R scope . The prompt is listed in Prompt 15. Since this step requires a strong understanding of the code, we use GPT-4 instead of GPT-3.5 as the LLM implementation.The code instantiation process converts general knowledge into task-specific code through a dedicated module. Here's the detailed implementation: I will give you an observation which you are facing in the simulator, your task is to instantiate a code that serves as a {knowledge_type} function, which is used for {knowledge_function}.Formally, the format of knowledge_type function is {knowledge_format}.To help you complete the task, I will provide you:1. Pieces of relevant knowledge from the tutorial, written in Python-style pseudocode 2. The observation space and action space of the target simulator 3. The current observation you are facing Python-style relevant knowledge from tutorial: {code_string}The observation space: {observation_space_desc} The action space: {ac-tion_space_desc} Current observation: {obs} About your code-instantiation task:• Please provide the PYTHON-style pseudo code as detailed as you can • Your task is to rewrite a code that describes a {knowledge_type} suitable to current observation• You should make the optimal decision based on analyzing current observation• After analysis, rewrite the pseudocode to make it most suitable for downstream tasks• Keep a main function named "{knowledge_type}" and add inner functions if necessary Important Notes:• You cannot call the pseudocodes provided in the prompt -Changes in active player -Ball position changes -Opponent position changesResponse format:{ "analyze": "Analysis of current observation, focusing on code instantiation approach", "code": "Rewritten code based on current observation analysis" }Finally, three LLMs are involved to generate the imaginary dataset D img , includingât = M π (ŝ t , M Inst (s t , |M|, R scope (ŝ, K S π ))), rt = M R (ŝ t , ât , M Inst (s t , |M|, R scope (ŝ, K S R ))) and ŝt+1 = M T (ŝ t , ât , M Inst (s t , |M|, R scope (ŝ, K S T ))), where K S π , K S R , and K S T are the scope of the knowledge database K π , K R , and K T respectively. I will give you an observation which you are facing in a simulator, your task is to response a correct results serving as a {knowledge_type} function, which is used for {knowledge_function}.For example, {knowledge_function_example}.Formally, the format of {knowledge_type} function is {knowledge_format}.To help you complete the task, I will provide you: The following configurations detail the task-specific parameters used in the previously described prompts for the Tic-Tac-Toe environment. These include the task description, knowledge type definitions, observation space descriptions, and other parameters that are substituted into the corresponding placeholders in the book content understanding prompts (Appendix F.1), knowledge rehearsing prompts (Appendix F.2).{task_info} "An expert Tic-Tac-Toe player and also an expert in python coding and Reinforcement Learning." {knowledge_type} • Policy Function:"The policy in Tic Tac Toe involves strategic placement of X's or O's on the game board to block the opponent's moves and create opportunities to complete a line. For example, "Always place your mark in the center if it is your first move, as it maximizes potential winning combinations." • Environment Dynamics Function: "This function describes the rules and changes in the game environment following player actions. For example, "If a player completes a row, column, or diagonal with the same symbol, they win the game. Otherwise, the game continues or ends in a draw if all spaces are filled without a winner." • Rewards Function:"This function outlines the incentives or penalties based on the game outcomes in Tic Tac Toe. For example, "Completing three consecutive symbols horizontally, vertically, or diagonally results in a win and positive reward, while failing to block an opponent's winning move results in a loss and a negative reward." "You can classify the outcomes as follows: 2: Winning move; 1: Blocking an opponent's winning move; 0: Neutral move; -1: Missing an opportunity to block an opponent's winning move; -2: Making a move that leads directly to a loss."{observation_space_desc} "The observation space consists of a 3×3 grid representing the game board. Each cell can be empty or contain an 'X' or 'O' symbol. We use a 1×9 array with number 0, 1, and 2 to represent empty, 'X', and 'O' respectively. The board state is updated after each player's move, reflecting the current game configuration. We will use another line to tell you whose turn it is, 1 for X and 2 for O."{action_space_desc} "The action space is discrete from 0 to 8, including 9 possible moves corresponding to the 9 cells on the game board. Players can place their mark ('X (1)' or 'O (2)') in any empty cell to make their move."• Policy Inference: "For example, if it is X's turn and you observe the following board state: [1, 2, 0, 0, 0, 0, 0, 0, 0] The strategic thought might be "Placing X in the center (cell 4) could block O's potential diagonal completion and creates multiple opportunities for X to win on subsequent turns. Therefore, the most appropriate action for X in this situation would be to place in the center." Expected output format: { "thought": "Given the current board state [0,1,2,0,0,0,0,0,0] and that it's X's turn...", "action": 4 }" "The behavior that is encouraged is when the forwards are restricted, the midfielder can support and take away the defenders. This is a very encouraging behavior because it allows the team to keep possession of the ball and control the game. You should only identify 5 types of rewards: 2: Optimal behavior; 1: Encouraging behavior; 0: Borderline behavior; -1: Punishing behavior; -2: Worest behavior."• Rewards Function:"When the direction of shotting is vertical to the goal, the ball will be easy to the goal." {observation_space_desc} "First, it provides information such as the time and score of the match. Second, which side has control of the ball, and the active player in the Left team you need to propose corresponding policies to him. Next is the position, direction and role information of each player:• In this text description, the football grass field, which is 120m * 90m in the real world, is divided into 240 zones in the simulator, where "x" ranges from 1 to 20 (left to right from the left team's perspective) and "y" ranges from 1 to 12 (bottom to top). • Based on this transformation, the concepts in real-world football field can be mapped to the simulator, For instance:the center circle is at (10, 6) , and the lower left corner from the left team's view is at (1, 1), and the upper right corner position of the Right team is (20, 12) the distance from (10, 5) to (11, 5) in the real-world is 12m, and the distance from (10, 5) to (10, 6) is 7.5m the left team's penalty area is zone (1, 4)-(1, 8)-(2, 8)- (2, 4) , and the right team's penalty area is zone (20, 4)-(20, 8)-(18, 8)-(18, 4) the center circle position of the field is zone (10, 6) , where the game start • The position of the player is defined by the player's moving in the corresponding direction, including north, northeast, east, southeast, south, southwest, west, northwest • Role information: for each team, player roles are fixed, including Goalkeeper, Forward, Forward, Defender, Defender, Defender, Defender, Midfielder, Midfielder, Midfielders, Forward NOTE: Venues never interchange or change. Following the position information is direction, meaning the direction the player is currently facing and the direction of future actions."{action_space_desc} "The action set includes:1. action_idle: A no-op action, sticky actions are not affected (player maintains his directional movement etc.) 2. action_left: Sticky action and will change the player's direction, player will continue to move left until another action is taken. Such as, from zone (11, 4) to zone(10,4) 3. action_top_left: Sticky action and will change the player's direction, player will continue to move top left until another action is taken. Such as, from zone (11, 4) to zone(10,5) 4. action_top: Sticky action and will change the player's direction, player will continue to move top until another action is taken. Such as, from zone (11, 4) to zone(11,5) 5. action_top_right: Sticky action and will change the player's direction, player will continue to move top right until another action is taken. Such as, from zone (11, 4) to zone(12,5) 6. action_right: Sticky action and will change the player's direction, player will continue to move right until another action is taken. Such as, from zone (11, 4) to zone(12,4) 7. action_bottom_right: Sticky action and will change the player's direction, player will continue to move bottom right until another action is taken. Such as, from zone (11, 4) to zone(12,3) 8. action_bottom: Sticky action and will change the player's direction, player will continue to move bottom until another action is taken. Such as, from zone (11, 4) to zone(11,3) 9. action_bottom_left: Sticky action and will change the player's direction, player will continue to move bottom left until another action is taken. Such as, from zone (11, 4) to zone(10,3) 10. action_long_pass: The player will long pass to their teammate in his current direction.Before you pass, you should release the sprint if the agent is doing sprint. A long pass covers a large distance on the field 11. action_high_pass: The player will high pass to their teammate in his current direction.Before you pass, you should release the sprint if the agent is doing sprint. A high pass sends the ball into the air, often over obstacles, to reach a teammate 12. action_short_pass: The player will short pass to their teammate in his current direction.Before you pass, you should tune make sure the direction is fine, using action 0-8 to change the direction. A short pass is a quick and close-range exchange between teammates, commonly used to maintain possession and build an attack 13. action_shot: Players will try to shoot. When near to the opponent penalty area, such as zone(x, y), x>18, 4<y<8, try to shoot 14. action_sprint: When the player will choose this action, the agent will sprint with sticky action's direction, it will make agent run faster 15. action_release_direction: Player will stop moving in the current direction. Choose it when you want this agent to change the direction, after this action, you should choose another action from 0-8 to change it 16. action_release_sprint: Player will stop sprinting, only when the agent's during the sprint 17. action_sliding: The player will try to slide the tackle. If your position is on the opponent's path with the ball, you can intercept it. However, if the sliding tackle fails, you will be separated by a large distance, allowing the opponent to ignore the defense 18. action_dribble: Players will try to dribble. When they have the ball, dribbling will greatly improve the success rate of dribbling, especially in multi-person double-teams and difficult-to-handle situations 19. action_release_dribble: Player will stop dribbling" This algorithm implements the URI method for policy learning from books. It consists of three main stages: 1) Understanding (L1-L2): knowledge is extracted and aggregated from the input book using CodeExtractor and CodeAggregator. 2) Rehearsing (L3-L14): An imaginary dataset is generated by iteratively applying knowledge retrieval, instantiation, and language model-based simulation.3) Introspecting (L15-L16): a policy is learned from the imaginary dataset using Conservative Imaginary Q-Learning (CIQL). The algorithm leverages language models M π , M R , and M T for action selection, reward estimation, and state transition, respectively, bridging the gap between textual knowledge and reinforcement learning. K S ← KNOWLEDGE SCOPE RETRIEVAL(ŝ t , K ′ ) Similar to the visualization analysis in the main text, we also conduct t-SNE [58] visualization for the Tic-Tac-Toe environment to examine the quality of imaginary data generation and uncertainty estimation. As shown in Fig. 8 , despite the discrete nature of the state space in Tic-Tac-Toe, we observe similar patterns as in the football environment. The imaginary data (separated into low and high uncertainty regions) largely follows the distribution of real data collected from optimal minimax gameplay. The uncertainty estimator successfully identifies deviant clusters (marked by yellow dashed circles) that lie outside the distribution of optimal play data. This consistency in behavior across both simple discrete environments and complex continuous ones supports the robustness of URI's data generation and uncertainty estimation mechanisms.To further validate the effectiveness of our approach, we implement three additional baselines: (1) RT-Policy: a policy trained using LLAMA following the RT-style architecture [61] , using the same 7,500 initial states as URI and rule-based actions for training; (2) URI w/o BK: a variant of URI that generates imaginary data without using book knowledge; and (3) CQL: a pure offline RL approach using the Conservative Q-Learning algorithm with real interaction data. We list more visualization results for each stage of URI in the links: https://plfb-football. github.io/, including examples of knowledge extraction, code aggregation, imaginary data, and videos of policy execution in GRF.We hope that this promising result will initiate more research on PLfB. We will first elaborate on the points mentioned in Sec. 7 on applying URI when textual data are insufficient of low-quality.• Multimodal Data Integration: Beyond purely textual data, additional modalities such as tutorial voices, demonstration videos, and replays can be incorporated. By employing advanced multimodal large language models, these diverse forms of data can be processed in a manner akin to the current handling within the URI, thereby augmenting the knowledge base and enhancing the robustness of policy learning. • Utilization of Real Interaction Data: During the introspection phase, incorporating real interaction data with the target environment, rather than relying solely on simulated data generated by large language models, can enhance policy learning. This mixed-data approach can be used to further fine-tune various modules within the URI framework, potentially boosting overall performance. It also mimics human learning behavior better as humans learn from both reading and actual experience. However, one might develop new techniques to utilize both sources of data for better policy learning. • Injection of Prior Knowledge: The URI framework allows for the integration of human expert knowledge at different stages of the pipeline. Experts can provide specific code knowledge representations/formulations/templates when generating code-based knowledge. We can also provide constraints during the rehearsal process to enhance stability and realism. • Evaluating the Quality of Textual Data: Most current studies in this fields [62] [63] [64] are still in primitives. These studies suggest frameworks and empirical strategies that can be adapted to evaluate the relevance and quality of textual and multimodal data for training purposes. However, given the significant modality gap between textual data and the neural network parameters in PLfB topics-and the largely unsupervised nature of our learning process-this is not a trivial problem. This complexity requires innovative approaches for assessment.We are also compelled to investigate other potential applications of the framework beyond the scope of the current experiment. Domains such as embodied AI for robotics, which require a large amount of data for foundation policy learning and the robotics dataset currently, could benefit significantly from the integration of the paradigm for data augmentation. This exploration opens exciting avenues for future research and application of the method.All experiments were conducted on a high-performance computing (HPC) system featuring 128 Intel Xeon processors running at 2.2 GHz, 5 TB of memory, an Nvidia A100 PCIE-40G GPU, and two Nvidia A30 GPUs. This computational setup ensures efficient processing and reliable performance throughout the experiments.Integrating Large Language Models (LLMs) with Reinforcement Learning (RL) in PLfB offers a groundbreaking approach to AI skill acquisition. This innovation has the potential to revolutionize industries reliant on automation, such as manufacturing and robotics, by enabling AI agents to learn complex tasks from textual sources, reducing the need for extensive real-world data. However, ethical considerations regarding bias, privacy, and transparency are paramount. Overall, PLfB promises enhanced efficiency and accessibility but requires careful navigation of its social implications.https://github.com/google-research/football/blob/master/gfootball/https://github.com/run-llama/llama_index 


Decision: The paper introduces a novel and timely approach to solve reinforcement learning problems by incorporating textual knowledge about the problems. The experiments convincingly show that the approach can be effective, and significantly better than RL baselines and ablations of the approach.

All of the reviewers agree that the paper is technically sound, clearly written, with high significance and novelty. The authors in their feedback provided additional experiments and discussion that will make the revised paper stronger.

Reviews: review: paper_summary: paper_summary: This paper introduces Policy Learning from Books (PLfB). This framework leverages the knowledge encoded in textual books and tutorials to train decision-making policies, specifically for playing football, without requiring direct interaction with the environment. This method is a three-stage framework Understanding, Rehearsing, and Introspecting -- **Unstanding**: extracts knowledge from books, uses it to **rehearse** decision-making trajectories in an imaginary dataset, and then **introspects** on the imagined data to distill a refined policy network.

They found that the URI approach significantly outperforms baseline methods in the Google Research Football (GRF) 11 vs 11 scenarios, and that the iterative process of code extraction and aggregation significantly reduces the number of code segments for dynamics, policy, and reward functions.
strength_weakness: strength_weakness: 1. The main weakness could the narrow scope of application in this paper. It is not sure how data quality from the collected textbook data could affect the performance the model. If there are more domains where the authors could experiment with different data sources, the readers would have better expectation of the model
2. The second weakness is the complex framework has many components. This on one hand is the novelty of this paper, however, it also adds to the difficulty of replicating this method, especially on other domains.
questions: questions: Can the URI framework be iterated -- collecting data online and summarize new knowledge. And the knowledge can be passed on to the next iteration of URI.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper introduces an intriguing approach to reinforcement learning (RL) through the concept of Policy Learning from Books (PLfB), which leverages textual resources like books and tutorials to derive policy networks. This methodology represents a interesting departure from traditional RL techniques that rely heavily on real interactions with the environment as recall by the authors.

The proposed URI framework outlines how the system first comprehends the textual content, then rehearses decision-making trajectories, and finally introspects to refine the policy network using an imaginary dataset. 

The practical validation of this method is demonstrated by training a football-playing policy and testing it in the Google Football simulation environment. The results are promising, with the agent achieving a 37% winning rate against the built-in agent without any interaction with the environment during training. This is a substantial improvement over the 6% winning rate achieved using an LLM.

In addition, the paper addresses the question of extracting policies without direct environment interaction by incorporating descriptions of MDP structures, transition functions, and reward functions within the textual data. This ensures the feasibility of the PLfB approach and adds depth to the methodology.

However, one aspect that could have been elaborated on is the influence of the prompting strategy used for generating the imaginary dataset. Detailing how different prompting techniques impact the quality and effectiveness of the dataset could provide valuable insights and enhance the robustness of the proposed approach.

Overall, the paper is a interesting contribution to the field, proposing an original perspective on utilizing textual knowledge for policy learning wel align with this current epoch of LLMs development. The results are encouraging, and the methodology is articulated and validated through practical experiments.
strength_weakness: strength_weakness: * Prompting Strategy Details: The paper lacks detailed discussion on the influence of the prompting strategy used for generating the imaginary dataset. Exploring different prompting techniques could provide valuable insights and improve the approach's robustness.
* Textual Resource Dependence: The success of the approach heavily relies on the quality and comprehensiveness of the textual resources, which might limit its applicability in domains with sparse or low-quality textual data.
* Generalizability: The generalizability of the method across different domains remains uncertain and needs further exploration to ensure its broad applicability.
* Complexity of Implementation: The methodology, while innovative, might be complex to implement and require significant computational resources, which could be a barrier for some researchers or practitioners.
questions: questions: 1. Prompting Strategy Exploration: Could you elaborate on the specific prompting strategies used to generate the imaginary dataset? How do you believe different strategies might influence the quality and effectiveness of the policy learned?

2. Generalizability Across Domains: What steps do you envision for testing the generalizability of the Policy Learning from Books (PLfB) approach in different domains or environments? Have you considered any preliminary experiments in varied contexts?

3. Quality of Textual Resources: How do you plan to address potential limitations related to the quality and comprehensiveness of the textual resources used? Are there specific criteria or methods you would recommend for selecting or evaluating these resources?
limitations: limitations: NA
paper_summary: paper_summary: The paper introduces a novel approach to policy learning, termed "Policy Learning from Books," which leverages existing textual knowledge, such as books and tutorials, to develop policy networks without the need for extensive real-world interactions. This method is inspired by how humans learn new skills from written resources. The authors propose a three-stage framework called URI (Understanding, Rehearsing, and Introspecting) to implement this approach. In the URI framework, the process begins with understanding the content of the books, followed by rehearsing decision-making scenarios based on the understood knowledge, and finally, introspecting on these rehearsed scenarios to refine a policy network. To demonstrate the effectiveness of this method, the researchers applied it to train a football-playing policy using the Google Football game. The trained agent, which did not interact with the environment during training, achieved a 37% winning rate against the built-in AI, significantly outperforming a GPT-based agent that only managed a 6% winning rate. This study highlights the potential of utilizing textual knowledge for enhancing decision-making processes in reinforcement learning.
strength_weakness: strength_weakness: 1. More datasets can be considered in the experiments.
2. I think the potential application of this work is beyond the football game. Why not validate its effectiveness on learning some other policies and validating their performance in some different challenging environments?
3. As for the baselines, the authors only compare their proposed framework with the LLM-based and rule-based policies. Why not compare it with some other policies learned with conventional RL algorithms, like PPO, DDPG, SAC, and so on?
questions: questions: 1. I am curious to the performance of distilled policy from the imaginary dataset generated directly by the GPT without the information from the books?
2. Why not provide the pseucode for this work in the anonymous repository? Maybe the reproducibility of this work can be further enhanced.
3. The caption and illustration of Figure 1 are a little bit confusing. Personally, I think book tutorial is also a kind of data, though it is not the real interaction data between the agent and the interaction. I suggest the authors revise this point in the future version.
4. The Figure 2 is somewhat redundant, considering the detailed 3-stage framework: understanding, rehearsing, and introspecting, has been clearly provided in the Figure 3.
limitations: limitations: See above weaknesses and questions.
paper_summary: paper_summary: The paper presents a method to distill knowledge about a given task or domain from text based knowledge into a form that can be used train a RL policy. The method extracts knowledge from text with a LLM, which is represented in a pseudocode-like textual form, and uses the knowledge to turn the LLM into a dynamics function, a reward function, and policy. Then the LLM is used to generate example trajectories by conditioning on the different types of extracted knowledge to turn it into a dynamics function, a reward function, and a policy. Finally conservative Q-learning is applied to the generated trajectories to learn a policy robust to the noise in the trajectory dataset. The method is evaluated on the Google Research Football environment against several baselines. The results demonstrate that the method improves performance relative to the baselines.
strength_weakness: strength_weakness: - It was difficult for me to follow what exactly what does in the method. The amount of notation is maybe obfuscating what the specifics of the method is.
- A class of methods that use LLMs as the backbone is missing, e.g. RT-2 or "Large Language Models as Generalizable Policies for Embodied Tasks"
- It would have been helpful to see how well URI does in a head-to-head match with the baselines (e.g. LLM-as-agent, LLM-RAG, and Random Policy). Given that the knowledge comes from books, which likely talk about semi-skilled strategies, it is unclear how important it is for the opponent to be of a similar quality as is discussed in the textual material. Head-to-head match ups would help understand how general and robust the policy is.
questions: questions: - How does this method relate to the use of background textual knowledge in "Motif: Intrinsic Motivation from Artificial Intelligence Feedback" and "Learning to Model the World With Language"?
- How beneficial is learning with a LLM versus learning a policy with the task reward and no LLM?
limitations: limitations: The limitations are not discussed in the main body of the paper

Comments: title: Response to Author's Rebuttal
comment: Dear Reviewers and ACs,

As the discussion period is almost over, we would like to deeply thank all reviewers and ACs for their efforts in evaluating the paper and for the timely responses during the discussion periods. We also appreciate all reviewers’ positive evaluations and recognition of our work's conceptual novelty, its value to the community, well-motivated methodology, extensive experiments, and promising results.

During the rebuttal periods, we received many valuable suggestions for improving the presentation of the paper, making it easier to follow and reproduce, and enhancing its potential impact. We summarize the revision plan we committed to as follows:

1. **Presentation:**
    - Add pseudocode to clarify the interactions of the components (R3, R4).
    - Polish Figures 1 and 2 (R3).
    - Include related work on Motif and Dynalang (R4).
    - Discuss the potential limitations when the quality of textual resources is low (R2).
    - Discuss the limitations of the paper in the main body instead of in the appendix (R4).
2. **Experiment:**
    - Add the results of the proof-of-concept benchmark based on the classic Tic Tac Toe game (TTT) to the paper (R1, R2, R3).
    - Add baselines, including URI-without-book-knowledge, pure RL-based algorithm, and RT-based method (R3, R4).
    - Include head-to-head match results between URI and the baselines (R4).
3. **Reproducibility:**
    - Open-source high-quality code, including the two environments, the full URI procedure, and a configurable implementation (R1, R2, R3, R4).
    - Discuss the details of the prompting strategy and highlight the important principles that make the solution work in the paper (R2).

Thank you again for your constructive feedback and recognition of our work. We are committed to revising the paper as planned to ensure it is well-represented and meets the reviewers’ expectations.

Sincerely,

Authors

R1: uFxN, R2: HEXR, R3: mSmU, R4: b6EF
Thank you for your constructive feedback and for maintaining your score after considering the reviews. We appreciate your recognition and will continue to address any aspects highlighted throughout the review process to improve our work further.
Thank you very much for your strong support and positive feedback on our manuscript. We are grateful for your recognition of the significance and contributions of our work. We will continue to address the key points discussed during the review process to further enhance the quality and impact of our research.
I do appreciate the authors' review. Considering other reviews, I keep my score unchanged also.
I appreciate the authors' review. After reading the other reviews, I will keep my positive score unchanged.
Thank you for your positive feedback on our work. We are pleased to have addressed your primary concern. We commit to put the discussion of the limitations to the main body of the paper in the revision, utilizing the extended page limit. Thank you again for your suggestions!
The authors have addressed my primary concern by committing to provide pseudocode and make their code open source. Please also find space to move a discussion of the limitations to the main body, and I will raise my score.
Thank you for your valuable feedback and for adjusting your evaluation of our work. We appreciate your acknowledgment of our efforts. We commit to adding the mentioned experiments to our revised paper.
Thanks for your detailed response. Most of my concerns have been addressed. I will raise the score.