## 3 Methodology
### Design of LLM-Based Agents  
Expressing the environmental state in natural language is crucial for LLMs to understand the situation and decide on appropriate actions. In this study, the get_environment_state function was used to create a state string that included structured sentences describing (1) the agent's current position and orientation, (2) the goal's location, (3) the cumulative reward up to that point, and (4) a summary of past learning experiences, enabling the LLM to comprehensively understand the environment.  
The prompt for the LLM was designed to clearly define the agent's role, current state, and expected output, as shown in Tables 2 and 3. The structure of the prompt is as follows: (1) definition of the agent's role, (2) presentation of the current environmental state, (3) description of the goal, (4) listing of possible actions, and (5) specification of the response format. This structured prompt guides the LLM to generate responses in a consistent format.  
The conversion of the LLM's output into actual agent actions is handled by the get_llm_action function. This process consists of the following steps: (1) analysis of the LLM output text and (2) conversion of this text into predefined action mappings. This action conversion process can be expressed mathematically as follows:  
\[\text{A(r)}=\left\{\begin{array}{ccccc}&\text{\emph{Move}}&\text{\emph{Forward}}& \text{\emph{if}}&\text{\emph{r}}&\text{\emph{contains}}\\ &\text{\emph{Turn}}&\text{\emph{Left}}&\text{\emph{if}}&\text{\emph{r}}& \text{\emph{contains}}\\ &\text{\emph{Turn}}&\text{\emph{Right}}&\text{\emph{if}}&\text{\emph{r}}& \text{\emph{contains}}\end{array}\right. \tag{1}\]  
In this context, \(r\) represents the response text from the LLM, and A(r) is the final selected action.  
\begin{table}
\begin{tabular}{l} \hline The agent is currently located at position (2, 3) and is facing right. \\ The goal is located at position (4, 4). \\ There is no wall in front of the agent, there is a wall to the left, and no wall to the right. \\ The current cumulative reward is 5. \\ \hline \end{tabular}
\end{table}
Table 2: Environmental state string.  
\begin{table}
\begin{tabular}{l} \hline You are an AI agent controlling a character in a 5x5 grid environment. You must choose the optimal action based on the current state and past learning memories. \\ Current state: [Environmental State String] \\ The agent must efficiently reach the goal location. Refer to the rewards and \\ punishments learned from past experiences to select one of the following actions in your \\ response: ‘Move forward’, ‘Turn left’, ‘Turn right’. No further explanation is needed. \\ \hline \end{tabular}
\end{table}
Table 3: LLM prompt design string.  
Additionally, the overall execution of the reinforcement learning episode is represented as shown in Equation (2).  
\[\begin{split}\text{Total reward}:& R_{total}=\sum_{t}^{T} =0\;r_{t}\\ \text{Action selection}:& a_{t}=\text{get llm action}(s_{t})\\ \text{State transition}:& s_{t+1},\;r_{t}=\text{ move agent}(s_{t},\;a_{t})\end{split} \tag{2}\]  
Here, \(R_{total}\) represents the total reward obtained during each episode, and \(a_{t}\) denotes the action selected by the LLM in the current state \(s_{t}\). \(s_{t+1}\), \(r_{t}\) indicates the new state and immediate reward after performing the action, respectively.  
This conversion process maps the natural language output of the LLM to executable actions within the MiniGrid environment.  
As shown in Figure 1, the environmental state is converted into natural language and fed into the LLM, and the output from the LLM is then transformed back into specific agent actions applied to the environment, demonstrating a cyclical process. Through this approach, this study effectively applies the language understanding and generation capabilities of LLMs to reinforcement learning problems. This presents a new paradigm for integrating natural language processing techniques into decision-making processes in complex environments.