Package: tabularmazemdp
TabularMazeMDP:This repository contains code to provide a simple Maze environment used as example MDP for tabular dynamic programming and reinforcement learning labs.If you want to do the corresponding labs, you need a Google account. Then you can copy-paste thedynamic programming colaband thereinforcement learning colab.DocumentationMDPs and mazesSome code is provided to create mazes, transform them into MDPs and visualize them together with policies or value functions.
It is contained into three files: *maze.py,mdp.pyandmaze_plotter.py.
The following sections give an overview of this code.Content of themaze.pyfileA maze is represented as an object of theMazeclass. It is defined as a grid ofwidth x heightcells, and some of these cells contain a wall.Thebuild_maze(width, height, walls, hit=False)function is used to create a Maze, wherewallsis a list of the number of the cells which contain a wall.
Thehitparameter has an impact on the MDP reward function: ifhitistrue, the agent is penalized each time it tries to move to a wall cell. Otherwise,
the agent is just rewarded when it reaches terminal states. In the provided function, the list of terminal states is a singleton corresponding to the last cell that the agent can visit.Apart from representing the two reward functions described above, theMazeclass contains a constructor whose only role is to create the MDP corresponding to the maze and the maze plotter used to display simulations. A key point is that only cells where there is no wall are considered as states of the underlying MDP. To facilitate the correspondence between mazes and MDPs, each free cell (i.e. with no wall) knows the number of its corresponding MDP state.The maze constructors also builds the action space, the initial state distribution, the transition function and the reward function of the MDP. Once all these data structures have been created, the resulting MDP is built.Abuild_maze()and acreate_random_maze()functions are provided to create mazes in the lab notebooks or python files.Content of themdp.pyfileThemdp.pyfile contains theSimpleActionSpaceclass and theMdpclass.TheSimpleActionSpaceclass contains the list of actions and a method to sample from this list. In our maze environment, the possible actions for the agent are going north, south, east or west (resp. [0, 1, 2, 3]).TheMdpclass is designed to be compatible with the OpenAI gym interface (https://gym.openai.com/). The main methods arereset(self, uniform=False), which resets the MDP into an initial state drawn from the initial state distribution, andstep(self, u, deviation=0)which is used to let the agent perform a step in the environment, sending and action and receiving the next state, the reward, and a signal telling whether a terminal state was reached. The functionrender(self)provides a visual rendering of the current state of the simulation.Content of themaze_plotter.pyfileThe code to display the effect of the algorithms in these environments is inmaze_plotter.py, in theMazePlotterclass.
In order to visualize the environment, you use thenew_render()function to initialize the rendering, thenrender(V, policy, agent_pos)to refresh the maze with either the newly calculated state values and the policy, or the state-action values, and eventually the current position of the agent. There is also arender_pi(policy)function which only displays the policy (useful forpolicy iteration). The functionsave_fig(title)is used to save the last render into a file.You can see examples of calls to these different visualizations in the functions defined in dynamic programming and reinforcement learning notebooks or python files.ToolboxThetoolbox.pyfile provide a few useful functions such asegreedy(),egreedy_loc()andsoftmax()which are used to perform exploration in reinforcement learning algorithms.