# Deep Reinforcement Learning Algorithms for Path Planning Domain in Grid-like Environment
## 3 Materials and Methods
### The Environment  
The data used in the learning process were obtained from the agent's interaction with the environment--an abstracted representation of the actual scenario, feasible for the automatic processing, provided in the form of a game. The design and implementation of the environment play a crucial role in the reinforcement learning process, as the agent's complete knowledge is derived from the interaction with it. Therefore, it is essential to abstract the problem without losing the parameters that are vital for the decision process. In this case, the delivery area map was abstracted into the form of a grid to form computationally feasible data (Figure 2). Such an approach has been previously used and tested in reinforcement learning scenarios with satisfactory results [8; 58; 59; 60]. The environment map consists of a \(10\times 10\) grid, with 10 objects of interest--6 fixed gas stations and 4 quest instances scattered through the map. The quest instances are generated randomly--at the beginning of every episode and upon interacting with one of them. Since only four can exist simultaneously, the next is spawned instantly upon completion of one of them. To complete the quest, the \"package\" has to be firstly retrieved from a quest location. Upon that, the random reward location with the corresponding monetary reward is generated--the further the destination point, the higher the \"in-game\" reward for completing that task. To make the learning process slightly more manageable, the quest's initiation provides the agent with a small reward--10% of the total reward (meaning both the reinforcement learning reward signal used in the learning process and in-game monetary reward) as a form of an advance, deducted from the total reward received at the end. This applied to the money obtained for completing the quest as well. The agent could later use this money to fill the gas tank while visiting the tiles containing the gas station to prevent his death. The maximum and start capacity of the gas tank was set to 30--forcing the agent to incorporate tanking into the plan quite often. Additionally, this action is not available to the agent at the very beginning of the episode, as initial funds (required to purchase the gasoline) are set to 0. In order to earn the required money, the agent has to begin or complete a task first. To achieve a smaller variation of the learning process, the magnitude of the rewards received by the agent was pretty small. For completion of the entire quest, the agent would receive a positive reward of size \(1\) (10% of which at the beginning) and \(-0.1\) while losing due to the empty gas tank. Such design of the environment allows us to ensure a sufficient level of complexity, as randomisation of the reward location generation and the fact that there can be multiple quest objects at once (each of them with a different reward for completion) requires the agent to constantly  
Figure 2: Visualisation of example environment layout. In this visualisation **O** is the agent, **X** are the quest instances, and dashed cells are the ones containing stations. This layout is an example of many different possibilities, as the quest instances are randomly scattered over the map, and their position differs from iteration to iteration—only the station locations are fixed and distributed as presented here. take multiple factors into consideration and forces it to reevaluate the plan on every quest completion, as the new locations cannot be predicted and planned earlier. The state vector provided by the environment is presented in Figure 3. To ensure that the agent possesses all the necessary information required for the learning process, the entire \"map\" of the game environment is represented as the vector of size 100 (Figure 3--**I**), where particular cells correspond to map tiles. Tiles values are set to 0 if empty; Otherwise, distinctive values based on the object inside--255 for the player, 140 for the reward location, 100 for a quest location, and 50 for a gas station.