## 3 Proposed Multi-Agent N-DQN
### Proposed Architecture
#### 3.1.1. Hierarchization and Policy
Observe \(S_{0}\) and choose \(A_{0}\sim\tau_{0}(S_{0})\) fort = 1 to T do  Observe \(S_{t}\), \(R_{t}\), \(r_{t}\) forp = 1 to N do  Store transition (\(S_{t-1}\), \(A_{t-1}\), \(R_{t}\), \(r_{t}\), \(S_{t}\) ) in \(H\) with maximal priority \(p_{t}=max_{i<t}\)\(p_{i}\) endfor ift \(\equiv 1\) mod K then forj = 1 to k do  Sample transition j \(\sim\) P(j) = \(p_{j}^{\alpha}\)/ \(\sum_{i}p_{i}^{\alpha}\)  Compute sampling weight \(\text{w}_{j}=(N\cdot P(j))^{-\beta}\)/ \(\text{max}_{i}\text{w}_{i}\)  Compute TD-error \(\delta_{j}=\text{R}_{j}+\text{r}\delta_{target}(S_{j},\text{argmax}_{a}Q(S_{ j},a))-Q(S_{j-1},A_{j-1})\) Update transition priority \(p_{j}\leftarrow|\ \delta\ |\)  Gather weight-change \(\Delta\leftarrow\Delta+w_{j}\cdot\ \delta_{j}\cdot\ \
abla_{\theta}Q(S_{j-1},A_{j-1})\) endfor  Update weights \(\theta\leftarrow\ \theta+n\cdot\ \Delta\), reset \(\Delta=0\)  From time to time copy weights into target NN \(\theta_{target}\leftarrow\ \theta\) endif forp = 1 to N do  Choose action \(A_{t}\sim\tau_{\theta}(S_{t})\) endfor endfor
```
**Algorithm 4**Prioritized Experience Replay

## 4 Evaluation Results and Analysis with Maze Finding

### Environment

Figure 6 refers to the structure for performing reinforcement learning in the maze game environment. The principle that the action at each state is performed is diagrammed. ### Training Features

The state is defined as the actor's current position coordinates (x/y) in the game. The actor is implemented to acquire information again each time an action is performed because the location changes in real time according to the action. Action is defined as the direct movement of an Actor. The actor can take one of the following actions (up/down/left/right) and cannot pass through blocked walls. Reward is defined as checking how close the state's location information is to the goal in proportion to the number of cells. Also, when reaching the goal, it is implemented to obtain a reward with a high weight. Table 3 shows the object of learning in the maze game. ### Implement and Performance Evaluation

#### 4.3.1 N-DQN-Based Implementation

Four sublayers and one main layer were used between the implementation of reinforcement learning in the maze game environment through N-DQN. The learning algorithm is based on the DQN and includes the structure and functions of the architecture described in Chapter 3. Figure 7 means the reward policy applied to the maze game. While performing learning and exploration for a certain period, it establishes a policy to subdivide the reward acquisition periods based on the time taken in each state and the number of steps to reach the goal. In the reward, the sum of the rewards until the game is finished is calculated and weighted by additionally implementing Equation (3), which is discounted by a factor of y at each stage. In addition to this process, the stage in which the reward rapidly deteriorates \begin{table}
\begin{tabular}{c c} \hline \hline
**Category** & **Contents** \\ \hline State & The coordinates of actor’s location (x/y) \\ Action & Up/down is related to movement, and left/right pushes the ball \\ Rewards & +1, on arrival at the goal \\  & For movement of each step—(0.1/number of cell) \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Object of learning in the maze game. Figure 6: Visualization of reinforcement learning of maze game. is identified based on the state. The identified stage as such is given a negative reward to the weight. \[R_{t}=\sum_{t^{\prime}=1}^{T}Y^{t^{\prime}-t}\ r_{t} \tag{3}\]

#### 4.3.2 Performance Evaluation and Discussion

In this section, to evaluate the performance of the proposed multi-agent N-DQN model, we compare and examine our results with Q-Learning and DQN algorithm under the same environments and conditions. Figure 8 is the result of applying Q-Learning to a 5 \(\times\) 5 maze environment. In conclusion, Q-Learning solved the problem within the targeted 400 steps, and it took less than 50 steps and showed good learning efficiency.