## 3 Action Selection and Replay Update for Non-Executable Massive Actions
### Replay Memory Updater  
Because the replay memory randomly extracts the states and target values of the experience replays by the batch size and updates with the Q-function, the following problems occur. First, different target values can be recorded for similar or identical states. Consequently, the Q-function update may be error-prone when the batch size becomes larger. Second, there are cases where the action with the highest Q-value cannot be executed. Particularly, in an environment where a massive non-executable action occurs, the experience replay of such an action that cannot be executed in practice is recorded in the \begin{table}
\begin{tabular}{c} \hline \hline
**PROCEDURE**_Agent_Process_ \\ \hline
**BEGIN** \\
**SET**\(D\)\(\leftarrow\)\(\varnothing\) \\
**FOR** each episode \\
**FOR** _each time_ \\
**SET**\(\operatorname{s}\)\(\leftarrow\)**CALL**_State_Receiver_ \\
**SET**\(\operatorname{Q}(s_{t},a)\)\(\leftarrow\)**CALL**_Action_Estimator_with_\(s_{t}\) \\
**SET**\(a_{t}\)\(\leftarrow\)**CALL**_Action_Filter_with_\(s_{t}\), \(\operatorname{Q}(s_{t},a)\) \\
**SET**\(s_{t+1}\), \(r_{t}\)\(\leftarrow\)**CALL**_Action_Executer_with_\(a_{t}\) \\
**CALL**_Replay_Memory_Updater_with_\(D\), \(s_{t},s_{t+1},a_{t},r_{t}\) \\
**END FOR** \\
**END** \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Pseudocode of Agent Process. \begin{table}
\begin{tabular}{c} \hline \hline
**PROCEDURE**_Action_Filter_ \\ \hline
**INPUT:**\(s_{t},\operatorname{Q}(s_{t},a)\) \\
**OUTPUT:**\(a_{t}\) \\
**BEGIN** \\
**Initialize** \(\operatorname{M}(s_{t},a)\) **by** \(0\) \\
**\(\forall_{a}\)****SET** \(\operatorname{M}(s_{t},a)\leftarrow-\infty\)**, where** \(a\) **is not executable at** \(s_{t}\) \\
**SET**\(a_{t}\)\(\leftarrow\)\(\operatorname{argmax}(\operatorname{Q}(s_{t},a)+\operatorname{M}(s_{t},a))\) \\
**END** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Pseudocode of action filter. replay memory. Therefore, if the batch size is limited, the experience replay corresponding to an executable action cannot be selected relatively in replay memory. The proposed DQN-based reinforcement learning model shares a single target value by clustering similar, dependent states using the ANN algorithm. This provides space for additional dissimilar experience plays to be recorded in the replay memory. Moreover, because target values are updated for similar states, the Q-values of the actions output from the Q-function that learns these target values are similar and dependent to each other. Regarding the image classification research field, a CNN demonstrates higher performance than other traditional machine learning algorithms [38] and can be used to cluster similar states; however, the use of a CNN requires relatively more computation and memory than an ANN, and ANN learning must be repeated. This can lead to forgetting problems [39], causing a significant loss in performance. In addition, similarity generally cannot be measured in a CNN. For this reason, the proposed method uses an ANN algorithm, instead of a CNN. Table 3 shows the procedure of the updated replay memory process with ANN. To identify similar states with the ANN, the maximum allowable difference in similarity is defined as the distance \(\delta\). The proposed DQN-based reinforcement learning model compares the individual states recorded in replay memory \(D\) to the state \(s_{t}\) using the ANN to find and process the most similar state. Similarity is calculated in terms of a Euclidean distance. When the similarity is greater than the distance \(\delta\) or corresponds to the first experience replay, a new state is assumed, and a new target value \(y_{t}\) is calculated for DQN learning. When the similarity is smaller than \(\delta\) and a similar state exists in replay memory \(D\), \(y_{i}\) is updated. Action \(a_{i}\) is not updated because the ultimately selected action \(a_{i}\) may be different each time through the action filter even when the state is similar. If \(a_{i}\) is updated, the Q-value of the Q-function will be different each time.