## 4 Configuration of DQN Agent and Environment
Control performance of the DQN agent is significantly affected by the number of hidden layers and nodes (neurons) in hidden layers in the agent neural network. Generally, the more hidden layers and neurons are used, the better the control performance of the agent that can be obtained. However, the use of networks that are too complex can induce overfitting, resulting in poor control performance of the agent for unlearned data. The number of hidden layers and neurons in the agent neural network were determined to be two and thirty, respectively, based on a parameter study. A hidden layer usually has an activation function for introduction of non-linearity into the outputs of a network. Therefore, the rectified linear unit (ReLU) activation function was applied to each hidden layer as shown in Figure 9, because it is simple to apply and effective compared to other activation functions [34]. Determination of hyperparameter values and proper function for learning are not easy to develop for the DQN model for the semi-active control algorithm. Table 2 lists the hyperparameter values and functions selected for the DQN training. An Adam (Adaptive Moment Estimation) optimizer with the learning rate of 0.001 was employed as an optimization algorithm. The Adam optimizer is popularly used in the field of deep learning and has shown stable and good performance. Overfitting is a critical issue to be considered with DQN learning. Because the dropout method is known as an effective regularization technique to decrease overfitting and improve deep learning performance, it was used in this study. The dropout rate means the rate of nodes used for network output calculation. A dropout rate of 1.0 implies that there is no dropout node, and it is generally used for the verification process. An appropriate dropout value is known to be in the range between 0.5 and 0.8. The dropout rate of 0.8 was selected in the DQN training. Figure 9: Network configuration of DQN agent. In reinforcement learning, the agent usually discovers which action provides the most valuable reward by trial and error. When an agent always takes the known action that provides a big reward and does not explore any other actions, i.e., greedy behavior or exploration, the agent cannot improve its knowledge and may fall into sub-optimal results. Therefore, the agent requires enough exploration to find a global optimal solution. However, if the exploration rate is too high, the known high-value actions are not effectively used, and thus, the improvement speed of the agent's performance is decreased in the RL training process. When the agent explores, it gets closer to global optimal solutions, and when it exploits, it could obtain more reward. Because there is a trade-off between exploration and exploitation, they cannot be selected simultaneously [35]. In this study, the epsilon-greedy method was used to balance exploration and exploitation. In the epsilon-greedy method, epsilon means the probability of exploration with random command voltage. Because the epsilon is calculated by Equation (10) in this study, the probability of exploration decreases as the episode increases, resulting in the decreased range of fluctuation of the DQN model control performance. \[\varepsilon=1/(episode/10) \tag{10}\]  
The Adam optimization algorithm that is a training algorithm for deep neural networks in DQN uses the idea of moving averages of the gradients and squared gradients to adapt the learning rate for each parameter. It also uses bias-correction to ensure that the algorithm starts with a fair estimate of the moving averages. The update rule for the weights is a combination of the gradient with respect to the loss and the bias-corrected moving averages of the gradients and squared gradients. The Adam algorithm uses two different moving averages of the gradients and the squared gradients, denoted as \(m(t)\) and \(v(t)\), respectively.