## 2 Reinforcement Learning Framework with DQN
(2015) [22] introduced two main techniques, i.e., experience replay and target network separation, into the DQN. When the training data is independent and equally distributed, the reinforcement learning results are good. However, when the reinforcement learning is performed, the series of experience data can be highly correlated to each other. For example, each dynamic response of every time step of the sample structure subjected to an earthquake does not differ much, because the numerical integration time step for time-history analysis is usually less than 0.01 or 0.02 s. The conventional Q-learning method that learns from each of these sequential simulation data can be influenced by the effects of this correlation. In the DQN algorithm, this problem was solved by using a large buffer for storing the past experience of the agent and selecting training data from it, instead of using the most recent experience. Figure 3 shows the concept of this replay buffer (experience buffer) technique. As shown in the figure, past experiences are stored in the replay buffer, and then a subset of these experiences are randomly selected to update the Q-network. In Q-Learning, the Bellman equation is expressed by an artificial neural network (ANN) as shown in Equation (1):  
\[Q_{\theta}(s,a)=r+\gamma\underset{at}{\max}Q_{\theta}(s,at) \tag{1}\]  
where the s, a, r, and \(\gamma\) mean the state, action, reward, and discount factor, respectively. The meaning of primed \(\theta\) variables is the variable's value in the next state. The variable \(\theta\) of the action-value function \(Q_{\theta}(s,a)\) represents the parameter vectors of ANN. In Q-Learning, the value of \(r+\gamma\underset{at}{\max}Q_{\theta}(s,at)\) is considered to be the target Q value. Therefore, a predicted value of ANN \(Q_{\theta}(s,a)\) should gradually reach the target value in the learning process. For this purpose, the loss function of Q-Learning can be expressed by the difference between the target value and the predicted value, as shown in Equation (2). When the learning starts, the parameters \(\theta\) of ANN for the action-value function \(Q_{\theta}(s,a)\) are updated to minimize the loss function:  
\[L(\theta)=(r+\gamma\underset{at}{\max}Q_{\theta}(s,at)-Q_{\theta}(s,a))^{2} \tag{2}\]  
In native Q-learning, a single ANN with the parameters \(\theta\) is used for both the predicted value \(Q_{\theta}(s,a)\) and the target value \(Q_{\theta}(s,at)\). Therefore, when an update of ANN parameters \(\theta\) to make \(Q_{\theta}(s,a)\) closer to the target value is preformed, the target value \(Q_{\theta}(s,at)\) is indirectly changed. It may cause the training to be unstable. To make the training more stable, the DQN uses a target network separation, by keeping a copy of the neural network (\(\overline{\theta}\)), and using it for the target value \(Q_{\overline{\theta}}(s,at)\). In this technique, the parameters of the target network are not changed during the main network training, but they are periodically copied from the parameters of the trained main Q-network, as shown in Figure 4. This idea can improve the stability of Q-learning training [28]. Figure 3: Experience replay of DQN.