## 4 Configuration of DQN Agent and Environment  
The sample building structure with the SMIS and the artificial ground motion developed in the previous section were used to construct the environment of reinforcement learning. When the time history analyses of the sample structure subjected to the artificial earthquake are performed for numerical simulation in the environment, a lot of dynamic responses of the sample structure can be obtained. Among them, the essential dynamic responses should be selected as the states for the agent to take the appropriate action for optimal semi-active control. In this study, four structural responses and a series of ground accelerations were selected for the states, as shown in Figure 8.  
First, the roof story dynamic responses, i.e., displacement and acceleration that can represent the overall structural behavior of the sample building, were selected. Because the inter-story drift is directly related to the structural damage of the building, the 11 th-story drift where the peak story drift occurs and the isolation story drift were also selected as the states to check if the maximum story and isolator drifts exceed their limit or not. Because the ground acceleration is one of the major causes of structural responses, it will give the agent good information to find the appropriate action to decrease the seismic responses of the sample structure. However, one ground acceleration value at a specific step is not enough to predict the seismic responses of the sample structure at that moment. The responses of the specific ground acceleration value are not identical all the time. For example, even if the ground acceleration is 0.2 g, the seismic responses at that moment vary, depending on whether the acceleration is increasing or decreasing. Therefore, not only the ground acceleration value at the current step, but also a series of ground acceleration values of the previous four steps, were selected as the states to give the agent sufficient information to select the optimal control command. As explained previously, the command voltage for the MR damper in the SMIS is determined by one of eleven voltage values, i.e., the voltage range between 0 and 5 V with a step of 0.5 V. Therefore, the agent selects one of the finite actions (command voltage) based on the input state. When the agent selects the control command for the SMIS, its control forces are transmitted to the sample structure, and the seismic responses are changed. Because the goal of this reinforcement learning is to minimize the seismic responses of the sample structure, the reward should be designed to reflect this goal. The inter-story drift is one of the most critical seismic responses related to building failure. Therefore, the 11 th-story drift and the isolation story drift were used to calculate the reward, as shown in Equation (9). Because the agent in reinforcement learning is trained to maximize the reward, the drift responses of the 11 th story and the isolation story are expressed as denominators in the  
Figure 8: Configuration of the DQN agent and environment. fraction in Equation (9). By squaring these drift responses, the reward decreases quickly as the drift values increase:  
\[\text{reward}=1/\{(11\text{th story drift})^{2}+(\text{Isolation story drift})^{2}\} \tag{9}\]  
In the DQN, the agent is modeled by an artificial neural network. Figure 9 shows that the structure of the agent neural network used for the DQN implementation in this study has two hidden layers. The neural network of the agent maps the input state to the output action. As explained previously, four seismic responses (roof story displacement and acceleration, and the 11 th story and the isolation story drifts) and five steps of the ground acceleration are used for the input state, and thus, the input size of the agent is nine. The output size of the agent neural network is eleven, because one hot encoding is used for the agent's action determined by one of eleven voltage values.