## 2 Reinforcement Learning Framework with DQN  
Machine learning generally means a computer learning from data using algorithms to perform a task [25]. When a machine learns, an algorithm acquires skills or knowledge from experience. There are many different types of machine learning techniques. Figure 1 presents the three main learning types in machine learning. Supervised learning makes a model learn a mapping between input data and the output targets [26]. Unsupervised learning develops a model that can extract relationships from input data without outputs or target data. Unlike supervised learning, unsupervised learning does not have a teacher and is generally used for clustering complex data. RL develops an agent that learns using feedback with an environment. An agent learns how to map states to actions in order to maximize a reward in RL. The agent is not guided on which action is good or bad but has to find out which action provides the most reward [22]. State, action, and reward are the main basic concepts in RL. The state shows the current situation in the environment. In this study, the states are the seismic responses of the sample structure and ground motions that can explain the characteristics of the excitation. Action means what an agent can do in each state. The semi-active control algorithm, which is the agent in this study, decides the control command for the SMIS based on the given state. The control command produced by the agent is the command voltage sent to the MRdampers in the SMIS. The MR damper used in this study has a voltage range of 0 to 5 V. The finite actions the agent can take are determined to be eleven voltage values in the range between 0 and 5 V with 0.5 V intervals. When an agent takes an action in a state, it receives a reward. Generally, an agent in RL is trained to maximize the reward. Because the purpose of the SMIS is to reduce the seismic responses of the sample structure, the maximization of the reward should be achieved by decreasing seismic responses of the sample structure. Therefore, it is required to use the seismic responses of the sample structure as negative values or denominators in fractions for reward design. Figure 2 illustrates the configuration of the environment and the agent, including the feedback data types used in this study. A lot of methods of RL have been proposed to date. Among them, the deep Q-network (DQN) is widely used because it has shown good performance in various control problems. DQN is a reinforcement learning algorithm that combines Q-learning with neural networks. The basic idea behind DQN is to use a neural network to approximate the Q-value function for a given state-action pair, rather than using a table as in traditional Q-learning. The DQN algorithm methodologically involves the following steps:  
1. Initialize the Q-network with random weights and a replay memory with a fixed capacity;
2. At each time step, the agent selects an action based on the current state and the Q-network's estimates of the Q-values;
3. The agent then receives a reward and a new state, and the transition (state, action, reward, next state) is stored in the replay memory;
4. Periodically, a batch of transitions is randomly sampled from the replay memory and used to update the Q-network's weights. This is performed by computing the target Q-value for each transition using the Bellman equation and then minimizing the difference between the target Q-value and the Q-network's estimate;
5. The Q-network's weights are also periodically updated with the weights of a fixed target network, which helps to stabilize the learning process. Figure 1: Machine learning categories. Figure 2: Relationship of the environment and agent in reinforcement learning. In the DQN, deep neural networks are used for mapping between input and output data, and it may cause to RL to be more unstable. A lot of data is required to train a deep neural network, but even then, the network weights sometimes cannot converge [27]. To solve this, [PERSON] et al.