## 4 System Design
### Performance of Deep Q-Network  
In Section 4.2, the state is described as the input parameters required by the agent to derive actions from the DQN neural network model. All elements were used as inputs to the neural network model, along with the amount of change before each element. For example, the bandwidth of the following nodes and switches is shown in Equation (5) \begin{table}
\begin{tabular}{c c c} \hline \hline
**Notation** & **Symbol** & **Definition** \\ \hline _Time_ & \(x_{1}\) & Time when network data was collected \\ \(G(V,\,E)\) & \(x_{2}\) & Location of node or switch \\ \(RTT\) & \(x_{3}\) & Round tip time (Ms) \\ \(MSS\) & \(x_{4}\) & Maximum segment size (Byte) \\ \(AB_{sw}\) & \(x_{5}\) & Available bandwidth currently available for transmission \\ \(PreBw_{sw}\) & \(x_{6}\) & Previous link bandwidth (Gbps) \\ \(Now_{sw}\) & \(x_{7}\) & Location of the current node or switch \\ \(NextBw_{sw}\) & \(x_{8}\) & Next link bandwidth (Gbps) \\ \(Preb_{sw}\) & \(x_{9}\) & Previous hop node or switch \\ \(Nexthop_{sw}\) & \(x_{10}\) & Next hop node or switch \\ \(TNode_{count}\) & \(x_{11}\) & The total number of node or switch \\ \(PNode_{count}\) & \(x_{12}\) & The number of parity nodes \\ \(DNodecount\) & \(x_{13}\) & The number of data nodes \\ \(Node_{line}\) & \(x_{14}\) & Current living nodes \\ \(Node_{eland}\) & \(x_{15}\) & Current dead nodes \\ \(Block_{ssc}\) & \(x_{16}\) & Source node that transmits blocks \\ \(Block_{ds}\) & \(x_{17}\) & Destination node that be received blocks \\ \(Block_{size}\) & \(x_{18}\) & Transmitted size of blocks \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Input parameters of DQN neural network. through the current time \(t\), which is the time information, and the previous time \(t\)-1, which is the amount of change. \[\Delta NextBw_{sw}=NextBw_{sw}(t)-NextBw_{sw} \tag{5}\]  
The action is passed on to the erasure coding network topology according to the policy configured by the agent based on any state value. In addition, when a proper action is applied according to the state value, the agent can obtain a positive reward value for nodes that need to transfer blocks in the erasure coding network topology, and a negative reward value for moving to the wrong node and switch. Thus, the entire _Action_set_ group performs \(k\) different actions for each time period \(t\), as expressed through Equation (6). \[Action\_set_{t}=\{a_{1},a_{2},\dots,a_{k}\} \tag{6}\]  
A reward is utilized to update the weights in the DQN neural network model, and unlike using labels for correct answers in supervised learning, each weight is updated through backpropagation using action-driven rewards. Therefore, the learning outcomes of neural network models may vary depending on how the reward is defined. In this study, the \(Reward_{total}\) values were configured in three forms. \(Reward_{1}\) has a positive value when data are sent using the correct link, \(Reward_{2}\) has a negative value when data are sent using the wrong link, and \(Reward_{3}\) has an extremely small negative value because it enters into a loop unless it chooses a switch link with another node. The overall reward value is shown in Equation (7). \[Reward_{total}=Reward_{1}+Reward_{2}+Reward_{3} \tag{7}\]  
\(Reward_{1}\) is shown in Equation (8) by setting it as a positive integer value when the current node and switch location (\(Now_{sw}\)), the node and switch location (\(Prehbo_{sw}\)) of the previous link, and the next node and switch location (\(Nexthbo_{sw}\)) are all moved throughout the entire path. \[Reward_{1}=Now_{sw}+Prehbo_{sw}+Nexthbo_{sw}(0<Reward_{1}\leq 1) \tag{8}\]  
\(Reward_{2}\) is shown in Equation (9) by setting the value to a negative integer when the link is moved from the current node and switch position (\(Now_{sw}\)) to the failed node and switch (\(Node_{dead}\)). \[Reward_{2}=Now_{sw}-Node_{dead}\ (-1\leq Reward_{2}<0) \tag{9}\]  
\(Reward_{3}\) is a negative value for exiting the loop state as a constant cycle is repeated.