## 4 System Implementation and Experimental Results  
This section explains in detail the cooperative human-machine evaluation framework introduced in Section 3. The implementation of the proposed framework and the numerical analyses using the proposed software program are presented in this section. The program software is developed using MATLAB(r) (made by MathWorks, Natick, MA, USA)and C++ language. The implemented software program includes several panels and two graph windows. The panels that make up the program deal with the network topology as shown in Figure 3. They form a network in two dimensions as shown in Figure 3a,b and in three dimensions as shown in Figure 3c and are expressed in a distribution to maximize visibility. \begin{table}
\begin{tabular}{l l} \hline \hline
1: & \(\delta\leftarrow\) constant/human intervene rate \\ \hline
2: & \(\gamma\leftarrow\) constant/discount rate \\ \hline
3: & \(\alpha\leftarrow\) constant/learning rate \\ \hline
4: & \(\textbf{for}\ 1\leq t\leq T\) \\ \hline
5: & **while** true **do** \\ \hline
6: & \(\overset{\rightarrow}{s}\gets getState()\) \\ \hline
7: & \(\textit{TakeAction}\Big{(}\overset{\rightarrow}{a}\Big{)}\) \\ \hline
8: & \(\textit{TakeSystemReurard}(r_{t})\) \\ \hline
9: & wait for next time step \\ \hline
10: & \(h\leftarrow\textit{gethumanQuantitativePrecise}\textit{Signal}()\) \\ \hline
11: & \(Rh_{t}\leftarrow\textit{getRandomlyGeneratedReurard}()\) \\ \hline
12: & **if**\(h=\)!0 \\ \hline
13: & Estimation for \(h\sim N\Big{(}\mu,\sigma^{2}\Big{)}\) \\ \hline
14: & \(\overline{h_{t}}=\frac{\sum_{i=1}^{n}h_{i}}{n}\) \\ \hline
15 & \(h_{t+1}=h_{t}-\frac{\eta}{\sqrt{h_{t}+\epsilon}}\cdot\Delta F(h_{t})\) \\ \hline
16 & \(h_{t}^{\prime}=\max\Big{(}\overline{h_{t}},h_{t}\Big{)}\) \\ \hline
17 & \(H_{t}^{\prime}=\max(h^{\prime}_{t},Rh_{t})\) \\ \hline
18 & \(\textit{TakeFinalReurard}\Big{(}H_{t}^{\prime}\Big{)}\) \\ \hline
19 & \(H_{t}^{\prime}=(1-\delta)r_{t}+\delta H_{t}^{\prime}\) \\ \hline
20 & \(\textit{Q}_{t+1}\Big{(}\textit{s}_{t}^{1},\textit{s}_{t}^{2},a_{t}\Big{)}=(1- \alpha)\textit{Q}_{t}\Big{(}\textit{s}_{t}^{1},\textit{s}_{t}^{2},a_{t}\Big{)}+ \alpha(H_{t+1}^{\prime}+\gamma\textit{max}\textit{Q}_{t}\Big{(}\textit{s}_{t+1} ^{1},\textit{s}_{t+1}^{2},a_{t}\Big{)}\) \\ \hline
21 & **end if** \\ \hline
22 & **end while** \\ \hline
24 & **end for** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Algorithm for Cooperative Human–Robot Evaluation Framework. The graph window and the windows that show the calculated values are shown in Figure 4. The windows have several functions. In a soccer game depicted in dynamically changing competitive network topologies as shown in Figure 4, the players' status changes and the system's reward is calculated and shown. At the same time, the user monitors two competitive network topologies depicted as a football game and inputs feedback on the behavior of the current state with a real number between 0 and 1. The evaluation feedback from the human is calculated by estimation and adaptive reward evaluation calculation, as shown in Table 2.  
Afterward, the reward from the system in Figure 4 and the human evaluation in Figure 4 are calculated according to the given level of human intervention, and the reward value according to the behavior of the learning object is updated. Figure 5 shows a program that implements the proposed cooperative human-machine evaluation framework for the reinforcement learning of competitive network topologies. The panels and windows that make up the program are described in detail in Table 3. For instance, in a soccer game, there exist both network topologies: one and its enemy, coexisting in the same environment. Figure 4: Final Reward of the Competitive Network with Dynamic Change Network Topology in Soccer. Game, (**a**) Automatically Calculated System’s Reward and (**b**) Human’s Feedback for Winning a Game. Figure 3: Players’ (Agent) Network Topology in Soccer Game Problem, (**a**) Soccer Game Network Topology.