## 3 Methodology of Cooperative Human-Robot Evaluation  
This study is concerned with the problem of artificial intelligence soccer game shown in Figure 1. This is an example of a special case that differs from those previously investigated. In this paper, to deal with the environment in which two or more dynamically changing multi-agents in networks compete and coexist with each other, designed an artificial intelligence soccer game similar to the artificial intelligence basketball game covered in [27]. Figure 1: Dynamically Changes Multi-Agent in Competitive Network mapped with a Soccer Game. The learning goal of the soccer game in this study is to pass the ball by avoiding the red team soccer players who are obstructed by the blue team soccer players. In real-life football games, players move dynamically on the field and try to pass the ball successfully to a player from the same team. As such, players in football games described as multi-agent form a network topology as shown in Figure 1, and these network topologies show the two teams competing against each other. Each team is given a network topology at time \(t\), and there is a state as a reference when it is expressed as a reinforcement learning problem considering the situation and environment, and there is another state that follows and competes similarly. To deal with this reinforcement learning problem in this special case, this study defines the Q-Function as Equation (2) and calculates the maximum human evaluation reward value by calculating every time cycle period, \(t\). \[Q_{t+1}\big{(}s_{t}^{1},\ s_{t}^{2},a_{t}\big{)}=(1-\alpha)Q_{t}\big{(}s_{t}^{ 1},\ s_{t}^{2},a_{t}\big{)}+\alpha\Big{(}h_{t+1}^{*}+\gamma maxQ_{t}\big{(}s_{t +1}^{1},\ s_{t+1}^{2},a_{t}\big{)}\Big{)} \tag{2}\]  
State \(s_{t}^{2}\) that coexists with state \(s_{t}^{1}\) is not an independent state, but it is affected by the same action in the same environment. Therefore, it can be defined as Equation (3). \[a_{t+1}\big{(}\overset{\rightarrow}{s_{t}^{2}}\big{)}=s_{t+1}^{2} \tag{3}\]  
Traditional reinforcement learning problems often involved a single network topology with the learning objects. However, this study covers two or more network topologies such as a dynamically changing competitive network. In a single network topology, the reward policy could be learned through a system reward update calculated from the learning object. Conversely, in this study, to establish a reward policy for a complex network in which two networks are learned in the same environment, the procedure for obtaining rewards through human evaluation and the reward of the system calculated from the learning object are properly applied. Here, a procedure is involved that updates rewards through human intervention, while existing studies [25; 26] have mapped human evaluation through a binary process; moreover, this study advances evaluation feedback in a more quantitative and accurate form. The human rating is entered as a real value between 0 and 1, and the reward policy is updated in a much more accurate form. The human evaluation reward, \(h_{t}\), obtained in the process of learning a number of times, can be modelled by a Gaussian distribution as shown in Equation (4),  
\[h_{t}\ \sim N\Big{(}\mu,\sigma^{2}\Big{)} \tag{4}\]  
where \(\mu\) is the evaluation mean, and \(\sigma\) is the distribution of standard deviation. In general, the mean of the Gaussian distribution is estimated \(\overline{h_{t}}\) using Equation (5),  
\[\overline{h_{t}}=\frac{\sum_{i=1}^{t}h_{i}}{n} \tag{5}\]  
where \(n\) is the sample size and is the number of quantitative rewards from human evaluation that has been learned for a number of times \(t\). The standard deviation of the Gaussian distribution is estimated using Equation (6).