## 3 Experimental Setting to Probe the Role of a Reward Using Football Scenarios
### Engineering Artificial Football Agents
In brief, _standard role-based players_ refer to a team following a standard formation in which two attackers, two defenders, and a goalkeeper play their own roles in pre-defined areas. _Gegenangriff (GGA)_ is a team following the most aggressive strategy, in which all football agents except the goalkeeper persistently focus on the attack regardless of the opponent's strategy. _Gegen pressing_ is a strategy in which three defenders press the opponents and two attackers focus on offensive behavior to score. _Becoming sprinter_ is the same strategy as that of _standard role-based players_, but with a faster moving speed. _Becoming defenders in the penalty area_ refers to a strategy whereby all agents become defenders and are located in the team's area. There is no offensive movement in this strategy. _Becoming man-to-man defenders_ is when agents become defenders and focus on the man-to-man defense. They only defend, and there is no offensive behavior. In _becoming ball-hunters_, there is neither defensive nor offensive behavior. All agents just track the ball's location and chase the ball without a consideration of the opponents' behavior. _All becoming goalies_ #1 and _all becoming goalies_ #2 involve the same strategy: all agents simply become goalkeepers; however, in the former, all agents stand in a row in front of the opponent's goal, whereas in the the latter, they stand in front of their team's goal. We used these _dynamic planning_ agents as training partners for the RL agents. **RL agent. ** This agent focuses on training agents to optimally perform a sequential decision in the environment to maximize a reward. The scalar-valued reward shapes the agent's behavior by providing a signal that the agent can use to estimate the value of the actions. The agent learns from past experiences by associating the reward signal with actions--higher rewards come from better actions and lower rewards come from worse actions when estimating the expected cumulative reward. Over time, the agent finally learns the optimal policy to achieve a goal that can maximize the cumulative reward. The scalar reward has played an important role in building artificial agents with human-level performance in several domains [36, 37, 38]. Now, the impact of rewards drastically affects the domain of MARL algorithms in which multiple agents in a team cooperate and collaborate to achieve their common goals [39, 40, 41, 42]. In the _AI World Cup_ environment, we focused on engineering MARL-based football agents. In particular, we simulated the football agents' behavior with regard to the team's main strategy, the so-called formation of the team, which can be exhibited persistently regardless of the opponent once the agents are well-trained. Two approaches were employed to this end: independent Q-learning (IQL hereinafter; Figure 6) [43] and QMIX [42]. The former, IQL, is a representative MARL algorithm based upon the principle of 'decentralized  
Figure 5: Nine sets of artificial football teams implemented by the _dynamic planning_ approach. training and decentralized execution'. In IQL, there is no centralized, unified Q-value for the entire team; each agent has a separate Q-value for each state-action pair to learn from past experiences instead. Thus, each agent learns the optimal policy subject to its own partially observed experience, which is shaped by a scalar reward for individuals. It is not expected that communications take place between agents to cooperate and/or coordinate their behavior. Without considering others, the agent is only governed by its own reward signal, and thus the team's performance is limited due to the partial overall observability (Figure 6A). In contrast, the latter approach, QMIX, is based upon the principle of 'centralized training and decentralized execution'. In QMIX, there is a centralized, total Q-value for the entire team. The total Q-value is actually a mixture of all individual agents' Q-values--specifically, a linear combination of them.