# The Important Role of Global State for Multi-Agent Reinforcement Learning
## 2 Materials and Methods  
### Multi-Agent Reinforcement Learning Methods  
A MARL can be defined by a tuple \(\mathrm{G}=\langle S,U,P,r,Z,O,n,\gamma\rangle s\ (s\in S)\) refers to the state in a true environment. At each time step, each agent selects an action \(a\ (a\in A\equiv U^{n})\) to form a joint action \(u\ (u\in U\equiv U^{n})\). When a joint action \(u\) is selected, the agent will reach a new state after interacting with the environment. This process is represented by the state transition function \(P(s^{\prime}|s,u)\colon S\times\ \mathrm{U}\times S\ \rightarrow\ [0,\ 1]\.\ r(s,u)\) is the reward function and \(\gamma\in[0,\ 1]\) expresses a discount factor. A partially observable scenario represents the environment observed by a single agent, not all the information in the environment. Each agent has an episode \(\tau\), and the episode is represented by action-observation pairs or action-state pairs. The joint action is obtained from the joint policy \(\pi^{a}(u^{a}|\tau^{a})\) and has a joint action-value function, \(Q^{\pi}(s_{t},u_{t})=E_{s_{t+1:\infty,s_{t+1:\infty}}}[R_{t}|s_{t},u_{t}]\), where \(R_{t}=\sum\limits_{i=0}^{\infty}\gamma^{i}r_{t+i}\) is the discounted return. Deep Q-learning [25; 26; 27; 28; 29; 30] is a classic deep reinforcement learning method. Deep Q-networks (DQNs) use a deep neural network to represent the action-value function Q \((s,a)\). In DQNs, the training data are the stored transition tuple \(\langle s,\ u,\ r,s^{\prime}\rangle\) and the squared TD error is used to train the parameters \(\theta\):  
\[L(\theta)=\sum\limits_{j=1}^{b}\left[\left(y_{i}^{DQN}-Q(s,u;\theta)\right)^{ 2}\right] \tag{1}\]where \(y^{DQN}=\tau+\gamma max_{u^{\prime}}Q(s^{\prime},u^{\prime};\theta^{-})\). \(\theta^{-}\) is the parameter of a target network periodically copied from \(\theta\) and remains unchanged for a number of trainings. In MARL, agents are trained based on action-observation history. However, this method cannot fully consider the overall factors of the environment. The work of this paper takes the global state into account in strategy training, hoping to improve the effectiveness of strategy. In this section, we describe our approaches for adding environmental information to multi-agent settings. In the RL community, real-time strategy (RTS) games have always been an area of concern. Some games, such as football games and StarCraft [23], offer a great opportunity to tackle competitive and cooperative multi-agent problems. Competitive and cooperative problems are the main concerns in behavioral decision-making. Solving these problems is also the main aim of RL. Here, our algorithm was applied in StarCraft. Thanks to its rich set of complex micro-actions, which allow the learning of complex interactions between collaborating agents, StarCraft has been widely used in the RL community. Previous work applied RL to the original version of StarCraft: BW [31]. We performed our experiments on the StarCraft II Learning Environment (SC2 LE) [32], which is based on the second version of the game and has been supported by the developers of the game. Environmental importance in MARL. In this work, we focus on the important role of environmental factors in RL. An important feature of reinforcement learning is to learn in constant interaction with the environment. Multi-agent reinforcement learning needs to cooperate according to different environmental states. In this case, the learning of the environmental state is particularly important. Attention to this detail prompted our engagement with this work. Independent Q-learning (IQL) decomposes a multi-agent problem into a collection of simultaneous single-agent problems that share the same environment. Value decomposition networks (VDNs) aim to learn a joint action-value function, and each agent conditions only on individual action-observation histories. QMIX estimates joint action-values as a complex non-linear combination of per-agent values. QTRAN proposes a new value decomposition algorithm.