# Reinforcement Learning for Mean-Field Game
## 2 Background
### Mean-Field Game  
In a game with a large number of players, we might expect that the distribution of agents over the action space carries more meaning than the actions themselves. It is intuitive that a single agent has a negligible effect on the game as the number of agents increases. The effect of other agents on a single agent's payoff is only via the action distribution of the population. This intuition is formalized in the _mean-field game_. We now formally define the mean-field game and equilibrium concepts. First, we define few notations. Let \(\alpha_{-,i,t}(a):\mathcal{A}\rightarrow[0,1]\) be the fraction of the agents (excluding agent \(i\)) that take action \(a\in\mathcal{A}\) at time \(t\). Mathematically, we have  
\[\alpha_{-,i,t}(a)=\frac{1}{n-1}\sum_{m\in[n]\setminus\{i\}}\mathbb{1}\left(a_ {m,t}=a\right), \tag{1}\]  
where \(\mathbb{1}(a_{j,t}=a)\) is the indicator function that the agent \(j\) takes action \(a\) at time \(t\). Since we assume an episodic framework, \(\alpha_{-,i,t}(a)\) can be different at each time index in an episode and also across the episodes. The episodic nature of the problem will be used later (in (9)) to define convergence to a value that depends only the time index in the episode. Further, since each agent selects exactly one action from \(\mathcal{A}\), we have  
\[\alpha_{-,i,t}(a)\geq 0\ \forall\ a\in\mathcal{A},\text{and}\ \sum_{a\in \mathcal{A}}\alpha_{-,i,t}(a)=1\]  
Similar to distribution of agents over actions, we define \(f_{-,i,t}:\mathcal{S}\rightarrow[0,1]\) as the distribution of agents (excluding agent \(i\)) over the state space \(\mathcal{S}\). \[f_{-,i,t}(a)=\frac{1}{n-1}\sum_{m\in[n]\setminus\{i\}}\mathbb{1}\left(s_{m,t}= s\right), \tag{2}\]  
In a mean-field game, every agent \(i\in[n]\) assumes that its next state \(s_{i,t+1}\) is randomly distributed according to the transition probability distribution \(P_{i}\) conditioned on agent's current state \(s_{i,t}\), the action taken \(a_{i,t}\) and other agents' distribution over actions \(\alpha_{-,i,t}\). Also, the reward is function of the agents current state and action and the next state. \[s_{i,t+1} \sim P_{i}(\cdot|s_{i,t},a_{i,t},\alpha_{-,i,t}) \tag{3}\] \[r_{i,t} =\phi(s_{i,t},a_{i,t},\alpha_{-,i,t},s_{i,t+1}) \tag{4}\]  
Thus the agent does not need to concern itself with the actions of the other agents, as the population action distribution \(\alpha_{-,i,t}\) becomes a part of the environment. This updated environment dynamics can now be used in decision-making. Note that the distribution of the population action \(\alpha_{-,i,t}\) could be explicitly taken into account for deciding an action as well. Note that the reward may also depend on \(f\) the state distribution of other agents. The analysis would have been similar. **Example 1**. : _We now provide an example drawn from a real-life application that simulates our setting. Suppose we consider the scenario of malware spreading. The state \(X_{i,t}=0\) means that an agent is vaccinated and can not infect at \(t\). On the other hand, \(X_{i,t}\) can vary between \(0\) and \(1\) with \(n\) quantization levels. The action space of an agent is \(a_{i,t}=\{0,1\}\). If \(a_{i,t}=0\) then the agent does not take any action. If \(a_{i,t}=1\), agent \(i\) takes action in order to protect itself. In order to simplify the model, we consider the following state evolution model_  
\[X_{i,t+1}= \begin{cases}\min\{X_{i,t}+\omega_{i,t}\alpha_{-,i}(0),1\}&\text{if }a_{ i,t}=0\\ 0&\text{if }a_{i,t}=1\end{cases} \]Note that if all the other agents have taken action \(1\) i.e. they protect themselves, then an agent has less chance to be infected. Thus, the state is smaller when the fraction of agents taking action \(0\) (i.e., \(\alpha_{-,i,t}(0)\)) is smaller. \(\omega_{i,t}\) is a noise who has its value in \(\{1/n,2/n,\ldots,1\}\). The reward function for agent \(i\) is defined as follows \(r_{i,t}=-X_{i,t}\alpha_{-,i,t}(1)-\lambda a_{i,t}\). The reward function increases if the number of agents who have protected themselves is higher.