## 4 Stationary Equilibrium
Therefore, the payoff of player \(p(n)\) given that in periods \(n+1\) through periods \(2n\), the players will use the strategies \((f_{1}^{1},\cdots,f_{n}^{1})\) is thus given by  
\[u_{p(n)n}(s_{\omega},a)+\delta\sum_{s_{\ell}\in S}v_{p(n)1}^{1}(s_{\ell})q(s_{ \ell}:s_{\omega},a),\]  
as the future expected payoff is now given by \(v_{1}^{1}:S\to\mathbb{R}^{n}\). Let \(f_{n}^{2}:S\to A_{n}\) denote the optimal period-\(n\) strategy of player \(p(n)\). The future expected payoff of player \(i\) from period \(n-1\) onwards, for each \(s_{\omega}\in S\), is then given by  
\[u_{i}(s_{\omega},f_{n}^{2}(s_{\omega}))+\delta\sum_{s_{\ell}\in S}v_{1}^{1}(s _{\ell})q(s_{\ell}:s_{\omega},f_{n}^{2}(s_{\omega})). \tag{4}\]  
Denote this expected future payoff by \(v_{i(n-1)}^{2}(s_{\omega})\) for each player \(i\), and for each \(s_{\omega}\in S\). Then, the payoff of player \(p(n-1)\) in period \(n-1\) is  
\[u_{p(n-1)(n-1)}(s_{\omega},a)+\delta\sum_{s_{\ell}\in S}v_{p(n-1)(n-1)}^{2}(s _{\ell})q(s_{\ell}:s_{\omega},a).\]  
Denote the optimal choice of player \(p(n-1)\) for this payoff to be \(f_{n-1}^{2}(s_{\omega})\), and the optimal period \(n-1\) strategy as \(f_{n-1}^{2}:S\to A_{n-1}\). Proceeding in this manner, one can find the optimal strategies \((f_{n-2}^{2},\cdots,f_{1}^{2})\), for periods \(n-2\) through \(1\). Therefore, the period \(1\) through period \(n\) optimal strategies, given that the future expected payoff in period \(2n\) is \(v:S\to\mathbb{R}^{n}\), is given by \((f_{1}^{2},\cdots,f_{n}^{2})\), and for periods \(n+1\) through period \(2n\), the optimal strategies are given by \((f_{1}^{1},\cdots,f_{n}^{1})\). It should now be clear that if the future expected payoff in period \(3n\) is \(v:S\to\mathbb{R}^{n}\), then the period \(2n+1\) through period \(3n\) optimal strategies are given by \((f_{1}^{1},\cdots,f_{n}^{1})\). Period \(n+1\) through period \(2n\) optimal strategies are given by \((f_{1}^{2},\cdots,f_{n}^{2})\). The period \(1\) through \(n\) optimal strategies are now derived on the basis of these period \(n+1\) through period \(3n\) optimal strategies. Denote these by \((f_{1}^{3},\cdots,f_{n}^{3})\). Proceeding in this manner, one can show that there exists a sequence \(\{f_{1}^{k},\cdots f_{n}^{k}\}_{k=1}^{T}\) for each \(n\)-period cycle, from period \(1\) through \(T\). Therefore, as \(T\to\infty\), the sequence of optimal strategies will converge to a limit \(\{f_{1}^{\star},\cdots,f_{n}^{\star}\}\), for each of the \(n\)-period cycles. We claim that the [PERSON] strategy \(f^{\star}=\{f_{1}^{\star}\}_{n=1}^{\infty}\), in which \(\{f_{1}^{\star}\}_{n=1}^{n}=\{f_{1}^{\star},\cdots,f_{n}^{\star}\}\), \(\{f_{1}^{\star}\}_{l=n+1}^{2n}=\{f_{1}^{\star},\cdots,f_{n}^{\star}\}\), and in general, \(\{f_{Markov equilibrium, by using the same argument as in Theorem 1. This establishes the result. In the stationary Markov perfect equilibrium, the value functions must satisfy the condition that when the players choose their optimal strategies, the value function generated is the same as the future expected value function used in defining the game in any given period. Hence, in the period in which player \(p(n)\) chooses, the stationary equilibrium value function has to satisfy the condition that the value function generated is the value function used for the future expected value. More specifically, the stationary equilibrium value function for a player \(p(n)\) choosing in period \(n\) satisfies  
\[v^{\star}_{p(n)n}(s_{\omega}) = \max_{a\in A_{n}}[u_{p(n)}(s_{\omega},a)+\delta\sum_{s_{\ell}\in S }v^{\star}_{p(n)(n+1)}(s_{\ell})q(s_{\ell};s_{\omega},a)], \tag{5}\]  
where \(v^{\star}_{p(n)n}(s_{\omega})\) is the stationary equilibrium value function of player \(p(n)\) in period \(n\) in state \(s_{\omega}\), and \(v^{\star}_{p(n)(n+1)}(s_{\ell})\) is the stationary equilibrium value function of player \(p(n)\) in state \(s_{\ell}\in S\) in period \(n+1\).