## 4 Stationary Equilibrium  
The model of asynchronous stochastic games analyzed until now is non-stationary and allows the parameters of the model to be different over the time periods. In many instances, however, the major parameters remain the same over the periods. In such a case, it is natural to raise the question as to whether there are equilibrium strategies that are not only Markovian but also stationary. In a stationary equilibrium, the players use the same function to make their choices in each period. In many cases, however, the utility functions as well as the action sets of the different players are different, and the idea of stationary strategies then needs to be adjusted to reflect this. In such cases, it would be natural to require that a stationary strategy of a player would be one in which the player uses the same function to make choices whenever that player is the active player. Definition 3: A strategy of a player \(i\) is a stationary Markov strategy if the choice function \(f_{it}:S\to A_{i}\) is the same in every period \(t\) in which \(p(t)=i\). We will say that an asynchronous stochastic game has a **fixed cycle** if players \(1,\cdots,n\) choose actions in a sequence \(1,2,\cdots,n\) followed again by the same sequence \(1,2,\cdots,n\). Theorem 2: _An asynchronous stochastic game with a fixed cycle has a stationary Markov perfect equilibrium in pure strategies._  
Proof: Consider the first cycle, in which the \(n\) players move in some sequence. Let \(v:S\to\mathbb{R}^{n}\) be some future expected payoff function of the \(n\) players from period \(n+1\) onwards. The payoff of player \(p(n)\) in period \(n\), for \(s_{\omega}\in S\) and \(a\in A_{n}\), is then given by  
\[u_{p(n)n}(s_{\omega},a)+\delta\sum_{s_{\ell}\in S}v(s_{\ell})q(s_{\ell}:s_{ \omega},a).\]  
Let \(f^{1}_{n}(s_{\omega})\) be the optimal choice of player \(p(n)\) for \(s_{\omega}\in S\) and \(f^{1}_{n}:S\to A_{n}\) denote the optimal \(n\)-period strategy of player \(p(n)\). The future expected payoff of player \(i\), from period \(n-1\) onwards for each \(s_{\omega}\in S\), is then given by  
\[u_{i}(s_{\omega},f^{1}_{n}(s_{\omega}))+\delta\sum_{s_{\ell}\in S}v(s_{\ell}) q(s_{\ell}:s_{\omega},f^{1}_{n}(s_{\omega})). \tag{3}\]  
Denote this expected future payoff by \(v^{1}_{i(n-1)}(s_{\omega})\) for each player \(i\), and for each \(s_{\omega}\in S\). Then, the payoff of player \(p(n-1)\) in period \(n-1\) is  
\[u_{p(n-1)(n-1)}(s_{\omega},a)+\delta\sum_{s_{\ell}\in S}v^{1}_{p(n-1)(n-1)}(s_ {\ell})q(s_{\ell}:s_{\omega},a).\]Denote the optimal choice of player \(p(n-1)\) for this payoff to be \(f_{n-1}^{1}(s_{\omega})\), and the optimal period \(n-1\) strategy as \(f_{n-1}^{1}:S\to A_{n-1}\). Proceeding in this manner one is able to obtain the optimal strategies for periods \(1\) through \(n\), given that the expected future payoff from period \(n\) onwards, is given by \(v:S\to\mathbb{R}^{n}\). Now, we go to period \(2n\), when the fixed \(n\)-period cycle ends after a second iteration. Let the future expected payoff from period \(2n\) be \(v:S\to\mathbb{R}^{n}\). Then, as in the case of the \(n\)-period, we now get the optimal period \(n+1\) to period \(2n\) strategies to be given by \(f_{n+1}^{1}:S\to An+1\) through \(f_{2n}^{1}:S\to A_{2n}\). As \(A_{n+1}=A_{1}\), \(A_{n+2}=A_{2}\) and so on, through \(A_{2n}=A_{n}\), we have that for each \(s\in S\), \(f_{n+1}^{1}(s)=f_{1}^{1}(s)\), \(f_{2}^{1}(s)=f_{2}^{1}(s)\), and so on through \(f_{2n}^{1}(s)=f_{n}^{1}(s)\). We now note that the expected future payoff of the players in period \(n\) is given by \(v_{1}^{1}:S\to\mathbb{R}^{n}\), which is the expected payoff of the players at the beginning of the \(n\)-period cycle, when the expected future payoff at the end of the \(n\) periods is \(v:s\to\mathbb{R}^{n}\), and the strategies from periods \(1\) through \(n\) is given by \((f_{1}^{1},\cdots,f_{n}^{1})\).