there is a stationary Markov equilibrium. From these results, we derive an algorithm to compute the Markov perfect equilibrium strategies. We use the algorithm to compute the pure strategy stationary equilibrium in three distinct experiments. The first experiment is an asynchronous dynamic game with two players who choose in alternate periods with three states and three actions per state. The second experiment is more involved and is a cybersecurity game from [1] with two players, i.e., an attacker and a defender with three actions and 18 possible states. The third experiment is a discrete version of a duopoly game in which two firms choose quantities in alternate periods as in [2]. The game has 101 possible states and each player has multiple actions for each state. In all three experiments, we first look for the convergence of the value functions. As indicated by the algorithm, the value function, which is the limit of the sequence of value functions, is the value function of a stationary equilibrium. We use the value functions of the stationary equilibrium to generate the optimal actions for each possible state. These then give us the stationary equilibrium strategies. The computation of Markov perfect equilibria in stochastic games has been studied by using various methods. The focus is on computing a stationary equilibrium that satisfies the condition that the equilibrium value functions of the players are stationary and do not change over periods. Specifically, a stationary equilibrium point satisfies the fixed-point condition that the value functions of the players, from playing the stationary equilibrium strategy in each period, are the same as the value functions used to compute the future expected payoff functions. [PERSON] and [PERSON] [3] used this property of a stationary Markov perfect equilibrium to show that the equilibrium payoff function is the solution of a nonlinear programming problem. This method of computing the stationary Markov perfect equilibrium is used in an important study in cybersecurity games [1]. Another method used to compute the stationary Markov perfect equilibrium of a stochastic game also relies on the fact that the stationary equilibrium payoff vectors satisfy the fixed-point property. However, unlike the nonlinear programming method used in [3], the stationary equilibrium value functions of the players are computed using an algorithm that directly computes the value functions that satisfy the fixed-point condition. This is done for example in [PERSON] et al. [4]. A closely related method to this is to use a Homotopy method to first compute the fixed point by perturbing the payoffs of the game by a small \(\epsilon\) and then finding the limit by letting \(\epsilon\to 0\). This method is used in [5] as well as by [PERSON] et al. [6]. In computer science, a different approach is used, which is based on reinforcement learning or machine learning. This method relies on \(Q\)-learning, in which the solution is found through an iterative process that converges to the stationary equilibrium value functions of the players. Such methods have been used very successfully in computing the stationary solutions of Markov processes. [PERSON] [7] used this approach to compute the Markov perfect equilibrium of a two-person stochastic game in which the single-period games are all zero-sum games. Subsequently, [PERSON] and [PERSON] [8] tried this approach for more general stochastic games. They showed that such a process can compute the stationary equilibrium of stochastic games in which the single-period games have a global solution. One should note that the stochastic games that we analyze here are asynchronous and not necessarily stationary. Therefore, the equilibrium value functions do not satisfy any obvious fixed-point property. Hence, one has to rule out the computational methods that are based on computing the fixed point that satisfies the conditions for the stationary equilibrium value functions, as well as the nonlinear programming method of [PERSON] and [PERSON] [3].