Infinite Horizon Markov Economies
Authors: Denizalp Goktas
Sadie Zhao
Yiling Chen
Amy Greenwald
Venue: ICLR 2026 Poster (2026)
Keywords: Algorithmic game theory
Equilibrium computation
Markov pseudo-games
Markov exchange economies
Primary area: other topics in machine learning (i.e., none of the above)

Abstract: In this paper, we study a generalization of Markov games and pseudo-games that we call Markov pseudo-games, which like the former, captures time and uncertainty, and like the latter, allows for the players’ actions to determine the set of actions available to the other players. In the same vein as Arrow and Debreu, we intend for this model to be rich enough to encapsulate a broad mathematical framework for modeling economies. We then prove the existence of a game-theoretic equilibrium in our model, which in turn implies the existence of a general equilibrium in the corresponding economies. Finally, going beyond Arrow and Debreu, we introduce a solution method for Markov pseudo-games, and prove its polynomial-time convergence.

We then provide an application of Markov pseudo-games to infinite-horizon Markov exchange economies, a stochastic economic model that extends Radner’s stochastic exchange economy and Magill and Quinzii’s infinite horizon incomplete markets model. We show that under suitable assumptions, the solutions of any infinite horizon Markov exchange economy (i.e., recursive Radner equilibria—RRE) can be formulated as the solution to a concave Markov pseudo-game, thus establishing the existence of RRE, and providing first-order methods for approximating RRE. Finally, we demonstrate the effectiveness of our approach in practice by building the corresponding generative adversarial policy neural network, and using it to compute RRE in a variety of infinite-horizon Markov exchange economies.

Published as a conference paper at ICLR 2026
INFINITE HORIZON MARKOV ECONOMIES
Denizalp Goktas'
Sadie Zhao' & Yiling Chen
Cornell Tech, Computer Science
Harvard University, Computer Science
Amy Greenwald
Brown University, Computer Science
ABSTRACT
We study Markov pseudo-games a framework that unifies Markov games and pseudo-gaman games by incorporating both dynamic uncertainty and action eetion-dependemt feasibility. We establish the existence of equilibria and present first-order solu- tion method with polynomial-time guarantees. As an application, we show that recursive Radner equilibria in infinite horizon Markov exchange economies can be formulated as equilibria of concave Markov aseudo-games thereby establishing their existence and enabling algorithmic approximation. Finally, we demonstrate the practical effectiveness of our method by implementing a generative adversarial policy network and computing equilibria in several infinite horizon economies.
INTRODUCTION
A central goal of economics is to understand how rational agents interact in dynamic, uncertain environments and how such interactions give rise to equilibria. From Walras' model of markets systems of supply and demand, to Arrow & Debreu's competitive economies as apeudo-games, general equilibrian theory has provided rigorous mathematical framework for modeling economies. Yet, classical formulations are inherently static: they capture only a single period of trade and, even when commodities are made contingent on future states, they rely on the unrealistic assumption of a complete market, meaning full set of cotteenontnnent assets. As result, they fail to capture the ongoing uncertainty and dynamic decision-making makinn that characterize real-world economies, especially those involving sequential trade of financial assets, intertemporal borrowing and lending, and evolving shocks to productivity or preferences.
Radner (1 (1972) introduced stochastic exchange economies, finite-horizon models in which agents trade commodities and assets under uncertainty, leading to the canonical notion of Radner equilibrium. These models provide a dynamic extension of Frrow-Debreu economies by linking a sequence of spot markets through asset trade, but they retain a finite time horizon. Beyond the finite, infinite horizon stochastic economies are particularly attractive for macrocconomic applications. as they are better suited to modeling long-run phenomena such assel bubbles (Huang & Werner, 2000), economic growth, and persistent aggregate shocks. However, infinite horizons introduce substantial thcoretical difficulties. For example, they permit the possibility Ponzi schemes asset markets, in which agents indefinitely roll over debt, thereby complicating equilibrium existence in incomplete markets. Magill & Quinzi (1994) extended Radner's framework to the infinite-horizot setting, albeit with financial assets rather than arbitrary assets, and provided conditions under which sequential competitive equilibria exist. Despite these advances, computational progress has been limited, with most solution methods still largely confined to finite-horizon environments (Sargent & L.uugqvist, 2000; Taylor& & Woodford, 1999; Fernández- Fernández-Villaverdde Villaverde. 2023). These difficulties highlight the need for new computational methods and theoretical frameworks in which to analyze infinite horizon models.
We address this challenge in context Markov greudo-games, model we develop that combines the dynamic uncertainty of Markov games with the action-dependent feasibility of pseudo-gand gammes. This framework not only extends Arrow & Debreu's pseudo-gaman to dynamic setting, but it is also
"quual contribution.

---

Published as a conference paper at ICLR 2026
sufficiently expressive to model infinite horizon Markov exchange economies with potentially incom- plete markets. In this way, our framework unifies the game-theoretic perspective and the economic perspective: it serves as Markov game model with computable equilibria, while simultaneously capturing the structure of general infinite horizon economies.
Contributions In Section 2, we introduce Markov pseudo-games (MPGs), and we establish the existence of (pure) generalized Markov perfect equilibria (GMPE) in concave Markov pseudo -ames (Theorem )) extending Arrow-Debreu's 's classical existence result for pseudo-gam settings. This result implies the existence of pure (or deterministic) Markov perfect equilibria in a large class of continuous- actioo Markov games for which existence, to the best of our knowledge, was heretofore known only in mixed (or randomized) policies (Fink, 1964; Takahashi, 1964). Although the computation of GMPE is PPAD-hard in general. by taking advantage of the recent progress on solving generative adversarial learning problems (e.g.. (Lin et al., 2020; Daskalakis et al., 2020)), we show that an approximate stationary point of the exploitability (i.c.. the players' cumulative maximum regret) be computed in polynomial time under mild assumptions (Theorem 2.2). This result implies that a policy profile that satisties necessary first-order stationarity conditions for a GMPE in Markov pscudo-games with a bounded best-response mismatch coefficient (Lemma 5) can be computed via policy gradient in polynomial time, result which is analogous to known computational results for zero-sum Markov games (Daskalakis et al., 2020).
to dynamic
In Section 3, we introduce extension of Magill & Quinzii (1994)'s infinite horizon exchange economy, which we call an infinite horizon Markov exchange economy. The Markov restriction allows us to establish the existence of recursive Radner equilibrium (RRE) (Theorem 3.1). Our proof reformulates the set of RRE of any infinite horizon Markov exchange economy as the set of GMPE of an associated Markov pseudo-game (Theorem 3.1). To our knowledge, ours is the first result of its kind for such general setting, as previous recursive competitive equilibrium existence proofs were restricted to economies with one consumer (also called the representative agent), one commodity, or one asset (Mehra & Prescott, 1977; Prescott & Mehra, 1980). The aforementioned computability results allow to conclude that an approximate stationary point of the exploitability of the Markov pseudo-gam game associated with any infinite horizon Markov exchange economy can be computed in polynomial time (Theorem 3.2).
Finally, in Section 4 we implement our policy gradient method in the form a generative adversarial policy network (GAPNet), and use it to search for RRE in three infinite horizon Markov exchange economies with three different types of consumer utility functions. Experimentally, we observe that GAPNet finds approximate equilibrium policies that are closer to GMPE than those produced by standard baseline for solving stochastic economies.
MARKOV PSEUDO-GAMES
We begin by developing our formal game model.2 The games we study are stochastic, in the sense of Shapley (1953), Fink (1964), and Takahashi (1964). Further, they are seeudo-games, in the sense of Arrow & Debreu (1954a), as the players' fcasible action sets are determined by other players' choices. We model stochastic souudo-game,, and dub them Markov pseudo -ames (MPGs),² the games are Markov that the stochastic transitions depend only on the most recent state and player actions.
Model An finfinite horizon discounted) Markov pseudo-gamy -amee M = (n, m. A,A, X, r, P. 7/ ()) is an n-player dynamic game played over an infinite discrete time horizon. The game starts at time t= in some initial state 5(() ~ μ E (SS) drawn randomly from a set of states S C Be. At this and each subsequent time period 2.2.... the players encounter a state g(t) E S, in which each player E [n]] simultaneously takes an action a X((s!)) d! from a set feasible actions X,((()) (9) S. A; S R.", determined by a feasible action correspondence X, : S x A- = As, which takes input the current state g(t) and the other players' actions a E A- and outputs a subset of player i's action space A, We define X(s,a) a) = Xxein) (((s,
'On the one hand, our model generalizes Magill & Quinzii's to setting with arhitrary not just financial assets; the other hand, restrict the transition model to be Markov. Explanations of notation and definitions of mathematical termnnology can be found in Appendix A. "following Littman (1994).
2

---

Published as a conference paper at ICLR 2026
Once the players have taken their actions (*)) = (a] a. ). each player E [rx] receives a reward r,(s(), a ( given by a reward function r: A R". after which the game either ends with probability| - ", where ? E (0, 1) is called the discount
or continues on to time period transition probability
+ transitioning function p S x S x A + R.,, where p(s(++1) | a()), al))) E [0, denotes the probability of transitioning to state 8 (++1) E S from state 8 E when action profile (* E A is played.
new state g((+1) ~ p(- g(0) a, according
Our focus is on continuous-state and continuous-action MPGs, where the state and action spaces are non-empty and compact, and the reward functions are continuous and bounded in each of 8 and a.
A history h- ((s(t), a (T s()) N = (S x A) x S of length E N U } is sequence of states and action profiles. Overloading notation, we define the history space 71 U.-o H" For any player i E [rn]. a policy π; : H - A; is a mapping from histories of any length to i's space of (pure) actions. We define the space of all (deterministic) policies as P; = {TT; H + Ar). A Markov policy (Maskin & Tirole, 2001) π, is a policy s.t. (T()(()) = TI (h) for all historics h E H" of length E N4, where s(F) denotes the final state of history h. As Markov policies
only state-contingent, we can compactly represent the space of all Markov policies for player E [xi] as P.maakov
[T. 55 + A.}.
Fixing player i E [n] and -= E P-jj we define the feasible policy correspondence FT( = { E: | 4h E H. (h) E (,( policy correspondence (Tu , (h))) given history E 'H',, and the feasible subelass { Prub 1 Vs E S, (, E X2ss, aT [T for any peub C <pmarkov Of particular interest is F.markor (T itself, obtained when Puub -mmakkor We also define the feasible policy profile correspondence F(TT) = X.e(n) F,(TT and the feasible subclass policy profile correspondence Jestl (TT = Xiejm) F." (T_.). for any paub ç ppmarkov.
Given a policy profile π E P and history h H", we denote the discounted history distributions Vs" and ui that originates at state s and given initial state distribution μ. respectively. Next, we define the set of realizable trajectories of length 7 under policy π as H. = supp(v,ī TT. Moreover, given a policy profile π E P. we denote the state-value function v S R" as nn (s) - R- [E>= r(St0, A19]] and the action-value function q"' S x A Rd as 9x (s. = r(s,a) a) + 1 [u (S)]. Finally, we define the discounted state-visitatiin distribution 07, (s) Ei=o Shent sit)
(att) Mp (f)) and the (expected) payoff' of policy profile π as u(T) = Es- (s)( ES (SS,
Solution Concepts and Existence Having defined our game model, we now define two natural solution concepts, and establish their existence. The first applics the usual notion of Nash cquilibrium (1950b) to MPGs. The second is based on the notion of boogame-perfeee equilibrium in extensive- form games, strengthening Nash equilibrium with the additional requirement that an equilibrium be Nash not just at the start the game, but at all states encountered during play.
An E-generalized Markov perfect equilibrium (E-GMPE) n E F.markoo (TT" is a Markov policy profile s.t. for all states E S and players i E [m], V. (8) 2 max (a,
( (s) An e-generalized Nash equilibrium (s-GNE) π E F(m") is policy profile s.I. for all states 8 E S and players E [n)) (,(mr) 2 max,, (xx)) (T,, mr c. We call 0-GMPE (0-GNE) simply GMPE (GNE). As GMPE is a stronger notion than GNE, every e- e-GMPE is an s-GNEE
To establish existence of GMPE, we introduce two assumptions: the first is the standard convexity and continuity assumption (see Assumption in Appendix D. the second, introduced as Condition I in Bhandari & Russo (2019), 9), ensures that the policy class under consideration (e.g.. ppsu C pinarkor) is expressive enough to include best responses (see Assumption 3 in Appendix D.1).
Theorem 2.1. If M is an MPG for which Assumption holds, and peub C ppmarkow is a subspace of Markov policy profiles that satisfies Assumption 3, then there exists a GMPF, T²" E ppau of M.
Equilibrium Computation Our approach to computing a GMPE in MPG M is to minimize a merit function associated with M, i.c., a function whose minima coincide with the pseudo-game 's GMPE. Our choice of merit function, common one in game theory, is exploitability i.c., the sum of the players' maximal unilateral payoff deviations, denoted (p(T) for policy π E P. Exploitability,
3

---

Published as a conference paper at ICLR 2026
however, is merit function for GNE, not GMPE; state exploitability at states 8 E is a merit function for GMPE, Nevertheless, as we show in the sequel, for a large class of MPGs, namely those with a bounded best-response mismatch coefficient, the set of Markov policies that minimize exploitability alone equals the set of GMPE, making our approach sensible one.
We are not of the woods yet, however, as exploitability is non-convex in general, even in one-shot finite games (Nash, 1950a), GNE computation is PPAD-hard, as NE computation is PPAD-hard (Chen et al., 2009; Daskalakis et al., 2009) and MPGs generalize one-shot games. Accordingly, we instead set our sights on computing a stationary point of the exploitability, i.e., policy profile x'' jumarkov (+²) s.t. for any other policy E j.märkoy
(T²"). )."" it holds that min (h, X'' T I 0, where (T((T) denotes the subdifferential of at π. Under suitable assumptions, such point satisfies
the necessary conditions of GMPE.
In this paper, we study MPGs with possibly continuous state and action spaces. As such, we can only hope to compute an approximate stationary point of exploitability in finite time. Defining notion of approximate stationarity for exploitability is, however, a challenge.
Given an approximation parameter 2 O, a natural definition of an E-stationary point might be a policy profile π¹ E Finaroo (πT) s.t. for any other policy π E FFmarkor (xT", it holds that minneDD(rr)) (h, n'' - TT S E. Exploitability is not necessarily Lipschitz-smooth, however, so in general it may not be possible to compute an E-stationary point in poly(/e) evaluations of the (sub)gradient of the exploitability.
To address this challenge, a common approach in the optimization literature (see, for instance Definition 19 of Liu et al. (2021)) is to consider alternative definition known as (s, -)-staiionar Given E, 5 2 0, an (s, -sttttintyy point of exploitability is policy profile
E F.mackov (I'' T''l I ó s.t. for any other policy E J.markor (x*), it holds that minngDy(st) (h, - -) S. The exploitability minimization method we introduce can indeed compute such an approximate stationary point in polynomial time: i.e. point in the neighborhood of an approximate stationary point of exploitability. Furthermore, asymptotically, our method is guaranteed to converge to an exact stationary point of exploitability.
for which there exists a 8-close policy x! E P with |r| -r'l
Exploitability Minimization Given an MPG .M and two policy profiles π n E P. we define the (=).==)) (8) v." (s) ; the expected
state cumulative regret at state a E S as (188, n,, X') ) = -iejj]] cumulative regrer for an arbitrary state distribution E A (S) as e0v x,x')= Es (V(S, x, T')]; and the cumulative regret as (TT, x²) = v11,, म, n') Additionally, for a policy profile n,, we define S as o(s, T))
(m)) (s) Enejal maxx/eFFaakeny (x_,) 1/:
the state exploitability at state
v. (s): the expected state exploitability for an arbitrary state distribution w E (SS as Φlv, TT) Es [ø(S, π }]; and the (global) exploitability as p((T) )- Eie(n) max m (r () (T( ग. _...
With these definitions in hand, we can reformulate the problem of computing a GMPE as the quasi- minimization problem of minimizing state exploitability, i.c., min Frmuckerr Φ π ). at all states s E S simultaneously. We can also compute a GNE and exploitability, analogously.
Lemma 1. Given an MPG M, a Markov policy profile nt' E J.marboo (x²) ) is a GMPE iff ((s 0, for all states E S. Similarly, policy profile "t" E F(TT") is GNE iff p(m²') 0.
This straightforward reformulation of GMPE (resp. GNE) in terms of state exploitability (resp. exploitability) does not immediately lend itself to computation, as exploitability minimization is non-trivial, because exploitability is neither convex nor differentiable in general. Following (Goktas & Greenwald, 2022), we can reformulate these problems yet again, this time as coupled quasi-min-max optimization problems (Wald, 1945), We proceed to do so now; however, we restrict our attention to exploitability, and hence GNE, knowing that we will later show that minimizing exploitability suffices to minimize state exploitability, and thereby find GMPE.
Observation 1. Given MPG M, (i) ((T)) ((x max, g (.Fastro V ((T, n
While the above observation makes progress towards our goal of reformulating exploitability mini- mization in a tractable manner, the problem remains challenging to solve for two reasons: first, a fixed point computation is required to solve the outer player's 's minimization problem; second, the
4

---

Published as a conference paper at ICLR 2026
inner player's policy space depends on the choice outer policy. We overcome these difficulties by choosing suitable policy parameterizations.
Policy Parameterization In coupled min-max optimization problem, any solution to the inner player's maximization problem is implicitly parameterized by the outer player's ' decision. We restructure the jointly feasible Markov policy class to represent this dependence explicitly.
Define the class of dependent policies R - X(s, a)} (ie (p. : 8 x A, + A- (Vs, E p,().a_)) E X(s, With this definition in hand we arrive at an uncoupled quasi-min-man optimization problem:
x - A V(s.a) E S A, p(s,a) E
Lemma 2. Given an MPG M,
EF()) max e (m) ((
= min maxpER (T.,p(. nT (-))-
It can be expensive to represent the aforementioned dependence in policies explicitly. This situation can be naturally rectified, however, by suitable policy parameterization. A suit- able policy parameterization can also allow us represent the of fixed points P s.t.
w E Jrmaitoo (Ir) more efficiently in practice (Goktas el al., 2023a). Define a parameterization scheme (T, p. R RR)) as comprising an unconstrained parameter space I2 and parametric policy profile function æ S x R + A for the outer player, and an unconstrained parameter space IR and parametric policy profile function p S x A x B + AA for the inner player. Given such scheme, we restrict the players' policies to be parameterized: i.e. the outer player's 's space of policies px क 8 R A | E RR)) S pmarkov while the inner player's space of policies R== {p= : A R A o E R=).
We further impose following assumptions on the parameterization scheme so that specifies the two structural properties: (I) enforces the fixed-point feasibility condition for the outer player's policy, and (2) it explicitly encodes the dependence of the inner player's ' policy on the outer player's through the dependent-policy architecture (i.e., takes π (a) as input).
Assumption 1 (Parameterization for Min-Max Optimization). Given an MPG M together with a parameterization scheme (T, p. R R²), assume I. fixing w E RR, (8: E Xss o (s:w)), all 8 E S; and 2. fixing a E R p(s, a) E X(s, a). for all (s,a) S X.A.A. A.
Under such a parameterization, we can express state exploitability minimization as the following min-max optimization problem: min maX (w. (T((:))) p( ([:)::
In summary, by choosing a suitable parameterization scheme, we resolve the two challenges high- lighted in Observation I. First, the unconstrained parameter space facilitates an efficient representation of the outer player's 's policy space, i.e., the set of fixed points { E pmmazkov π E Finakkor (Tr }} Second, the parameterization provides an explicit representation of the class of dependent policies, thereby eliminating the dependence of the inner player's s policy space the outer player's policy.
Now, given an unconstrained parameter space, we are able to simplify our definition of (e, 8)- stationary point of exploitability, namely, a policy parameter w Rd for which there exists a 8-close policy parameter wi e 2 with |w' - w'' < 8 s.t. minpeDe(wwt) ||/] S
State Exploitability Minimization Returning to our objective, namely state exploitability mini- mization, we now turn our attention to obtaining a tractable characterization of this goal. Specifically, we prove two lemmas, which together establish that it suffices to minimize exploitability. rather than state exploitability, as any policy profile that is a stationary point of exploitability is also stationary point of state exploitability across all states simultaneously, under suitable assumptions.
Our first lemma (Lemma 4, Appendix D.I) states that a stationary point of exploitability is almost surely also stationary point of state exploitability at states, Our second lemma (Lemma 5, Appendix D.1) upper bounds gradient of state exploitability in terms of gradient of exploitability,
UUsing these parameterizations, we redefine v" ((w)), 9t q (iww,, u(w) uu ())), and U/p # ((w), and ( 2 (lom)( ((( 2 (ow))))) u((((w)) - ulp(( ()) (ww) [- ; and so on.
'In Lemma (Appendix D.I), showed that such parameterization is guaranteed exist for large class of Markov pseudo-gumes, ie.. any Markuv pseudo games with DAG dependency structure, and this class includes any Exchange Economy Markov Pseudo-Game.
5

---

Published as a conference paper at ICLR 2026
when the best-response mismatch coefficient is bounded. Given M with initial state distribution j and alternative state distribution v E A (S), and letting (T(;,) = arg max p " ",(TT, __;) denote the set of best response policies for player i when the other players play policy profile --- we define the best-response mismatch coefficient for policy profile π as (T, =
maXie(n] maxn/e0.(r ,) (1/1>)) 1 (// " //1looll
Algorithm and Convergence Next, we present our algorithm for computing an approximate stationary point of exploitability, and thus state exploitability. The algorithm we use is two time-scale stochastic simultancous gradient descent-ascent (TTSSGDA) ) Appendix D.I, Algorithm 1), first analyzed by (Lin et al., 2020; Daskalakis et al., 2020), which, for polynomial typically requires that the objective be Lipschitz smooth in both decision variables and gradient dominated in the inner one. We thus impose regularity assumptions to ensure these conditions.
convergence,
In particular, (ww,, or) being Lipschitz smooth in (w, .r) is ensured by the assumption that policy parameterization, reward function, and transition function are all twice continuously differentiable (Assumption 4); while (ww. being gradient dominated in a ensured by assumption that parametrized policy classes are closed under policy improvement, and cach player's 's action value function is concave in a (Assumption 5).
As the gradient of cumulative regret involves an expectation over histories, we assume that we simulate trajectories of play h ~ ~2 according to the history distribution v/ w, and that doing so provides both value and gradient information for the rewards and transition probabilities along simulated trajectories. That is, we rely on differentiable game simulator (Suh et al., 2022). meaning stochastic first-order oracle that returns the gradients of the rewards and transition probabilities, which we query to estimate deviation payoffs and cumulative regrets.
" for any policy profile
Under this assumption, estimate these values using realized trajectories from the history dis- tribution h" Vi induced by the outer player's policy, and the deviation history distribution (a.(w_.))--))
h Xigln M/
induced by the inner player's 's policy."
Our main thcorem requires one final definition: the equilibrium distribution mismateh coefficient |006 /aullxx. defined as the Radon- Madoo-Nikodym derivative of the statevviiitation distribution of the GNE x² w.r.t. the initial state distribution μ. This coefficient, which measures the inherent difficulty of visiting states under the equilibrium policy x¹ without knowing πt is closely related to other distribution mismatch coefficients used analyze policy gradient methods (Agarwal et al., 2020).
We now state our main theorem, namely that, under the assumptions outlined above", ", Algorithm I computes values for the policy parameters that nearly satisfy the necessary conditions for a GMPE in polynomially many gradient steps, or equivalently, calls to the differentiable simulator.
Theorem 2.2. Given MPG M and parameterization scheme (π P, R R²) satisfying As- sumption assume Assumptions 4, and 5 hold. For any 5 > 0, E 81] )lla) If Algorithm is run with inputs that satisfy Tw Ao x poly(e, 11067 fawll T Tlo Eee)), then 11965" foulloe, b Lv. diam((P) x R2)) (2n). and k S beatt = ((()) is an (c, (2,5/22) (1/2co))stsee )-stationary point of exploitability, i.e., there exists w" E R s.t. s.t. w (T)
there exists T E poly
w best w0 S s/2t, and minneDe("")) [|h]| S s. And, for any distribution V E A(S), if (Vv..))
"More specifically, for all policies E -pmariov and histories h 'HT,, the payoff estimator for player i [x] is given by ti (T;:)) Ei (( x x'()(0)) II > (( ( Morcover, for all w E RR, E RT, h" ~ vi, and h² = -h, .
........... hi)) .h.)).)))))))) XigIn M/ (o)
the cumulative regret estimator is given by o(w, (rhhh"hh) = EiE C.pp( , n (=()::)))) T x-
w): w):h") ((([:));))) while the cumulative regret grudient estimator is given by (V_, Vo or; h ,k"),, V. & (w, ht" ())).
"The Lipsehitz smoothness and bounded mismatch assumptions imposed to obtain polynomial-time convergence guarantces and are standard in modern analyses of policy-gradient methods. They should be viewed leaming-theorrtic stability conditions rather than cconomic restrictions; empirically, method performs well even in settings with mooth primitives (e.g.. Levntief utilities), where neural parameterizations provide effective smooth approximations and practical exploration reduces state-distribution mismarch.
6

---

Published as a conference paper at ICLR 2026
is differentiable at w", then |urll,,0) I ó. ie. w is an (stttstttioonary point of expected state exploitability (v.,)).
In other words, by running Algorithm 1 on M,, compute a policy profile w (T) in the neighborhood of w",, an approximate stationary point of exploitability. By Lemma 5, w" also an approximate stationary point of state exploitability at all states in M simultaneously, and therefore approximately satisfies the necessary conditions of a GMPE. Therefore, Algorithm I converges to point w best in the neighborhood of a point w' that approximately satisfies the necessary conditions of an GMPE, While arguably relatively weak theoretical conclusion, our experiments demonstrate that in practice our method succeeds at approximating GMPEs exchange economy MPGs. Moreover, in the limil, Algorithm 1 converges to a point that exactly satisfies the necessary conditions of an GMPE.
(2)
3 INCOMPLETE MARKOV ECONOMIES
Having developed a mathematical formalism for MPGs, along with a proof of existence of GMPE as well as an algorithm that computes them, we now move on to our main agenda, namely modeling incomplete stochastic economies in this formalism. We establish the first proof, our knowledge, of the existence of recursive competitive equilibria in standard incomplete stochastic economies, and we provide a polynomial-tine algorithm for approximating them.
Infinite Horizon Markov Exchange Economies Classic Arrow Dew-ee economies provide the static foundation for our framework; full details appear in Appendix C.2. Building on these static models, we introduce our infinite- model.
An infinite horizon Markov exchange economy (MEE) I = (n, m. comprises n E N consumers who, over an infinite discrete time horizon t= = 0, 1,2, repeatedly encounter the opportunity to buy a consumption of m E N commodities from space X; C R of consumptions and portfolio of I E N assets from space V C R of asset portfolios (or investments), where Va. 2 0 denotes units of asset & bought (long) by consumer i, while y/x < denotes units that are sold (short). Assets are assumed to be short-lived (Magill & Quinzii, 1994). meaning that any asset purchased at time t pays its dividends in the subsequent time period + and then expires. The consumers' collective decisions lead them through the state space S = O x (Ex T), which comprises world state space O and a spot market space & x T, the product of the consumers' endowment and type spaces. The spot market space is a collection of spot markets, each one a static exchange market EE, θ) E x S RM x IR..
R. p)),
Each asset l E [1] is generalized Arrow security, i.e., a divisible contract that transfers to its owner a quantity of the jth commodity at any world state E O determined by matrix of asset returns R, = (Fol.....,Pol .ro) E I2'xm s.t. Pooj E R denotes the quantity of commodity j transferred world state for one unit of asset k. The collection of asset returns across all world states is given by R = {R. Joeg. Assets allow consumers to insure themselves against future realizations of the spot market (i.e., endowments and types). by allowing them to transfer wealth across world states.
The economy starts time period 0 in an initial state 510) j determined by an initial state distribution E (S)). At each time step 8 = 0, 0,,, the state of the economy is s(9) = (ol(), E00, ( E S. Each consumer E [n]], observes the world state (f1) E O. and participates in a spot market (E(), (()), where it purchases a consumption x. E X, at commodity prices p E Am, and an asset market y = Xxeijn 3% by which it invests in an asset portfolio y!
(1) E ), at assets prices q0 E IR'. Every consumer is constrained to buy consumption x² E X, and invest in an asset portfolio y!" (e) E 3/ with total cost weakly less than the value of its current endowment c E &, Formally, the set of consumptions and investment portfolios that consumer can afford with its current endowment c E, at current commodity prices ( e Am and current asset prices (()) E RR,, i.c., its budget set B, (e (!)
,p()), g is determined by its budget correspondence
23, (e, P.P. q) = ((,, v.) E & x D: xx p y. S e. p))
After the consumers make their consumption and investment decisions, they each receive reward 10:
ra(=::: )(h) as a function of their consumption and type, and then the economy either collapses with probability 1 7, survives to see another day with probability 7, where 7 E (0. 1) is called
7

---

Published as a conference paper at ICLR 2026
the discount rate. If the economy survives, then a new state is realized, namely (0', (O','", 0') ~ p(-)
(t), ( according a transition probability function p: 8 S x [0, that depends on the consumers' investment portfolio profile y = (vl") ..... " E 3 after which the economy transitions to new state 5(t-1) = (0', E' (+) Ro Ro, 0"), where the consumers' new endowments depends on their returns Y Ro E _ on their investments.
A history h (( X y( p00, q y a()) E "²' is a sequence of tuples comprising states, consumption profiles, investment profiles. commodity price, and asset prices. Overloading notation, we define the history space H = - H and then consumption, investment, commodity price and asser price policies as mappings x;
+ X. V, : H + Ve p + A... and : H1 + R' from histories to consumptions, investments, commodity prices, and asset prices, respectively, s.t. (æ (;)) is the consumption-inee investment decision of consumer
[ni]. and (p. ())) are commodity and asset prices. both at history h H. A consumption policy profile (resp. investment policy profile) X (h) = [
cessumption-ineesteentt
(h))(h"" (resp. (h) = (UI..... v,))h) (1) is a collection of consumption (resp. investment) policies for all consumers, A consumption policy x; S + X. is Markov if it depends only on the last state the history, i.e., x;(h) x, 2,(s("), for all histories h E "H" of all lengths T E N. An analogous definition extends to investment, commodity price, and asset price policies.
Given = (X, Y, PP q) and history h E HH", we denote the discounted history distribution assuming initial state distribution μ by vy "". Overloading notation. we define the set of all realizable trajectories 7T of length under policy profile π as HT - supp(y""")
Solution Concepts and Existence An outcome (X, EXYYPpP p. H + x Am x R' of an infinite horizon MEE is a tuple consisting of a commodity prices policy, an asset prices policy, consumption policy profile, and an investment policy profile. An outcome is Markov if all its constituent policies are Markov.
While Radner equilibrium is the canonical solution concept for finite-horizon stochastic economics, its history dependence becomes infinieediimensionall in infinite-horizz settings, rendering equilibria analytically and computationally intractable. Following standard practice in macroeconomics (e.g.. Stokey et al. (1989); Prescott & Mehra (1980)), we restrict our attention to Markov outcomes.
Given Markov consumption and investment profile (X,Y), the consumption state- value function u! (X.Y.PP .P.))
S R for consumer i is defined as o.X.Y.PP.)) E .... p.oo D>eo r'r, (s)( A Markov outcome (X',Y",p) ""',"",'') is perfect for i if maximizes its consumption state-value function over all affordable consumption and investment { (",""".".""" p²'*", (s)) E..
=
policies, i.e. (a), vi) E arg max 1. "").
A Markov consumption policy X is said to be feasible iff' for all time horizons T E N and state (7)) Eiele (x(7))) -ie((n) e!") s Om. where !!7) E & is consumer i's endowment at state (rr), Similarly, investment policy is feasible if Eie(n v, (x(*)) S O,, If all the consumption and investment policies associated with an outcome are feasible, we say the outcome is feasible as well.
A Markov outcome (x, Y, T.P. q) is said to satisfy Walras' law iff for all time horizons r E N and state (tr) E S.)((s ( (E(et))(((""") ()") Exetnj e(*)) (((((t))
( (Erepn) 2,(6(1)) 0, where,
as above, e!t) E E, is consumer i's endowment at state g(r),
refinement of Radner equilibrium the infinite horizon setting is recursive Radner equilibrium.
Definition 1 (Recursive Radner Equilibrium). recursive Radner (or Walrasian or competitive) equilibrium (RRE) (Mehra & Prescont, 1977; Prescott & Mehra, 1980) of an infinite horizon MEE I is a Markov outcome (X" Y ,pp", (4")) that is 1. Markov perfect, for all consumers i E [n]; 2. feasible; and 3. satisfies Walras' law.
We establish the existence of RRE in infinite horizon MEES under standard economic assumptions: convexity, boundedness, no-satiation (see, for example, Geanakoplos (1990). To do so, we first associate an exchange economy MPG M with given infinite horizon MEE I.
8

---

Published as a conference paper at ICLR 2026
Definition 2 (Exchange Economy Markov Pseudo-Game). Let I be an infinite horizon MEE.
The corresponding exchange economy MPG M = (m + 1, + S, (eei x Va x (P x
2). B'''''' ...rr.l .... x',p') is defined as
-The n + players comprise n consumers, players ,,...,7,, 1I, and one auctioneer, player n + 1.
TThe set of states S O E x T. At each state (o, E, O) E.
-each consumer i [n) chooses an action a; = (T. E B (s C X Y
from a set of feasible actions B,(8, a_:) = B,(e,, am+1) n (1),,,,) Eiejul 2 S Σie(n) c: Eie(v V. S Om (..(X.Y. Y) E X' X) and receives reward rewardri(s,0) (2,10;;0))
the auctioneer + chooses an action ax+1 (p.q) E Blat1 (s.a (.ta = x Q where
p = Am and Q [0, MAXKEE EnE(N) Eye(m) Ctyl', and receives reward ry r as) = p (Eiejn) T. Eiejm) e.) + +9- (Eie(w) U.)
The transition probability function is defined as gs'ss s, s, = p(s' ss The discount rate y and the initial state distribution p'
Our existence proof reformulates the set of RRE of any infinite horizon MEB as the set of GMPB of the exchange economy MPG.
Theorem 3.1. Consider an infinite horizon MEE I. Under Assumption 6, the set of RRE of I is equal to the set of GMPE of the associated exchange economy MPG M.
We can define the exploitability (resp. expected state exploitability) of any infinite horizon MEE I satisfying Assumption 6 as the exploitability (resp. expected state exploitability) its associated exchange economy MPG M. By Theorem 3.1, an outcome that minimizes the exploitability of the economy I is a Radner equilibrium (RE), while outcome that minimizes state exploitability at all states simultaneously is RRE, The following corollary now follows from Theorem 2.1.
Corollary 1. Under Assumption 6, the set of RRE of an infinite horizon MEE is non-empry.
Equilibrium Computation Since an iRE infmite-dimensiionall when the state space is contimuous, its computation is FNP-hard (Murty & Kabadi, 1987). As such, it is generally believed that the best we can hope to find in polynomial time is solution that approximately satisfies the necessary conditions of an RRE, Since the set of RRE of any infinite horizon MEE I is equal to the of GMPE of the associated exchange economy MPG M (Theorem 3.1), we can apply Theorem 2.2 to compute a policy profile with the following computational complexity guarantees for Algorithm 1, when run the exchange economy MPG associated with an infinite horizon MEE.
Theorem 3.2. Given an infinite horizon MEE I for which Assumption 6 holds, and the associated exchange economy MPG M. If (T, R. R,, R²) is a parametrization scheme for M such that Assumptions 4 and 5 hold, then the convergence results in Theorem 2.2 hold, meaning Algorithm I converges point in the neighborhood of a point that approximately satisfies the necessary conditions of a GMPE M, which is likewise a point that approximately satisfies the necessary conditions of an RRE of TI. Moreover, beyond its finite-ime guarantees, in the limit, Algorithm converges to point that satisfies these conditions exactly.
4 EXPERIMENTS
Given an infinite horizon MEE I, we associate with it an exchange economy MPG M, and we then construct a deep learning network to solve M. To do so, we assume a parametrization scheme (T.P, IT., ). where the parametric policy profiles (T, p) are represented by neural networks with (RS2 "R the corresponding network parameter spaces. Computing an RRE via Algorithm I can then seen as the result of training& a generative adversarial neural network (Goodfellow et al., 2014; Goktas et al., 2023a), where и (resp. p) is the output of the generator (resp. adversarial) network. We call such neural representation generative adversarial policy network (GAPNet).
Convergence in Base Economies Following our proposed approach, we built a GAPNet to ap- proximate an RRE in two types of infinite-horizon MEEs: the first with deterministic transitions, and the second with stochastic. In each, we experimented with three randomly sampled economies
9

---

Published as a conference paper at ICLR 2026
with 10 consumers, 10 commodities, asset, 5 world states. three distinct classes of reward functions, linear, Cohb-Douglas, and Leontief, each of which imparts different smoothness properties on their state-value functions."
The consumers were characterized by
We compare our GAPNet results with classic neural projection method (NPM), also known as deep equilibrium nets (Azinovic et al., 2022), which macroeconomists and others use to solve stochastic economies. This method seeks pol- icy profile that minimizes the norm of the sys- tem first-order necessary and sufficient con- ditions that characterize RRE (e.g.. Fernández- Villaverde (2023)). We use the same network architecture for both methods, and select hyper- parameters for each through grid search. In all experiments, we evaluate the performance of the ensuing policy profiles using three metrics: total first-order violations. average Bellman errors," and exploitability.
Urea Economy
Erith Fronorry
Loontie Bcoromy
Experiments in economies with deterministic transitions are presented in Section E. While NPM performs well under metrics it is designed to minimize, GAPNet performs well under all three metrics, highlighting its ability to approxi- mate the necessary conditions of an RRE. Moreover, stochasticity further hinders NPM's performance (see Figure 1), while GAPNet successfully minimizes all three metrics across all economies.
Figure 1:
Economies with Stochastic Transitions.
Economic Interpretation of Learned Equilibria Beyond measuring convergence, it is also necessary to verify that the learned policies exhibit economically meaningful behavior. We thus analyze equilibrium consumption and investment patterns across different preference specifications and discount factors. The results illustrate that GAPNet recovers classical economic predictions across all settings. We include the results of these experiments in Appendix E.
Utility specifications Under lincar prefcrences, agents exhibit almost no asset demand and allocate roughly 97-98% of wealth to current consumption. Under Cobb Cobb-Douglas preferences, consumption shares fall 38-90% and agents take positive asset positions, as Cobb-Douglas utilities are strictly quasi-concave and exhibit diminishing marginal utility in our experiments, motivating positive intertemporal smoothing. Under Lcontief preferences, spending rises to nearly 99% of wealth and asset demand approaches zero. This is because Leontief utility is limited by the scarcest good, any reduction in present consumption extremely costly. Agents therefore avoid intertemporal substitution and concentrate spending on bottleneck goods, leading to low investment. These patterns are consistent with well-known predictions from standard consumer theory and dynamic consumption- savings models under different curvature assumptions.
Discount-rate effects. We also vary the discount factor in the Cobb-Douglas economy. As expected, impatient agents (small discount rate 7) devote nearly all wealth current consumption,, while more patient agents invest more heavily and smooth consumption intertemporally.
Scalability to Larger Economies To assess the scalability of our approach, we extend our ex- periments to substantially larger exchange economies: 20 consumers, 20 commodities, 5 assets. and 10 world states. This setting increases both the dimensionality of the joint action space and the complexity the endogenous state transitions, making equilibrium computation significantly more challenging. Due to the higher variance introduced by larger economies, we observe slightly increased instability and greater sensitivity to learning-rate choices. Nevertheless, our extended exper- iments consistently show clear convergence toward near-zero exploitability (Figure 5, Appendix E). demonstrating that our approach remains effective even the scale of the economy grows.
*Qur code can be found here.
"The definitions of these two metrics can be found in Section E.I.
10


Reviews:
[{"review_id": "gWlZeMZo1F", "reviewer": "Reviewer_9wkb", "summary": "This paper introduces Markov pseudo-games (MPGs), a novel framework that unifies Markov games and pseudo-games to model dynamic, uncertain environments where agents’ feasible actions depend on others’ choices. The authors prove the existence of generalized Markov perfect equilibria (GMPE) in concave MPGs (Theorem 2.1), extending Arrow–Debreu’s pseudo-game existence theorem to dynamic settings. They further show that recursive Radner equilibria (RRE) in infinite-horizon stochastic exchange economies can be characterized as equilibria of certain MPGs (Theorem 3.1), thereby establishing the existence of equilibria in a broad class of infinite-horizon incomplete markets. Algorithmically, they propose a first-order exploitability-minimization method—implemented via a generative adversarial policy network (GAPNet)—that computes approximate equilibria in polynomial time under standard smoothness assumptions (Theorem 2.2).", "questions": "1. Can you provide examples of equilibrium behavior in the tested economies? This would help connect the mathematical constructs to economic meaning.\n2. Can you explain more on the high-level intuition of how does the \"pseudo game\" aspect of the formulation affects or restricts the development of the entire theory?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The notion of Markov pseudo-games is conceptually elegant and unifies concepts in game theory and general equilibrium theory in a dynamic setting. The theoretical results are rigorous and intense, and build on a well-established foundation.\n2. The mapping from recursive Radner equilibria to GMPE of MPGs appears novel, extending previous equilibrium existence results that were limited to representative-agent or finite-horizon settings. \n3. The application of deep multi-agent RL to macroeconomic equilibrium computation is interesting.", "weaknesses": "1. Experiments are limited to synthetic, small-scale economies. The experimental section focuses on numerical metrics (exploitability, Bellman error) but lacks economic interpretation of the learned equilibria (e.g., consumption paths, asset prices, welfare implications). This limits the insight for economics audiences.\n2. Notational issue:  $\\mathcal{F}(\\boldsymbol{\\pi})$ is used without giving definition, where $\\boldsymbol{\\pi}$ is the joint policy profile of $n$-agents. The only definition I found is given other player's policy $\\boldsymbol{\\pi}_{-i}$ the meaning of $\\mathcal{F}(\\boldsymbol{\\pi}\\_{-i})$. Since the actions of the policies are mutually constrained in a pseudo-game setup, how does this further imply the definition of  $\\mathcal{F}(\\boldsymbol{\\pi})$?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TjsCmvIR1Z", "reviewer": "Reviewer_9YuJ", "summary": "The paper introduces Markov Pseudo Games, a class of stochastic games that combine the standard Markov games and pseudo-games with the flexibility to capture dynamic uncertainty and action-dependent feasibility constraints. This class of games is particularly suitable for modeling infinite-horizon stochastic economies with incomplete markets, where agents face evolving uncertainty and limited strategic feasibility. The main contribution of the work is the proof of existence of a recursive Radner equilibrium within this generalized setting. Authors achieve this by reformulating the problem as a Generalized Markov Perfect Equilibrium (GMPE) of the corresponding Markov Pseudo Game, using a suitable exploitability function. Furthermore, a polynomial-time first-order solution approach is proposed that uses deep neural networks and minimax optimization; this enables an efficient computation of equilibria in high-dimensional, dynamic environments. Some computations are illustrated.", "questions": "Apart from the key weaknesses mentioned above, some clarification and more discussion on the following points would be useful:\n\nWhile the idea of infinite economies is theoretically sound and offers strong generalization, it is not very clear how the proposed theoretical results would scale when applied to large agent populations or more complex utility functions. Are there any known limitations or additional challenges that might arise in such scenarios? \n\nMost of the analysis rely on assumptions such as Lipschitz smoothness and bounded best-response mismatch coefficients. Also, the algo needs gradients of the rewards and transition probabilities. This seems to a strong requitement. Please clarify; if so, some discussion is needed. \n\n What would happen if these assumptions were violated. For instance, if these regularity conditions do not hold, which may often be the case in real-world infinite economic settings, how would that affect convergence, stability, or the overall validity of the theoretical guarantees?\n\nThe computational experiments currently focus on linear and Cobb-Douglas utility functions, which are standard but relatively well-behaved. \n\nThe plot of exploitability for the Cobb-Douglas seems to have not converged to zero during the reported horizon of the computation; some discussion is needed. \n\nOverall, to me it looks like the discussion about the computational results is very short and can be enhanced. \n\nCould the approach be extended to more general utility forms where these assumptions do not hold? Would such cases introduce potential stability challenges such as mode collapse or convergence failure, and if so, what mitigation strategies could be considered?\n\nThe computational architecture is generative-adversarial network (GANs) and they is known to be prone to issues like mode collapse; alternatives like W-GAN are prompted by such considerations. Some discussion on the possible mode collapse would help.  \n\nIt would strengthen the paper if the Authors could discuss the sensitivity of the results, particularly regarding the equilibrium convergence and approximation quality to the choice of hyperparameters, such as the learning rate, batch size, or regularization coefficients.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper considers a novel and challenging problem. Also, it has a good motivation. The proposed solution approach is also novel. The proposed formulation is particularly interesting, as it unifies infinite-horizon stochastic economies (relevant for macroeconomics area) with incomplete markets using a common theoretical model. The notion of recursive Radner equilibria in this framework is good; it offers an  understanding of the market dynamics. It seems that Roy Radner was not awarded a Noble Prize; while the broad area of general equilibrium seems important enough in econ literature that this sub-area had at least a couple of those awards. \n\nThe work has a good theoretical rigor in my view; the theorems and proofs are carefully done. This is a good contribution to macroeconomic modelling. The reformulation of the problem as a Generalized Markov Perfect Equilibrium (GMPE) is a key contribution that allows the authors to effectively address challenges in prior literature. Furthermore, the paper opens promising directions for future research in economic theory and computational economics; this is possibly via deep reinforcement learning and economic equilibrium analysis.", "weaknesses": "The paper is primarily theoretical, with limited experimental validation. Thus, we have very limited view of the performance of the proposed approach across networks of varying sizes both in terms of the number of agents and the size of the state space. Hence, the important aspect of scalability and robustness remains underexplored.\n\n\nThere are many assumptions that are part of the paper’s analysis (to be expected in a theoretically oriented paper). For example, Assumption 6. But, it is deferred to the appendix. It is better to make the main paper self-contained by summarizing or at least highlighting such essential assumptions within the main text. Similarly, many theorems and results are presented with minimal discussion of their intuition or motivation. Including brief insights/intuition into the meaning and implications of these results would significantly improve the paper’s readability. In general, some discussion on the need or motivation for assumptions is needed. \n\nSimilarly, some of the key definitions such as v^\\pi  and u^\\pi are currently given in the appendix, even though they are central quantities that are used throughout the paper. Bringing these definitions into the main text would enhance clarity and help readers follow the theoretical development more easily.\n\nThe polynomial-time convergence and approximation guarantees are built on regularity assumptions such as Lipschitz smoothness and bounded best-response mismatch coefficients. These conditions may not hold in many practical economic settings. A more detailed discussion of their limitations and how potential violations might impact convergence or solution quality would make the contribution more transparent and guide future research in this direction.\n\nFinally, while the authors claim polynomial-time convergence to approximate equilibria, the paper provides limited information on approximation bounds or exploitability levels achievable in practice. The paper needs a clarification or acknowledgment  regarding any possible trade-off between computational effort and the quality of equilibrium approximation proposed in the paper; this would strengthen the practical relevance and interpretability of the results. Also, giving some background for those who are less familiar with the literature would enhance the readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "scUY85eSSC", "reviewer": "Reviewer_giHS", "summary": "This work introduces the notion of *Markov pseudo-games*, an extension of Markov games where each agent's action space depends on the actions of other agents. The authors then introduce two solution concepts: GMPE and GNE. Under Assumption 3, the authors reformulate this problem as a min-max optimization problem. By further assuming the concavity with respect to the max-player, the paper proposes a two-timescale GDA algorithm that provably converge to an approximate stationary point. In Section 3, the authors show that infinite horizon Markov economy can be modeled as a Markov pseudo game and provide an existential proof of RRE, along with an algorithm that efficiently computes it.", "questions": "See Weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The framework the authors propose is novel. It is nice to see that the authors model a real-world economic scenario using their proposed framework.\n\n- The paper extends the existential result of RRE in infinite horizon Markov exchange economies.\n\n- The paper provides a provable efficient algorithm that converge to a GMPE in Markov pseudo-games and RRE in infinite horizon Markov exchange economies.\n\n- Numerical results are provided where the authors model agents as deep learning neural networks, validating the theorems.", "weaknesses": "- The notations of this paper is quite dense and difficult to follow. Some notations are used before being properly defined (i.e. $D_{\\varphi}(\\pi)$ in line 158).\n\n- One of the interesting feature of Markov pseudo-games is that each agent's action space depends on actions of other agents. However, Assumption 3 largely simplifies that the problem into a standard min-max optimization problem without any coupling dynamics. This assumption appears overly strong and undermines the main motivation of introducing Markov pseudo-games.\n\n- From an optimization perspective, the problem reduces to a min-max optimization problem with hidden structures, which has been extensively studied in the optimization and game theory literature [1].\n\n- Theorem 2.2 does not provide an explicit bound on the convergence time, making it difficult to compare the algorithm’s efficiency with other min-max optimization methods.\n\n- Section 3 is heavily focused on economics, which may limit its appeal to the broader ICLR audience. I also suggest that the authors include additional background on Markov exchange markets in Section 3 to make the section more accessible to readers without an economics background.\n\n-----\n\n[1] Flokas, L., Vlatakis-Gkaragkounis, E.-V., and Piliouras, G. Solving min-max optimization with hidden structure via gradient descent ascent. arXiv preprint arXiv:2101.05248, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JWgJo59ar4", "reviewer": "Reviewer_hHrX", "summary": "This paper investigates the Markov pseudo game and develops a first-order solution method with polynomial-time guarantees for the existence of equilibria. One key contribution is that the authors demonstrate recursive Radner equilibria in infinite-horizon Markov exchange economies can be reformulated as concave Markov pseudo-games (MPG). Once formulated as an MPG, new solution concepts can be applied to solve for equilibrium. Empirical results are provided via a GAN policy network representing players in an exchange economy, adapting the learning parameters via computation of recursive Radner equilibrium (RRE) via Alg. 1 (TTSSGDA).", "questions": "Does the statement in 164-166 regarding $\\epsilon$-stationarity imply that no FPTAS solution exists for $\\epsilon$-stationarity point in the MPG? Is this tight, or just a claim?\n\nThe fact that the reduction to RRE exhibits polynomial-time convergence guarantees are compelling - nevertheless could the authors be transparent about the limitations specifically w.r.t. this reduction?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "This paper proves the existence of the generalized markov perfect equilibria(GMPE) for concave Markov pseudo game, which extending Arrow-Debreu's existence theorem. \n\nBased on a two-time scale stochastic gradient descent ascent algorithm, they provided a algorithm for finding the approximate GMPEs. They further provided the convergence rate under this algorithm. \n\nThe authors demonstrate recursive Radner equilibria in infinite-horizon Markov exchange economies can be reformulated as MPG and in consequence, proved the existence of recursive Radner equilibria\n\nI also really like the scope of experiments and find them highly relevant - and I appreciate the authors motivations to address non-incremental problems.", "weaknesses": "Some of the critical assumptions are relegated to the appendix (e.g. line 79).\n\nDespite being somewhat knowlegeable in Markov game theory - this paper is highly complex, and also subject to a lot of notational overload - for example in line 88 the definition for M, and line 310 the notation for I. I would really suggest being more concise with the notation and even some notational abuse where necessary to reduce the overload and convey meaning faster. I could only understand the scope of the contributions and what the ramifications of these contributions are, but I failed to get an intuitive grasp of the solution (proof) method overall nor been successful to follow through on the proofs entirely (although the sections I did review seem correct).\n\n---\nTherefore the result of my assessment hinges on hoping that in the rebuttal the authors could provide a simpler explanations to how they arrived at their conclusions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)