Title: Grouped Satisficing Paths in Pure Strategy Games: a Topological Perspective
Venue: NeurIPS 2025
Accepted: no
Keywords: game theory

multi-agent reinforcement learning

strategic dynamics
Abstract: In game theory and multi-agent reinforcement learning (MARL), each agent selects a strategy, interacts with the environment and other agents, and subsequently updates its strategy based on the received payoff. This process generates a sequence of joint strategies $(s^t)_{t \geq 0}$, where $s^t$ represents the strategy profile of all agents at time step $t$. A widely adopted principle in MARL algorithms is "win-stay, lose-shift", which dictates that an agent retains its current strategy if it achieves the best response. This principle exhibits a fixed-point property when the joint strategy has become an equilibrium. The sequence of joint strategies under this principle is referred to as a satisficing path, a concept first introduced in [40] and explored in the context of $N$-player games in [39]. A fundamental question arises regarding this principle: Under what conditions does every initial joint strategy $s$ admit a finite-length satisficing path $(s^t)_{0 \leq t \leq T}$ where $s^0=s$ and $s^T$ is an equilibrium? This paper establishes a sufficient condition for such a property, and demonstrates that any finite-state Markov game, as well as any $N$-player game, guarantees the existence of a finite-length satisficing path from an arbitrary initial strategy to some equilibrium. These results provide a stronger theoretical foundation for the design of MARL algorithms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA satisfying path is a sequence of strategy profiles such that 1) any player with best response strategy in some time does not update her strategy in next time, and 2) ends with an NE. Existing work[39] proves the existence of satisfying path with arbitrary initial point in a finite-player, finite-action (allowing mixed NE), normal-form game.\n\nThis work generalizes the result to a class of games with many theoretical assumptions. Among them, (I think) the most insightful assumptions are: 1) the payoff function is partially analytic, 2) any sub-game exhibits an NE.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\nThe problem studied is well-motivated. The insights behind all results are clearly explained. The generalization over finite-action games is a theoretical contribution.\n\n**Weaknesses**:\n\n1. I think the most crucial weakness is that the techniques used in this paper are similar to those in [39], which makes the contribution of this paper seemingly incremental. There indeed exist some new techniques, such as the usage of Baire category theorem to justify the existence of an inner point in a set (Lemma 1). However, I found the high-level idea similar: 1) find $x^ t$ that makes the satisfied players minimum; 2) find the NE $x^ *$ of sub-game with unsatisfied players; 3) prove that the strategy profile $x^ {t+1}$, which equals $x^ t$ for satisfied players and $x^ *$ for unsatisfied players, substitutes an NE and is a feasible successor of $x^ t$. I would like to increase my score if (part of) the techniques used to prove the main result were found essentially different from existing works, especially [39].\n\n2. Some sentences have grammar flaws, such as:\n\n* line 163 \"while maximum means cannot increase\"\n* line 205-208, \"... will (not/still) achieve\"\n* line 517 \"a important\"\n\n3. See questions below.\n\nQUESTIONS:\nI would appreciate if the authors could response following questions:\n\n1. Since partial analytic assumption seems unnatural, I have some questions w.r.t this assumption:\n\n* Does there exist some functions which is continuous, partially analytic but not (globally) analytic?\n* Does there exist some games that exhibit at least one NE, but fail to have a satisfying path with some initial point (probably violating partial analytic property)?\n\n2. (Resolved) What is the precise definition of \"sub-game\"? This concept appears in the statements of Theorem 2 but the authors give no definition before. Though it is not hard to speculate the definition of \"sub-game\", a precise definition before its first usage would improve the clarity.\n\n3. I'm also confused about the concept of \"grouped satisfaction path\" over \"(individual) satisfaction path\" as in [39]. The author claims that this concept is useful in the proof of Corollary 3. In the proof, the author first construct a new pure-strategy game $H$, in which the players in the new game consist of all $(i,x)$ pair where $i$ is a player and $x$ is a state in the original Markov game $G$. A group set $A$ is used to construct the existence of NE in any sub-game of $H$. However, since any sub-game of $H$ is exactly a Markov game with singleton action set in some (player, state) pair, the existence of NE should be natural, which might then indicate the concept of \"grouped satisfaction path\" is unnecessary.\n\nLIMITATIONS:\nSee weaknesses and questions above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the concept of a \"grouped satisficing path\", which extends the existing game-theoretic concept of a \"satisficing path\" (one in which strategies that are already best responses don't change). It studies the properties of this grouped satisficing path from a topological perspective. Finally, it proves sufficient conditions for the existence of such a path from any initial strategy profile to some equilibrium. This supplements existing research on convergence in MARL, and provides theoretical support for stochastic update approaches.\n\nSTRENGTHS AND WEAKNESSES:\nThe theoretical result looks interesting and relevant, and seems to have fairly wide applicability (per the 4 corollaries on page 8).\n\nThe grammar and phrasing used in the paper could be improved at some points, which I've described below. This also contains a few points where clarity could be improved. Since the term \"best response\" is used very often, sometimes multiple times in the same sentence, I suggest shortening it to \"BR\" (after introducing the acronym somewhere near the beginning of the paper.)\n\nI also suggest amending the somewhat-awkward phrase \"has achieved best response\" to simply \"is a best response\", and future tenses (\"will\") to present tenses (\"is\"). Mathematics is usually written in an atemporal/timeless way, and usually reads better that way.\n\nExample to illustrate (from page 6):\n\n- Before: \"the group which has achieved best response in $s$ will still achieve best response in $t$.\"\n\n- After: \"the group that is a BR in $s$ is still a BR in $t$.\"\n\nI suggest improving the structure/presentation of the argument on page 7. Perhaps it can be broken down into smaller, clearly-labeled pieces. This also makes it clearer which assumptions are applicable to which parts. For example, if one is doing a sub-proof by contradiction (from a false assumption), the end of that sub-proof should be clearly marked.\n\nI also suggest using singular instead of plural, wherever possible. It's easier to contemplate a single object in mind at a time, rather than multiple objects, especially when juggling multiple quantifiers. Example (from page 7):\n\n- Before: any player does not change their strategies\n\n- After: no player changes its strategy\n\nSuggestions, going page-by-page:\n\nPage 1:\n\nChange \"this process generates a strategy path\" to \"this process generates a strategy profile path\".\n\nPage 2:\n\nChange \"The path generated\" to \"A path generated\" (there might be more than one possibility).\n\nReword the grammatically-awkward \"and finally there comes to the conclusion of this paper\" to \"and the conclusion of this paper is presented\" (or the like).\n\nChange \"limited to the specific game structures\" to \"limited to specific game structures\".\n\nChange \"analyzing the convergence of the game\" to \"analyzing the convergence of the dynamics\".\n\nShouldn't the update equation on line 81 use $s_{\\leq t}$ rather than $s_t$? That is, it can depend on the entire history, as described in line 41. Fictitious play, for example, depends on the historical average strategy profile, not just the previous strategy profile.\n\nChange \"the game will be not convergent to an equilibrium generically\" to \"the dynamics will not generically converge to an equilibrium\".\n\nChange \"the agent has the probability to update the strategy randomly\" to \"an agent may update its strategy randomly\".\n\nUnclear what \"the conditional update mechanism\" refers to.\n\nPage 3:\n\nChange \"proved the path existence\" to \"proved path existence\".\n\nThe following is unclear: \"[39] demonstrated universal existence in all N-player games. While these works successfully addressed the existence problem in specific game classes, they could not extend the result in a more general form directly.\" How is \"universal existence in all N-player games\" a \"specific game class\"? Please clarify what's specific about that result.\n\nReword \"So it is necessary to establish the theory of the satisficing path. Since this paper studies the topological properties of the satisficing path, the existence results can be successfully extended in a more general form\" to something like \"Thus it is necessary to establish the existence of a satisficing path more generally. This paper studies the topological properties of a satisficing path and successfully extends the previous existence results to a more general form\".\n\nChange \"A mixed strategy game can also be defined by\" to \"A mixed strategy game can also be constructed from\".\n\nChange \"However, each player $i \\in I$ can choose\" to \"In such a game, each player $i \\in I$ chooses\".\n\nChange \"Construct a pure strategy game\" to \"Consider the pure strategy game\".\n\nPage 4:\n\nDefinition 4: The usual definition of a partition requires that the parts be inhabited (nonempty). Is this condition missing?\n\nPage 5:\n\nChange \"it is called that any sub-game has an equilibrium\" to \"it is said that any sub-game has an equilibrium\".\n\nChange \"Due to space limitation\" to \"Due to space limitations\" or \"Due to limited space\".\n\nIn Lemma 1 part 2, change \"satisfying:\" to \"satisfying both of the following:\" (or the like), if that's what you meant.\n\nIn Lemma 1 part 2a, change \"will not achieve\" to \"that will not achieve\".\n\nIn Lemma 1 part 2b, change \"will achieve\" to \"that will achieve\".\n\nIn Lemma 2, change \"the player k\" to \"player k\".\n\nIn \"to describe the property of the best response\", clarify what property you're referring to.\n\nIn general, consider replacing the somewhat awkward phrase \"achieve[d] [ϵ-]best response\" with simply \"is a[n] [ϵ-]best response\", or \"is [ϵ-]BR\" for short (introducing the acronym before that).\n\nPage 6:\n\nThe following sentence is grammatically unintelligible: \"Either any joint strategy in T(s) makes the group which has achieved best response in s will still achieve.\"\n\nLikewise for part 2 of \"Necessity\":\n\n- \"achieved best response in s will achieve in any joint strategy\"\n\n- \"there exists one group which has achieved best response in s will not achieve.\"\n\nChange \"Assume group\" to \"Assume groups\".\n\nChange \"This is a contradiction with\" to \"This contradicts\".\n\nRemark 3 is unclear, perhaps because the second sentence is too long (run-on?) and should be split. Consider rewording it.\n\nChange \"Due to space limitation\" to \"Due to space limitations\" or \"Due to limited space\".\n\nPage 7:\n\nDelete \"Reduction to absurdity needs to be used here.\" and change \"Assume on the contrary that\" to \"Assume, for the sake of contradiction, that\".\n\nChange \"any player does not change their strategies\" to \"no player changes its strategy\".\n\nChange \"the group who has achieved best response will still achieve, and the group who not will still not\" to \"any group that has achieved best response will still achieve it, and any group that has not will still not\".\n\nChange \"So (at)t≥0 is actually a grouped...\" to \"Therefore, (at)t≥0 is a grouped...\".\n\nChange \"which is a contradiction with\" to \"which contradicts\".\n\nChange \"is local minimum\" to \"is a local minimum\".\n\nChange \"without violating the property of grouped satisficing path\" to \"that does not violate the property of a grouped satisficing path\".\n\nThe following is grammatically unintelligible: \"makes the group which has achieved best response in sx will still achieve.\" Change it to \"makes the group that has achieved best response in sx still achieve it\", if that's what you meant.\n\nChange \"Assume group p1, ..., pn\" to \"Suppose groups p1, ..., pn\". Also, consider adding a paragraph break right before this sentence, or connecting it to the previous sentence somehow. The structure of this argument should be made clearer.\n\nChange \"will [still] achieve in u\" to \"will [still] achieve it in u\".\n\nPage 8:\n\nChange \"so only need to check\" to \"so we only need to check\".\n\nChange \"with finite players and finite sets\" to \"with finitely many players and states\".\n\nChange \"Then, check whether\" to \"Then, we check whether\".\n\nChange \"Contraction mapping theorem will be used in this proof.\" to \"A contraction mapping theorem is used in this proof.\", if that's what you meant.\n\nChange \"to the reinforcement learning with historical records\" to \"to reinforcement learning with history-dependent policies\", if that's what you mean. Otherwise, clarify what you mean.\n\nChange \"there exists satisficing paths\" to \"there exist satisficing paths\".\n\nChange \"from arbitrary initial joint mixed strategy to a mixed equilibrium\" to \"from any arbitrary initial joint mixed strategy to some mixed equilibrium\", if that's what you mean.\n\n\"finite history records\" Again, define this term somewhere before it's first used.\n\nRemove the colon from \"implies that: it is possible\".\n\nChange \"from arbitrary initial joint strategy\" to \"from an arbitrary initial joint strategy\".\n\nChange \"while keeping the player who has achieved best response not change its strategy\" to \"while keeping the strategies of players who have achieved best responses unchanged\".\n\nChange \"for the player who has not achieved best response\" to \"for players that have not achieved best responses\".\n\nChange \"have the finite-length satisficing paths\" to \"has a finite-length satisficing path\".\n\nQUESTIONS:\nIn the Discussion section, you allude to the possibility of extending Theorems 1 and 2 from ϵ = 0 to ϵ > 0. What obstructions, if any, are there to such an extension with your current proof techniques?\n\nLikewise, you allude to the possibility of extending Theorem 2 to the case where non-BR players are restricted to selecting strategies only from a neighborhood of their current strategies. This would indeed be very interesting, because many MARL techniques used in practice are based on infinitesimal/incremental/gradient-based dynamics. What obstructions, if any, are there to such an extension with your current proof techniques?\n\nIs there anything quantitative that can be said about the minimal length of a satisficing path from any strategy profile to some equilibrium? Would that require a much deeper or more difficult analysis?\n\nLIMITATIONS:\nYes (see Discussion section).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers a problem in game theory about the reachability of equilibrium points through a process related to “win-stay, lose-shift” strategy updates. The authors address open questions from a recent work [39] and show that the reachability results of [39] extend to a larger class of games that are relevant to multi-agent reinforcement learning.\n\nThe techniques used in this paper generalize the existing definition of “satisficing paths” to “grouped satisficing paths”, which encompass multi-agent coalitions, with the special case of single agent coalitions recovering the original definition. The authors study the existence of grouped satisficing paths to equilibrium by studying the topological properties of such paths, defining a notion of local minimum for the set of strategies and drawing conclusions about what this means for coalitions that are best responding at such a local minimum.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\nThe topic covered here seems fundamental to game theory and multi-agent reinforcement learning. I have some concerns about the proofs of some lemmas and Theorem 1, but if they are correct (or if they can be fixed), the main result of this paper contributes nicely to this research area. \n\nSpecifically, Theorem 2 and Corollary 4 are good contributions. The authors generalize the best previously known result, going from finite games and special cases of Markov games to general Markov games.\n\nIn addition to finding more general results, the analytic techniques used are interesting, as they translate some ideas and analysis present in the earlier papers [39] and [40] into topological language and pave the way for applying new tools to understand the game theoretic landscape.  Some caveats to this strength will be described in the weaknesses section, but these caveats might be easy to fix.\n\n\nWeaknesses:\n\nThis paper’s main weakness is about the proofs of Lemmas 1 and 2, which may or may not be correct (or fixable) but left me with some concerns. These are outlined below. If these concerns about the topological proofs can be addressed in the discussion period, then I will raise my score. If the issues seem to persist after the discussion, I may lower my score. Some secondary weaknesses include how the paper cites existing work and small issues in the writing. \n\nTechnical concern 1: The openness of $O \\subset T_{\\epsilon} ( s )$ in part 2 of Lemma 1 hasn’t been proven to my liking. It may still be true, but I’d like some clarification and probably some re-writing. My issues are with equality (14) and the final sentence of the proof of Lemma 1. Please see the associated questions.\n\nTechnical concern 2: How can we be sure that the open set $O \\subset U \\subset \\prod_{i \\in I} S_i $ in Lemma 2’s proof can be decomposed into the product of open sets $O = \\prod_{i \\in I} O_i $ with each $O_i \\subset S_i$ an open set. I think $O$ can be written as an arbitrary (i.e. possibly uncountable) union of such product sets, but it’s not clear to me why we can do it using a single product as claimed in equation (15). Consider, for example, the following open set in the plane {$ (x,y) \\in \\mathbb{R}^2 : 0<x<1  \\text{ and } x < y < x + 1 $}. This is open in $\\mathbb{R}^2$ but I don't think you can write it as $O_1 \\times O_2$ where $O_1, O_2 \\subset \\mathbb{R}$ are open sets. In game theoretic terms, these constraints relating allowable values of x and y are interdependent, meaning that the strategies of each player shouldn’t be arbitrary but should depend on $s_{-k}$, no? In short, I'm skeptical that $t_k$ can therefore be truly arbitrarily chosen, and Lemma 2 may have a flaw. I’m not saying it’s certainly incorrect, but I need to be convinced.\n\nOutside of these technical concerns, there are secondary weaknesses related to how this paper contextualizes itself in the literature. If I am understanding the proofs of this paper and the preceding works [39] and [40] correctly, the main line of argument used in this paper basically mirrors earlier lines of argument to a large extent. This point should probably be acknowledged more clearly, since it better explains how the innovation of this paper is its analytic techniques studying the local minima in more complex game settings. For example, it appears that Markov games were also discussed in [40], and a two-player result for Markov games was obtained using basically the same line of argument as used here. The analysis of the present paper captures the case where $| I | \\geq 3$ in a way that [40]’s analysis did not, which is certainly a strength, but in my opinion it’s worth clarifying.\n\nRelated to the point above, the authors may wish to cite another paper by the authors of [39], which seems to relate to the idea of well-defined sub-games used in Theorem 2 of this paper. The paper: B. Yongacoglu, G. Arslan, L. Pavel and S. Yüksel, \"Generalizing Better Response Paths and Weakly Acyclic Games,\" 2024 IEEE 63rd Conference on Decision and Control (CDC), 2024, pp. 5262-5267. \n\nBesides the points above about the topological proofs and the literature, I think the abstract and first paragraph of this paper should probably be re-written. In their current state, they parallel the abstract and first paragraph of [39] to an alarming degree. The remainder of the paper and technical findings are sufficiently different that this is only a minor weakness, in my opinion. \n\n\nMinor notes:\n- Definition 8 has some minor grammatical issues.\n- Line 239 says “Reduction to absurdity needs to be used here”. If this is intentional, I’d recommend re-writing it for style. There are other somewhat awkward phrases as well as some grammatical issues elsewhere in this proof.\n- While not a mistake in the strict sense, the symbol $x$ is used inconsistently, which was a bit distracting. In some places, $x$ was an integer (e.g. proof of Theorem 2), in others $x$ was a strategy, in others (e.g. equation (13)) $x$ was a player, in others $x$ was a real number in [0,1] (Line 507, equation (16)), and in still others $x$ was a state for a Markov game (p. 18, Line 568). \n- Similar to the last point, the sets $A$ and $A_k$ were used with several unrelated meanings, which was also a bit distracting.\n\nQUESTIONS:\n1. (Resolved) Can you explain the equality in (14)? I don’t see why this follows from the previous sentence.\n\n2. (Resolved) In Line 496, you say “Obviously, $s_J \\times C $ is an open set in $T_{\\epsilon} (s)$”. But even if $C$ is an open set, $s_J$ is only a singleton — how can this be open in the topology of the ambient space (where players in J can also vary their strategies)?\n\n3. (Resolved) How can we be sure that the open set $O \\subset U \\subset \\prod_{i \\in I} S_i$ in Lemma 2’s proof can be decomposed into the product of open sets $O = \\prod_{ i \\in I} O_i$ with each $O_i \\subseteq S_i$ open in $S_i$?\n\n4. Can you clarify the wording of Proposition 5 (Line 563)? It uses “there exists” and “when $p$ goes to infinity” together in a confusing way.\n\n5. Can you clarify the subscript of the last expectation in Line 586? It’s quite confusing.\n\n6. Can you clarify the wording before Line 587? Where did this $c_k a^k$ appear from, and how does it relate to the lines before it?\n\n7. Repeatedly (e.g. Lemma 1, Line 178; Theorem 1, Line 199; proof of Theorem 1, Line 204;) you refer to “the group” rather than “a group” or “any group”. Is this intentional? Is it unique and that’s why we say \"the\"?\n\n8. Can you comment on the relationship, if any, between the satisficing paths of this paper and the satisficing equilibrium of the following paper: Pradelski, Bary SR, and Bassel Tarbush. \"Satisficing Equilibrium.\" arXiv preprint arXiv:2409.00832 (2024)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper study learning dynamics arising by the simple principle of “win-stay, lose-shift” in pure strategy games. In particular they refine the notion of satisficing path (a sequence of strategies converging to an equilibrium and satisfying conditions encoding the principle above) to the one of group satisficing path. They first show sufficient conditions under which a group-satisficing path in finite time exists for all starting strategies in pure games. Then, as corollaries, they show the same for satisficing paths in different and more complex type of games.\n\nSTRENGTHS AND WEAKNESSES:\n**Positive points**:\nThe topic per-se in interesting: it is well known that studying independent dynamics in MARL is non-trivial. The question of sufficient conditions is also interesting.\n\n**Negative points**:\nI think that the paper would need to be revised strongly from a presentation perspective. The english is sometimes informal or too much formal, the ‘Formalization’ section (which would be Preliminaries to me) lack a lot fo concepts that are named without any explanation in the other sections of the paper. To mention one: Baire spaces: it is very likely that many AGT researchers would either not know or remember such definition and thus it would be important to include it in the main paper, before mentioning it. Same goes for topological conditions like Haussdorff etc. On the other hand, in Definition 4, the authors redefine what a partition is! The difference between the group satisficing paths definition and the satisficing paths ones should be discussed a bit more in depth since they are very similar. Also, I think that the authors mix a lot in the presentation between pure games and Markov games and in general, models for MARL and that is very confusing.\nAll in all the paper is not ready for acceptance, in my opinion.\n\nQUESTIONS:\nWhat is the advantage in using group satisficing paths in your proofs?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}