Title: Solving Zero-Sum Markov Games with Continuous State via Spectral Dynamic Embedding
Venue: NeurIPS 2024
Accepted: yes
Keywords: zero-sum Markov game

reinforcement learning

dynamic programming
Abstract: In this paper, we propose a provably efficient natural policy gradient algorithm called Spectral Dynamic Embedding Policy Optimization (\SDEPO) for two-player zero-sum stochastic Markov games with continuous state space and finite action space.
  In the policy evaluation procedure of our algorithm, a novel kernel embedding method is employed to construct a finite-dimensional linear approximations to the state-action value function.
  We explicitly analyze the approximation error in policy evaluation, and show that \SDEPO\  achieves an $\tilde{O}(\frac{1}{(1-\gamma)^3\epsilon})$ last-iterate convergence to the $\epsilon-$optimal Nash equilibrium, which is independent of the cardinality of the state space.
  The complexity result matches the best-known results for global convergence of policy gradient algorithms for single agent setting.
  Moreover, we also propose a practical variant of \SDEPO\ to deal with continuous action space and empirical results demonstrate the practical superiority of the proposed method.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the two-player zero-sum stochastic Markov games (2p0s-MGs) with large scale or continuous state spaces. These problems have a large cardinality and function approximation methods are needed. The paper consider a spectral dynamic embedding method and proposed SDEPO. This methods utilized the transition dynamics in the construction of the state-space value function. SDEPO is able to converge with order $1/\\epsilon$, which matches the optimal rate in single agent RL.\nTheorems are provided for the last iterate convergence of the SDEPO algorithm. The effectiveness of the algorithm has been verified in games against baseline methods.\n\nGenerally the paper is well structured, but section 3-4 should be better explained while section 5 and 6 focused on the \"practical algorithms\" is stretched quite far from the analytical results in the previous sections.\n\nSTRENGTHS:\nThe function approximation approach for markov games is a necessity for those problems with large/infinite state cardinality. It is indeed true that the dynamics of the problem was not utilized in previous methods. This algorithm seems to be the first work addressing this.\n\nThis work adapted the spectral dynamic embedding for stochastic nonlinear control problems and proposed SDEPO, the motivation is clear and well stated.\n\nWEAKNESSES:\nThe only evaluation for the proposed algorithm is a rate of success in playing games with baseline algorithms. To the reviewer, this seems to be very limited. For a submission with strong theoretical focus, current result fails to validate the convergence properties of the proposed algorithm.\n\nThe sections for main results are not very well-written and is a bit difficult to read, more explanation would be appreciated. Although this could be due to the page limit.\n\nAssumption 5 is in fact quite strong, a brief discussion on the impact and reasoning should be provided.\n\nSection 5 and 6 seems a bit rushed and is intended to bring out the neural networks, the prior sections discussed the setting with tabular actions, where in these sections the action space is seen as continuous and more algorithms have been added, with no analytical results. I suggest the authors focusing on the existing setting with better presentation, explanation and more experiments.\n\nAnother problem this paper did not address is what are the current existing algorithms involving dynamics and function approximation in the single agent setting. The single agent RL with function approximation literature should be somewhat addressed in general.\n\nQUESTIONS:\nWhat is the effect of truncation w.r.t. the later regularity condition assumptions? Does a more limiting truncation negatively impact these assumptions on the problem?\n\nWhat is the reason for the consideration of one-sided $\\epsilon$−optimal Nash equilibrium? The author stated that many existing works also consider this, but an explanation would be appreciated.\n\nLIMITATIONS:\nThe authors did not fully address the limitations of this paper. The authors mentioned that the algorithm is non-independent and relies on some assumptions. There are some other limitations of this work, which has been raised in the previous sections.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new algorithm named Spectral Dynamic Embedding Policy Optimization (SDEPO) to solve the zero-sum Markov games with continous state and finite actions. The convergence analysis indicates that the proposed method achieves the best-known sample complexity as the case of finite-state space; this paper is the first theoretical result in handling the continuous state space with known dynamic and infinite horizon.\n\nSTRENGTHS:\nThis paper is the first result for solving the NE of infinite-horizon two-player zero-sum Markov games  with continuous state space when the dynamic is known. Moreover, this paper resents sufficient introduction on the technical backgrounds and preliminaries. All assumptions are clearly listed. Lastly, the theoretical results are verified using emprical experiments.\n\nWEAKNESSES:\n1. The Assumption 1 is not reasonable. It says, whatever the state $s$ and the action $(a,b)$ are, the agent can move to any state $s'$ with a positive probability. Please correct me if I am wrong.\n\n2. I am confused about what is new in the Spectral Dynamic Embedding method. It seems that both Bochner and Mercer theorem are well-known. This paper simply applies them to represent the transition probability and the reward using some kernels. Then everything is the same as traditional method in RL. \n\n3. A mild comment on Assumption 3: Since the optimal policy might be deterministic, it means that $\\pi(a|s)$ is likely to be zero for some $a$. During the training, the policy $\\pi_k$ will tend to the optimal policy; the mass at non-optimal action will also approach to $0$. It means if  $\\underline{c}$ is larger than $\\epsilon$, then $\\pi_k$ will never converge to the optimal action in the sense of $L_\\infty$ norm. From my understanding, the author needs to set $\\underline{c}$ to be $\\epsilon$ and it won't affect the complexity.\n\nQUESTIONS:\nQ1:  Can the author justify the use of Assumption 1? It seems to be unrealistic in RL. \n\nQ2: This paper seems to be a simple combination of linear MDP + policy gradient. What is the novel part of this work? I feel hard to consider representing $\\mathbb{P}$ and $r$ using kernel methods as a new thing.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

comments: SUMMARY:
The authors introduce an innovative approach to solving 2p0s-MGs with continuous state spaces, providing both theoretical guarantees and practical improvements over existing methods. The SDEPO algorithm and its variants offer efficient and scalable solutions for complex Markov games, potentially applicable to various domains in reinforcement learning.

STRENGTHS:
1. This paper proposes a new Spectral Dynamic Embedding Policy Optimization algorithm that effectively addresses two-player zero-sum Markov games with continuous state space and finite action space.
2. To handle the finite action spaces, a practical variant of SDEPO is proposed to manage continuous action spaces, with empirical results showcasing its superior performance.
3. The complexity result of SDEPO matches the best-known results for policy gradient algorithms in the single-agent setting, proving its efficiency.

WEAKNESSES:
1. The spectral embedding methods can be computationally intensive in practice due to the complexity of handling spectral dynamic embeddings.
2. Why were these specific feature generation methods chosen? Is the proposed method sensitive to feature generation methods?
3. The experiments are somewhat limited, expanding the empirical section to include more complex and diverse scenarios would significantly strengthen the paper.

QUESTIONS:
Please see the weakness part.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-14