Title: MALinZero: Efficient Low-Dimensional Search for Mastering Complex Multi-Agent Planning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Monte Carlo Tree Search

Reinforcement Learning

Multi-Agent System
Abstract: Monte Carlo Tree Search (MCTS), which leverages Upper Confidence Bound for Trees (UCTs) to balance exploration and exploitation through randomized sampling, is instrumental to solving complex planning problems. However, for multi-agent planning, MCTS is confronted with a large combinatorial action space that often grows exponentially with the number of agents. As a result, the branching factor of MCTS during tree expansion also increases exponentially, making it very difficult to efficiently explore and exploit during tree search. To this end, we propose MALinZero, a new approach to leverage low-dimensional representational structures on joint-action returns and enable efficient MCTS in complex multi-agent planning. Our solution can be viewed as projecting the joint-action returns into the low-dimensional space representable using a contextual linear bandit problem formulation. We solve the contextual linear bandit problem with convex and $\mu$-smooth loss functions -- in order to place more importance on better joint actions and mitigate potential representational limitations -- and derive a linear Upper Confidence Bound applied to trees (LinUCT) to enable novel multi-agent exploration and exploitation in the low-dimensional space. We analyze the regret of MALinZero for low-dimensional reward functions and propose an $(1-\tfrac1e)$-approximation algorithm for the joint action selection by maximizing a sub-modular objective. MALinZero demonstrates state-of-the-art performance on multi-agent benchmarks such as matrix games, SMAC, and SMACv2, outperforming both model-based and model-free multi-agent reinforcement learning baselines with faster learning speed and better performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper, MALinZero, proposes a new way to tackle a big problem in multi-agent planning with Monte Carlo Tree Search (MCTS): the action space gets huge very fast. Their idea is to represent joint-action returns in a low-dimensional space, treating it as a contextual linear bandit problem. They introduce a new LinUCT (Linear Upper Confidence Bound applied to trees) for MCTS that helps explore and exploit this lower-dimensional space. They also use a special \"loss function\" that's convex and smooth to make sure they don't miss out on good actions. The paper claims MALinZero has good theoretical guarantees, like regret bounds, and they show that picking actions is like maximizing a submodular function, which has a known approximation algorithm. Plus, they say they made the back-propagation part more efficient. They tested MALinZero on matrix games, SMAC, and SMACv2. They report it beats other model-based and model-free methods, learning faster and performing better.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- **Tackles a Real Problem:** They're going after the exponential action space issue in multi-agent MCTS, which is a genuinely tough nut to crack. That's a solid motivation for the work.\n- **Novel Approach for MCTS:** Using a contextual linear bandit and LinUCT to project into a low-dimensional space for *multi-agent MCTS* is pretty neat. While these ideas exist elsewhere (like low-dimensional representations in MARL), their specific combination and theoretical backing for MCTS exploration is a notable contribution.\n- **Theoretical Backing:** They've got regret bounds and they've shown the action selection is a submodular optimization problem, which gives it a good theoretical foundation.\n- **Good Experimental Results:** The empirical results look strong across the benchmarks they chose. Outperforming existing methods, especially on SMACv2 where they claim to nearly double winning rates, is impressive. The speed-up claims are also interesting.\n\n### Weaknesses\n\n- **The \"General $f$\" Loss Function Needs More Detail:** They introduce this fancy convex and smooth loss function $f$ to avoid underestimating good actions. They give an example with $w_{+}$ and $w_{−}$. But how do you *choose* these weights in practice? Is it fixed? Is it tuned? Is it learned? This is crucial for understanding its real-world applicability and why it helps so much in the ablation study.\n- **Dynamic Node Generation (DNG) Sampling:** Algorithm 1 says DNG uses \"$\\beta$ and $P$ as Sampled MuZero\" for sampling new nodes. That's pretty vague. What exactly are $\\beta$ and $P$ in their context? How does this sampling work? Clarity here is needed.\n- **Computational Overhead of Submodular Maximization:** They proved action selection is submodular, which is cool, but they don't really talk about the practical speed impact of using an approximation algorithm for it during the MCTS rollout. MCTS relies on fast tree traversal. Does this add a noticeable overhead? How does it compare to the simpler UCB-like selections in other MCTS methods?\n- **Hard to Follow WITHOUT Appendix:** It's impossible to properly review this paper without the appendices. They refer to them *constantly* for proofs, architecture details, and experimental settings. Without these, I can't check their math, understand their \"low-dimensional representation\" fully, or see how their models are structured.\n\nQUESTIONS:\n1. **How are \"latent per-agent action rewards\" learned?** Section 3.1 talks about these rewards, $\\theta^{\\ast}$, but the high-level representation model $h_{\\theta}$ (mentioned for MuZero) isn't clearly connected to how these specific latent rewards are actually derived from observations. Is there a neural network learning this projection? If so, what's its structure?\n2. **Details on the General Loss Function $f$:** You introduce the generalized loss function $f$ with $w_{+}$ and $w_{−}$. How are these values determined for different rewards ($X_s\\ge\\langle\\theta,A_s\\rangle$ vs. $X_s<\\langle\\theta,A_s\\rangle$)? Are they fixed, hand-tuned hyperparameters, or are they learned during training? This is important for practical application.\n3. **Clarify Back-Propagation Notation in Algorithm 1:** Please clarify the $\\theta(s)$ update: \"$\\theta(s)\\leftarrow V(s)^{-1}r_{k}a$\". How does this specific $r_{k}$ relate to $X_s$ in the sum for $\\hat{\\theta}_{t}$? A consistent and precise update rule would be much better.\n4. **More in Main Text, Less in Appendices, please!** The paper refers to Appendices A, B, and D constantly. Without them, it's impossible to properly evaluate the theoretical claims, understand the model's architecture, or verify the experimental setup. Could you explain them more clearly in the MAIN TEXT, or at least provide a sketch explanation?\n\nMy score could increase if you address my concerns and clarify the core mechanisms (representation learning, loss function parameters, back-propagation notation). It would decrease if these fundamental aspects remain unclear.\n\nLIMITATIONS:\nThe authors mention two limitations: the current linear bandit formulation and the open problem of fully decomposable representations. These are fair.\n\nHowever, I think they should also explicitly acknowledge:\n\n- **Computational Overhead of Submodular Maximization:** While theoretically elegant, the practical speed implications of solving a submodular maximization problem (even approximately) during each MCTS selection step need to be considered and perhaps quantified, especially for time-sensitive applications.\n- **Hyperparameter Sensitivity of the Generalized Loss Function:** The novel loss function $f$ with its potentially tunable parameters ($w_{+}$, $w_{−}$) could introduce new challenges for hyperparameter tuning. It would be good to discuss the sensitivity of the method to these parameters as a potential limitation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes MALinZero for efficient low-dimensional search in multi-agent planning. The approach projects joint-action returns into a low-dimensional space via a contextual linear bandit formulation, and solves this contextual linear bandit problem with convex and $\\mu$-smooth loss functions. It reduces the tree search complexity from the exponential in the number of agents to linear in the number of agents. Results on SMAC demonstrate that MALinZero outperforms state-of-the-art methods in this multi-agent tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The paper proposes a novel method for multi-agent planning that reduces the complexity of tree search from exponential to linear in the number of agents, which is theoretically sound.\n- The proposed method is evaluated on the SMAC benchmark and compared against both model-free and model-based baselines.\n\nWeaknesses\n- The writing can be improved. There are many repeated phrases across the abstract, introduction, and related work sections, which makes the paper feel redundant. \n- It would be helpful to clearly state at the beginning of the paper that the proposed method applies only to the cooperative multi-agent setting.\n- The provided code repository lacks a README file, making it difficult to install dependencies and reproduce the results.\n\nminors:\n- line 149, linear bandit problem []\n- line 178, UCB1 and LinUCB []\n\nQUESTIONS:\n- Are all agents considered within a single MCTS process? If so, is MCTS applied during test time? If it is used, does the method still comply with the Centralized Training and Decentralized Execution (CTDE) paradigm?\n- The paper introduces a strongly convex and $\\mu$-smooth distance measure f, and discusses its benefits. However, it is unclear how f achieves the desired behavior. For example, line 186 states: “For higher observed rewards X_t that are likely optimal, the distance measure f will have a larger acceleration”. Does this behavior also apply to higher observed rewards that are not optimal? Further clarification and justification would strengthen the argument.\n- In high-dimensional multi-agent tasks, will the same state be encountered multiple times? I’m curious whether the MCTS is truly maintaining a tree structure, or just some independent trajectories, especially in this multi-agent setting.\n- see Weaknesses.\n\nLIMITATIONS:\nThe authors acknowledge the limitations of their work and outline directions for future improvement.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the problem of the exponential action space blowup in multi-agent model based RL algorithms that rely on tree-search (like MAzero). The authors learn a low dimensional linear approximation of the reward function (for every hidden state).  The authors then propose using the UCB algorithm for a linear bandit, to select actions for expanding the tree search. This strikes a balance between exploration of new actions and exploitation of high rewarding actions, instead of expanding the tree for all possible actions. In benchmark tasks like MatGame and SMAC, the algorithm shows consistent improvement over baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The strengths of the paper are a clear and detailed theoretical analysis of the MALinZero algorithm.\n2) The empirical performance on all benchmark tasks considered seem strong when compared to the baselines.\n\nWeaknesses:\n1) The paper seems to have a lot of technical details about UCB, linear bandits and regret analysis. This makes it a little difficult to read for practitioners / non-theory researchers. It also makes it less clear if the main contributions are theoretical /  empirical. I do not work in multi-agent RL /  multi-agent RL theory, so it is a bit difficult to gauge how strong the theoretical / empirical results are.\n\n2) How relevant do these algorithms remains when compute becomes orders of magnitude cheaper? Every agent can consider other agents as a part of the environment and use algorithms like sampled muzero to plan.\n\nQUESTIONS:\n1) Do you need to learn a reward function new approximation for different latent states? How computationally efficient is this? How is data generated to learn this function?\n\n2) At every-step during expansion, you need to run the linUCB algorithm to select an action, what additional computational and memory overheads does this cause?\n\n3) What happens when compute becomes orders of magnitude cheaper? Do you think the low rank approximation will still achieve better performance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MALinZero, a novel algorithm for multi-agent planning that incorporates low-dimensional representations of joint-action returns into Monte Carlo Tree Search (MCTS). The method models joint action values as a contextual linear bandit problem and derives a variant of Upper Confidence Bounds for trees, dubbed LinUCT, to guide efficient exploration. The paper presents regret analysis, derives an approximate algorithm to minimize it, an evaluates it over against existing baselines on MatGame, SMAC, and SMACv2.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper is very well written, ideas and results are presented clearly, and related work is well covered. \nI am not very familiar with the subject, though, so I may not be aware of missing references.\n\nWhile the idea of low-dimensional decomposition has been explored in previous MARL work (the authors cite QMIX and VD), the idea of transforming joint action return estimation in MCTS into a contextual linear bandit problem seems novel to me, and useful especially in MA problems where the joint action space grows exponentially. \n\n- How does MALinZero differs in terms of how it uses confidence bounds for exploration in contrast to value factorization? As someone not very familiar with the subject, I would appreciate a concise and clear conceptual comparison against baselines, and a recap of contributions at the end of the paper.\n\n- How does MALinZero scale to larger problems? The problem of scalability is briefly mentioned in the related work section, but not addressed later. Is MALinZero better than baselines in this regard? \n\n- How would MALinZero perform with different losses? E.g., Huber or exponential.\n\nThe authors also provide an extensive empirical evaluation, showing that MALinZero consistently outperforms all baselines across linear and nonlinear reward environments. \n\n- All algorithms are evaluated over 3 seeds only. I am not familiar with the environments, but in classic RL 3 seeds are usually unacceptable. I strongly encourage the authors to run more seeds. Also, please indicate what the shaded are and the +/- in the table are (standard deviation? confidence interval?).\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nSee above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}