Title: Effective Policy Learning for Multi-Agent Online Coordination Beyond Submodular Objectives
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Agent Coordination

Policy Learning

Continuous Relaxation

Online Non-Submodular Maximization

Surrogate Functions
Abstract: In this paper, we present two effective  policy learning algorithms for multi-agent online coordination(MA-OC) problem. The first one, **MA-SPL**,  not only can achieve the optimal $(1-\frac{c}{e})$-approximation guarantee for the MA-OC problem with submodular objectives but also can handle the unexplored  $\alpha$-weakly DR-submodular and $(\gamma,\beta)$-weakly submodular scenarios, where $c$ is the curvature of the investigated submodular functions, $\alpha$ denotes the diminishing-return(DR) ratio and the tuple$(\gamma,\beta)$ represents the submodularity ratios.  Subsequently, in order to reduce the reliance on the unknown parameters $\alpha,\gamma,\beta$ inherent in the **MA-SPL** algorithm, we then introduce the second online algorithm named **MA-MPL**. This **MA-MPL** algorithm is entirely *parameter-free* and simultaneously can maintain the same approximation ratio as the first  **MA-SPL** algorithm. The core of our **MA-SPL** and **MA-MPL** algorithms is a novel continuous-relaxation technique term as policy-based continuous extension. Compared with the well-established multi-linear extension, a notable advantage of this new policy-based continuous extension is its ability to provide a lossless rounding scheme for any set function, thereby enabling us to tackle the challenging  weakly submodular objective functions. Finally, extensive simulations are conducted to demonstrate the effectiveness of our proposed algorithms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deals with a multi-player cooperative online learning problem with monotone submodular or non-submodular objectives. The ground set is partitioned into $n$ groups, and each player $i \\in [n]$ chooses one element from the $i$th group in each round. The goal is to upper-bound on approximate dynamic regret with approximation ratio $\\rho$. The paper's contributions are three-fold. The first one is to introduce a generalization of the multilinear extension, which they call policy-based continuous extension. The second one is to apply the boosting technique to improve the approximation ratio (for example, to $1-c/e$ for monotone submodular functions with curvature $c$). The third one is to combine this offline technique with existing dynamic regret minimization algorithms to obtain dynamic regret upper bounds. The proposed algorithm are also empirically validated.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Among this paper's contributions, the boosting technique is most interesting to me. This was originally proposed by Zhang, Deng, Chen, Hu, and Yang in ICML 2022 for continuous submodular maximization. This method designs an optimal weight and maximizes a surrogate function that is the weighted average of the objective function. This paper applies the boosting technique to curvature-bounded submodular functions, $\\gamma$-weakly DR-submodular functions, and $(\\gamma, \\beta)$-weakly submodular functions to obtain good approximation ratio guarantees.\n- The problem setting of multi-agent online coordination is interesting and opens the door to new applications of submodular maximization, although it was already proposed in Zhang, Wan, Yang, Shen, and Tao in ICLR 2025. If the authors provided more practical applications in which parameters $c$, $\\gamma$, and $\\beta$ can be bounded, the significance of this paper is clearer.\n\n---\nWeaknesses:\n- I am not a big fan of the concept of approximate regret. Since the solution in each round can exceed the approximation ratio $\\rho$, theoretical guarantees on approximate regret do not really represent the \"regret\" for not playing the optimal action in hindsight. It just guarantees convergence of the approximation ratio of the total reward in hindsight. This paper considers dynamic regret, and its interpretation is more complicated.\n- The authors claim that the policy-based continuous extension is their contribution. This is a generalization of the multilinear extension to partition matroid constraints (or $k$-submodular functions in context). However, the same idea already appeared in existing papers. For example, Sahin, Bian, Buhmann, and Krause called it \"generalized multilinear extension\" and used it for submodular maximization on the integer lattice. Wang and Zhou used in ArXiv 2021 paper for $k$-submodular maximization.\n- I understand that all the factors of this paper (multi-agent coordination, dynamics regret minimization, and the boosting technique) are independently interesting to some extent, but these combinations make the paper complicated and blur the paper's focus in my opinion.\n\nQUESTIONS:\nPlease see the weaknesses discussed above. If my understanding contains serious mistakes, I would be happy to reconsider my score. For example, the proposed multilinear extension is essentially different from existing ones, I would appreciate your clarifications.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the multi-agent online coordination (MA-OC) problem where multiple agents jointly select actions to maximize a time-varying, submodular or weakly submodular reward function under combinatorial constraints. The authors introduce a new relaxation called the policy-based continuous extension and develop two algorithms:\n\n- MA-SPL: An online learning algorithm with tight approximation ratios for monotone submodular, weakly submodular, and weakly DR-submodular objectives. \n\n- MA-MPL: A parameter-free variant that does not require knowledge of problem-specific constants (like curvature or weak submodularity parameters), yet still achieves similar performance.\n\nThe algorithms come with theoretical dynamic $\\rho$-regret bounds and approximation guarantees. Empirical evaluations on multi-target tracking scenarios demonstrate that both methods outperform existing baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- The introduction of the novel ``policy-based continuous extension'' is a significant contribution. Its ability to provide a lossless rounding scheme for weakly submodular functions (not just submodular ones) is a clear advantage over traditional existing multi-linear extensions and addresses a critical limitation in the field. \n- Theoretical guarantees are strong and tight. The paper provides rigorous theoretical analysis for both MA-SPL and MA-MPL algorithms, demonstrating near-optimal approximation ratios and dynamic regret bounds. \n- The parameter-free MA-MPL algorithm is novel. The development of the MA-MPL algorithm to eliminate the dependence on possibly unknown parameters (such as $\\alpha$, $\\beta$, $\\gamma$) is also practical.  In real-world scenarios, these parameters are often difficult to estimate, making the parameter-free approach a valuable direction to study.\n- Overall, the manuscript is overall nicely written and easy to follow. Moreover, the theoretical analysis is rigorous, and the proofs look correct and sound, built upon existing results and tools on submodular optimization and online optimization. \n\nWeaknesses\n\n- The most significant weakness and largest concern with this paper lies in the assumption on access to a local feedback oracle. The paper assumes access to local marginal feedback oracles $ \\mathcal{Q}_{i}^{t} $ for marginal gains which are further to be used for calculating surrogate gradients (Lines 111--115). While the authors claim that such a local marginal feedback oracle is ``common'' (Lines 116--118), and it is used as a core assumption in [111] cited by the authors, which is the work of this manuscript built upon, more discussions on how the oracles are connected to real-world scenarios and used to model them (for example, how the limitation of UAV's perception range can be captured by and modeled as local marginal feedback oracles, and possibly with one more concrete example beyond UAVs) and how such oracles might be realized and learned in practice (instead of merely citing [23]) would be helpful and could enhance clarity for a broader audience.\n- Although the novelty of the manuscript is acknowledged, it seems to me that the novelty of the parameter-dependent algorithm MA-SPL is rather limited. The algorithm mostly follows the structure of the algorithm MA-OSEA in [111] and inherits the results from it, such as action sampling, information exchange, information aggregation, surrogate gradient estimation and policy update. The only difference is the use of policy-based continuous extension instead of multi-linear extensions. Hence, the framework of the algorithm MA-SPL overall isn't entirely novel, a fact that should be clearly acknowledged in the manuscript. \n- While the algorithm MA-MPL is parameter-free, the paper acknowledges that it ``typically incurs greater communication complexity'' (Line 1505 and Table I) in the order of $\\mathcal{O}(T^{5/2})$. Although it states future work will explore reducing this, a more in-depth discussion or initial analysis of this complexity compared to MA-SPL or other methods would strengthen the paper. It is also worthwhile to highlight the trade-off between a parameter-free algorithm and a greater running time in the manuscript. \n- The experimental validation is limited to simulated environments for UAV target tracking. \n    Evaluation on real-world systems or physical hardware (e.g., robotic coordination) would strengthen the empirical claims and showcase practical viability. Moreover, for weakly submodular objectives (Figure 1d-1f, Appendix A.2), the comparison is primarily against a random baseline, which seems to be a weak and low benchmark. While the authors state that no previous work explores MA-OC with weakly submodular objectives (Line 836), it would be beneficial to discuss if there are any offline weakly submodular maximization algorithms that could conceptually be adapted or contrasted (even if not directly online) to provide more context for the performance of the proposed algorithm.\n\nQUESTIONS:\n- Regarding the first point in weakness about the assumption on local feedback oracles, could the authors explain and discuss how the oracle can exactly and effectively model the limitation of UAV's perception range (Lines 106--110) and how such oracles might be realized and learned in practice?\n\n- Could the authors elaborate on the practical implications of the increased communication complexity of MA-MPL compared to MA-SPL? Are there specific real-world scenarios where this trade-off would make MA-MPL less desirable despite its parameter-free nature?\n        \n- The authors claim that the approximation ratio achieved by the algorithms MA-SPL and MA-MPL is tight. \n    Can the authors comment on the tightness of the regret bound achieved by MA-SPL and MA-MPL as well? \n\n- The authors claim that ``new techniques are required to verify Theorem 3'' (Line 284). \n    Can the authors briefly summarize the new techniques required in the proof for Theorem 3? \n\n- In Figures 1(d)--1(f), the x-axis \"Time Step\" goes up to $500$, and the average utility does not seem to converge, nor the curves tend to become flat. \n    By contrast, in Figure 1(a)-1(c), the time step goes up to 1200, and the average utility converges to a steady value. \n    Is there a specific reason why the authors did not use a larger time scale and let the average utility converge for Figures 1(d)--1(f)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a novel continuous-relaxation technique to achieve multi-agent cooperation under submodular and weakly submodular objectives, and they theoretically prove the optimality of the algorithm while analyzing performance metrics such as communication cost and regret. The paper is well-structured and clearly written. However, further discussion is needed regarding problem modeling, assumptions, and the practical feasibility and deployability of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\nThe authors have provided thorough modeling, algorithm design, and theoretical analysis for multi-agent cooperation problems with submodular and weakly submodular objective functions. The paper is well-written, the algorithms are comprehensive, and the proofs are convincing.\n\n**Weaknesses**\n\n1. I have some concerns regarding the modeling of the problem. The author requires the action sets of each agent to be disjoint, which is rather puzzling, since identical UAVs or UGVs would typically have the same action set. Furthermore, the stated problem (1) appears to be independent of the system state, whereas, in my view, the UAVs’ tracking tasks are highly dependent on target locations and related state information. In addition, if there are two equivalent decisions at time t (for instance, agent a moving right or agent b moving left), it is unclear which will be selected. Has the heterogeneity in the decision outcomes of different agents been fully considered?\n2. Although the paper focuses primarily on the theoretical aspects, it would be beneficial to include some discussion on practical considerations, such as the feasibility of communication, the possibility of asynchronous operations, and computational feasibility under constrained onboard resources, in order to strengthen the persuasiveness of the algorithm’s deploy ability.\n\nQUESTIONS:\n1. How do you justify the requirement that each agent's action set must be disjoint, especially in scenarios where agents (such as identical UAVs or UGVs) would naturally share similar or identical action sets?\n2. Could you discuss the practical feasibility of your approach in terms of communication requirements, asynchronous operation, and onboard computational constraints to better demonstrate the deployability of your algorithm?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 1, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the multi-agent online coordination problem, where multiple agent choose their actions in a finite action set to maximize a set function. This is an optimization problem with discrete support. The authors propose to learn stochastic polices for the agents so that the problem is converted to a continuous optimization problem, and this framework is named policy-based continuous-relaxation. Under this framework, the authors characterizes the suboptimality of stationary points of the relaxed continuous function (Theorem 2). Moreover, a surrogate objective function is proposed to improve the suboptimality gap (Theorem 3). Effectiveness of the algorithm is verified by simulations.\n\nSTRENGTHS AND WEAKNESSES:\n**This is a nice submission with the following strengths:**\n- A new continuous relaxation is proposed for set function optimization, which enables provably efficient optimization for weakly submodular functions. As a comparison, previous relaxation approaches relies heavily on the ``strong'' submodular assumption.\n- The authors theoretically characterize suboptimality of stationary points and proposes a method to improve suboptimality. \n- An end-to-end optimization algorithm is proposed and analyzed.\n- Empirical evaluation verify the effectiveness of the algorithm.\n\n**That said, the reviewer do have the following concerns/question:**\n- After policy-based relaxation, the problem is essentially reduced to a multi-agent identical interest game (a subset of potential games (PG)). There exists a huge amount of literature that designs provably efficient algorithms that learn nash equilibrium (a subset of stationary points) and provide price of anarchy lower bounds (or suboptimality in the context of this paper) for PGs. Are results for PGs also applicable to this setting? If so, how are the results in this paper compare to results for potential games? \n- Consider the surrogate function In theorem 3. Are stationary points (denoted by $x^s$) of the surrogate function also stationary points of the policy-based relaxed objective $F$? If not, what will happen if we start from $x^s$ and do gradient descent using gradients of function $F$? Will this strategy increase or decrease the sub-optimality gap? In case this is hard to characterize theoretically, I would like to see at least come empirical evaluations and discussions.\n- In line 5 of Algorithm 1, the authors normalize the policy and sample actions. Is it necessary for the empirical performance or theoretical analysis? As far as I am concerned, function $F^s$ seems to be supported on the entire probability simplex and gradient descent seems to work without this projection step. If this step is indeed necessary either theoretically or empirically, ablation simulation studies and discussions would be helpful. \n\n**Minor comments**:\n- The current abstract has involved too many specific terms (e.g. submodularity ratio, weakly submodularity ratio) for the general audience to understand. It would be helpful to talk about them more high-levelly.\n- It would be helpful to give a high-level introduction about how previous extension, e.g. multi-linear extension, works.\n\nQUESTIONS:\nQuestions are included in ``Strengths and Weaknesses''.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}