Dependency Structure Search Bayesian Optimization for Decision Making Models
Authors: Mohit Rajpal
Lac Gia Tran
Yehong Zhang
Bryan Kian Hsiang Low
Venue: Accepted by TMLR ()

Abstract: Many approaches for optimizing decision making models rely on gradient based methods requiring informative feedback from the environment. However, in the case where such feedback is sparse or uninformative, such approaches may result in poor performance.
Derivative-free approaches such as Bayesian Optimization mitigate the dependency on the quality of gradient feedback, but are known to scale poorly in the high-dimension setting of complex decision making models. This problem is exacerbated if the model requires interactions between several agents cooperating to accomplish a shared goal.
To address the dimensionality challenge, we propose a compact multi-layered architecture modeling the dynamics of agent interactions through the concept of role.
We introduce Dependency Structure Search Bayesian Optimization to efficiently optimize the multi-layered architecture parameterized by a large number of parameters, and show an improved regret bound. Our approach shows strong empirical results under malformed or sparse reward.

Published in Transactions on Machine Learning Research (10/2024)
Dependency Structure Search Bayesian Optimization for Decision Making Models
Mohit Rajpal
mohitr@comp. wwtpmmpppp......i.om...... edu..g
Deportment of Computer Science National University of Singapore
Lac Gia Tran
trunlac@ccomp.uus www..u.....uuuu.......
Department of Computer Science National University of Singapore
Yehong Zhang Peng Cheng Lab
zhangyh02@pcl.occcom
Bryan Kian Hsiang Low
lowkh@comp. nus. edu.sg
Deportment of Computer Science
National Universily of Singupore
Reviewed on OpenReview: https://ppauevviee.ettt///oo htt.://.ewe...otm//////////////a
Abstract
Many approaches for optimizing decision making models rely on gradient based methods requiring informative feedback from the environment. However, the case where such feedback sparse or uninformative, such approaches may result in poor performance. Derivative-Tree approaches such as Bayesian Optimization mitigate the dependency on the quality of gradient feedback, but are known scale poorly in the high-dimension setting of complex decision making models. This problem is exacerbated if the model requires interactions between several agents cooperating to accomplish a shared goal. To address the dimensionality challenge, propose a compact multi-layered architecture modeling the dynamics of agent interactions through the concept of role. We introduce Dependency Structure Search Bayesian Optimization to efficiently optimize the multi-layered architecture parameterized by a large number of parameters, and show an improved regret bound. Our approach shows strong empirical results under malformed or sparse reward.
1 Introduction
Decision Making Models choose sequences of actions to accomplish a goal. Multi-Agent Decision Making Modcls choose actions for multiple agents working together towards a shared goal. Multi-Agent Reinforcement Learning (MARL) has emerged as a competitive approach for optimizing Decision Making Models in the multi-agent setting [ MARL optimizes policy under the partially observable Markov Decision Process (POMDP) framework, where decision making happens in an envirormmntt determined by set of possible states and actions, and the reward for action is conditioned upon the partially observable state of the environment. A policy forms a set of decision-making rules capturing the most rewarding actions in a given state. MARL utilizes gradient-based methods requiring informative gradients to make progress. This approach benefits from dense reward, which allows reinforcement learning methods to infer a causal relationship between individual actions and their corresponding reward. This feedback may not be present in the scenario sparse
We include overview of approaches in Decision Making Models Section

---

Published in Transactions on Machine Learning Research (10/2024)
reward Pathak et al. 2017 Qian & Yu 2021 In addition, tradient-based methods are susceptible to falling into local maxima.
In contrast to optimization by MARL, Bayesian Optimization (no) offers an alternative approach to policy optimization. Since BO is a gradient-free optimizer capable of searching globally, applying BO to multi-agent policy search (MAPS) both ensures global searching of the policy, and overcomes poor gradient behavior the reward function (Qian & Yu, 2021). The chief challenge in BO for MAPS is the high dimensionality of complex multi-agent interactions.
A significant degree of high-dimensional multi-agent interactions exist in MAPS. For example, considering an autonomous drone delivery system, several agents (i.e., drones) must work together to maximize the throughput of deliveries. In doing 80, these agents may separate themsel ves into different roles, for example, long-distanee or short-distance deliveries, The optimal policy for each role may be significantly different due to distances to recharging base stations (c.g., drones must conserve battery). In forming the optimal policy, the interaction between agents must be considered to both optimally divide the lask between the drones, well as coordinate actions between drones (e.g., collision avoidance). These interactions may change over time. For example, drone must avoid collision with nearby drones, which changes as it moves through the environment. With many agents, these interactions become more complex.
However, we propose the usage of Bo for MAPS on memory-constraineed devices which necessitates very compact policies which enables the possibility of overcoming the above limitation. In the context of memory- constrained devices such as Internet of Things (IoT) devices (Merenda et al.] 2020), small policies must be used. Secondly, in environments with sparse reward feedback, training these networks with RI. presents significant challenges due to unhelpful policy gradicnts. Finally, the possibility of globally optimizing a compact policy for memory-cconstrained systems is appealing due to its strong performance guarantees.
To allow for the construction of compact policies, we utilize specific multi-agent abstractions of role and role interaction. In role-based multi-agent interactions, an agent's policy depends on its current role and sparse interactions with other agents. By simplifying the policy space with these abstractions, we increase its tractability for global optimization by bo and inherit the strong empirical performance demonstrated by these approaches. We realize this simplification of the policy space by exprossing the role abstraction and role interaction abstractions as immutable portions of the policy space, which not searched over during policy optimization. To achieve this, we use a higher-order model (HOM) which generates a policy model. The HOM is divided into immutable instructions (i.e., algorithms) corresponding to the abstractions of the role and role interaction and mutable parameters that are used to generate (GEN) a policy model during evaluation.
To optimize our proposed HOM, we specialize BO by exploiting task-specific structures. A promising avenue of High-dimensional Bayesian Optimization (UDBO) is through additive decomposition. Additive decomposition separates high-dimensional optimization problem into several independent low-dimensional sub-problems (Duvenaud et al. 2011 Kandasamy et al. 2015 These sub-problems are independently solved thus reducing the complexity of high dimensional optimization. However, a significant challenge in additive decomposition is learning the independence struclure which is unknown a-priori. Learning the additive decomposition is accomplished using stochastic sampling such as Gibbs sampling (Kandasamy et al. 2015 Rolland et al. 2018 Han et al. 2020) which is known to have poor performance in high dimensions (Johnson
ct al. 2013 Barbos et al. 2017)).
In our work, we overcome this shortcoming by observing the GEN process of the HOM. In particular, can measure surrogate Hessian during the GEN process which significantly simplifies the task of learning the additive structure. This surrogate Hessian informs the dependency structure of the optimization problem due to the equivalence between a zero Hessian value, and independence between dimensions due to the linearity of addition. We term this approach Dependency Structure Search GP-UCB (DSS-CP-UCB) and visualize our approach in Fig. 2 Our proposed BO approach is also applicable to policy-search in the single-agent setting, showing its general-purpose applicability in Decision Making Models. In this work, we make the following contributions:
We propose a paramcter-offeient HOM for MAPS which is both expressive and compact. Our approach is made feasible by using specific abstractions of roles and role interactions.
2

---

Published in Transactions on Machine Learning Research (10/2024)
We propose DSS-GP-UCB, a variant of BO that simplifies the learning of dependency structure and provides strong regret guarantees which scale with ((log(D)) under reasonable assumptions.
We validate our approach on several multi-agent benchmarks and show our approach outperforms related works for compact models fit for reward behavior in the reward function in multiple settings showing its elfectiveness Decision Making Models both in the single-agent and multi-agent settings.
scenarios. Our DSS-GP-UCB also overcomes sparse
2 Background
Bayesian Optimization: Bayesian optimization (BO) involves sequentially maximizing an unknown objective function V O - R. In each iteration ( t= -=,,..., an input query 0g is evaluated to yield a noisy observation y/ _ v(()) + e with i. d. Gaussian noise e NN, x²). BO selects input queries to approach the global maximizer 0r A arg maxage v()) as rapidly as possible. This is achieved by minimizing cumulaatiee regret Rr & Σi= r(0)), where r(f,)= a v(0*)- - v(t)). Cumulative regret is a key performance metric of Bo methods.
The probability distribution of v is modeled by a Gaussian process (GP), denoted GP (n(O), k(0,0') that is, every finite subset of {u(0)]ace follows multivariate Gaussian distribution (Rasmussen & Williams 2006). A GP is fully specified by its prior mean uu0) and covariance k(0,, 0*) for all 0, 0' O, which are, respectively, assumed w.l.o.g. to be p(0)= = 0 and k(0,00 0') S 1. Given a vector yr _ [_]][_ of noisy observations from
evaluating v at input queries 0,,..., 0r E θ after T iterations, the GP posterior probability distribution of v at some input 0 E
θ a Gaussian with the following posterior mean (0) and variance (o/13(0):
(0) & (0) T (KŞ + o I yr,
2 (0) - k(0, - (0) (KŞ + - [or]
(1)
where K A [k(,,, 0z. 11......... and (0) A 1).,).) In each iteration of BO, an input query 8g θ is selected to maximize the GP-UCB acquisition function, 0, - arg maxege (c-1(0) + (B,or-1 (Srinivas el al. 2010) where Be follows well defined pattern.
3 Related work
Decision Making Models: Decision Making Models (Rizk et al., 2018 Roijers et al. 2013 determine actions taken by agent agents in order to achieve goal. We focus the POMDP setting and optimizing
policy to accumulate maximum reward while interacting with a partially observable environment (Shani et al.) 2013 Many approaches exist which can be broadly categorized into direct policy search and reinforcement
learning methods. Direct policy search (Heidrich-
& Igel 2008 Lizotte et al., 2007
2017
Papavasileiou et al., 2021 Wierstra el. al.] 2008) searches the policy space in some efficient manner.
Reinforcement learning (Arulkumaran et al. 2017 Fujimoto et al. 2018 Haarnoja et al.) [018 Lillicrap
ct al. 2015 Lowe et al. 2017 Mnih et al.] 2015 Schulman et al. 2017) starts with randomly initialized
policy and reinforces rewarding behavior pallerns to improve the policy.
Bayesian Optimization for Decision Making Models: во has been utilized for direct policy search in
the low dimensional setting (Lizotte et al. 2007 Wilson et al. 2014 Marco et al. 2016 Martinez- Cantin 2017 von Rohr et al.) 2018). However, these approaches have not scaled to the high dimensional setting. In more recent works, BO has been utilized to aid in local search methods similar to reinforcement learning (Akrour et al., 2017 Eriksson et al. 2019aa] Wang al. (0200a) Fröhlich al.) [202]] Muller al. 2021). However, these approaches require evaluation of an inordinate number of policies typical of local search methods and do not provide regret guarantees. Recently, combinations of local and global search methods have been proposed (McLeod et al. 2018 Shekhar & Javidi 2021). However, these approaches rely on informative and useful gradient information and have not been shown to scale to the high dimensional setting.
MARL for multi-agent decision making; A well-known approach for cooperative MARL is a combination of centralized training and dccentralized exccution (CTDE) (Olichock et al.] 2008). The multi-agent interactions of CTDE methods can be implicitly captured by learning approximate models of other agents
(Lowe et. al. 2017
Foerster et. al. 2018: or decomposing global rewards (Sunehag et al.] 2017 Rashid et al.
3

---

Published in Transactions on Machine Learning Research (10/2024)
2018
2019 However, these methods do not focus how interactions performed between agents. In MARI, the concept of role is often leveraged to enhance the flexibility of behavioral representation while controlling the complexity of the design of agents (Lhaksmana et al. 2018 Wang et al. 2020b 2021b;
Son et al.
Li et al.; ) Our approach is related to the study of (Le et al., 2017] where the interactions are also captured by role assignment. However, the approach operates on an imitation learning scenario, and the role assignment depends on the heuristic from domaín knowledge. Another related field is Comm-MARI (Zhu ett al., 2022 Shao al., 2022; Liu et al.) 2020 Peng et al.) 2017 Das et al., 2019 Singh et al. 2019), where agents are allowed to communicate during policy execution to jointly decide
action. In contrast, our approach utilizes both abstractions of role and role interaction in HOM for a decision making model.
4
Design
We consider the problem of learning the joint policy of a set of n agents working cooperatively to solve a common task. During each interaction with the environment, each agent is associated with state st E S with the global state represented A |s*) Each agent cooperatively chooses action a A with the global action represented by - [a la]]:= Each state, action pair is associated with reward function: p(s, In order to achieve the common task, a policy parameterized by 0: x² ف A governs the action taken by the agents, after observing state s E S. The goal of RL is to learn the optimal policy parameters that maximizes the accumulation of rewards during a predefined number of interactions with the environment
2 u()) In contrast to RI, which receives feedback on the reward of action with every interaction, treat v()) as opaque function measuring the value of policy. We utilize Bo to optimize 0 using solely the accumulated reward, v(f), as feedback from the environment.
4.1 Architectural design
To achieve a compact and tractable policy space, we consider policies under the useful abstractions role and role inferaction. These abstractions have consistently shown strong performance multi-agent tasks. Therefore, we can simplify the policy space by limiting to only policies using these abstractions, but still have powerful and expressive policies suitable for multi-agent systems.
As role and role interaction are immutable abstractions within our policy space, we express them as static algorithms which are not searched over during policy optimization. These algorithms take as input parameters which are mutable and scarched over during policy optimization. This combination of immutable instructions, and mutable parameters reduces the size of the search space conform to the role and role interaction abstractions.
3 yet is still able to express policies which
We term this approach a higher-order model (Ном) which generates (GEN) the model using instroctions and parameters into a policy model during evaluation. This HOM is separated into role assignment, and role interaction stages. We visualize an overview of this approach in Fig. I left. The HOM parameters are interpreted in context of the current state by the instructions (Alg. 1, Alg, 2, Alg. 3) of the ноо to form the policy model which dictates the resultant action. In our work, each HOM component of role assignment and role interaction is implemented as a neural network.
4.2 Role assignment
Following the success of role based collaboration in multi-agent. systems, we assume the interaction and decision making of each agent is governed by its assigned role. For example, in drone delivery, roles could be short-distance deliveries, and long-disse dist-ncee deliveries. In filling these roles, the state of each of the agents are considered. E.g., drone with low battery may be limited to only performing short-distance deliveries. A straightforward approach to implement role based interaction is to permute agents into an equivalent number of roles g We assume that an optimal policy can be decomposed as follows:
FFurther RL overview can be found in Arulkumaann al. 2017
Thiis approach to efficiency is similar in spirit work Lee 1986 4This is common assumption multi-agent systems, see, e.g.. el.
4

---

Published in Transactions on Machine Learning Research (10/2024)
T(al ,...,,"" a" "1s'....."") a (a(()) ,..,, a (a.,a()...,
(2)
where a a permutation function dependent on the state, s!....,s" s" Our approach to role assignment is simple and general purpose, which is also well studied and theoretically principled.
To capture this behavior, we utilize a per role affinity function: (+) which is the affinity to take on role and is paramcterized by 0,, This function evaluates the affinity of agent l taking on role using the state of agent: s². The optimal permutation maximizes the total affinity of an assignment: Σ (st(())()) where a represents a permutation. This problem can be efficiently solved using the Hungarian algorithm (Kuhn 1955). We integrate the Hungarian algorithm in our IIOM approach during the CEN process. We formalize this Algorithm [
which forms the instructions in the role assignment HOM.
Given Algorithm 1 during GEN process, the agents' state, s! ,....., s" is contextually interpreted yield permutation model: Going forward, we consider the problem of determining the joint policy (a an(a) (o()),...s(())) which enables collaborative interactions.
4.3 Role interaction
Capturing multiple roles working logether is an important part of an effective multi-agent policy. For example in drone delivery, drones must both divide the available task among themselves, as well as use collision avoidance while executing deliveries. Modeling role interactions must accomplish two goals. Firstly, agent interactions may change over time. For example collision avoidance strategies involve the closest drones which change as the drone moves within the environment. Secondly, elficient parameterization ion is needed as the number of interactions can scale exponentially due to considering interactions between many agents.
To overcome these challenges, we propose HOM which generates (GEN) graphical model. The us- age of graphical model decomposes the exponen- tially scaling interaction problem into a pairwise interaction model, along with a message passing ap- proach to facilitate complex interactions between many agents The GEN process is conditioned on
GEN 100, 8 8.
g
Role Assigmennt
(n(n)
Role Interaction
8..
the agents' state, thus enabling dynamic role in-
teractions; in addition the GEN process allows for more compact policy space with far fewer parameters. The resultant generated graphical model captures the state-dependent interaction between roles and yields the resultant actions for each role. After GEN, the interaction between roles are captured by the resultant conditional random field. This is presented in Fig. right. The MRF (Markov Random Field) represents arbitrary undirected connectivity between nodes a ,.... aa()) which is denoted by G. This connectivity allows different roles to collaborate together to determine the joint action. To generate graphical models of the above form, our HOM uses edge affinity functions, Afxx(.), (.), which enables dynamic arbitrary connectivity between roles. For all pairs of roles with state, sa(1), sa(() an edge is generated if the affinity between these two states is sufficiently high (i.e., > 0). This dynamic edge generation approach overcomes the quadratic parameter scaling if all pairs of agents were separately modelled. The graphical model GEN process is presented in Algorithm 2 which yields a graphical model.
Graph N
2gmm 0g.0
a a
MPNN
MRP
Figure 1: Left: HOM architecture. GEN uses 0, and 0g during evaluation to yicld model which represents the policy. O, and 0g are optimized by BO. Right: Inferring a" given sa.
To cooperatively determine set of actions for roles given the graphical model, we perform inference over the graphical model presented in Fig, | using Message Passing Neural Networks Giimer et al. 2017) (MPNN). We present iterative message passing rules to map from s to a":
We refer readers
Wang al. 2013 for additional overview on graphical models.
5

---

Published in Transactions on Machine Learning Research (10/2024)
(()) Σ M 12i0 po(i) & U (sa(), 12 mfff): a m+1
h2(00)
(3)
((E)
where M is the message function parameterized by ggn which enables interaction between connected nodes, U is the action update function parameterized by fg. which updates the node's internal hidden state conditioned on the messages reccived, and No(i) denotes the neighbors of a(i). The message passing procedure allows for cooperative determination of all roles' actions using pairwise message passing, Roles which are not immediate neighbors each other influence each other's behavior through intermediary connecting nodes. The message passing procedure concludes after T iterations of message passing with the policy actions indicated by the hidden states, (0) v=1,...,m
Finally, Algorithm 3 drives the GEN process. The GEN process consists of permuting agents into roles, creating the graphical model to enable interactions between agents taking on their respective roles, and finally performing inference over the graphical model using a MPNN.
4.4 Additive decomposition
Although our HOM policy representation is compact, it is still of significant dimensionality which makes optimization with Bo difficult. HDBO challenging due to the curse dimensionality with common kernels such as Matern or RBF This curse of dimensionality stems directly from the difficulty of finding the global optima of a high-dimensional function (e.g., a value function v(0) determining the value of a policy some unknown environment). A commmn technique to overcome this is through assuming additive structural decomposition on v: v(0) è EM (B(+) where (( are independent functions, and g E e (Duvenaud ct al. 2011). The additive decomposition simplifies a high-dimensional optimization problem since the optima of function constructed through addition of subfunctions can be found by independently optimizing each subfunction visualized in Fig. 2 In the context of BO, additive decomposition significantly simplifies the optimization problem due to the properties of Multivariate Gaussian variables.
In additive decomposition we denote the domain of the optimization problem, θ - @ x x... x O for some dimensionality D, that the domain is constructed through the Cartesian product of each of its dimensions. Fach subfunction to optimize, (0), corresponds to subdomain restricted to some subset of these dimensions, (1) C (O),..., Oº}. Typically, it is assumed that each ( is of low dimensionality (i.e., v(() is defined on only a few di- mensions for each i). This structural assumption is combined with the assumption that each (1)) is sampled from a GP. Due the properties of Mul- tivariate Gaussians, if v(() ~ GP 40 (B(+) g00)
maxa [6]]]]]]] jo
Hessian
+
Hessian Aware
b max or l0m700
0c
@
max y 0r
Dependency Graph
GP-UCB
Figure 2: Left, above, plot of f(x, v) = x"; below, plot of f(x,y) x + y. The curvature of additively con- structed functions is zero; non-zero curvature indicates dependency among input variables. Right, examining the Hessian learns the dependency structure which decomposes complex problems into simpler problems solved by GP-UCB.
then GP Σ, (((")((()) (0((),)0))) Rassmussen &
Williams 2006). which follows from the addition of two Gaussian random variables is also a Gaussian random variable. This assumption decomposes a high dimensional GP surrogate model of into set of many low dimensional GPs, which casier to jointly learn and optimize.
To contextualize an additive decomposition, represent the decomposition by dependency graph between the dimensions: Ga A (Va, Ea) where V & {0l (ol....,c e"} and E, {(0", θ²) I a. E O for some i}. A simple decomposition of an additive function and its associated dependency graph is visualized in Fig. 2
A parallel in HDBO is of computational efficiency of acquisition which outside the scope of this work. We refer readers the works of Mutay Krause (0018) Wilson al. (2020), and A Ament & Gomes (022).

---

Published in Transactions on Machine Learning Research (10/2024)
Algorithm 1 Assggmmntn
Algorithm 2 RoleInteraction
Require: s' ,....,55" 1: return arg max Ei= (
Require: s²(1), su(1)...., 1: for + I,...,n do for L 4 1411...,5 do Edge affinities. if (80,0(80(1) sa(() 0 then Na(t), (ppend(((()) 5: return Na(1), "all))...,N. Na(n)
2:
Algorithm 4 DSS-GP-UCB
Require: v. H.
1: for I,..., To do
B Sample Hessian To x C1 times for dependencies. b. Randomly sample over the domain.
2: Beh ~ M()))
3: for f + I,....CC C1 do h + H(O.,n)
4: Ed < |> chi
(0) Ed)
Discriminate dependencics Algorithm 3 GEN- GEN-Policy
5: [0(" M + Max- Max-eeliquesSaall (Ga): En k Compute Max-Cliques Require: s!....,s"
6: for f
To...., do
Run GP-UCB with dependency structure 1: a <- RoleAssigrmment(s s")
7: 0: 4 arg maxe 1² (0) + (0)
>. Max-Cliques additive kernel 2: < RoleInteraction (sa(l) sa(n)
8: Query 6e observe y e((e) + +2)
a t- Ms N)
See Eq.
9: Update posterior, μ,, o, with 0z, ye
4: return [aa (=1.......
We highlight that this graph is between the dimensions of the policy parameters, O and is unrelated to the graphical model of role interactions presented in earlier sections. It is possible to accurately model u by a kernel & _ Σ; k where each ( corresponds to a mazimal clique of the dependency graph (Rolland et al, 2018 Knowing the dependency graph greatly simplifies the complexity of optimizing v.
However, learning the dependency graph in additive decomposition remains challenging as there are O(D*) possible edges each of which may be present or absent yielding 20(D²) possible dependency structures. This difficult problem is often approached using inefficient stochastic sampling methods such as Gibbs sampling.
4.5 Dependency Structure Search Bayesian Optimization
We propose learning the dependency structure during the GEN process. Our proposed approach is based on the following observation, which is illustrated in Fig. 2
Proposition 1. Let Ga (Vd, Ea) represent an additive dependency structure with respect v(0), then the
a²v following holds true: Va, ae- ag- + - ( E Es which is a consequence of formed through addition
8² of independent sub-functions v()), at least one of which must contain 8o, ,0,0" as parameters for ae # 0 which implies their connecticity within Ea.
In practice, observing the Hessian of the value function, H., is not possible due to being an opaque function. However, during the GEN process we can observe the Hessian of the policy, H This surrogate Hessian is closely related to the H, as v06) is determined through interaction of the policy with an unknown environment. Because the value of a policy is a function of the policy; it follows by the chain rule that H₂ is an important sub-component of H.. We utilize the surrogate Hessian in our work and demonstrate its strong empirical performance in validation. Following this reasoning, we consider algorithms with noisy query access to the Hessian, H.. Note that we assume that the surrogate Hessian, H can well serve as noisy surrogate for the true Hessian, H
Assumption 1. Let Ga = (Va, be sampled from on Erdös- Rényýi model with probability Py 1: Ga GgD. That is, each edge With [0()) representing the maximal cliques of Ga, we assume that v ~~ GP (0. E, k (0()) ,005 for some kernel k taking an arbitrary number of arguments (e.g., RBF). Noisy queries can be made to the Hession of v, Hy. We define H())= A 88, (a.b) has corresponding regret of r(0).
is i.i.d. sampled from binomial distribution with probability, Pu-
100 + c/ ( Ja.bml,....D where cn (a.b) ~ NO ai) i..dd Each query to H
Under this assumption, we show that it's possible to learn the underlying dependency structure of G. (Va, Ea) with a polynomial number of queries to the noisy Hessian. We present DSS-GP-UCB in Algorithm !| and prove theoretical results regarding its performance. In the first stage of DSS-GP-UCB, we perform C₁ queries to the
We revisit the validity of this assumption in Appendix H

---

Published in Transactions on Machine Learning Research (10/2024)
Hessian if To- These Hessian queries are then averaged and compared to cutoff constant ej to determine the dependency structure F.a. We show that after CiTo queries to the Hessian, with high probability we have Ex = Ed, where Eg is the unknown ground truth dependency structure for v. This argument is formalized in the following theorem.
Theorem 1. Suppose there erists oh s.t. Vi,j Po- [koios (0, 2 o 2 and Vi,j,0,0
kāspj (0,0) 0' 2 Then for any ò,, 82 E (0, 1) after z To steps of DSS-GP-UCB we have: Mig P(E = 16D2 p.
E 2 1 51 62 when To- =
log 2D o²
D 252 ch - Too 2log 01
Our Theorem 1 relies repeatedly sampling the Hessian to determine whether an edge exists between O", and @ in the sampled additive decomposition. The key challenge is determining this connectivity under a very noisy setting, and for extremely low values of of < o where the Hessian is zero with high probability. We are able to overcome this challenge using a Bienaymé's identity, a key tool our analysis. We defer all proofs to the Appendix.
In the second stage of DSS-GP-UCR, extract the maximal cliques depending Es and construct the GP kernel, k = E, k8th) the sum of the aforementioned kernels and inference and acquisition proceeds same as GP-UCB (lines 6-9).
To bound the cumulative regret, R₂ a Ete Cir(Ben)) + Eizr, r(O,), we follow the following process. First, we bound the number and size of cliques of graphs sampled from the Erdös- Erdös-Rényi model with high probability. Second, we bound the mutual information of an additive decomposition given the mutual information of its constituent kernels using Weyl's inequality. Third, use similar analysis as Srinivas et al. 2010 to complete the regret bound.
Theorem 2. Let k be the kernel as in Assumption and Theorem 1 Let 7 (d) N R be a monotonically increasing upper bound function the mutual information of kernel & taking d arguments. The cumulative regret of DSS-GP-UCB is bounded with high probability as sollows:
RT T== TBr
Dlog Dlog D4 D c.)
(4)
where by is an appropriately picked constant and the base of the logarithm is 1
Ps
Whereas for typical kernels such as Matern and RBF, cumulative regret of GP-UCB scales exponentially with D, our regret bounds scale with exponent O(log D). This improved regret bound shows our approach is theoretically grounded approach to HDBO.
5
Validation
We compare our work against recent algorithms in MARI on several multi-agent coordination tasks and RL. algorithms for policy search in novel settings. We also perform ablation and investigation of our proposed HOM al learning roles and multi-agent interactions. We defer experimental details to Appendix A
All presented figures are overage of 5 runs with shading representing + Standard Error, the y-aris represents cumulative reward, the v-aris displayed above represents interactions with the environment in RI, z-aris displayed below represents ilerations of Bo. Commensurale with our focus memoryy-onnstmimed devices, all policy models consist of < 500 parameters.
5.1 Ablation
We investigate the impact of Role Assignment (RA) and Role Interaction (RI) as well as model capacity on training progress. We conduct ablation experiments on Multiagent Ant with 6 agents, PredPrey with 3 agents, and Heterogenous PredPrey with 3 agents (Peng et al., 2021). Multiagent. Ant is a MuJoCo (Todoroy et al.
2012 locomotion task where each agent controls an individual appendage. PredPrey is a task where
8RBF kernel satisfies these assumptions when PP.
8

---

Published in Transactions on Machine Learning Research (10/2024)
Manyagent Ant
PredPrey
Het. PredPrey
40
30
Cumulative Reward
30
-2000
20
-4000
RA-
Med RA- RI
-6000
10
Med RA
5
Med RI
-8000
0
0
200
400
200
400
200
400
r,
0
θ g.
g.e
e,,
θ 9.
0 g.n 9g
0
gg.
i gg.n 0g.
r,,
r
0,, 17
Hessian
0g.
0 g.
θ
0g.n
0
0
0g.e
0g.e
0g.e
Figure 3: Ablation study. Training curves of our HOM and its ablated variants on different multi-agent environments.
Drone Delivery
Drone Delivery
Het. PredPrey 6
1e6
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
100
100
140
80
120
Cumulative Reward
60
60
DSS-GPUCS (HOM)
FACMAC MADDPG
DSS-GPUCE (CTDE)
CDS-OPLER
SOG
DSS-GPUCB
40
G2ANET
RODE
COMIX
60
TurBO
TreeBO
20
40
LineBO
GIBO
0
0-
20
AleBO
250
500
750
1000
250
500
750
1000
500
1000
1500
Figure 4: Left two plots: Sparse reward drone delivery task. Rightmost: Comparison with HDBO approaches. The left two plots validate the same approaches on different environments.
predators must work together to catch faster, more agile prey. Het. PredPrey is similar, except the predators have different capabilities of speed and accelcration. In ablation experiments, our default configuration is Med RA R/ which employs components of RA and RI parameterized by neural networks with three layers and four neurons on each layer (medium sized neural network). The Sm, small, model is instead parameterized with neural networks of layer with 2 neurons cach. When RA ablatod, the agents interact diroctly without taking on any role based specialization. When RI ablated, the agents' action is determined without any coordination between agents. We present ablation in Fig.

---

Published in Transactions on Machine Learning Research (10/2024)
Table DSS-GP-UCB typically outperforms RL with higher sparsity (e.g., Sparse-100 or Sparse-200
Ast-e3
Hupper-e3
Swimaaate
Waller2ioo3
DNPG
SAC
Intriner
DNPG
PPO
S4C
DD3C
PPQ
SAC
Intriiaae:
Baseline
-90.77
1305 AM1
2015..2 24
2006
2:04:00
014
1790/80
2175
1810. 76
1734.00
01:45
58.30
75
08.72
1100.00
3012 81
4159/40
19511-00
2/30
Spune
-32.88
1007 50
2063.97
1407
1564.00
577.98
1057.34
8380.60
1570.84
2074.00
85.00
99.50
06.75
47.23
1758.50
1471.62
1472.33
1077.4 .6
2297.45
19f22000
Spaae:
-5877.97
961.31
711.96
762.61
1915,00
S14.39
1616.70
8229 20
2258,07
1972.00
25.65
08.00
43.84
40.32
1836.00
951.31
607 93
1097
2582.27
1924,00
Speree 20
-2400.00 89
0731.07
091.3)
579--12
1656 00
783.95
1029-28
2535-11
1450.53
1537 /20
19:12
54.83
57.78
57.03
2108.0
963.01
365. 30
1010443
270
1840.00
Spuree 50
-9007.3 -7
-67.43
003 25
253/60
1091.20
816.25
1010.73
123%
661.45
622 (0)
23:75
51.52
38.
78
50.01
8:2.00
572.12
128 29
318.
298.28
831-75
Spiase 100
33123
-021 58
679.30
115
450 40
988.36
121.51
200.32
342.5 85
400. 90
9.64
21.09
27.08
30.30
376,0 81
523.50
205
200. 10
147:22
483.00
Spuze 200
37
-9367.98
100 14
-547.8
238 60
763.05
222. 76
300.36
281.00 65
350 99)
-9.97
21,09
38.30
30.48
342 50
182.84
193.
157. 15
148.06
353.20
1347
1009
173.78
1008.90
For a simpler coordination task such as Multiagent Ant, we observe limited improvement through RA or RI. In contrast, RI shows strong improvement in PredPrey and Het. PredPrey. It is because, in PredPrey, predators must work together to catch the faster prey. Since the agents in PredPrey are homogeneous, ablating RA makes the optimization simpler and more compact without losing expressiveness. Thus, ablating RA leads to a performance increase. In Het. PredPrey, the predator agents have heterogeneous capabilities in speed and acceleration. Thus, RA plays a critical role in delivering strong performance. We also show that overly shrinking the model size (Sm - RA - RI) can hurt performance as the policy model is no longer sufficiently expressive. This is evidenced in the Multiagent Ant task. We observed that using neural networks of three layers with four neurons each be sufficiently balanced across a wide variety of tasks.
In Fig. 3 present the detected Hessian structure by DSS-GP-UCB in the respective tasks. The de- tected Hessian structures generally show strong block-diagonal associativity in the HOM parameters, i.c., 10r,;, O (g.e:00.. 9gel. This shows that our approach can detect the interdependence within the sub-parameters, but relative independence between the sub-parameters. We observe more off-diagonal connectivity in the complex coordination tasks of PredPrey and Het. PredPrey. The visualization of Hessian structure on ProdPrey shows that our approach can detect the importancc of jointly optimizing role assignment and interaction to deliver a strong policy in this complex coordination task. We investigate the learning behavior of the HOM further in Appendix B
5.2 Comparison with MARL
We compare our method with competing MARL algorithms on several multi-agent tasks where the number of agents is increased. We validate both the пом with DSS-GP-UCB (DSS-GP-UCH (MM)) and neural network policies trained in the CTDE paradigm (DSS-CP-UCB (CTDE)). In the CTDE paradigm, both RI and RA are ablated reducing the policy model to neural network which is identical across all agents. We observe that on complex coordination tasks such as PredPrey and Het. PredPrey our approach delivers more performant policies when coordination is required between a large number of agents. This is presented 9 in Fig. 5 Although SOG (Shao et al. 2022), a Comm-MARL approach shows compelling performance with a small number of agents, with 15 agents, both DSS-GP-UCE (CTDE) and DSS-GP-UCB (MM) outperform this strategy. We highlight that DSS-GP-UCH (CTDE) outperforms Comm-MARL. approaches without communication during execution. We also note that DSS-GP-UCB (MM) outperforms DSS-GP-UCR (CTDE) showing the value of our HOМ approach in complex coordination tasks. We defer further experimental results in this setting to Appendix B
5.3 Policy optimization under malformed reward
We compare against several competing RL and MARL algorithms under malformed reward scenarios. We train neural network policies with DSS-CP-UCB and competing algorithms. We consider sparse reward scenario where reward feedback is given every S environment interactions for varying S. Table α shows that the performance of competing algorithis is severely degraded with sparse reward and DSS-GP-UCH outperforms competing approaches on most tasks with moderate or higher sparsity. Although intrinsic motivation (Singh et al. 2004 Zheng et al. 2018 has shown evidence in overcoming this limitation, we find that our approach outperforms competing approaches supported by intrinsic motivations at higher sparsity. This improvement
We plot with respect total environment interactions for I, and total policy evaluations for See Appendix P Appendix and Appendix L. for alternate presentations of data more favorable Rt. and MARI under which our conclusions still hold.
K
10

---

Published in Transactions on Machine Learning Research (10/2024)
PredPrey 6
Het. PredPrey 6
1e5
1e6
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
1000
800
Cumulative Reward
600
600
400
400
200
200
PredPrey 9
Het. PredPrey 9
700
1200
600
Cumulative Reward
1000
500
800
400
300
400
200
200
100
PredPrey 15
Het. PredPrey 15
2000
1750
1500
DSS-GPUCB (HOM)
Cumulative Reward
1500
1250
DSS-GPUCB (CTDE)
CDS-QPLEX
1000
G2ANET
750
COMIX
FACMAC
500
500
MADDPG
250
SOG
RODE
1000
2000
3000
4000
5000
0
1000
2000
3000
4000
5000
Figure 5: Scaling analysis. Training curves DSS-GP-UCB and competitors with increasing number of agents. The left column shows PredPrey with 6, 9, and 15 agents. The right column shows Hct, ProdPrey with 6, 9, and 15 agents.
is important as sparse and malformed reward structure scenarios can occur in real-world tasks Aubret et al., 2019). We repeat this validation in Appendix B with MARL algorithms in multi-agent settings and consider a delayed feedback setting with similar results.
5.4 Higher-order model Investigation
We examined policy for Multiagent Ant with agents for the role based policy specialization. The policy modulation plots were generated by examining the PredPrey and Het. PredPrey environments respectively.
11

---

Published in Transactions on Machine Learning Research (10/2024)
1.0
0.8
0.6
0.4
0.2
0.0
Ant
PredPrey
Het. PredPrey
Figure 6: Left: Action distributions of different roles showing diversity in the Multiagent Ant environment with 6 agents. Right. above: Policy modulation with role interaction in PredPrey and Het. PredPrey environment with 3 agents. Arrows represent change after message passing. These plots are visualizations of the two principal components after Principal Component Analysis. Right below: Mean connectivity ratio between agents and standard deviation role interaction Multiagent Ant with agents, PredPrey with agents, and Het. PredPrey with agents.
In Fig. 6 we investigate the learned HOM policies. Our investigation shows that role is used to specialize agent policies while maintaining a common theme. Role interaction modulates the policy through graphical model inferences. Finally, role interactions are sparse, however noticeably higher for complex coordination tasks such as PredPrey.
5.5 Comparison with HDBO algorithms
We compare with several related work in HDBO. This is presented in Fig. 4 rightmost plots. We compare against these algorithms optimizing our HOM policy. For more complex tasks that require role based interaction and coordination, our approach outperforms related work. TreeBO (Han et al., [021] is also additive decomposition approach to HDBO, but uses Gibbs sampling to learn the dependency structure. However, approach of learning the structure through Hessiian-Awarenee outperforms this approach. Additional experimental results are deferred to Appendix B
5.6 Drone delivery task
We design drone delivery task that is well aligned with motivation of considering policy search in memoory-oosslriined devices on tasks with unhelpful noisy gradient information. In this task, drones must. maximize the throughput of deliveries while avoiding collisions and conserving fuel. This task is challenging as a positive reward through completing deliverics is rarely encountered (i.c., sparse rewards). However, agents often receive negative rewards due to collisions or running out fuel. Thus, gradient- based approaches can easily fall into local minima and fail find policies that complete deliverios!" To We compare DSS-GP-UCR against competing approaches in Fig. 4 leftmost two plots. We observe that MARL based approaches fail to find a meaningfully rewarding policy in this setting, whereas our approach shows strong and compelling
10 her details this task be found in Appendix
12

---

Published in Transactions on Machine Learning Research (10/2024)
performance. Furthermore, DSS-GP-UCB (MM) outperforms DSS-GP-UCB (CTDE) through leveraging roles and role interactions.
6 Conclusion
We have proposed a HOM policy along with an effective optimization algorithm, DSS-GP-UCB. Our HOM and DSS-GP-UCB are designed to offer strong performance in high coordination multi-agent lasks under sparse or malformed reward on memory- memory-constrained devices. DSS-GP-UCB is a theoretically grounded approach to Bo offering good regret bounds under reasonable assumptions. Our validation shows DSS-GP-UCD outperforms RL and MARL at optimizing neural network policies in malformed reward scenarios. Our HOM optimized with DSS-GP-UCB outperforms MARI, approaches in high coordination multi-agent scenarios by leveraging the concepts of role and role interaction. Furthermore, show through our drone delivery task, our approach outperforms MARL approaches in multi-agent coordination tasks with sparse reward. We make significant progress on high coordination multi-agent policy search by overeoming challenges posed by malformed reward and memory-constrained settings.
Acknowledgements
We thank Jonathan Scarlett for pointing out small mistake in our proof.
References
Martín Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jelfrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, GeofTrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dandelion Mané, Rajat Monga, Sherry Moore, Derek Murray, Chris Olah, Mike Schuster, Jonathon Shlens, Benoit Steiner, Ilya Sutskever, Kunal Talwar, Paul Tucker, Vincent Vanhoucke, Vijay Vasudevan, Fernanda. Viégas, Oriol Vinyals, Pete Warden, Martin Wattenberg, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. TensorFlows Large-scale machine learning on heterogeneous systems, 2015. URL https://www tensorflow.org Software available from tensorflow.org.
Riad Akrour, Dmitry Sorokin, Jan Peters, and Gerhard Neumann. Local bayesian optimization of molor skills. In Proc. ICML, 2017.
Sebastian E. Ament and Carla P. Gomes. Scalable first-order bayesian optimization via structured automatic 2022.
differentiation. In Proc. 1CML, pp.
Kai Arulkumaran, Mare Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Process. Mag., 34(6):26 38, 2017.
Arthur Aubret, Laëtitia Matignon, and Salima Hassas. A survey intrinsic motivation in reinforcement learning. CoRR, htes/1908.09970 2019.
Andrei-Cristian Barbos, Francois Caron, Jean-François Giovannelli, and Arnaud Doucet. Clone MCMC: parallel high-dimensional gaussian gibbs sampling. In Proc. NewrIPS, 2017.
Joel Berkeley, Henry B. Moss, Artem Artemev, Sergio Pascual-Diaz, Uri Granta, Hrvoje Stojie, Ivo Couckuyt, Jixiang Qing, Nasrulloh Loka, Andrei Paleyes, Sebastian W. Ober, and Victor Picheny. Trieste, 7 2022. URL
com/secondmind- hww.iiiidndaabtaattttteeee
Béla Bollobés and Paul Erdös. Cliques in random graphs. In Proc. Cambridge Philosophical Society, 1976.
Greg Brockman, Vicki Cheung, Ludwig Pottersson, Jonas Schneider, John Schulman, Jic Tang, and Wojeiech Zaremba. Openai gym, 2016.
Shuyu Cheng, Guoqiang Wu, and Jun Zhu. On the convergence of prior-guided zeroth-order optimization algorithms. In Proc. NeurIPS, pp. 14620-14600 14631, 2021.
13

---

Published in Transactions on Machine Learning Research (10/2024)
John T Chu. On bounds for the normal integral. Biometrika, 42:263 260, 1953.
Abhishck Das, Théophile Gervet, Joshua Romoff, Dhruv Batra, Devi Parikh, Mike Rabbat, and Joelle Pincau. Tarmac: Targeted multi-agent communication. In Proc. ICML, pp. 1538 1538-1546, 2019.
Christian Schroeder de Witt, Bei Peng, Pierre-Alexandr Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Deep multi-agent reinforcement learning for decentralized continuous cooperative control. arXiv preprint arXic:2003.067 2020.
Carlo D'Eramo, Davide Tatco, Andrea Bonarini, Marcello Restelli, and Jan Peters. Mushroomrl: Simplifying reinforcement learning research. 2021.
Kevin Dorling, Jordan Heinrichs, Geoffrey G. Messier, and Sebastian Magicrowski. Vehicle routing problems for drone delivery. IEFE Trans. Syst. Man Cybern. Syst., 47(1):70 85, 2017.
David Duvenaud, Hannes Nickisch, and Carl Edward Rasmussen. Additive gaussian processes. In Proc. NewrIPS, pp. 226-22 236-234,22 2011.
David Eriksson, Michael Pearce, Jacob R. Gardner, Ryan Turner, and Matthias Poloczek. Scalable global optimization via local bayesian optimization. In Proc. NeurIPS, 2019a.
David Eriksson, Michael Pearce, Jacob R. Gardner, Ryan Turner, and Matthias Poloczek. Scalable global optimization via local Bayesian optimization. In Proc. NewrIPS, pp. 5497-500 5508, 2019b.
Natalia Y. Ermolova and Svsn-Gustav Häggman. Simplified bounds for the complementary error function.
Proc. Eurasip, pp. 1087-1090, 2004.
William Feller. An introduction to probability theory and its applications, volume 81. 1991.
Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Coun- terfactual multi-agent policy gradients. Tn Proceedings of the AAAT conference on artificial intelligence, volume 32, 2018.
Lukas P. Fröhlich, Melanic N. Zeilinger, and Edgar D. Klenske. Cantious bayesian optimization for efficient and scalable policy search. In Proc. L4DC, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural message passing for quantum chemistry. In Proc. TCML, pp. 1263-1272, 2017.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference on machine learning, pp. 1871-870. PMLR, 2018.
Eric Han, Ishank Arora, and Jonathan Scarlett. High-dimensional bayesian optimization via treo-structured additive models. arXiv preprint arXiv:2012. 13088, 2020.
Eric Han, Ishank Arora, and Jonathan Scarlett. High-dinensional Bayesian optimization via tree-structured additive models. In Proc. AAAI, pp. 7630-7638, 2021.
Verena Heidrich-Meisner and Christian Igel. Evolution strategies for direct policy search. In Proc. PPSN, pp- 428 437, 2008.
Matthew J. Johnson, James Saunderson, and Alan S. Willsky. Analyzing hogwild parallel gaussian gibbs sampling. In Proc. NewrIPS, 2013.
Kirthevasan Kandasamy, Jeff G. Schneider, and Barnabás Póczos. High dimensional bayesian optimisation and bandits via additive models. In Proc. TCML, pp. 295 304, 2015.
14

---

Published in Transactions on Machine Learning Research (10/2024)
Johannes Kirschner, Mojmir Mutny, Nicole Hiller, Rasmus Ischebeck, and Andreas Krause. Adaptive and safe Bayesian optimization in high dimensions via one-dimensional subspaces. In Proc. ICML, pp. 3429-3438, 2019.
Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly, 2 (1-2):83-97, 97, 1955.
Hoang Minh Le, Yisong Yue, Peter Carr, and Patrick Lucey. Coordinated multi-agent imitation learning. In Proc. ICML, pp. 1995-20033 2017.
YC Lee, Gary Doolen, HH Chen, GZ Sun, Tom Maxwell, and HY Lee. Machine learning using a higher order correlation network. Tochnical report, Los Alamos National Lab (LANL.) Los Alamos, NM (United States); Univ. of Maryland, College Park, MD (United States), 1986.
Benjamin Letham, Roberto Calandra, Akshara Rai, and Eytan Bakshy. Re-cxamining lincar embeddings for high-dimensional Bayesian optimization. In Proc. NewrIPS, 2020.
Kemas M Lhaksmana, Yohei Murakami, and Toru Ishida. Role-based modeling for designing agent behavior in self-organizing multi-agent systems. International Journal of Software Engineering and Knowledge Engincering, 28(01):79 2018.
Chenghao Li, Tonghan Wang, Chengjic Wu, Qianchuan Zhao, Jun Yang, and Chongjic Zhang. Celebrating diversity in shared multi-agent reinforcement learning. In Proc. NewrIPS, pp. 3991- 4002,400,22000 2021.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement. learning. arXiv preprint NT::1509.0000 2015.
Yong Liu, Weixun Wang, Yujing Hu, Jianye Hao, Xingguo Chen, and Yang Gao. Multi-agent game abstraction via graph attention neural network. In Proc. AAAI, pp. 7211-7218, 2020.
Daniel J. Lizotte, Tao Wang, Michael H. Bowling, and Dale Schuurmans. Automatic gait optimization with gaussian process regression. In Proc. IJCAT, 2007.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-comptitiv environments. Advances in neural information processing systems, 30, 2017.
Eduardo Magalhaes. On the properties of the hessian tensor for vector functions. viXra preprint viXm::20550004 0044, 2020.
Alonso Marco, Philipp Hennig, Jeannette Bohg, Stefan Schaal, and Sebastian Trimpe. Automatic LQR tuning based on gaussian process global optimization. In Proc. ICRA, 2016.
Ruben Martinez- Cantin. Bayesian optimization with adaptive kernels for robot control. In Proc. ICRA, 2017.
Alexander G. de G. Matthews, Mark van der Wilk, Tom Nickson, Keisuke. Fujii, Alexis Boukouvalas, Pablo Lcón-Villagee Zoubin Ghahramani, and James Hensman. GPflow: A Gaussian process library using TensorFlow. Journal of Machine Learning Research, 18(40):1 6, apr 2017. WTLLhttp://jml. v18/16-537 html
David W Matula. The largest clique size in a random graph. Department of Computer Science, Southern Methodist University Dallas, Texas, 1976.
Mark McLeod, Stephen J. Roberts, and Michael A. Osborne. Optimization, fast and slow: optimally switching between local and bayesian optimization. In Proc. ICML, 2018
Massimo Merenda, Carlo Porcaro, and Demetrio Iero. Edge machine learning for AI-Enabled IoT devices: A review. Sensors, 2099);3333 2020.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529 533, 2015.
Sarah Muller, Alexander von Rohr, and Sebastian Trimpe. Local policy search with Bayesian optimization. In Proc. NeurIPS, pp. 20708-20722 2021.
Mojmir Mutny and Andreas Krause. Efficient high dimensional bayesian optimization with additivity and quadrature fourier Teatures. In Proc. NewrIPS, pp. 9019 9030, 2018.
Frans A Oliehoek, Matthijs TJ Spaan, and Nikos Vlassis. Optimal and approximate q-value functions for decentralized pomdps. Journal of Artificial Intelligence Research, 32:289-353 2008.
Evgenia Papavasileiou, Jan Cornelis, and Bart Jansen. systematic literature review of the successors of "neuroevolution of augmenting topologies". Evolutionary Computation, 29(1): 2021.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Muriosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-277777 PMLR, 2017.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Pierre-Aexxanie Kamienny; Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Facmac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Syslems, 84:12208-12288 2021.
Peng Peng, Ying Wen, Yaodong Yang, Quan Yuan, Zhenkun Tang, Haitao Long, and Jun Wang. Multiagent bidirectionally-corddiae combat games. arXiv preprint arXiv:1
nets: Emergence of human-level coordination in learning to play starcraft 2017.
Victor Picheny, Henry B. Moss, Léconard Torossian, and Nicolas Durrande. Bayesian quantile and expectile optimisation. In Proc. UAI, pp. 1623 1633, 2022.
Hong Qian and Yang Yu. Derivative-free reinforcement learning: a review. Frontiers of Computer Science, 15 (6):1-13 19, 2021.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 4295-4304. PMLR, 2018.
Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes for machine learning. MIT Press, 2006.
Yara Rizk, Mariette Awad, and Edward W Tunstel. Decision making in multiagent systems: A survey, IEEE Transactions on Cognitive and Developmental Systems, 10(3):514-529, 2018.
Diederik M Roijers, Peter Vamplew, Shimon Whiteson, and Richard Dazeley. A survey of multi-objoctive sequential docision-making. Journal of Artificial Intelligence Reseorch, 48:67-113, 2013.
Paul Rolland, Jonathan Scarlett, Ilija Bogunovic, and Volkan Cevher. Hign-dimensional bayesian optimization via additive models with overlapping groups. In Proc. AISTATS, pp. 298-307, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. Xáv preprint ar arXiv:17070060 2017.
Guy Shani, Joelle Pineau, and Robert Kaplow. A survey of point- based POMDP solvers. Autonomous Agents and Auett-Agent Systems, 27:1-51, 2013.
Jianzhun Shao, Zhiqiang Lou, Hongchang Zhang, Yuhang Jiang, Shuncheng He, and Xiangyang Ji. Self- organized group for cooperative multi-agent reinforcement learning. Proc. NewrIPS, pp. 5711-5723, 2022.
Shubhanshu Shekhar and Tara Javidi. Significance of gradient information in bayesian optimization. In Proc.
A ATSTATS, 2021.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. In Proc. TCLR, 2019.
Satinder Singh, Andrew G. Barto, and Nuttapong Chentanez. Intrinsically motivated reinforcement learning. In Proc. Newr/PS, pp- 1281-1288, 2004.
Maciej Skorski. Chain rules for hessian and higher derivatives made easy by tensor calculus. arXiv preprinl arXiv:1911.13992 2019.
Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Earl Hostallero, and Yung Yi. Qtran: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 5887 5896. PMLR, 2019.
Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. Gaussian process optimization in the bandit setting: No regret and experimental design. In Proc. ICML, 2010.
Peter Sunehag, Guy Lever, Audrunas Gruslys, Wojeiech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. Value-decomiosition networks for cooperative multi-agent learning. arXiv preprint arXiv: (aee1770.05220 05296, 2017.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026 5033. IEEE, 2012.
Alexander von Rohr, Scbastian Trimpe, Alonso Marco, Peer Fischer, and Stefano Palagi. Gait Icarning for soft microrobots controlled by light fields. In Proc. IROS, 2018.
Chaohui Wang, Nikos Komodakis, and Nikos Paragios. Markov random field modcling, inference & learning in computer vision & image understanding: A survey. Comput. Vis. Image Underst., 117(11):1610-18)) 1627, 2013.
Jianhao Wang, Zhizhou Ren, Terry Liu, Yang Yu, and Chongjie Zhang. QPLEX: Duplex dueling multi-agent Q-Learning. In Proc. ICLR, 2021a.
Linnan Wang, Rodrigo Fonseca, and Yuandong Tian. Learning search space partition for black-box optimiza- tion using monte carlo tree search. In Proc. NewrIPS, 2020a.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjic Zhang. Roma: Multi-agent reinforcement learning with emergent roles. arXiv preprint arXiv:2003.0800 08039, 2020b.
Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. RODE: Learning roles to decompose multi-agent tasks. In Proc. ICLR, 2021b.
Daan Wierstra, Tom Schaul, Jan Peters, and Jürgen Schnidhuber. Fitness expectation maximization. Proc. PPSN, pp. 337-346, 2008.
Aaron Wilson, Alan Fern, and Prasad Tadepalli. Using trajectory data to improve bayesian optimization for reinforcement learning. JMLR, 15(1), 2014.
James T. Wilson, Viacheslav Borovitskiy, Alexander Terenin, Peter Mostowsky, and Marc Peter Deisenroth. Efficiently sampling functions from Gaussian process posteriors. In Proc. ICML, pp. 10292 10302, 2020.
Zeyu Zheng, Junhyuk Oh, and Satinder Singh. On learning intrinsic rewards for policy gradient methods. Proc. NewrIPS, pp. 4649 4659, 2018.
Changxi Zhu, Mehdi Dastani, and Shihan Wang. A survey of multi-agent reinforcement learning with communication. Xix preprint arXiv:2208 08975, 2022.
17

---

Published in Transactions on Machine Learning Research (10/2024)
A Experimental Details
We used Trieste (Berkeley et al. 2022), Tensorflow (Abadi al., 2015;; and GPFLow (Matthews et al. 2017 to build our work and perform comparisons using MushroomRL (D [D'Eramo et al., 2021 MultiagentMMJJoo jde Witt et al., 2020), OpenAT Gym (Brockman et al. 2016), and Multi-agent Particle environment (Lowe ct al., 2017). When comparing with related work, we used neural network policies of equivalent size. All of our lested policies are < 500 parameters, however the XL, models are constructed using 3 layers of 400 neurons each.
To estimate the Hessian, we used the Fessian-Vecctr product approximation. We relaxed the discrete portions of our HOM policy into differentiable continuous approximation for this phase using the Sinkhorn-Knopp algorithm for the Role Assignment phase. For role interaction network connectivity, we used sigmoid to create differentiable "soft" edges between each rolc. We pragmatically kept all detected edges in the Hcssian while maintaining computational feasibility. We observed that our approach could support up to 1500 edges in the dependency graph prior to experiencing computational intractability. We used the Matern- as the base kernel in all our models.
A.1 Ablation and Investigation
In the ablation, we perform experiments on MultiagentMMuJooo with environments Multiagent Ant with 6 segments, Multiagent Swimmer with 6 segments, Predator Prey with 3 predators, and Heterogeneous Predator Prey with 3 prodators. In the Predator Prey environment, multiple predators must work together to capture faster and more agile prey. In Heterogeneous Predator Prey, each Predator has differing capabilities of speed and acceleration. This modification is challenging as policy must not only coordinate between the Predators, but roles based specialization must be considered given the heterogeneous nature of each predator's capabilities.
To generate Fig. G we examined policy for Multiagent Ant with 6 agents for the role based policy specialization. The policy modulation plots were generated by examining the PredPrey and Het. PredPrey environments respectively.
A.2 Comparison with MARL
For the MARL setting, we compare against MADDPG Lowe ct al., 2017 FACMAC (Pong et al. 2021
COMIX (Peng et al. 2021), , RODE (Wang et al. 2021b and CDS Li et al. 2021 using QPLEX et al.) 2021a) as a base algorithm. We also compare against Comm-MARL MARL approaches SOG (Shao et al. 2022). and G2ANet [Liu et al. 2020). RODE and QPLEX are limited to discrete environments, thus we are unable to provide comparisons on continuous action space tasks such as Multiagent Ant or Multiagent Swimmer. All MARI environments were trained for 2, 000, 000 timesteps. The neural network policies were 3-layers each with 15 neurons per layer, and were greater than or equal to the size of the compared ном policy. For Actor-Critàs approaches, did not reduce the size exprossivity of the critic. All used hyperparameters and Algorithmic configuratinns were advised by the authors of the work.
Wang
In the MARL setting we use Multiagent Ant, Multiagent Swimmer, Predator-Prey. Heterogeneous Predator- Prey. Multiagent Ant, and Multiagent Swimmer are MuJoCo locomotion tasks where each agent controls a segment. of an Ant or Swimmer. Predator-Prey (PredPrey N) environment is a cooperative environment. where N of agents work together to chase and capture prey agents. In Hotcrogencous Predator Prcy, cach Predator has differing capabilities of speed and acceleration. This modification is challenging as policy must not only coordinate between the Predators, but roles based specialization must be considered given the heterogeneous nature of cach predator's capabilities. We also validated related work on the drone delivery task under which a drone swarm of N agents (Drone Delivery-N) must complete deliveries of varying distances while avoiding collisions and conserving fuel. The code of which is available in supplementary materials and will be open sourced.
We used batching (Picheny et al. 2022 in our comparisons with MARL to allow for a large number of iterations of BO. We used batch size of 15 in our comparison experiments. In this setting, all MuJoCo environments
18

---

Published in Transactions on Machine Learning Research (10/2024)
use the default epoch (total number of interactions with the environment for computing reward) length of 1000, for Predator-Prey environments, epoch length was 25, for Drone Delivery environment, epoch length was 150.
A.3 RL and MARL under Malformed Reward
For single agent RL compared against SAC (Haarnoja et al. 2018), PPO (Schulman et al. 2017 TD3 (Fujimoto et al., 2018). and DDPG (Lillicrap et al.) 2015) as well as an algorithm using intrinsic motivation (Zheng et al. 2018). In single agent setting, we trained related work for 200, 000 timesteps. In the MARL setting, we trained for 2, 000, 000 timesteps. In both single-agent setting and multi-agent setting all policy networks for both DSS-GP-UCH and related work was layers of 10 neurons each. The tested environments were standard OpenAl Gym benchmarks of Ant, Hopper, Swimmer, and Walker2D.
In the MARL setting compared against COVDN (Peng et al. 2021), COMIX, FACMAC, and MAD- DPG. Comparisons were not possible against other approaches these do not support continuous action environments and are restricted to discrete action spaces.
For all environments and algorithms, we used the recommended hyperparameter settings as defined by the authors.
A.4 Comparison with HDBO Algorithms
For this comparison, we compared with several related works in HDBO. We compared with TurBO (Eriksson
et al. 2019b), Alebo [etham et al. 2020), TreeB0 (Han et al. 2021 LineBO (Kirschner et al. 2019 and recent variant of BO for policy scarch, GIBO (uüler et al. 2021).
For computational efficiency, the epoch length for MuJoCo environments reduced to 500.
A.5
Drone Delivery Task
The experimental details follow that of comparisons with MARL.
A.6
Compute
All experiments were performed on commodity CPU and GPUs. Each experimental setting took no more than 2 days complete on a single GPU.
19

---

Published in Transactions on Machine Learning Research (10/2024)
Hopper-v3 (MARL. Swimmer-rs (MARL.) Walker2d-v3 (MARL.
478
393
353
353
PrwllPryy
477 2013
Table 2: Policy model sizes. Unfilled entries mean this enviromment was not considered during validation.
Table 3: Policy model sizes. Unfilled entries mean this environment was not considerod during validation.
ProdFPryy
478 373
267
267
8 ProdPPey
478 391
478 8FF
267
267
6
478
372
Ant-v3 Hopper-v3 Swimmer-ys Walker2d-v3 Ant-v3 (MARI.)
2096
4354
310
310
12 Mikkligent-he Mullingest
2006
454
356
356
222
222
207 264
MARI lnghtiigttt...... lngitr laggrr 8 Maltlggente
263
263
207 263
478
478
RL (Single Agent)
MARI (CTDE)
DSS-GI-UCH (Single )gent)
USS-GP-UCB (CTDE)
261
210
DRS DCB (CTDE)
20

---

Published in Transactions on Machine Learning Research (10/2024)
A.7 Policy Sizes
We list the policy sizes of our models in Table and
Of note is in each environment, the compared against policy of RL or MARL is greater than or equal to in size vs. the policy optimized by DSS-GP-UCB.
A.8 Hyperparameter for Higher-Orde Model
For our HOM we utilized simple grid search in order to pick the hyperparameter settings. Overly large neural networks suffered from difficulty of optimization by BO, whereas, overly small neural networks suffered from performance difficulty on several environments. We found that neural networks of 3 layers, and 4 neurons each performed well across wide number of tested environments.
21

---

Published in Transactions on Machine Learning Research (10/2024)
Multiagent Ant
Multiagent Swimmer
PredPrey
Het. PredPrey
40
30
Cumulative Reward
30
25
-2000
20
-4000
15
10
Med RA RI
6000
-50
Med RA
Med
-8000
200
400
0
200
400
200
400
200
400
r,.1
gg.
8
0g
0,,1
0g
8
0g.e
0,,1
8g.
9 8g
0,,
8 g.n 0g.
,,,7
0,,7
Hessian
9g
6 g.V
8g.
8
9g.7
0 q.n
9
0g.7
0g.0
0g.0
0g.e
0g.e
Figure 7: Ablation study. Training curves of DSS-GP-UCB and its ablated variants on different multi-agent enviromments.
B
Additional Experiments
B.1
Ablation
We present an expanded version of Fig. 3
Fig. 7 Zincluding the ablation for Multiagent Swimmer. Multiagent Swimmer shows similar behavior the simpler task Multiagent Ant, with stronger block-diagonal Hessian structure.
B.2 Comparison with MARL
We present an expanded version of Fig. 5 in Fig. including the results for Multiagent-Ant and Multiagent- Swimmer. We observe that in this relatively uncomplicated task not well-suited for our approach with dense reward, our HOM approach shows comparable performance to MARL approaches and far outperforms DSS-CP-UCD (CTDE). This shows the overall value of our ноо approach.
B.3 RL and MARL under Malformed Reward
We present additional experiments under malformed reward for both Rt. and MARL. We formally define the Sparse reward scenario. Let v(0) & Σt= FT where the value of the policy is determined through f interactions with some unknown environment and each interaction is associated with the reward, rr. Typically, RL algorithms observe the reward, Fr after every interaction with the environment. We consider a sparse reward scenario where reward feedback is given every S steps: ri & ΣF-s Tr if = 0 mod S and 0 o.w. In addition to the sparse reward setting described carlier, we also consider the sctting of delayed reward. The delayed reward scenario is defined: + A rr-n if r D and 0 o.w. Thus in the delayed reward scenario, feedback
action taken is delayed. This scenario is important as it arises in long term planning tasks where the value of an action is not immediately clear, but rather is ascertained after significant delays. We present. the complete lable comparing related works in RL with DSS-GP-UCB in Table 4 As can be seen, similar to the Sparse reward scenarios, significant degradation can observed across all tested RL algorithms with DSS-GP-UCB outperforming RL. algorithms with moderate to severe amount of sparsity or delay. This degradation cannot be overcome by increasing the size of the policy, as we verify with the "XI"" models which are orders of magnitude larger with 3 layers of 400 neurons.
22

---

Published in Transactions on Machine Learning Research (10/2024)
Swimmer 4
Ant 8
PredPrey 6
Het. PredPrey 6
1HE
1H
0.0
1.0
2.0
0.0
0.5
1,0
1.5
0.0
0.5
1.0
1.5
2.0
0.0
1.0
2,0
140-
1500
1000
800
120
250
1000
600
80-
750
500
400
40-
200
20
250
-500
Swimmer 8
Ant 12
PredPrey 9
Het. PredPrey 9
700
1200
60-
2250
600
000
1000
20
750
800
400
500
600
300
250
-20-
400
200
-40
100
250
500
Swimmer 12
Ant
PredPrey 15
Het. PredPrey 15
600
2000
750
20
1500
1500
DSS-GPUCE (HOM)
DSS-GPUCB (CTDE)
10
CDS-OPLEX
1000
1000
G2ANET
FACMAC
200
500
500
MADDPG
-10
250
SOG
4400
RODE.
-20
1000
2000
3000
1000
2000
3000
2000
4000
0
2000
4000
Figure 8: Comparison with MARL approaches with varying number of agents.
23

---

Published in Transactions on Machine Learning Research (10/2024)
We repeat these experimental scenarios in the MARL setting with similar results in Table 5 where MARL approaches are compared against DSS-GP-UCB in the CTDE setting. Thus our validation shows that in both RL and MARL strong performance requires dense, informative feedback which may not be present outside of simulator settings. In these settings, our approach of optimizing small compact policies using DSS-GP-UCB outperforms related work in both RI and MARL.
24

---

Published in Transactions on Machine Learning Research (10/2024)
Table 4: RL under sparse reward. Sparse n refers Lo sparse reward. Delay ni refers to delayed reward. Averaged over runs. Parenthesis indicate standard error.
IIS EITE TH
ON SPR PR WHININ TOR ME
THES BIR WE THE
ENONS THE
Table 5: MA under sparse reward. Sparso n refers to sparse roward. Delay n refers to delayed reward. Averaged over 5 runs. Parent indicate standard error.
FACMACC
545-71 (ISO
207.000 L1S
200-2 (0)
211
120
ING 222 215 77
86 477 220 5O
357 220 ISS (81
G41 LIS O8S
102 2113 T7
280 2N1281 215 771
267-300
444.11
220
102
20T
OR &U
1200 1ETTLL0 NO ROCRIOROO 130 1301 312 TEALIS
ALLAS IL 41SH PAIIING
MADIPG
30022 00)
02 88)
42N00 260
(4-55)
10 G1)
122 0901 14 35)
1ST 37 (070000000 61)
312 R3 (2 00)
SX2. 11 88
1ST. 11 700
140 04 50)
155 1S1-5 10 01)
191 PATER 50)
A00 01)
1400.24444
121.5
37
AE
Willerdde
250
230
220
1
5.00
200 5N 08
194-00 (1 36)
277 20) 4.201
2N 2N.2211 36)
S 90)
316-20 (0
29 98
14F-00 ST 21)
100 36)
COMIX
121 15 (
AN M0.3 OT BIT SETORUTT A0
1OR AO SUR AUTH LIMTS SUEEE
200 251
316
280
1
227
HIU IN THI ON MINA THE MUA
60)
541 801.88 (22
100 57)
106 106.24 (6.06)
155 105.50 (7.12)
1 00 S1.
12)
4.0 2012 655
(22)
8 ST
S0
12
COVDN
102 026
170.01
97.29
240
31
56
S0C0R1M 141 10000100 1100 100010011 700
1000 O0 (000TU
MI
FACMAC
14.00001 11
15
(320)
(2.00(100
1.0
01.0
100
14.00
11:20001
(3)
14.340411
(1)
22.00
34 11.16)
29 11.00
11)
(113)
10 (9)
5,36
(9)
SWER BELW BE
14
OF IT3
MADIIIG
11 21.00 30)
11. 11.994033 30)
12.0550.33 34)
1111111222 201
14.4 44.350775 73)
12 00 29)
0. 20)
100000330 39)
14.5000
(3)
12:23320 20)
70)
2213
BIRI PAY
SEAAN MANK
Swinmmess
12.
10.1
28
10
13
28
13
10
18
OOMO
13 034330300 200
1,1 1.13 5.00)
1 20
14.553 02)
4877 20
9750 36
4184 72)
6.2 2000
20
0.88 0.0
1.00
31.080
7. 10
7009
AH
53 SRATI
SITE ANN FOR AN LIN MERMS
COVDN
01000 20)
12/2 (6)
8.40 (2)
55,44 3220
0,15) 8.20
0.313328)
3.5
1009 (6)
0.791222) 52)
10. 10.110.20 28)
0.2919
5 6-69
S000 200
TH
DO 20 20)
RACHAC
005
352200 (5
27.100 260
72
71311130
38551C24 000
852311C ())
495 20 (K)
3557 209
28 000
28 97L 280
SILD INE II UON TH PRETTT 101 VIV II 2TM MOTTON SU BRTUR RESUMISS 35
600 00N
172-171
56
MOILT TO THE BRES AUTO
BE BVA ONLT WHE SITH TOE SWAWE TH INLATT DE
MADDPG
(1R 45)
44)
39)
47.47
50.7400.00 800
28 01 360
113 10 OZ 15)
SPE (2) 44)
912 20)
275.0550.000 200
28 04/0 800
530 10.1101100 06
273
Happer--t
300
5201
3622
210. 20..0220
Begge
ANLD IN AI HOR TE AU SEKI OR ON
COMI
0995800000
08-08 18)
21.24 0.22)
26.220
60 00)
300 (002
(2000
1N-5610220
15.98 (3 18)
28 (2)
00
ON HIS MAS IND LA SHIM
TO 203T
AANIE OO WILD UNN TH SON
ANCMISTOO TO HOR THE OO SHEST
38
2002
355
000000000
000.88 22)
222222000111 11)
815 N03 77 88)
0966 55/20 (I)
55 (0)
620 008.04722 22
183 21 78)
097 511E70 88)
(1)
COVDN
687
58
087
092
LAUNO
FACMACC
40 06)
852 90000 35)
200 F1EL 76
564 200
384 2012834
341
302 2813 30)
563 2018 311
536 77000000 06
910 30 35
76
80
341
314 2009 80)
ANDE 54)
THE 10
MAII SA
900
855
541
722
801
TTO
MADDPNG
58154. 10)
912.61 (2.45
772 201-2-441.
400 1942220
6012 00
04:
652 22)
4700 011077000
(00
901 14154 (9)
573 48
48
575 20
544 (2)
674 TEIOHTT (M))
(809922
611 PRONTT (4))
TO
AN
Aun-va
THIS MEEGE
RESHAORANN HARRAN SUMIN DAAN AD
COMIX
900 43)
2212 21)
700 71
1977 2004..0
200 201
02254001000 (00
(28 83)
83)
322 401449
272 21)
900
59255
10.21
133
200
200
Boseine 910, 50.. 00)
Sparo 5 ST7 0w7.0.22000 (6)
Sgore fit 242 00 252234
Spaino ini 40% 405 200. 18)
Spatse 2011 65.041 (2) 32)
Spone XL. 100 168 00 201 18)
Spenee XI 2014 11200000000 30)
Ing 1 650 OOIS (N)
Lag 5 265 26)
362223 041
112 80 28)
521
(8)
AT 62)
CONTIN
02100
Sptan WNIE SENEEN
Spear 30 ME
THE THE SON SARE AN
Lag N1. 369
860
667
DRINZ
Spart TAM ANTO
Lag fM
Lig 100
Lag 200
Log NL. 200
22

---

Published in Transactions on Machine Learning Research (10/2024)
Multiagent Ant
PredPrey 3
Drone Delivery 3
Het. PredPrey 3
Het. PredPrey 6
40
150
30
75
100
200
50
Tur80
TreeBO
25
LineBO
10-
GIBO
AleBO
0
0
500
1000
500
1000
0
500
1000
500
1000
0
500
1000
1600
Figure 9: Comparison with BO algorithms. DSS-GP-UCB outperforms on complex multi-agent coordination lasks.
B.4 Comparison with HDBO Algorithms
We compare with several related work in High-dimensional BO including TurBO EEriksson et al. 2019b).
AleBO (Letham et al. 2020). LineBO [Kirschner et al., 2019). TreeBO (Han et al. 2021 and GIBO (üüller
el al. 2021h. This is presented in Fig. We experienced out-of-memory issues with AleBO after approximately 100 iterations, hence the AleBO plots are truncated. We compare against these algorithms at optimizing our HOM policy for solving various multi-agent policy search tasks. We validated on Multiagent Ant with 6 agents, PredPrey with agents, Het. PredPrey with agents, Drone Delivery with 3 agents, and also Het. PredPrey with agents. We observe that these competing works offer competitive performance for simpler tasks such as Multiagent Ant and PredPrey with agents. However for more complex tasks that require approach outperforms related work. This is evidenced in Het. PredPrey 3, Het. PredPrey 6 as well as the Drone Delivery task with 3 agents.
role based interaction and coordination,
Thus our validation shows that for simpler task, competing related works are able to optimize for simple policies of low underlying dimensionality. However, for more complex tasks which require sophisticated interaction using both Role and Role Interaction, related work is less capable of optimizing for strong policies due to the complezity of the high-dimenssooal Bo task. In contrast, our work offers the capability of finding stronger policies for these complex tasks and scenarios.
26

---

Published in Transactions on Machine Learning Research (10/2024)
Table 6: Summary of key notations.
Description
optimisation
The domaiu for the objective function
92
point domatn that ploked at sime
The postecior (Inferred
limee
The differeace between the musimm the function domnin 6, v(*"), and c(02)
Pr
The cowolaativ regret Dt=1 ((0p)
g
Ga
A gropb showing the dependencies
dimensions
dimensions they dependent
Va
In the graph indicared by Ey the ras dimenstons corneponding to B graph indicated by Pa the
Ed
correspording dependencies between
e(
Collection dimcasion indieated
concsponding moximel clique the
Gausstan procese correspond
masimal clique (t)
The Gauzstan process kernal for inference corracponding to the sum of Le(1)
: = *B(f)
k
.0)
Under the additive assmmption, it is assamed that Σ v(t) where each sampled from gB())
u(e)
A uniform random dutribution over the domain 8
((
A query to the Besoiun at f,
Gd
The graph correggonding to the desected depandency atrmcture by querying the Hossian
Max (S
A function computing the maximal cliques in tāe gruph fd
The set of stases of the cooperative multi-agest system where [=']i=1..... and denotes the index of the agens talinn
The
ageul agent taking the role axf)
(a (el)..n.....
index ageat
xa(s)
The
The action taken agent taking the role (4))
A Pru
An affinty function for taking on role where denotos belonging to the part of the sou for role assignment
uffiaity determining
un cdge exiata during interaction of
unst policy
The action update function parameterized by dgge for the rale Interaction masage passing neural network
fg.n
C Table of Notations
Table provides summary of notations that are used frequently in paper.
27

---

Published in Transactions on Machine Learning Research (10/2024)
D On the Applicability of Our Assumptions to RBF and Matern Kernel
We show that assumption is satisfied by the RBF Kernel when θ [0, 1]2, and is quasi-satisfied by the Matern-3 kernel. We also show that in the setting where θ r0 for some bounded our assumptions are quasi-satisfied as although these kernels may take on small negative values, these values decay exponentially with respect to the distance. These Lemmas show that our assumptions are reasonable.
Lemma Let & (0,0) - expl ( be the RBF kernel with - 110 0'1, then
beoes (0, 0' = k (0,00) (1 (0 - 0²))) (1 (1- (0' -0))
Proof. As shown in [Rasmussen & Williams 2006 Section 9.4, the derivative of a Gaussian Process is also a Gaussian Process. Let GP(O0 k: (0, 0)) be the GP from which f is sampled. This implies:
0s ag
8 (0,00)
GP 0, 08
Applying this rule more for the Hessian, have:
8S
0*k (0,0')
GP 0,
00h0n
80 000 a0"
Given the above identities, we compute the partial derivatives for the RBF kernel:
8 (0,0")
(- 110--011 0'11²)
(1 (0 )2)
exp
2
Deriving once more we have:
8*k (0,0) exp (- |00 - 0-112) -010)
- (1-(00- -0--0000000 -199)) (1- (00 - 814)2)
00 "0 000 00
2
with a ||0 8'11. This completes the proof noting that 1: (0,8") a exp( =f
Corollary 1. Let k (0 & exp(=f), and 0,0' e [0, 1), then k00 (0, 0') 2 0.
Proof. The above is straightforward to exp (+) 0 and with 0' [0. 11dd we have (1 (0" 0(a)2) 2 0 (1 (1- grb(2) 2 0.
Corollary Let &s (0,6") A exp(-xt), and 0, [0,r] then t Biss (0.0) cexp(-d') for some constant dependent r.
Proof. The above is straightforward given the above Lemma. We note that although the RBF kernel may take on negative values the domain θ r]d, this values experience strong tail decay showing the quasi-satisfaction of our assumptions.
The above Lemma and Corollary shows that our assumptions are satisfied by the RBF Kernel when [0, 1) and quasi satisfied when θ [0, ri D after choosing a suitable ph and of We show how these assumptions are quasi-satisfid by the Matern- 5 kernel.
28

---

Published in Transactions on Machine Learning Research (10/2024)
Lemma 2. Let le (8,8") - (1 v5d + exp) be the Matern- kernel with d - ||0 8'11, then with d _ 6 gr we have
hioss =exp(-v))
5V5
25
d
25
25v5
de +
3d3 dadg)
25
3
3d
3d
3d
we state the partial derivatives of the Matern- kernel: Proof. Following the proof of Lemma
(0,00, 08-00ra
exp (-15110-00 011)
(3
55
5v5
100
811
25
1 g())²
3
3
3
Differentiating one more we have
0 (0, )')
(exp(-v5) (-v510 -0'l)
exp
00 00 08008m
5/5
25
(0
0²0
3d 10h
25
0r0g
25
(0
)00) 0xh/²
3d
302
25
(0
(30 0mhy3
3d3
This completes the proof noting that d: - 0i rr and d - |10 1-
Corollary 3. Let le (0,0')
(1 + v5d + dd exp( 55d) and 0, 0 E [0,
Then Rooos (0,0') N
exp(-V5d)
5y5 25
25
25
Proof. The above is immediate consequence of Lemma 2 and noting that ||&]l s 1.
Corollary 4. Let k (0,0" (') (1 v5d _dd) exp( v5d) and o' E [0, Then koios (0, cexp(-d) for some dependent
Proof. The above is immediate consoquence of Lemma 2 and noting that ||]]| Sr.
Although the above corollary shows that the Matern- 2 kernel may take negative values, note that these values experience strong lail decay due the presence of the exp (-v5d) term. Thus, the negative values are likely to be extremely small, thus quasi-satisfynng our assumptions, In our experiments, observed no shortcoming in using the Matern- kernel in DSS-GP-UCD.
29

---

Published in Transactions on Machine Learning Research (10/2024)
E Proof of Proposition 1
We restate Proposition 1 for clarity.
Proposition 1. Let Ga (Vd, Ea) represent additive dependency structure with respect to v(0), then the
8²v following holds true: Va, b 06- # = (0", @²) E Fi which is a consequence of formed through addition of independent sub-functions v(), at least one of which must contain 0 00 parameters for 50" which implies their connectivity within Ea-
f # O
Proof. The above follows from the linearity of addition, which naturally implies lack of curvature, In the multivariate case, this corresponds to zero or non-zero entries in the Hessian.
To be precise, we prove the contrapositive
0.v
(0",0") Ea =
0.
00h
Let a, be arbitrary dimensions with (@", @0) Ea- As consequence of the dcfinition of the dependency
graph, 1061 s.t. ( 0²) C (+)), That is, no subfunction vl0 lakes both 0 and 0 as arguments.
By the linearity of the partial derivative, we see that:
a
M
M
8
80 (0)
a0
00
1 (0())
=
80
00
v 000
(0(+))
=
where the last equality follows from no subfunction (10 taking both 0a and 0 as arguments.
30

---

Published in Transactions on Machine Learning Research (10/2024)
F Proof of Theorem 1
Our proof of Theorem 1 relies in being able to determine whether an edge does does not exist in the dependency graph. To be able to do this, we examine the Hessian. As we have shown in Proposition [ examining the Hessian answers this question. The challenge of Theorem D is detecting this dependency under noisy observations of the Hcssian, as well as in domains where the variance of the second partial derivative is often zero, i.e, 10505 (") = 0 with high probability. To overcome this challenge, we sample the Hessian multiple times to both find portions of the domain where (0 00) 2 oh and also reduce the effect of the noise on learning the dependency structure. To proceed with the analysis, we first prove helper lemma showing that if we can construct two Normal variables of sufficiently different variances, then it's possible to accurately determine which Normal variable has low, and high variance by laking a singular sample from each. This helper lemma will be used later to help determine edges in the dependency graph. As we shall soon show, If an edge exists, we are able to construct a Normal variable with high variance. Correspondingly, if an edge does not exist, we are able to construct a Normal variable with low variance.
Lemma 3. Let X1 ~ of) and Xx. ~ o0) be two random univariate ganssian variables. For any 6 E 10, 3ch s.t. |X|] S en I [xal with probobility 1 1-6 8 when L σnd > > oiy 2 2 8
> de log & and precisely when of
Proof. First we note that |Xi] and |xx] are Half-Normal random variables, with cumulative distribution function of Fi(x) erf ب.ب and (x) erf v. respectively. Thus to show that |Xi] S / log and [& 2 6 with high probability, we utilize well known bounds on the erf and erfe function. The proofs of the below can be found in several places, e.g.. Chu (1555) and Ermolova & Häggman (2004) respectively.
2
af v1 11-exp exp -2x²; erfex S exp =x..
Given the above, we show that p(cn S (i)) s 1 and plcn (xx) s 3 and utilizing the union bound completes the proof.
2
2
c
8
2
Cp > 011
2 log
=
o%
20
log
5
=
207
7
> log
2
=
207
log
2
exp-
207 2
4
2
<
8
erfe
V20
Ch
8
=
- erf V201
21 1 -
5
=
Fi(cx) 2 1 1 5
2
5
= p(c $ (x11)
2
Following a similar line of reasoning we have:
σnd Ch <
8²
82
2
2
2
8²
2
Th
4
4
Th
2
c
log
4
exp
0
?
>
1
4
- - exp-
c
h
82
4
exp
exp
0
2
8
2
erf
σh On
Ch
2
8
2
(c()
<
8
8
p(cn
2 [xl)
2
Finally, to complete the proof, we show that the interval (or (01 / log 2, 220) is not the empty set when
c > 82
log 3.
31

---

Published in Transactions on Machine Learning Research (10/2024)
0 8
2v2
2
σh.
2 8
log
log
2 2log
of
8
6
2
We are now ready prove Theorem
Theorem 1. Suppose TT there exists o. s.t.
Vi.j Po-u(o) [1000 2 o 2 Ph and Vi,j,0,8 0
kisoi (0,0') 0. Then for any 81, 82 (0, after To sleps of DSS-GP-UCH we have: Mi, P(F
Edd) 2 1 51 - 82 when To = C1 16D
p log 2D o2
2 log 2D D - Toon v
282
b1
Proof. We prove the above for single pair variables, ie., Aae and utilize the union bound to complete the proof. The first challenge to overcome is to sufficiently sample enough points in the domain such that we are able find enough points 0 E θ where (0, 2 oh. To achieve this we sample To different 0 in the domain. After sampling To points if there exists edge between Oo and 6% then with probability & we have sampled Topa D
2 26, points where (B 8) 2 o% To show the above we use bounds on the cumulative distribution of the Binomial distribution. A bound is given To trials, with pa probability of success, the probability of having fewer than successes is upper bounded as follows:
ToPa 2s
The above bound derives from the following well known tail bound Feller [1911]
(To s)ph if S ToPA (ToPh s)2
(S,, 5))
where Sn denotes the number of successes. The above bound can be loosened by the following process:
(To s)pn
Toph
Topn
Topn
S
(ToPh s)²
(Toph s)2
Tórn + 8² 2Topps
Tori 2Topus
ToPh.
Toph (ToPh. 2s)
Tuph
which yields the bound that we utilize. We note the above bound requires s S Tops 1 2
however it the
2
case that > Taps then the worst case tail analysis is unnecessary since Taps results still hold.
1 Taps
D2
and our
2
28
Given the above, we use 62 and derive:
82
S
Toph
Topa
D2
2
25g
Given the above, with least Tapn 2
26, points where Bio (..)) 2 of. as well our assumption 0000 (0,0) 2 0,
D2
apply Bienaymé's identity which restate for convenience:
Ca
C
C
Var
Cov (he,e, h..)).
E=1
HRBF kernel satisfies these assumptions when P..
32

---

Published in Transactions on Machine Learning Research (10/2024)
Noting each of the Tops D successes is sampled C1 To times with Cov (he,e, hv,e) 2 of for each of the 26g
successes and Cov (,,,, )e,e)) 2 0 for all samples by our assumption. Applying Bienaymé's identity and the sum D²
of (correlated) Normal variables is also a normal variable, we have Var [ E he,e 2 TT 252 oT
Compare this quantity with the variance if no edge exists between @" and 0², where the variance results from i..d. noise: Var D D Re.s] = To Comparing these two quantities, with an appropriately picked en determines the edge between @a and @ using Lemina 3 By Lemmna 3. letting Toon v 2log 202 ensures
or
that p(h < c+) D3 if edge E exists, and p(h* cx) D' if edge Eyd does not exist. Applying the union bound over D pairs of variables completes the proof with N. P(E E 2 1 61 82.
33

---

Published in Transactions on Machine Learning Research (10/2024)
G Proof of Theorem 2
Our proof of Theorem 2 is presented under the same sctting and assumptions the work of Srinivas et al. (2010).
To prove Theorem 2 we rely on several helper lemmas. The high-level sketch of the proof is to use the properties of Brdös-Rény' graph bound both the size of the marimal clique as well as the number of marimal cliques with high probability. Once these two quantities are bounded, we are able to analyze the mutual information of the kernel constructed by summming the kernels corresponding to the marimal cliques of the sampled Erdös-Rényi graph indicated in Assumption | Finally, once this mutual information is bounded, we use similar analysis as Srinivas et al. (2010) to complete the regret bound.
We begin by bounding the size of the maximal cliques.
Lemma 4. Let Ge = (Va, Ed) be sampled from Erdös-Rényí model with probability pg: Ga ~ G(D, Pg), then vg E 10 the largest clique of Ga is bounded above by
Va
Max- Clique 2 2og 1 P5
[Val + 2 log 1 Pg
with probability at least 1- 5.
Proof. The above relies well known upper bounds on the maximal clique size graph sampled from
Erdös-Rényi model. As shown in (Bollobás & Erdös 1976 and MMalula 1976) the expected number of Cliques of size k, E [Ck] is given by:
[Va k:
MC 1)
log (
|Val |Va|-R+1)
E [C*]
<
VV
Fg
Pg
Pg
Pg
In the sequel, we omit the base of the log: L for clarity. To bound the size of the maximal clique, we find
Pa
suitable such that [Cx] < & and utilize the union bound over [Cile=k.... where we have |[Ca]]=k...... S n. Finally, we utilize Markov's incquality to complete the proof.
V
Let l: 2 log Val 2 log
1.
We utilize the above bound on E [Cx].
k 2 22og IVll k +1)
2
[og log/Val
+ 1 log n /)
(2)(2222))) (2log[Val -2log Val 2
log
n
z +
1 +
S <-log| log n
8
1 < log
8
n
log
[[C <
Pg
The proof is complete by noting that by Markov inequality, p(C, 2 1) S EE [Ck] and taking the union bound over at most n members of [C.)i=k......
34

---

Published in Transactions on Machine Learning Research (10/2024)
Next, bound the total mumber of maximal cliques:
Lemma 5. Let Ga - (Va, Ea) be sampled from Erdös-Réngý model with probability p: Ga G(D, Pg), then vg E (0, 1) the number of total marimal cliques in Ga is bounded above by
1
log |Va++5
Val
Pg
8
with probability least 1 8.
Proof. We prove the above by bounding max; Cs with high probability and noting that the number of maximal cliques is bounded by Σn C& S n max; Ck with high probability, To bound max Cke we first consider max; E [Cx]-
1 (2l0g |Va [Val-kt1)
max (2 [Vll-k+1
max E [C&] max
Pg
Py
Taking the partial derivative of &/2 (2l0g)) |Val - 1s x+1) with respect to k we determine the maximum:
k IVal +11
arg max
(2log Pg
log Pg
[Vl + 1.
Thus we are able to bound:
log VVl +1
(2log)
[Va log Pg
VVl 1 1+1
2
2g
log 1 pg
|Val +1
(log
[Vl = log
[Va + log Va
2
Which yields the bound:
1 Hlog?,
(Val+ IVal
Tox Val+1
[Cx] S
Pg
Pa
Val
Pg
log Val+1
complete the proof, we utilize Markov's inequality with p Cx the union bound over n choices of k:
Val
Va
S [
and utilize
Va
log F3
|Va++1
1
log Val++5
Σ
Val
Va
Flg
with probability 1-6. 6.
Now that we have bounded both the number of cliques, as well as the sizes of the maximal cliques with high probability, we now consider the mutual information of the kernel constructed by summing the kernels corresponding to the maximal cliques of the dependency graph.
Lemma 6. Define 1y)
H(yA) H(ya v) the mulual information belween ya and log 2 as the entropy function. Define री maxAce:/A|=T" I a) when
with H(N(u,,) Σ)) & GP(O, l (6, ()).
Let [kijit.....M be arbitrary kernels defined on the domain O with upper bounds [ M; then the following holds true:
mutuall information
35

---

Published in Transactions on Machine Learning Research (10/2024)
E. S M max (7
To prove the above, first state Weyl's inequality for convenience:
Lemma 7. Let H,P P E Raxn two Hermilian matrices and consider the matrix M H + P. Let Pi, v4 Pi. 1,...,m. n be the eigenwabues of M, H, and P respectively decreasing order. Then, for all r we have
py S Up + Ps-
The above has an immediate Corollary as noted by Rolland et al. 2018):
Corollary 5. Let K2 Raxx be Hermitian matrices for 3...., M with K - M
EN K; Let [ [x Ie=
denote the eigenvalues of K2 in decreasing order. Then for all & E No such that CM + 1 S we have
EM+1
We are now ready prove Lemma G using Weyl's inequality and its corollary as key tool.
Proof. Given the definition of ((yaiv) 1 log| I ²K of + Σ K- using the cigenvalues of [T + o 2 Corollary we see that:
Srinivas et al. 2010 we bound the eigenvalues where ΣM ky Using the above
M
MT+a
KK
1ta 1ta
1
Given the above, we see that 2 mmx(11
li=1.... M 2
loggI + o 2 as ΣM MY 2
llgg|MI +
M Kkl
j
Finally, we require an additional helper lemma to bound the supremum and infimumm of a function sampled from a GP. This helper lemma helps bound the regret during the first phase of DSS-GP-UCE where we randomly sample the Hessian over the domain.
Lemma 8. Let &s 1') be four times differentiable on the continuous domain @ 2 r0 for some bounded r (i.e., compact and conver) with f ~ GP(O, &s (0, 8)) then for all 5 E (0,1) 1) the following holds true:
f cov Covav/Dog Dlogo-1 = 11log--1=00 =0 (Doogoo1)) -V(VDlogo1)
sup 8E
inf f2 -c-c Dlogo 8- 81== (-VV-Dlog @
for some constant dependent on and with probability 1 8.
Proof. The proof of the above is contained in Srinivas et al. 2010 Lemma 5.8, We restate the key parts of the proof herein. In the sctting of Srinivas et al. (2010) there exist constants a, b,, such that
dv < ae bb
P sup Ice
00,
36

---

Published in Transactions on Machine Learning Research (10/2024)
Letting L [log( (og(6))))) min, b,]"/2 have that ae -1 6/(2D) for all i 1....,,, D, that for K1 D./21, by the mean value theorem, we have
P[]x(0)- - u(0)) & Kill0 - 8'11 48, E 0 6/2.
The proof is complete by noting that K1 ((log (1)//-1))1/2) and picking an appropriate a dependent on and r.
We are ready to prove Theorem 2
Theorem 2. Let k be the kernel as in Assumplion and Theorem Let (d) R be a monolonically increasing upper bound function the mutual information of kernel & taking d arguments. The cunulaatve regret of DSS-GP-UCB is bounded with high probability as follows:
Rr RT=
T D log D + +c)
(4)
where Cyy is appropriately picked constant and the base of the logarithm is
P5
37

---

Published in Transactions on Machine Learning Research (10/2024)
We restate the above theorem with more precision:
Theorem 2. Let be the kernel as in Assumplion
7 and Theorem 7 and for some conslands a, b,
P
sup LEEe
dv
L S ae (L/b))
= Li.,.... D.
002
Let (d) R be monotonically increasing upper bound function the mutual information of kernel k laking d arguments. Let k 0') be four times differentiable on the continuous domain θ [0,r] for some bounded (i.e., compact and convex). For any δ1, 82, 83, 81, 85, ò6 E (0.1). Let, & t ToC1 and let
B. log ( + 2D log( DDr (log(4DD/a))))
The cumulative regret of DSS-GP-US UCB is bounded:
P Rr s 20/cu 0logo5' + VCIIBrTT++ +2 VT 2
82 62 63 84
5g
6
when Ci = 1612
pnoj
log 20 o²
on
+
25
D
1, C2 8/ log(1 + t)-2), and
YT D 1OR//y5 D4 ( (20gg//
D + 2 log1/ps / D/63 +1 where is some consland dependent ö5.
Proof. The proof is consequence of the helper lemmas and theorems we have proved. First consider Phase 1 of DSS-GP-UCB where t I To. By Theorem | at most TaC; Cf queries will be made during Phase 1, and Lemma 8 indicates the maximum regret for any query. Consulting the respective Theorem and Lemma, we are able to bound the cumulative regret during Phase 1 by:
20 0 O log D).
Considering Phase 2, we utilize Lemma Lemma Lemma to bound the mutual information of the sampled kernel with high probability, The number cliques is given by:
1081/99 D+5
81
The size of the largest clique is given by:
2logi/p, D + 2 1ogi/n. D/8s + 1 2(l0g1/pg
D + logi/r, D/6g + I 4log1/// /rv D/6g +
Following Lemma see that
E.
ki < 2 max ) S
D loss/vs
D+5 2 logi/ps D/6g + 2).
TT
Let 0, = =4log_ Pg
1/53 + 2 which yields the following information on the mutual information:
Σ. hi < D 1081/p D (4log1/p, D c.). TT
The proof is complete by leveraging the connection between mutual information and cunulative regret as shown by Srinivas et al. (2010) where O the same as with the log factors suppressed.
38

---

Published in Transactions on Machine Learning Research (10/2024)
H On the Surrogate Hessian, H
In Section 4.5 we remarked that although we cannot observe He, we can observe surrogate Hessian, H, which is related to H by the chain rule. We justify our choice here with showing how H2 is an important sub-component of H, (Skorski) 2019). Although the reasoning we give is in one dimension, an analogous argument can be made arbitrary dimensions using the chain rule for vector-valued functions yielding the Hessian tensor (Magalhães) 2020 We have + R is a function of the policy x and can be expressod as a
composition of functions:
v. θ - - 0 (π (0)).
(5)
In the above we use ι (0) as shorthand for a (s", a 0) with é representing some unknown function. Using the definition of the Hessian we have:
0²
H
00h a.b=I.....D
00:
0 ( (0))
b,b=l....D
Where the above identity follows from the definition of in Eq.
We now apply chain rule to express:
0² 0 (TT (0))
Ôm 006
θT n
H. (0))
(0)
(0)
+
(0)
Vi(T(())
(6)
a0
00a
80e 00-00.
00b
5(1)
r(0)
H2
As we see in the above as a consequence of the chain rule, Given the above, we can simplify the above the following manner:
8"
forms an important sub-component
8
30
= H. og
where r, and H, arise from the corresponding highlighted terms in Eq. 6 with representing some unknown remainder term and o representing the Hadamard product. Given the above, it is straightforward to see how H₂ serves as a surrogate Hessian for H,- Indeed if r + -H, H. # = H. * O. In our use case, we are most concerned with non-zero entries in the Hessian, H,, and the surrogate Hessian, H, is well served for determining H, * 0 due to the above.
and has no zero entries then
Since π (0) is shorthand for as;,,, 0), to approximate H2 we average H (0))) over state action pairs, (s a²) formed through interaction of the policy with the unknown task environment.
A possible avenue of overcoming this limitation is considering Hessian estimation through zero'th order queries. Several works along this direction have recently appeared using Finite Differences (Cheng el al., 2021), as well as Gaussian Processes (Müller et al. 2021 We consider removing this dependency on the surrogate Hessian for future work.
39

---

Published in Transactions on Machine Learning Research (10/2024)
Drone Delivery Task
Our drone delivery task was inspired by recent rescarch work in studying unique problems in drone delivery vehicle routing problems (Dorling et al. 2017).
Drones fly from delivery point to delivery point where completing a delivery gives a large amount of reward, but running out of fuel and collisions give a small amount of negative reward. After completing delivery, the delivery point randomly removed within the environment. collision gives small amount of negative reward and momentarily stops the drone. Completing delivery refills the drone fuel and allows it to continue to make more deliveries. The amount of reward given increases quadratically with the distance of the delivery to highly reward long distance deliveries which require long term planning. To compound this requirement for long term planning, fuel consumption also dramatically increases at high velocities to encourage long-term fuel efficiency planning. In this complex scenario requiring long term planning, RL approaches can casily fall into local minima of completing short distance, low reward deliveries and fail to sufficiently explore (under sparse reward) policies which complete long distance deliveries with careful planning.
Implementation code of this task can be found in supplementary materials.
40

---

Published in Transactions on Machine Learning Research (10/2024)
Swimmer 4
Ant 8
PredPrey 6
Het. PredPrey 6
1ue
1ue
0.0
1.0
2.0
0.0
0.5
1,0
1,5
0.5
1,0
1.5
2.0
0.0
1.0
2.0
120
1000
800
1250
100
1000
800
600
80
750
600
60
600
400
40
20
0
200
250
500
Swimmer 8
Ant 12
PredPrey 9
Het. PredPrey 9
700
40
1200
600
0000
1000
750
800
0-
400
500
600
300
-20-
250
400
200
-40
100
250
-500
Swimmer 12
Ant
PredPrey 15
Het. PredPrey 15
30
800
2000
7550
20
400
1500
1500
CDS-QPLEX
G2ANET
200
10
1000
1000
FACMAC
750
MADDPG
SOG
200
500
RODE
-10-
250
DSS-GPUCE (HOM)
400
DSS-GPLCB (CTDE)
-20
Figure Comparison with MARL approaches with varying number of agents.
J
Replot With Timesteps
We replot the relevant figures in Fig. 12 and Fig. 13 while maintaining total environment interactions the singular independent variable. We note that there is significant change to our conclusions consequence of this replotting. We also highlight that although total environment interactions is considered the important independent variable in RL. and MARL, Bo typically the total evaluated policies is considered the more important independent variable as each evaluation is assumed to be costly.
41

---

Published in Transactions on Machine Learning Research (10/2024)
Drone Delivery 6
Drone Delivery 9
1e6
te5
0.0
0.5
2.0
0.0
0.5
1.0
2.0
100
100
80
80
CDS-QPLEX
60
60
G2ANET
COMIX
FACMAC
40
40
MADDPG
SOG
20
20
RODE
DSS-GPUCE (HOM)
DSS (CTDE)
Figure 11: Comparison with MARL approaches the drone delivery task.
K
Replot With "best found policy so far" in RL
We replot the relevant figures in Fig. 12 and Fig. 13 where both BO and MARL approaches show the value of the "best found policy so far.' We note that there is no significant change to our conclusions as consequence of this replotting.
42

---

Published in Transactions on Machine Learning Research (10/2024)
Swimmer 4
Ant 8
PredPrey 6
Het. PredPrey 6
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1,5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2,0
1500
800
120
250
000
600
100
750
500
400
60-
250
40-
0-
200
200
250
0-
500
Swimmer 8
Ant 12
PredPrey 9
Het. PredPrey 9
1500
700
60
255
1200
600
40-
0000
500
750
400
500
600
400
200
0-
-40
200
100
250
-500
Swimmer 12
Ant 16
PredPrey 15
Het. PredPrey 15
2000
1500
20
400
1500
1500
DSS-GPUCB (HOM)
250
DSS-GPUCB (CTDE)
10
200
CDS-OPLEX
1000
1000
G2ANET
0-
750
COMIX
200
500
800
MADDPG
SDG
4000
RODE
0-
-20
2000
2000
2000
Figure 12: Comparison with MARL approaches with varying number of agents.
Drone Delivery 6
Drone Delivery 9
1e6
fe6
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
100
100
80
DSS-GPUCB (HOM)
DSS-GPUCE (CTDE)
60
60
CDS-QPLEX
G2ANET
40
40
COMIX
FACMAC
20
20
MADDPG
SOG
RODE
Figure 13: Comparison with MARL approaches on the drone delivery task.
43

---

Published in Transactions on Machine Learning Research (10/2024)
L Tables With "best found policy so far" in RL
We generate new tables investigating RL and MARL under sparse or malformed reward. In Table and we show the value of the best found policy during the training process for RL and MARL. Our observations and conclusions remain the same where RL and MARI performance severely degrades under sparse and malformed reward and is often outperformed by our DSS-GP-UCD approach.
44

---

Published in Transactions on Machine Learning Research (10/2024)
Table 7: RL under sparse reward. Sparse n refers Lo sparse reward. Delay ni refers to delayed reward. Averaged over runs. Parenthesis indicate standard error.
DU SEAIDNN de
CO EOLIN II LOLLNNER all
20 EOTTRMA 1
THE STID TU AN FOR B9 DE ALL 20200 LOD will 30
THM BY HIS MA TH REC METO MEN I AN SHILENN all
THE ARIIE HIS B BONIIU (ND PRI THE UND 20
20EAAN TH
Table 8: MARL under sparse reward. Sparse n refers to sparse reward. Delay n refers to delayed reward. Averaged over runs. Parenthesis indicate standard error.
7:MM 30 Lue 750
101 1 004
300 SILETT 0S
RACMAC
71 (100
077
2001
1070
DAN THE SN
MADDPG
0M (4
10 (5)
(3)
00L990 15390)
189 10 18)
300 0ND 110 19)
128 228033 D0 WAE 50 2024 OU 3T0 2033E30 2001 20L0 40K DANEL 30 10U0
310
Wel-ohl
SAK
A30000 SE
Walleraa of md
31) 435
21
307
301
405.000
201
481
463
AWTLA 42)
00 50
(0)
100
04 50
13550
40
10 30.00
COMEX
SET PO WN 10133-20
112.000
237
227
377
257
310
280
LIMES SOTT BALON
471 911.9011 2I1T--T00 20100
1H106 201
089
UN)
770
BO SON AN
COVION
OT
300h 2015
200
FED 87100
2010
002
500
250
270
200
NO
90 AND HOS
THM SIREOO
6534 QUISUE 300 NO PHU
IN FES AO II TH WIL AOIOO
PACSAC
3013 48) 23 50000000
02 01/01000 50)
050000
30
365500 70) 25-04:202
26. 11) 24 220
36.5000000 37 33 18
101 30
37) 24 20
25
20.00 30.0 68) 29
22,200 (0) 27
SADDPP
20/0000 501
79)
19)
41 LELB AU MATHO RU
DO
BAD AN SA RALA DOD
22/00
2010
1133
Setmesere Kd
28.
22.
25
20.00
28.
22
33.50
COMIX
30)
20)
200
31 (2)222077
(2.0
23.0100 40)
36)
40)
IT ANT 300
2071
03.58 0110
06 5005
259R
25
30.
22
1943
2110 BI)
2E 20
CDVIN
22.72
3
28.20 20
2551
25.07 03.07.03
24.4
60
22.3811 79
000
23.36
36 26
4200
52.
20
15 311
IN BATTM
IN (0 9000
112
FACSICC
2000 60)
000
2810
1022 5000
351 6R 9R0
3F7 UET LU SSI OPL 100000 30
711
05 MU251
50
310 US (00 301 SSN 00 71)
1OR 30 S3ROWNOSS 20)
3AH BAILAN ANO TH TIR IA ())
100 SOVER BH00000 O0
001 090 090.00
SANE
200
217
03
DR DN BAN
MADDIG
5000003 17)
T7. 2112007000 00)
011 02000011
41 2.200088 1T)
6L 2HCNT 00)
ON DATT DND 19 TH
200200222000
00 US1
1100
Tipger--s
7200
110
20)
2317 2IN 371
COXEX
21/2240
2313000 02)
46)
216.00222000 75)
206 20 (2)
200 201.00
8C. 1.12221000 (7200)
241 45
20)
600 20066344AA A8
7M)
SA0
47.88
753
479
700
1130 T0 2013 PA CILAAY
TWN THE ALL Ae PH
(6)
35
20100 20)
206 62028220
900. (2172)176660 ON
1014 101445457 (413
1008 2314 23)
061 30
1008 83 10
VPA 201 30)
1116 SAAT AU 3200 D9
THIS MOIL 1IM
TEO HORR THE
CONDN
707
000.0777
07178
3214
10.5
1007 1007.901
AH
MAS DE ROIMER DOD TIL COODAL
PACMAC
32 13)
085 1015 57)
56L07102666
63)
57)
507 201122
09 02
00)
96 96)
681
9F7 8030000000
65)
DN AOLLERII 201
1176.3
1077
371
307
300
962
MODE STO 20 LINKU SONIANS
AND
MADDIN
1001000000
(01151110 10)
22)
477099
37777
100 (0)
01 40
505 72)
901 10)
(5)
927 5517787 07)
10
07)
40
AND DI LIDO
Tag S1 LI 30 HE BY WE HIR HE 20T
Aut-v0
1105
1999
1002
960
980
990
COMEX
241 5m
264.41 5.53
0001 (1
00)
200 2713 320
94
623641155
5A2
NE
057
013
515
364
111
600
F50
Barte FOR DELIIIINN 1334 60031
BouIna -1 MO
BRECS3 HIL WE MESOOUT AN
COVDN
1932 3113377200 077
902 02
176
Spacs 700.547 07) 07)
Tag. (43.04(20-0)
Lag 90 20.09 21.09214.35
OT
Los 100 U11 22122000000
30
19
510 00
50
Lag XL 200 799 MSSTI0
DERR
Lng ML 1OUTMTT
Baniin
Space
Spao
Spars 100
Lay 200
Lag XL 100
46


Reviews:
[{"review_id": "vDRjMGexpI", "reviewer": "Reviewer_TYug", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The authors tackle an important problem that is likely to become more important as increased automation requires more lightweight and efficient solutions for MAPS.\n- The authors present a promising solution that addresses the problems they associate with alternative methods. They provide empirical results as well as theoretical findings that support the usefulness of their method.\n\n### Weaknesses\n- The authors' method assumes the existence of a small number of roles that cover the roles of the agents. Further discussion into when this assumption is reasonable and when it is not is important to have in the main text. Their experiments indeed show that for some tasks it might be preferable to not have this component. The maximum number of agents they include in their experiments is 15, one would expect their method to be relevant in scenarios where num. roles << num. agents.\n- A similar assumption is made for the value function. A better justification for this decision in realistic MAPS problems should be provided, and the resulting behavior when this assumption does not hold should be discussed.\n- Hyperparameter selection for the two central assumptions described above is not described in detail. More guidance regarding the choice of number of roles and number of additive components should be provided.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors utilize Bayesian optimization (BO) for high-dimensional multi-agent policy search (MAPS), as gradient-based methods can suffer from excessive computational requirements and sparse rewards, as well as the existence of local maxima. The authors mitigate the computational complexity of such an undertaking by relying on the abstractions of \"role\" and \"role interaction\"; they also assume an additive structure for the reward function, the exact structure of which they estimate using a surrogate for the parameters' Hessian.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Addressing the weaknesses mentioned above would strengthen the authors' arguments and exposition.\n- The introduction is hard to follow and it takes a while to figure out the authors' contributions. E.g. the connection between the paragraph starting with \"To optimize the HOM...\" and the previous paragraph is not immediately clear. Figure 1 is not very helpful without the relevant context. I think a slightly longer introduction that includes a mathematical overview of the authors' method would be more helpful (Table 1 can be moved to Appendix), but I am happy with this structure as well if it does a better job of onboarding the reader.\n- The concept of belief $\\nu$ is not introduced.\n- The letter $r$ is used to denote both the reward function and role.\n- How are the cutoff constant $c_h$ determined?\n- Please explicitly introduce that \"Med\" in \"Med - RA - RI\" refers to medium size.", "broader_impact_concerns": "In my estimation the present work does not require a broader impact statement.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hAu8S58CDK", "reviewer": "Reviewer_kavf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThe paper presents a BO setup and method with an improved regret bound. The experiments demonstrate that BO can be competitive with (or even better than) RL on at least some non-trivial environments.\n\nWeaknesses\nI understand there’s a lot of background information, and BO is not my general area of expertise, but I found the paper hard to follow. On the positive side, the authors do present examples and diagrams giving an overview in the spots where I would expect to see this, but the text / diagrams are following different conventions than I would expect or are assuming knowledge I don’t have.\nFor one concrete example, Figure 2 has state, Algorithm 2, and theta_r feeding into GEN to produce s^alpha(i). This, Alg 2, and theta_g are then feeding into GEN to produce pi. My immediate expectation from that diagram is that GEN is the same. At first impression, it’s also not clear what it means for Alg 1 to be an input of GEN. The text talks about a two stage process with an algorithm for the lower layer, so seeing a diagram with an algorithm as an input is not unexpected, even though it is in fact just the fixed step taken to generate the permutation. The diagram I would have expected to see is maybe something like\n\n```\n(s^1, …)      theta_r\n    \\            /\n    Role Assignment\n            |\n       (s^alpha(1), …)\n      /             \\\nRole Interaction    |\n     |              |\n  Graph N  theta_g  |\n       \\     |     /\n           MPNN\n             |\n             pi\n```\n\nwith the whole process being labeled as GEN.\n\nAt the point in 4.1 where Figure 2 is included, the choice to have “role” be entirely synonymous with permutation has not yet been introduced. Given an expectation of everyday use where we might expect to have a many to one mapping from index to some set of roles, having a permutation (s^alpha(1),...) is unexpected.\n\nI found much of the explanation to have a similar problem: unexpected way of describing things, using things before describing them, or incomplete description – again noting that some of that incomplete description might just be assumed knowledge for someone more familiar with BO.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents DSS-GP-UCB, a role-based hierarchical Bayesian optimization (BO) method for cooperative multi-agent tasks. The authors introduce a role assignment and interaction model using role affinities and a graphical model, and a learning process for learning an additive decomposition of the observation dimensions. The authors state a regret bound, noting an improvement in the asymptotic complexity w.r.t. the number of dimensions. They also include experimental results on a few open multiagent environments and a synthetic drone delivery environment, comparing DSS-GP-UCB to a number of existing Bayesian Optimization and RL algorithms.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Appendices don’t seem to be included.\n\nIntroduction: “This restriction requires the usage of large gradient-friendly policy representations”.  Do the authors have a specific sense of large they mean here, and an argument to support that? Otherwise, I’m not sure it requires large representations: I think it would be reasonable to say many RL algorithms -do- use larger NN to get empirically stronger performance, but not that they require large models. Smaller models – either small NN or linear weights on something like tile coding – can also have surprisingly competitive performance on some non-trivial problems.\n\n“... and informative reward feedback from the environment.” This should be expanded to be more clear: without more context, observing the rewards for the trajectory an agent takes seems like a reasonably low requirement.\n\nSection 4.3: “the number of interactions scales quadratically due to considering interactions between all pairs of agents.”  Isn’t pairwise interaction already a generalization, as interactions are possible between all agents? E.g., something doesn’t happen unless three (or more) agents coordinate on a decision.\n\n“To generate graphical models of the above form, our HOM uses edge affinity functions… Edge affinity functions Lambda…”  Provide a (short description) of how: this is part of the contribution of this work, don’t just point at the algorithm.\n\nSection 4.4: “Specifically Theta defeq Theta^1 x … x Theta^D for some dimensionality D, and Theta(i) … is of low dimensionality.” This definition seems to be general and allows for decomposition into multi-dimensional spaces. For the subsequent theory and bounds, are each of Theta^i assumed to be 1-dimensional spaces?\n\nSection 4.5: “We utilize the surrogate Hessian in our work” Given the introductory motivation noting a limitation of MARL methods being they require gradient-friendly policy representations, it seems odd to see the proposed alternative computing any sort of Hessian.\n\nSection 5: “interactions with the environment in l”  what is l?\n\nSection 5.1: References for Multiagent Ant, PredPrey, and Mujoco\n\nWhat exactly are the ablated agents? E.g., what does it mean to consider role interaction without role assignment?\n\nFigure 4: label doesn’t seem to match. There are left, middle, and right, and all seem to be comparisons. Left plot is unlabeled. Middle plot is largely obscured by the labels.\n\nWhat exactly is Sm compared to Med?\n\nSection 5.2: What exactly is DSS-GP-UCB (CTDE)?\n\nSection 5.3, Table 2: What is the setup? How long are agents trained? What does running DSS-GP-UCB mean if there are no rewards received on some steps? Given there is only a single line in Table 2 for DSS-GP-UCB, the authors are presumably saying there is absolutely no difference whatsoever between receiving rewards every step, or every 200 steps. How is this possible?\n\nFigure 6: What are the axes?", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0NXsfOx2Fr", "reviewer": "Reviewer_LAWU", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nThe content of this paper is substantial, and its structure is clear. The proposed DSS-GP-UCB method addresses the scalability issue in high-dimensional settings of complex decision-making models, which is a significant challenge for traditional Bayesian Optimization approaches. Moreover, the experiments presented in this paper are also comprehensive.\n\n**Weaknesses:**\n\nThere are some confusing aspects in this paper, primarily in the algorithm description and theoretical guarantees. Regarding the algorithm description, Algorithm 4 is not described in sufficient detail, making it unfriendly to those unfamiliar with the GP-UCB algorithm. As for the theoretical guarantees, the use of O-notation twice in Theorem 2 is quite unusual.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces Dependency Structure Search Bayesian Optimization (DSS-GP-UCB), a novel method that efficiently optimizes multi-agent decision-making models by leveraging role-based abstractions. This approach addresses the scalability issues in high-dimensional settings, making the optimization process more tractable and effective. Their method demonstrates strong empirical performance, particularly in environments with malformed or sparse rewards, and provides an improved regret bound under reasonable assumptions.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The formatting of Algorithms 1-4 needs to be revised, and additional descriptions should be provided for these algorithms. \n* In the experimental results, the legend in Figure 4 should not obscure the Sparse reward curve.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers all agreed that the paper tackles an important problem and makes a useful theoretical contribution.

There were some concerns on the clarity of the paper, which have been addressed by the authors. In the final discussion all reviewers agree on an accept decision.
Decision: Accept (poster)