Multi-agent Dynamic Algorithm Configuration
Authors: Ke Xue
Jiacheng Xu
Lei Yuan
Miqing Li
Chao Qian
Zongzhang Zhang
Yang Yu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Multi-agent reinforcement learning
Multi-objective optimization
Evolutionary algorithm
Automated machine learning
Dynamic algorithm configuration.

Abstract: Automated algorithm configuration relieves users from tedious, trial-and-error tuning tasks. A popular algorithm configuration tuning paradigm is dynamic algorithm configuration (DAC), in which an agent learns dynamic configuration policies across instances by reinforcement learning (RL). However, in many complex algorithms, there may exist different types of configuration hyperparameters, and such heterogeneity may bring difficulties for classic DAC which uses a single-agent RL policy. In this paper, we aim to address this issue and propose multi-agent DAC (MA-DAC), with one agent working for one type of configuration hyperparameter. MA-DAC formulates the dynamic configuration of a complex algorithm with multiple types of hyperparameters as a contextual multi-agent Markov decision process and solves it by a cooperative multi-agent RL (MARL) algorithm. To instantiate, we apply MA-DAC to a well-known optimization algorithm for multi-objective optimization problems. Experimental results show the effectiveness of MA-DAC in not only achieving superior performance compared with other configuration tuning approaches based on heuristic rules, multi-armed bandits, and single-agent RL, but also being capable of generalizing to different problem classes. Furthermore, we release the environments in this paper as a benchmark for testing MARL algorithms, with the hope of facilitating the application of MARL.

Multi-agent Dynamic Algorithm Configuration
Ke Xue'; Jiacheng Xu'; Lei Yuan 1.2
Miqing Li,, Chao Qian'! Zongzhang Zhang', Yang Yu
State Key Laboratory for Novel Software Technology, Nanjing University Polixir Technologies
3 CERCIA, School of Computer Science, University of Birmingham (xuek, xujc, yuanl @lammaa njj. edu. en, m. ww.1..88biii 8wbhambbbmmm......... ac. uk, (qianc, zzzhang, yuy junju @nj..edu. cn
Abstract
Automated algorithm configuration relieves from tedious, trial-and-error tun- ing tasks. A popular algorithm configuration tuning paradigm is dynamic algorithm configuration (DAC), in which agent learns dynamic configuration policies across instances by reinforcement learning (RL). However, in many complex algo- rithms, there may exist different types of configuration hyperparameters, and such heterogeneity may bring difficulties for classic DAC which uses a single-agent RL policy. In this paper, we aim address this issue and propose multi-agent DAC (MA-DAC)) with one agent working for one type of configuration hyperparameter. MA-DAC formulates the dynamic configuration of complex algorithm with mul- tiple types of hyperparameters a contextual multi-agent Markov decision process and solves by a cooperative multi-agent RL (MARL) algorithm. instantiate, we apply MA-DAC to a well-known optimization algorithm for multi-objective optimization problems. Experimental results show the effectiveness of MA-DAC not only achieving superior performance compared with other configuration tuning approaches based on hcuristic rules, multi-armed bandits, and single-agent RL, but also being capable of generalizing different problem classes. Furthermore, release the environments in this paper as a benchmark for testing MARL algorithms, with the hope of facilitating the application of MARL.
Introduction
Finding right configurations of hyperparameters is critical for many learning and optimization algorithms [18]. Automated methods, such as algorithm configuration (AC) [17] 27), emerge to search for right configurations, with the aim of relieving users from tedious, trial-and-error tuning tasks. However, static configuration policies obtained by AC may not necessarily yield optimal performance (compared with dynamic policies) since algorithms may require different configurations at different stages of their execution
38
Dynamic algorithm configuration (DAC) configuration policies. Unlike AC, DAC can dynamically adjust the algorithm's configuration during the optimization process, through formulating it contextual Markov decision process (MDP) and then solving it by reinforcement learning (RL) 33 DAC has been found to outperform static methods many tasks, including the learning rate tuning of deep neural networks (7), step-size adaptation of evolution strategies [35] and heuristic selection of Al planning (38).
3
well-known paradigm for obtaining dynamic
'Equal Contribution Corresponding Author
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

The task of DAC typically focuses on single type of configuration hyperparameter, such as tuning the step-size in CMA-ES [35]. However, due to the increasing complexity of real-world problem modeling, there are many algorithms whose performance rests on multiple types of hyperparameters. Tuning one type while fixing the rest may not produce promising results. Take a popular evolutionary algorithm for multi-objective optimization problems, MOEA/D [51], as example. It has four types of configuration hyperparameters: weights, neighborhood size, reproduction operator type, and parameters associated with the reproduction operator used. All of them are critical and significantly affect the performance of the algorithm [4). Finding a near-optimal configuration combination for each part of these hyperparameters requires massive manual effort [12]. However, how to jointly
adjust multiple types of configuration hyperparameters is still an open problem [.]
In this paper, we attempt to extend DAC to deal with tasks with multiple types of configuration hyperparameters. We model this as cooperative multi-agent problem 29,, with each agent handling one type of hyperparameter, for shared goal (i.e., maximizing the team reward). Specifically, the proposed multi-agent DAC (MA-DAC) method formulates the task as contextual multi- multi-agent MDP (MMDP) [5], and solves it by common pay-off cooperative multi-agent RL (MARL) algorithm. As an instantiation, we apply MA-DAC to the multi-objective evolutionary algorithm MOEA/D [51] to learn the right configurations of its four types of hyprraaraeteess by the classic MARL algorithm called decompoccooooiiionnn networks (VDN) 40
Experiments on well-established multi multi-objecttive optimization problems show that MA-DAC outper- forms other configuration tuning methods based on heuristic rules [I, multi-armed bandits [5] and single-agent RI. (28). Furthermore, we demonstrate the generalization ability of the learned MA-DAC policy to both inner classes (different instances with the same number of objectives) and outer classes (different instances with different numbers of objectives). Ablation studies also demonstrate the importance of tuning every type of hyperparameter.
Our contributions are three-fold:
1. To the best of our knowledge, the MA-DAC method is the first one to address the dynamic configuration of algorithms with multiple types of hyperparameters.
2. The contextual MMDP formulation of MA-DAC is analyzed, and experimental results show that the presented formulation works well and has good generalization ability.
3. The instantiation configuring MOEA/D in this work can used as benchmark problem for MARL. The heterogeneity of MOEA/D's hyperparameters and the stochasticity of its search can promote the research of MARL algorithms. Besides, the learned policies are useful for specific type of optimization task multi-objective optimization, which will facilitate the application of MARL.
2 Background
2.1 Dynamic algorithm configuration
Different from the static configuration of AC, DAC aims at dynamically adjusting the configuration hyperparameters of an algorithm during its optimization process. Biedenkapp et al. [ formulated DAC a contextual MDP Mz MM-z and applied RL to solve it. I represents the space of problem instances, and each M, := (S. A. Tr, r.)
B 10 corresponds to one target problem instance i I. The notion of context I allows to study generalization of policies in principled manner (20]. Given a target algorithm A with its configuration hyperparameters space O a DAC policy II maps the state 8 S (e.g., history of changes in the objective value achieved by the target algorithm A) to the action a E A (i.e., hyperparameter E θ of the target algorithm A). DAC aims improving the performance of A on set of instances (e.g., optimization functions). Given a probability distribution over the space I of problem instances, the objective of DAC is to find an optimal policy x². That is,
/ p(i)c(T, ildi,
a' arg min
(I)
TEll JEZ
where E I an instance to be optimized, and e(TT, i) E TR is the cost function of the target algorithm with policy и the instance i. It has been shown that DAC outperforms static policies in learning rate

---

adaptation in SGD 77. step-size udaptation in CMA-ES B3), and heuristic selection in planning [38] These applications all only involve single type of hyperparameter. Dynamic configurations of complex algorithms with multiple types of hyperparameters have been found to be difficult for current DAC methods TO 1.
2.2 Multi-agent reinforcement learning
multi-agent system [48] under fully observable cooperative situation can be modeled as an MMDP [5], which can be formalized M :- (N,S, {Ay}}= T,r), where N is set of agents, S is the state space, and A, is agent j's action space. At each time-step, agent E N acquires 8 E S and then chooses the action aQ) E A,.- The joint action a = (a(1) (a!")....,a ....,.(m)) leads to next state s' T( a) and all agents get shared global reward r(s,a).
The goal of an MMDP is to find a joint policy that maps the states to probability distributions over joint actions, + A(Ai A x .... x An), where ((Ai x A2 x ... x An) stands for the distribution over joint actions, with the goal of maximizing the global value function:
= B
rrr( a.) 0.00
3 Multi-agent DAC
This section is devoted to the MA-DAC method, where Section 3.1 is concerned with formulating MA-DAC contextual MMDP and Section 3.2 explains components of MA-DAC.
3.1 Problem formulation
We propose MA-DAC as paradigm for solving the dynamic configuration of algorithms with multiple types of hyperparameters. We formulate MA-DAC as a contextual MMDP MM := (Mi)azz, where M; == (N S, (A,)* r,) is a single MMDP as defined in Section 2.2 The
notion of context induces multiple MMDPs 3 [O]. Each M, stands for a specific instance i sampled from given distribution over I, where instance E I corresponds to function f to be optimized. Different M,s have shared state and action spaces, but with different transitions and reward functions. Algorithms are often tasked with solving different problem instances from the same or similar domains. Searching for well-performing parameter settings on a specific instance might achieve strong performance on that individual instance but might perform poorly in new instances. Therefore, we explicitly take the instance distribution I as context into account to facilitate generalization [3].
Given parameterized algorithm A with a configuration space θ = {0, jeNs one agent in MA-DAC handle type of configuration hyperparameter 0g E O and all the agents work with the same goal, i.e., maximizing the team reward. The action space of each agent j E N is the hyperparameter space of the corresponding fy. Note that MA-DAC can be seen common-payon cooperative MARL problem and can be solved by any cooperative MARL algorithm.
3.2 Components of MA-DAC
Next, we introduce several important components of MA-DAC, including the state, action, transition, reward, and instance set.
State. The state is used to describe the situation of the target algorithm, which is a key component in MA-DAC. We suggest that the state should have the following properties.
1. Accessibility. The state should be accessible in each step during the optimization process.
2. Representation. The state should reflect the information in the optimization process.
3. Generalization. The leamned policy is expected generalize to inner and outer classes of instances. Thus, the state should consist the common features across different instances.
Besides, is better the state can be easily obtained, reducing the computational cost. Note that some of the above properties of state formulation have also been suggested for DAC 24 [0].

---

Action. Each agent in MA-DAC focuses on one type of configuration hyperparameter. Its action the value of the hyperparameter that should be adjusted. which can be discrete or continuous. MA-DAC agents are heterogeneous because they have completely different action spaces and affect different types of hyperparameters of the target algorithm.
Transition.
The transition function describes the dynamics of the problem. For an iterative algo-
rithm, each iteration can be defined a step. Given state st, each agent j acts at and the probability of reaching state St+1 can be expressed as T(s++1 St, ag). Different from the state and action spaces, the transition function depends on the given instance f E F. Thus, the policy should identify the characteristics of the current instance and take the optimal action under it.
Reward. The reward used to guide the policy's 's learning process, whose quality has significant impact on the policy's & final performance. Many RI. benchmark environments include a well-defined reward function. However, many application scenarios, we must define the reward function manually based on a specific metric that can indicate the final performance.
Since the goal of an iterative optimization algorithm iteration is to find a solution with the best objective function value so far rather than to just find a better solution than the solution at the - 1 step (since the quality of solutions in the last step may be poor), would make sense to give the policy a reward when finding better solution than the best solution so far (rather than finding a better solution than the solution the last step). Another factor one may need to consider when designing reward function is that with time, it is getting harder to find better-quality solutions. In the beginning. may be easy to achieve rapid improvement of solution quality, but later stages, improvement could be harder. As such, a reward function that rewards more an agent who can find a better solution in later stages can encourage the agent to find a very good solution in the end. Considering these two factors, we propose the following reward function at step t:
(1/2) (p2+1 pt) if f(4411) ft 0
Tt
otherwise
where
[(en) ((ess)
if f((42) < fi otherwise
P++1
[(eo]
(4)
Pr
and f? is the minimum metric value found until step t.
As shown in Figure [ the isosceles right-angled triangle indicates the maximum return we can get, where p" is the largest value of Pt. At step f, the current reward we obtain the area of the triangle with side length P++1 minus the area of the triangle with side length Pt. The proposed triangle-based reward function has two properties: 1) it only rewards the agent when a new solution better than the best solution found far and 2) the reward increases in later stages. The comparative experiment is provided in Appendix B.2. This design may be applicable to a wide range of similar tasks.
Pt Pe+1
p'
Curullative rewards already obtained
Current reward obtained
Future cumulative rewards available
Instance set. Instance set defines the optimization prob- lem instances that have to be solved by the target algo- rithm. Note that it is possible to learn policies even across very heterogeneous problem instances by using the context information instances have some similar properties, the learned policy can generalize better [0]
Figure 1: Illustration of the reward function.
3 35
Admittedly, if the
4 Applying MA-DAC to dynamic configuration tuning of MOEA/D
As an instantiation, we apply MA-DAC to a well-established evolutionary algorithm for multi- objective optimization problems, multi-objective evolutionary algorithm based on decomposition (MOEA/D) [51]. As one of the most widely used multi-objective evolutionary algorithms, MOEA/D

---

has several heterogeneous types of configuration hyperparameters (e.g., weights, neighborhood size, and reproduction operator type), which good fit for the proposed MA-DAC. We first briefly introduce multi-objective optimization and MOEA/D in Section followed by detailed explanations of applying MARL to MOEA/D Section 4.2] which results in an MARL. benchmark, denoted by MaMo. Finally, we compare MaMo with other MARL benchmarks in Section 4.3
4.1 Multi-objective optimization and MOEA/D
Multi-objective optimization refers to an optimization problem with more than one objective to be considered simultaneously. It is very common in the real world. For example, in deep neural networks, apart from the accuracy, one may also care about the model size, latency, and/or energy consumption 26 14.
prominent feature of multi-objective optimization problems (MOPs) is that unlike single-objective optimization problems, in MOPs there is no single optimal solution, but a set of Pareto optimal solutions. solution is called Pareto optimal if there no other solution in the search space that dominates
i Unfortunately, the number of Pareto optimal solutions for given MOP is typically prohibitively large or even infinite. Therefore, the goal of a multi-objective optimization algorithm is to find good approximation that well represents all the Pareto optimal solutions.
Evolutionary algorithms have demonstrated their effectiveness to solve MOPs. Their population- based nature can approximate the problem's 's optimal solutions within one execution, with each solution in the population representing unique trade-off among the objectives. MOEA/D (51) is a representative multi-objective evolutionary algorithm. Unlike other mainstream multi-objective evolutionary algorithms like NSGA-II (8), which compare solutions based on their Pareto dominance relation and density in the population, MOEA/D converts an MOP into number of single-objective sub- uub-problems through a number of weights, where neighboring solutions work cooperatively for the optimal solutions for the single-objective sub- nub-problems. As such, MOEA/D entails different heterogeneous types of hyperparameters, e.g., weights and neighborhood size. Such hyperparameters significantly affect the performance of the algorithm. For example, different weight distributions are suitable for MOPs with different Pareto front shapes as the weights are used to control the distribution of the final population 233 A large neighborhood has high search ability in the objective space while small neighborhood is beneficial for diversity maintenance in the decision space [19]. In short, finding right configurations for these hyperparameters requires massive manual effort and different configurations work well on different MOPs.
Note that some recent efforts have been made to adjust the hyperparameters of MOEA/D by heuristic rules [3I,, multi-armed bandit (2I], and RL [43]. However, they all focus on a single type of hyperparameter, and also very few of them work on the dynamic algorithm configuration well.
4.2 MA-DAC for MOEA/D
In this section, we show how to apply MA-DAC learn the configuration policy of MOEA/D and also introduce the resulting environment MaMo. Each generation in the evolutionary process of MOEA/D is one step the MaMo.
The state of MaMo can be divided into three parts.
1. To describe the general properties of the optimization problems, the first part is the features of the problem instance, including the number of variables and objectives.
2. To emphasize the general information of the algorithm, the second part includes the features of the optimization process, i.c., how much computational budget has been used and how many steps of the algorithm have not made any progress.
3. To show the properties of the population and that how the population evolved, we use several indicators, i.c., hypervolume [53], ratio of non-dominated solutions in the population and average distance of the solutions, in the third part. For each indicator, we also use the gap between the current value and the value corresponding to the last population to
FFrr two solutions r and y. is called to dominate y. x not worse than y on all objectives and better than y on at least one objective.
2 "ypeevvolume is quality indicator that reflect both convergence and diversity of solution set 22
5

---

reflect the immediate evolutionary progress. Besides, we use statistic metrics (the mean and standard deviation) of the indicators in several recent steps and all steps from the beginning to measure the short and long histories of the optimization, respectively.
To facilitate the generalization of the learned MA-DAC policy, we have pre- pre-proeesse some state features. More details about state are provided in Appendix A.2 due to space limitation.
We consider four heterogeneous types of configuration hyperparameters in MOEA/D as the actions of different agents of MA-DAC.
1. Weights. In MOEA/D, weights are used maintain the diversity of the converted single- objective sub-problems (31]. The action space is discrete with two dimensions, i.e., adjusting (T) and not adjusting (N) the weights. Furthermore, we limit the frequency of adjustment, because too frequent adjustment will lead to drastic changes in the sub-problems and is detrimental to the optimization process.
2. Neighborhood size. The neighborhood size is to control the distance between solutions in mating selection; small size helps the search exploit the local area while large size helps
the search explore a wide objective space [47]. We discretize the action space into four dimensions. That is, the neighborhood size can be 15, 20, 25 or 30.
3. Types of the reproduction operators. We consider four types of differential evolution (DE) operators. Each type has different search ability [2] 36
4. Parameters of reproduction operators. The parameters (e.g., scaling factor) of the reproduc-
tion operators in MOEA/D significantly affect the algorithm's 's performance [3 and the action space has four discrete dimensions.
The detailed definitions of the actions are given in Appendix A.3 due to space limitation.
We use the MOPs with similar properties from the well-known MOP benchmarks DTLZ and WFG [16] as the instance set of MaMo. These MOPs can have different number of objectives, and those with the same number of objectives can be seen as inner classes because the number of objectives considerably affects the properties and difficulty level of MOPs. We use the inverted generational distance (IGD) as the metric in Eq. B3) resulting in the reward function.
To learn the policy, we use a classical MARL algorithm named VDN [40], which widely used in cooperative multi-agent systems. VDN learns to decompose the team value function into agent-wiss value functions, alleviating the exponential growth of the action space. It follows the Individual- Global-Max (IGM) principle (37), i.e., the consistency between joint and local greedy action selections by the joint value function Qu.t(s: a) and individual value functions [Q,(s, a())) j=1 All parameters
in MA-DAC are updated using the standard TD loss from the global Q-value Qiot. which follows additivity to factorize the global value function, i.e., Qua(s, a) Σÿ= Qj (s, al)).. In the testing phase, each agent acts greedily with respect to its Qy.
4.3 Comparison among MARL benchmarks
There are many benchmarks 30
emerged in recent years. In order to clearly show the character- istics of MaMo, we compare it with some MARL benchmarks, as shown in Table Heterogeneity refers to agents having different action spaces or skills 45). For example, in MaMo, there are four entirely different action spaces, i.e., weights, neighborhood size, types of the reproduction operators and their parameters. Stochasticity means that performing the same action in the same state may lead to different next state. The high stochasticity of MaMo comes from the randomness of MOEA/D.
The primary benchmarks are designed to investigate different aspects of multi-agent systems. For example, Matrix Games [6] and Multi-Agent Particle Environments (MPE) [25] are popular and classical testbeds, to investigate cooperative or competitive behaviors in small scale settings (with 2-3 agents). MAgent [52] is used to test the scalability of MARL methods, where the number of agents up to 1000. SMAC (34] has attracted wide attention in recent years to test the coordination ability in cooperative MARL [I]
49 30). For the first time, Active Voltage Control [46] creates an exciting yet challenging real- weal-world scenario for the application of MARL. To the best of our knowledge, none of the known benchmarks has focused on highly heterogeneous and stochastic scenarios in MARI.. We hope our new MaMo benchmark can offer good supplement that could benefit the MARL community.

---

Table Overview of MARL benchmarks and their properties.
Benchmark
Heterogeneous
#agents
Stochastic
Application scenarios
Matrix Games
2
Low
Game
MPE 25
2-3
Low
Game
MAgent
2-1000
Low
Game
SMAC
34
2-30
Low
Game
Active Voltage Control
46
3-38
Low
Control
MaMo (Ours)
2-4
High
Optimization
Experiments
To examine the effectiveness of MA-DAC, conduct experiments using MaMo. We investigate the following research questions (RQs). RQI: How does MA-DAC perform compared with the baseline and other tuning algorithms? RQ2: How is the generalization ability of MA-DAC? RQ3: How do the different parts of MA-DAC affect the performance? We introduce the experimental settings in Section 5.1
and investigate the above RQs in Section 5.2
5.1 Experimental settings
We consider several well-established multi-ohjective test functions (DTL.72, DTLZZ and WFG4- WFG9) with 3, 5 and 7 objectives following the practice in Bezerra et al. 22 to examine our method. In all of the following experiments, several arbitrary functions (here DTTZ22, WFG4 and WFG6) from the function set are used as the training set, and all the other functions are considered as the testing set. Note that "train" and "test" in the tables mean that the comparison is based on the training (i.e., DTLZ2, WFG4 and WFG6) and testing (i.e., DTLZ4, WFG5, WFG7, WFG8 and WFG9) problems, respectively. IGD 4 is considered as the metric to measure the performance of the algorithms. The mean and standard deviation of the IGD values obtained by each algorithm on each MOP for 30 independent runs are reported. We apply the Wilcoxon rank-sum test with significance level 0.05. For fair comparison, other parameters (such population size and computational budget) are set to be the same across all the compared algorithms. More details about the settings be found in Appendix B.1 due space limitation. Our code available https: thtp::
t.ub.. com/ lamda bbo/ maad/maaac
5.2 Experimental results
RQ1: How does MA-DAC perform compared with the baseline and other tuning algorithms? We compare the MA-DAC policy with the original MOEA/D (5T), DQN 28 and MA-UCB (T5). DQN is single-agent RL algorithm that shares the same state, reward, and transition as our MA-DAC but has a different action space that is the concatenation of the four types of hyperparameters. DQN can be seen an instantiation of DAC in MaMo. MA-UCB is simplified algorithm that uses four UCB agents to adjust the four types of hyperparameters. For MA-DAC and DQN, we perform the training and testing the problems with the number of objectives.
The results are shown in Table 2 As can be seen in the table, MA-DAC significantly superior to the other algorithms on almost all the 24 problems. Furthermore, MA-DAC (which is trained only on DTLZ2, WFG4 and WFG6) shows excellent performance on previously unseen problems during the training process, demonstrating its good generalization ability. DQN policy is significantly worse than MA-DAC on all the problems, which is consistent with the previous observation that DQN performs poorly in MARL tasks [33. DQN suffers from the exponentially increasing action space with the number of agents 229 50] while MA-DAC effectively decomposes the action space, making each agent much easier to find their near-optimal policy and then leading to a good joint policy, The superiority of MA-DAC over MA-UCB also demonstrates the necessity of learning configuration policy by MARL algorithm.
We also compare our algorithm with two adaptive MOEA/D algorithms, i.e.. MOEA/D- FRRMAB [21 and MOEA/D-AWA B1] which adjust reproduction operator types and weights based on MAB and heuristic rules, respectively. The results clearly show the superior performance of MA-DAC, and are provided in Appendix B.3 and Appendix B.4 due to space limitation.
7

---

Table 2: IGD values obtained by MOEA/D, DQN, MA-UCB and MA-DAC different problems. Each result consists of the mean and standard deviation of 30 runs. The best mean value on each problem highlighted in bold. The symbols '4', and "Re indicate that the result is significantly superior to, inferior to, and almost equivalent to MA- MACC respectively, according to the Wilcoxon rank-sum test with significance level 0.05.
Problem
M,
MOEA/D
DQN
MA-UCB
MA-DAC
4.605B-02 (3.54F-04)
4.628F--2 (2 9..66---0 (1)
4.671E-02 (3.70F 04)
3.807E:02 (5.05F-00 04) 2.442E-01 (1.26E-02)
DTL22
3.006E-01
(1.55E-03)
3.016E-01 (L34E-03)
3.041E-01 (1.69E-03)
4.455E-01 (1.41E-02)
4.671E-01
(1.15E-02)
4.826B 01 (9.59E: 03)
3.944E-01 (1.17E-02)
5.761E-02 (5.410)04)
6.920E-02
(1.20E-03)
7.165E-02 (1.83E-03)
5,200E-02
(1.19E-03)
WFG4
3.4426-01 (1.2 (1.21E-02)-
2.810E-01
(6.86--03)
2.859E-01 (6. (6.777-03)
L86SE-01 (2.81E-03)
4.529F: 01 02)
3.725E 01 (1.14:02)
3.868B-00 01 (1.54E-02)
3.0338-01
(3.66E (3)
6.938E-02 (5. -0E))---)
6.834E-02 (L.7E--02)
6.601E-02 (1.00E-02)-
4.83E-022 (8. (8.95E-03)
WFG6
3.518E-01 (2. 2.822-03)
3,160E-01
(2.40E-02)
3,359E-01 (1 (1.47E-02)
1.942E-01 (6.90E-03)
4.869E-01 (3.03E-02)
4.322E-01 (2.95E-02)
4.389E-01 (3.41E-02)
3,112E-01 (4.93E-03)
Train: +- 105
0/9/
0/9/0
0/9/0
6.231B-02 (8.85E-02) 02) F
5.590E-02 (5.77E-03)
6.011B-02 02 (S.08E 03)
6,700E-02 02 (6.14E-02 02--02)
DTLZ4
3.133E-01 (4.45E-02)
3,457E-01
(L61E-02)
3.492E-01
(169E-02)
2.995E-01 (2.10E-02)
4.374B-01 (2.57E-02)
4,552E-01
(1.47E-02)
4.756E-01
(2.01E-02)
4.182E-01 (1.21E-02)
6.327B-02 (1.10F-03)
6,212E
(5.54F 04)
6.118E-02 02 (7.03E-04)
4.7300E-02 (7.89E-08 04)
WFGS
3.350E-01 (9.77E-03)
3.077E-01 -11 (6.36E-03)
3.036E-01 (8.83E-03)
1811E-01 (3.02E-03)
4.101E-01 (2.08E-02) 02)
4.996E-01 01 (L.32--02) 02)
5.024B-01
(1.38E-02) 02)
3,206E-01 (8.04E-03 (3)
5.811E-02 (6.31E-04)
5,930E-02 (7.32E-04)
6.014E-02
(7.11E-04)
4.066E-02 (5.31E-04)
WFG7
3.572B-01 (5.471--3))
2.993E-01
(143E-02)
3,207E-01
(1.71E-02)
18888-01 (2.12E-03)
5.236B-01 01 (2.19E-02) 02)
4.576E 01 (2.38E 02)
4,879E-01
(2.75E 02)
3,258E-01 (1.25E 02)
8.646E-02 (3.44E-03)
9.280E-02 (L06E-03)
9.612E-02 (L48EE-03)
7.901E-02 (1.19-003
WFG8
4.258E-01 (8.42E-03)
3.969E-01 (1 (.226E-02
3.956E-01
(1.32E -2)-0))
2.479E-01 (7 (0-20)-3))
5.816E-01 (1.30E-02)
5.575E-01
(1.39E-02)
5.642E-01
(1.38E-02)
4.127E-01 (5 (5.93E-03)
5.817F-02 (1 (1.24F-03))
5.628F-02 (7. (7.29F.04)
7.953F-02 (2.45E-122)
4.159F-02 (6.10F-04)
WFG9
3.633B-01 (1.20E-02)
3,258E-01 (1.61E-02)
3.396E-01
(1.SSE 02)-
1.832-001 (7.10E-03)
5.538E-01 (2.63E-02)
5.115E-01 (2.15E-02)
5.227E-01 (1.79E-02)
3.278E-01 (7.21E-03)
Test: +/-
0/13/22
0/15/00
0/15/0
How is the generalization ability of MA-DAC? We compare MA-DAC policies trained from different training sets to test the generalization ability of MA-DAC policies. MA-DAC (M) denotes that the policy is trained on the problems (i.e., DTI.72, WFG4 and WFG6) with all 3, 5 and objectives. Meanwhile, MA-DAC (3), (5) and (7) denote that the policies are trained on the problems (i.e., DTLZ22, WFG4 and WFG6) with 3, and objectives, respectively.
RQ2:
As shown in Table 3 it is unsurprising that the results of average rank show that the MA-DAC (3), (5) and (7) policies have excellent performance on the problems with 3, 5 and 7 objectives, respectively. For example, MA-DAC (7) has the best average rank (i.e., 1.0) on the test problems with 7 objectives, but has the worst average rank (i.e.. 4.0) on the problems with objectives. To obtain more robust policy, we mix the problems with different number of objectives as the training set, resulting in the MA-DAC (M). As can be seen in Table 3 MA-DAC (M) demonstrates its robustness. Among all the policies, MA-DAC (M) takes the first, third, and second places in terms of its performance in the three types of problems, respectively. Lastly, the last row of the table shows the out- out-performance of these MA-DAC policies compared with other policies- the IGD values of all the four MA-DAC policies perform significantly better than the best result obtained by the three peer algorithms MOEA/D, DQN and MA-UCB in Table 2
RQ3: How do the different parts of MA-DAC affect the performance? We conduct ablation studies to show the importance of different types of hyperparameters, and the results are given in Table 4
We use MA-DAC (M) w/o to denote the MA-DAC policy that does not include the i-th agent, and we use a reasonable setting as the default for each ablated agent; the detailed settings are provided in Appendix B. That is, MA-DAC (M) w/o 1, 2, 3 and represent MA-DAC (M) without tuning weights, neighborhood size, types of reproduction operators, and parameters of reproduction operators, respectively.
As can be seen in Table MA-DAC (M) outperforms all the ablations, demonstrating the importance of tuning every type of hyperparameter. Note that the column of MA-DAC (M) in Table 4is just as
8

---

Table 3: IGD values obtained by MA-DAC (M), MA-DAC (3), MA-DAC (5) and MA-DAC (7) on different problems. Each result consists of the mean and standard deviation of 30 runs. The best mean value on each problem is highlighted in bold. The symbols '_', and "zy" indicate that the result is significantly superior to, inferior to, and almost equivalent to the best value except for MA-DAC in Table respectively, according to the Wilcoxon rank-sum test with significance level 0.05.
Problem
M
MA-DAC (M)
(3)
(5-DAC
MA-DAC.(7)
3,839--12 (5. (5.356-04)
3,807E-02 (5.055-044)
3.8306-02 (7.24E-04)
3,837E-02 IS 15.698-04) +
D1T.22
2.468F-01 (7.55F-03) +
2.472F-011 (1 (1.56F-02) +
2.442F-01
(1.26F-02) +
2,509F-01 (1.391-02) +
3.921E-01 (8.84E-03)
3.880E-01 (1 (1.02E.00 02)
4,081E:01 (1.521-02)
3,944E-01 (1.17B-02)
5.220F-02 (9:83F-04)
5.2001E-02
(1.19F.03)
5.236F.02 (1. (0F-03)
5.302F-02 (9. (8788-04) +
WFG4
1.800-001 (3.14E.03)
1.867E-01 (3.01E-03)
1.868E001
(2.811-03)
+
1.853E-0 01 (2.67B-033 +
3.091E-01 (5.80E-03)
3.104E-01 (7.14E-03)
3.100E-01
(5.89E-03)
3,033E-01 (3.66E-03)
5.078E-02 (1.20E-02)
4.831E-02 (8.95E.03)
4.599E-02 (9.48E.03) (8) I
5,206E-02 (1 (.64--02) 1
WFG6
1.971E-01 (6.40E-03)
+
2,003E-01 (6.26E-03)
1.92E-01
(6.9UE-03) +
1,957E-01 (6.67B-03) +
3.114E-01 (5:08B-03)
3,2426-01 (9.24B-03)
3.129E-01
(5.71B-03)
3,112E-01 (4.93E-03)
6.171E-42 (3:67E-02)
6, 6,700E-02 (6. 14B--02)
6.618E-02 (4. (4.628-02)- +
8.088E-02 (7. 128-02)- +
DTLZ4
3.04-1-01 (1.66B-02) Rd
2.974E-011 (1. (1.94--02) +
2.995E-01 (2. (2.101-02)) Ra
3.036E-01 (1.69E-02) RS
4.271E-01 (1.458-02) +
4.313 (1.39E-02)
4,327E-01 (2. (58-05)2))
4.182E-01 (1 (18-16-2) +
4.721E-42 (7.15E-04)
4.780F-02 (7.89 (7.89F0()
4.733F.02 (8. (QF-D4) +
4.746F-02 (S (.F-0-0)) +
WEGS
1.811E-01 (2.591-03)
1.817-.01 (2.966-03)
18811.001 (3.021-03) +
1808E-01 (3833-03) 4
3,256F-01 (5.496-03)
3,266F-01
(8.98F-03)
3,263F-01
(9.73F-03) 4
3.206F-01 (8.04..0) +
4.076E--02 (5. (5.33E-04)
4.066E-02 (5. (5.31E.04)
4.077E.02 (5. (2.12)-04) 1t
4.124E 02 (4, (4,98I. (4) F
WFG7
1.839E-01 (2.38E-03)
1.81E-01 (3.70E-03)
1.85E-01 (2.126-03)
+
1836E-01 (2.21E-03) +
3.368E-01 (1.54E-02)
3.461E-01 (1.97602)
3.
(1.381-022)
3.258E-01 (1.25E.0)
7. 7,828E-422 (1 (1.46E-03)
7,901E-02 (1.19E-03 (3)
7,921E-02 02 (1.36G-02 03) +
7,944E 02 (1.30B 30B-03) +
WFG8
2.506E-01 (1.118-02)
2.653E-01 (1.51B002)
2.479E-01
(23200-03)) +
2.532E-01 (9. (9.28E-03)
4.303E-01 (1.496-02)
4.364E-01
(1.388-02)
4,2426-01
(9.086-03)
4.127E-01 (S (3.936-03)- +
4.324E-02 (7.07E-04) +
4.159E-02
(6.10E-04) +
4.359E-02 (1.00E-022 (2)E-02)-0 +
6.415E-02 (2.641B02) +
WFG9
1888E-01 (3.636-05)-
1.814E-01
(4.596-03)-
1.832E-01 (7. 10B-033)
1,9185-01 (1.138-02)
3.3286-01
(1.02F-02)
3:298F-01 (L.03-002)
+
3.307F-01
(1.376-002
+
3.278F-01 (1 (7.218-03) +
1.4
L88
2.8
4.0
Test: average rank
2.8
2:2
2.4
2.6
3.4
3.0
1.0
SAVD
SAVD
S/00
S/V0
Test: 1///
4AV1
S/VD
4/V/
4/V1
5/00
4/01
4/0/1
5/00
same as that in 3 On the other hand, we notice that the importance of hyperparameters varies. For example, adaptive weights are in general more important as the performance of MA-DAC (M) w/o drops significantly.
Further studies Due to the space limitation, more experiments on MaMo and DACBench 10 are provided in Appendix and Appendix C, respectively.
To show the effectiveness of the proposed triangle- based reward function, we compare it with different reward functions in Appendix B.2.
We give a detailed analysis of the reproduction operators and the adaptive weights in Appendix B.3 and Appendix B.4, respectively.
To show the optimization process of each method, we plot the curves of IGD value of different methods (i.c., MOEA/D, MOEA/D-FRRMAB, MOEA/D-AWA, DQN, MA-UCB and MA-DAC) in Appendix B.5.
To improve the compared baseline, we use DQN to dynamically adjust each type of hyper- parameter of MOEA/D in Appendix B.6.
We use two more MARL algorithms as the implementations of policy networks, i.e., Inde- pendent Q-Learning (IQL) 41 and QMIX (32 in Appendix B.7.
We also conduct experiments on Sigmoid from DACBench [10] in Appendix C, which can generate instance sets with a wide range of difficulties. We conduct experiments on the 5 5D-Sigmoid and 10DD-igmoii (i.e., there are 5 and 10 agents in MA-DAC, respectively) with action space size (i.e., each agent has a 3-dimensional discrete action space). The experimental results show the versatility and scalability of MA-DAC.

---

Table 4: IGD values obtained by MA-DAC (M) w/o I, MA-DAC (M) w/o 2, MA-DAC (M) w/o 3 and MA-DAC (M) w/o on different problems. Each result consists the mean and standard deviation of runs, The best mean value on each problem is highlighted in bold. The symbols '4' and 'z' indicate that the result is significantly superior to, inferior to, and almost equivalent to MA-DAC (M), respectively, according to the Wilcoxon rank-sum test with significance level 0.05.
Prefllm
M
MA-DAC (M) w/o
MA-DAC (M) w/o
MA-DAC (M) wlo
MA-DAC w/o
MA-DAC
4.656B 02 (3.80E 03)
3,914E-02 43E (3E-00)
3.935E-02 (6.72) (6.226-04)
3.919E-02 (5.91E-01)
3.839E-02 (S (5.35E-04)
DTL22
3,086E-01 (7.24E-03)
2,619E-01 (8 99E-099 99E-03)--
2.303E-01 (1 10600--0)-
2.433E-01 (1.59E-02) Ri
2-468E-001 (7 (7.55E-03)
4,500E-01 (1.26B
4.067E-01 (1.206-02)
4.003E-01
(1.19-02))
4.2286-01 (1.25B
3,921E-01 (8.84E-03)
7.2238-002 (1.93F-03)
5.844-02 (1 (1F-03)
5.410F-02 (8.85 35P555-44
5.410F-02 (8. (3F59--4)
5,2208-02 (9 (9.83F-04)
WFGH
2,8088-01 (1.01E.02)
18899-001 (3. 768--6)))
1.845-001 (2.178-03)
1,846E-01 (2.29E-08) +
1880E-01 (3. (3.148-433
3.75882401 (1.336-02)
3.102E:01 (6.34B-03)
3.020E-01 (3.9HE-03)
3.052E-01 (4.326-03)
3,091E-01 (5.00E-03)
6.86455-02.00 (8. 14F--0))
5.338E-02 (1. 176---00)
6.5436-02 (1 60F-02)
6.0677-022 (2. 1..11E 1IE-22) Ps
5.0788-02 (1 20200077)
WFF16
3,4808-001 (1.34F-02-
2.005F-01 (1.118-03)
1.996E-01
(6.51P-433)
1.9795-01 (6, (6F26F--))
1,9718-01 (6 (6.40F-03))
4.78415-01 (3:37E-02)
3,147E-01 (5.856-03)
3.162E-01 (6.155-03)
3.147E-01 (5.736-03)
(5.06E-03)
Train +/-/F/
05KO
0/8/1
1771
1/533
6,4335-02 (3.8 (3855-02)
6.2426-02
076-02)
4,496E-02 (2.45E-03)
4,496F-02 (2. (2445-001) +
6.171E-02 (3677--07)
1YTL.44
3,4977-0011 (1 (LA1E-02)
3.061E-01
(2.126-02) Fs
3.545-011 (1 55B-412) Rs
3.1505-01 (1 81F-22)
3.0448-01 (1 (1.6F-02)
4.8538-01 (28
4.275E:01 (1.908 (.)00-02)
4,208E-01
(1.528-02)
4,2898-01 (2.07E-02)
4,271E:01 (1455E402)
6.18998-02 (7.80E-03)
4.82/E-02 (6.318-04)
4.766E-02 (S (5.722:-44) Pa
4,7008-02 (5.72) (5.72E-04) =
4,721E-02 (7.15E-04)
WIGGS
3.2021B-0 01 (9,77E 03))-03)---
1821E-01 01 (2.73B 03)
L835E-00 OL (2.90g- 03)
1.82E-00
(2.31E 03)
1811E
(2.59E (13)
4.948E-01 (1.47E-02)-
3,290E-01
(8.87E-03)
3.310E-01 (7.00--03)
3.261E-01
(9,26E-03)
3,256E-01 (5/49E-03)
6.0048--02 (9.45E-04)
4.250E-02 (5 828-04)
4.150E-02 (6 (6)00-444)
4,1508-02 (6.60E--04)
4.0768-02 (5 (3333--04)
WEG?
3,402E-00 01 (2.49E 02)
L873E-01
(3.67E) 03)
1826E-01 (2,60E 03)
1887E-01 (2.80E 02) F
L8899 01 (2.28E-03)
4,8771--0 01 (5.70E 02)
3.393E-01 (1.23E 02)
3.373E 01 (1.16E-02)
3.377E-01 01 (1.59E-02 02)-02)
3.368E-01 (1 (1.54E-15 02)
9.66/E--22 (1.87--03)
8.374E-02 (1. 2.700E-00 70E-03)--
1029E-0221 (1 296-03)
8.029E-02 (1. (1229--03)
7,828E-02 (1 (1,66--33)
WFG8
4.119E-01 (1. 30E-02)-
2.695E-01 (1 (96--02)
2.571E-01 (1 (08-000)
2.632E-01 (9. (696-6-0)-
2.5066:00 (1.11E-02)
5.8305-00 (1.59F--02)-
4.345E-01 (9.276-03)
4.260R-001
(8.715-03)
4.3225-011 (1.125-02)
4.303F-01
(.400-02)
5,891E 02 (9. 01)
4.321E-02 (7 50.00--00 04)
4650E-02 (1 AOG 02)
4.650E-02 0200--2240 (1. 40E 02)
4.324E-02 (7.07E-04))
WFG9
3,148E-01 (2.06E-02)
1.875--01 (5.1 (.14--0))
1.941E-01 (6.45B-03)
1.865--01 (9 (9022-003)
L888E-01 (7,63-033)
5,069E-01 (2.53E-02)
3.433E-01 (L.35B-02)
3.402E:01
(1.006-02)
3.368E-01 (LO5E-02)
3.328E-01 (1.02E-02)
Test: +/-/#
0V1500
(/12/3
2/10/
1/12/2
Conclusion
This paper considers the dynamic configuration of algorithms with multiple types of hyperparameters. We propose MA-DAC to solve it, where one agent works to handle one type of configuration hyperparameter. Experimental results show that MA-DAC works well and has good generalization ability. The instantiation of configuring MOEA/D forms the benchmark MaMo for MARL, with the hope of facilitating the application of MARL.
Considering the superior performance of MA-DAC in empirical studies, an interesting future work to perform theoretical analysis, to better understand why MA-DAC can work. Particularly, many MARL algorithms follow the IGM principle that assumes the global Q is factorizable, while it is not yet clear whether DAC problems are (approximately) factorizable. In addition, we will try to propose better contextual MMDP formulation as well better cooperative MARL algorithms, based on the heterogeneity and stochasticity of MA-DAC. It is also interesting to include more real-world optimization problems into MaMo.
Acknovledgements
We thank the reviewers for their insightful and valuable comments. We thank Chenghe Wang and Hao Yin for providing helpful comments. This work was supported by the National Key Research and the NSFC (62022039, 62276124, 618761 19),
Development Program of China the Fundamental Research Funds for the Central Universities (0221-1438000) 0221-14380014), and the program B for Outstanding Ph.D. candidate of Nanjing University.
References
[1] Steven Adriaensen, André Biedenkapp, Gresa Shala, Noor Awad, Theresa Eimer, Marius Lindauer, and Frank Hutter. Automated dynamic algorithm configuration. arXiv preprint 13881 2022.
10

---

[2] Leonardo C. T. Bezerra, Manuel López-lbáñeez, and Thomas Stützle. A large-scale experimental evaluation of perhorperorringg multi- and many-objective evolutionary algorithms. Evolutionary Computation, 26(4):621 -656, 2018.
[3] André Biedenkapp. H. Furkan Bozkurt, Theresa Eimer, Frank Hutter, and Marius Lindauer. Dynamic algorithu configuration: Foundation of new algoriiggii framework. In Pro- ceedings of the 24th European Conference on Artificial Intelligence, pages 427-4344 Santiago de Compostela, Spain, 2020.
[4] Peter A. N. Bosman and Dirk Thierens. The balance between proximity and diversity in multiobjective evolutionary algorithms. IEEE Transactions on Evolutionary Computation, 7(2): 174 188, 2003.
[5] Craig Boutilier. Planning, learning and coordination in multiagent decision processes. In Proceedings of the 6th Conference on Theoretical Aspects of Rationality and Knowledge, pages 195-210, De Zeeuwse Stromen, The Netherlands, 1996.
[6] Caroline Claus and Craig Boutilier. The dynamics of reinforcement learning in cooperative multiagent systems. In Proceedings of the 15th National Conference on Artificial Intelligence and 10th Innovative Applications of Artificial Intelligence Conference, pages 746-752, Madison, WI, 1998.
[7] Christian Daniel, Jonathan Taylor, and Sebastian Nowozin. Learning step size controllers for robust neural network training. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, pages 1519-1525, Phoenix, AZ, 2016.
[8] Kalyanmoy Deb, Samir Agrawal, Amrit Pratap, and T. Meyarivan. A fast and elitist multiob- jective genetic algorithm: NSGA-IL, IEEE Transactions on Evolutionary Compulation, 6(2): 182-197, 2002.
[9] Kalyanmoy Deb, Lothar Thiele, Marco Laumanns, and Eckart Zitzler. Scalable multi-objeccivv optimization test problems. In Proceedings of the 2002 Congress on Evolutionary Computation, pages 825 830, Honolulu, HI, 2002.
[10] Theresa Eimer, André Biedenkapp. Maximilian Reimer, Steven Adriaensen, Frank Hutter, and Marius Lindauer. DACBench: A benchmark library for dynamic algorithm configuration. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, pages (668-1674) Montreal, Canada, 2021.
[11] Rihab Gorsane, Omayma Mahjoub, Ruan de Kock, Roland Dubb, Siddarth Singh, and Arnu Pretorius. Towards a standardised performance evaluation protocol for cooperative marl. arXiv preprint arXiv:2209. 10485, 2022.
[12] Alessio Greco, Salvatore Danilo Riccio, Jon Timmis, and Giuseppe Nicosia. Assessing algo- rithm parameter importance using global sensitivity analysis. In Proceedings of International Symposium on Experimental Algorithms Special Event, pages 392-407, Kalamata, Greece, 2019.
[13] Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. A survey and critique of multiagent deep reinforcement learning. Autonomous Agents and Multi-Agent Systems, 33(6):750-797, 2019.
[14] Wenjing Hong, Peng Yang, Yiwen Wang, and Ke Tang. Multi-objective magnitude-based pruning for latency-aware deep neural network compression. In Proceedings of the 16th International Conference on Parallel Problem Solving from Nature, pages 470-483, Leiden, The Netherlands, 2020.
[15) Safwan Hossain, Evi Micha, and Nisarg Shah. Fair algorithms for multi-agent multi-armed bandits. Advances in Neural Information Processing Systems 34, pages 24005-24017, Virtual, 2021.
[16] Simon Huband, Philip Hingston, Luigi Barone, and Lyndon While. A review of multiobjec- tive test problems and scalable test problem toolkit. IEEE Transactions on Evolutionary Computation, 10(5):477-506, 2006.

---

[17] Frank Hutter, Holger H. Hoos, Kevin -eyton-Brown, and Thomas Stützle. Paramils: An automatic algorithm configuration framework. Journal of Artificial Intelligence Research, 36: 267-306, 2009.
[18] Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated Machine Learning: Methods, Systems, Challenges. Springer, 2019.
[19] Hisao Ishibuchi, Naoya Akedo, and Yusuke Nojima. Relation between neighborhood size and MOEA/D performance on many-objective problems. In Proceedings of the 7th International Conference on Evolutionary Multi Multi-Criterion Optimization, pages 459 474, Sheffield, UK, 2013.
[20] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of generalisation in deep reinforcement learning. arXiv preprint arXiv:2111.0999 2021.
[21] Ke Li, Alvaro Fialho, Sam Kwong, and Qingfu Zhang. Adaptive operator selection with bandits for a multiobjective evolutionary algorithm based on decomposition. IEEE Transactions on Evolutionary Computation, 18(1): (8(1):114-1))) 10::11143,,, 2013.
[22] Miqing Li and Xin Yao. Quality evaluation of solution sets in multiobjective optimisation: A survey, ACM Computing Surveys, 52(2):1-33, 2019.
[23] Miqing Li and Xin Yao. What weights work for you? Adapting weights for any Pareto front shape in decomposition- based evolutionary multiobjective optimisation. Evolutionary Computation, 28(2):227-253, 2020.
[24] Marius Lindauer and André Biedenkapp. Algorithm configuration: Challenges, methods and perspectives. Tutorial on PPSN 2020 and IJCAI 2020.
[25] Ryan Lowe, Yi Wu, Aviv Tamar, Jean Harb, Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-competiee environments. In Advances in Neural Information Processing Systems 30, pages 6379-6399 Long Beach, CA, 2017.
[26] Zhichao Lu, Kalyanmoy Deb, Erik D. Goodman, Wolfgang Banzhaf, and Vishnu Naresh Boddeti. Nsganetv2: Evolutionary multi-objective surrogate-assitted neural architecture search. In Proceddings of 16th European Conference on Computer Vision, pages 35-51, Glasgow, UK, 2020.
[27] Manuel López-Ibáñez, Jérémie Dubois-Lacoste, Leslie Pérez Cáceres, Mauro Birattari, and Thomas Stützle. The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives, 3:43-58, 2016.
[28) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Pe- tersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 5 (1(75500):5295533 2015.
[29] Afshin OroojlooyJadid and Davood Hajinezhad. A review of cooperative multi aulti-agent deep reinforcement learning, arXiv preprint arXiv: 1908. 039603366, 2019.
[30] Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Benchmark- ing multi-agent deep reinforcement learning algorithms in cooperative tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, pages 1-13, Virtual, 2021.
[31] Yutao Qi, Xiaoliang Ma, Fang Liu, Licheng Jiao, Jianyong Sun, and Jianshe Wu. MOEA/D with adaptive weight adjustment. Evolutionary Computation, 22(2):231-264 2014.
[32] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, pages 2995-4304, Stockholm, Sweden, 2018.
12

---

[33] Sutton Richard S. and Andrew G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
[34] Mikayel Samvelyan, Tabish Rashid, Christian Schroeder de Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon White- son. The starcraft multi-agent challenge. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 1186-2188 Montreal, Canada, 2019.
[35] Gresa Shala, André Biedenkapp, Noor H. Awad, Steven Adriaensen, Marius Lindauer, and Frank Hutter. Learning step-size adaptation in CMA-ES. In Proceedings of the 16th International Conference on Parallel Problem Solving from Nature, pages 691-706, Leiden, The Netherlands, 2020.
[36] Mudita Sharma, Alexandros Komninos, Manuel López-Ibáñezz and Dimitar Kazakov. Deep reinforcement learning based parameter control differential evolution. In Proceedings of the 21 Genetic and Evolutionary Computation Conference, pages 709-717, Prague, Czech Republic, 2019.
[37] Kyunghwan Son, Daewoo Kim, Wan Ju Kang, David Hostallero, and Yung Yi. QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Proceedings of the 36th International Conference on Machine Leurning, pages 5887-5896, Long Beach, CA, 2019.
[38] David Speck, André Biedenkapp. Frank Hutter, Robert Mattmüller, and Marius Lindauer. Learning heuristic selection with dynamic algorithm configuration. In Proceedings of the 31th International Conference on Automated Planning and Scheduling, pages 597-605, Guangzhou, China, 2021.
[39] Jianyong Sun, Xin Liu, Thomas Bäck, and Zongben Xu. Learning adaptive differential evolution algorithm from optimization experiences by policy gradient. IEEE Transactions on Evolutionary Computation, 25(4):666-68) 2021.
[40] Peter Sunchag, Guy Lever, Audrunas Gruslys, Wojciech Marian Czarnecki, Vinícius Flores Zambaldi, Max Jaderberg, Marc Lanctot, Nicolas Sonnerat, Joel Z. Leibo, Karl Tuyls, and Thore Graepel. Value-decomposition networks for cooperative multi-agent learning based on team reward. Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pages
Stockholm, Sweden, 2018
[41] Ardi Tampuu, Tambet Matüisnn, Dorian Kodelja, Ilya Kuzovkin, Kristjan Korjus, Juhan Aru, Jaan Aru, and Raul Vicente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12())=01772395, 2017.
[42] Justin K. Terry Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sullivan, Luis S. Santos, Clemens Dieffendahl, Caroline Horsch, Rodrigo Perez-Vicente, Niall L. Williams, Yashas Lokesh, and Praveen Ravi. Pettingzoo: Gym for multi-agent reinforcement learning. In Advances in Neural Information Processing Systems 34, pages 15032-15043, Virtual, 2021.
[43] Ye Tian, Xiaopeng Li, Haiping Ma, Xingyi Zhang, Kay Chen Tan, and Yaochu Jin. Deep reinforcement learning based adaptive operator selection for evolutionary multi-objective opti- mization. IEEE Transactions on Emerging Topics in Computational Intelligence, in press.
[44] Anupam Trivedi, Dipti Srinivasan, Krishnendu Sanyal, and Abhiroop Ghosh, A survey of multi- objective evolutionary algorithms based on decomposition. IEEE Transactions Evolutionary Computation, 21(3):440-45 462, 2017.
[45] Ceyer Wakilpoor, Patrick J Martin, Carrie Rebhuhn, and Amanda Vu. Heterogeneous multi-agent reinforcement learning for unknown environment mapping. arXiv preprint arXiv:2010.02663 2020.
[46] Jianhong Wang, Wangkun Xu, Yunjie Gu, Wenbin Song, and Tim C. Green. Multi-agent reinforcement learning for active voltage control on power distribution networks. In Advances in Neural Information Processing Systems 34, pages 3271-3284, Virtual, 2021.
13

---

[47] Zhenkun Wang, Qingfu Zhang, Aimin Zhou, Maoguo Gong, and Licheng Jiao. Adaptive replacement strategies for MOEA/D. IEEE Transactions on Cybernetics, 46(2):474 2016.
[48] Yaodong Yang and Jun Wang. An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv preprint arXiv:201 0at::2011.000000 1.00583, 2020.
[49] Lei Yuan, Chenghe Wang, Jianhao Wang. Fuxiang Zhang. Feng Chen, Cong Guan, Zongzhang Zhang, Chongjie Zhang, and Yang Yu. Multi-agent concentrative coordination with decentral- ized task representation. In Proceedings of the 31 International Joint Conference Artificial Intelligence, pages 599-605, Vienna, Austria, 2022.
[50] Kaiqing Zhang, Zhuoran Yang, and Tamer Başar. Multi-agent reinforcement learning: A selective overview of theories and algorithms. Handbook of Reinforcement Learning and Control, pages 321-384, 2021.
[51] Qingfu Zhang and Hui Li. MOEA/D: A multiobjective evolutionary algorithm based on decomposition. IEEE Transactions on Evolutionary Computation, 11(6):12-731, 2007.
[52] Lianmin Zheng, Jiacheng Yang, Han Cai, Ming Zhou, Weiman Zhang, Jun Wang, and Yong Yu. Magent: A many-agent reinforcement learning platform for artificial collective intelligence. In Proceedings of the 32th AAAI Conference on Artificial Intelligence, pages 8222-8223, New Orleans, LA, 2018.
[53] Eckart Zitzler and Lothar Thiele. Multiobjective optimization using evolutionary algorithms A comparative case study. In Proceedings of the 5th International Conference on Parallel Problem Solving from Nature, pages 292-304, Amsterdam, The Netherlands, 1998.
Checklist
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] See Abstract and Sectionl.
(b) Did you describe the limitations of your work? [Yes] We describe the limitations in Section 6.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] The authors do not see obvious negative societal impacts of the proposed method
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? Yes]] Our paper conforms to them.
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed reproduce the main ex- perimental results (either in the supplemental material or as URL)? [Yes] Yes, see Section 5, Appendix and Supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Yes, see Section 5 and Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] Yes, see Section 5 and Appendix.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] Yes, see Appendix.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See References.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
See supplemental material.
14

---

(d) Did you discuss whether and how consent was obtained from people whose data you' re using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "jIJj1HsXF5z", "reviewer": "Reviewer_Y4Zz", "summary": "The work presents a multi-agent extension to the Dynamic Algorithm Configuration (DAC) framework. As such, the paper discusses how to extend the formulation of a contextual MDP to a contextual (cooperative) multi-agent MDP. Further, the work provides a benchmark in the form of MOEA/D which allows to study 1) dynamic algorithm configuration in the case of various heterogeneous hyperparameters, and 2) MARL algorithms. ", "questions": "The questions are not ordered in any particular order.\n* Why did you not consider any benchmarks from DACBench?\n* As you target handling of different parameter types, how would you handle ordinal or categorical hyperparameters, as well as hyperparameters that are using some form of transformation (i.e. log-scaled learning-rates).\n* Do you believe your approach is also helpful in the multi-hyperparameter case with hyperparameters of the same type?\n* To me it seems that the proposed reward might give higher weight to crossing from positive function values to negative ones. Is my interpretation of that correct? If so, is there any particular benefit from that?\n* How did you discretize the neighborhood size?\n* How did you normalize the state features?\n* How did you limit the frequency for adjusting the weights?\n* Are all hyperparameters adjusted at the same frequency?\n* How did you select the instances? What exactly are MOPs with similar properties?\n* When testing, it is stated that \"[...] all the functions are considered in the testing[...]\" (line 266). Does that mean that you test on the training set?\n* Why do you differentiate between three types of features in the state? Aren't indicators such as hypervolume and other statistics part of the optimization process?\n* Why is the reward scaled by $\\frac{1}{2}$?", "limitations": "The limitations are not particularly addressed. For example, a limitation of the work is that it only considers the multi-agent configuration on a single benchmark. Further, from the text it seems that the test set includes the training set which will result in inaccurate assessments on the generalization performance.\n\n----\n\n**During the rebuttal the authors remedied these shortcomings.**", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Taking Author Feedback into account (during rebuttal)\nI am very happy how the authors addressed my and other reviwers concerns. Thus **I already increased my score to strong accept**\n\n-----\n-----\n\n# Original Review\nOverall I'm very excited about this paper. I believe that dynamic configuration is an important problem that is not yet studied broadly enough. I am particularly excited about the extension to multiple hyperparameters of different types. In the following I will list the strengths and weaknesses.\n\n-----\n-----\n\n## Strengths\nThe work combines two interesting and important fields of research (Multi-agent RL and DAC). This combination enables interesting research opportunities in both fields. For example, to the best of my knowledge, MARL methods have not had a particular focus on learning general policies. By following the DAC framework and using contextual MDPs (or rather contextual (cooperative) multi-agent MDPs) MARL algorithms can be evaluated with regards to their generalization capability. In particular, the notion of context allows to study generalization of policies in a principled manner (see e.g. [Kirk et al. 2021](https://arxiv.org/abs/2111.09794)), as was also discussed in the original work on DAC (referenced as [2] in the paper).\nOn the DAC side for example, as most methods employ some form of RL, the use of (cooperative) multi-agent methods allows to deal with very different types of hyperparameters (as shown in the paper). Thus I believe the combination of MARL and DAC is of importance to the NeurIPS community. \n\n-----\n-----\n\n## Weaknesses\n### Differentiation with DAC\nThe work calls the proposed multi-agent approach to DAC a novel framework, however I do not see large enough differences to talk about a novel framework. In particular, the original DAC work considers also tuning of multiple hyperparameters. As the go-to solution methods for DAC are vanilla deep RL approaches (e.g. DQN) the so far considered DAC scenarios are limited by the capabilities of vanilla RL, which typically does not deal with \"heterogeneous\" action spaces. Thus, from my point of view, the work proposes a novel *solution approach* to the DAC problem based on multi-agent RL, which might be superior as it allows to make efficient use of human knowledge (e.g. certain hyperparameters are distinct from each other).\n\nTo me, this weakness is particularly apparent in section 3.\nFor example in Section 3.1 (line 103-104) it is said that the work \"... explicitly take[s] the instance distribution [...] as context into account to facilitate generalization.\" This same motivation was stated by Biedenkapp et al. 2020 (cited as [2] in the paper) when formulating the DAC problem as cMDP. The work presented here does not make this connection explicit.\nFurther, in Section 3.2 the work details the components of the cMMDP. In the paragraph about the state, desiderata are listed that are not unique to the multi-agent setting. For example, [Lindauer and Biedenkapp 2020](https://www.automl.org/wp-content/uploads/2020/08/AC_Tutorial_2020.pdf) presented a similar set of 3 desiderata for the state.\n\nIn Section 4.2 when the different hyperparameters are discussed, selection of different operators for differential evolution are mentioned. I believe in the main paper there should be a reference to Sharma et al. 2019 (cited as [10] in the appendix) as their work proposed to adapt DE operators via DQN.\n\nFinally, although a benchmark collection for DAC problems, DACBench, is cited (as [9] in the paper), none of the existing benchmarks were used for evaluation. This seems strange to me as I believe there are benchmarks in the collection that allow for configuration of multiple hyperparameters as well as multiple hyperparameters of different types. At least the former seems to be covered as the artificial benchmarks from the ECAI paper by Biedenkapp et al. 2020 are contained. It would have given a good experiment that could have shown if a multi-agent approach can outperform  the DQN results by Biedenkapp et al.\n\nThus, overall I believe the work would benefit if from a closer alignment with DAC.\n\n-----\n\n### Notation\nThe work is somewhat inconsistent with the notation. In Section 2.1 contextual MDPs are discussed, though context is not formally introduced. Later in Section 3.1 context is denoted as $i\\in\\mathcal{I}$, however the connection to problem instances is not given. Later on when the transition function is discussed the connection between context $i$ and problem instances $f\\in\\mathcal{F}$ is not obvious. Thus for readers not familiar with DAC it might make it difficult to follow what context is and how it is used in this work. Further, the index variable $i$ is overloaded. On line 82 it is first introduced as the agent index and later reused to denote context. Thus in line 126 it was at first confusing as it seemed that the actions were context dependent rather than $a^{(i)}$ giving the agent index.\n\nThe way the reward function in equation 2 is written up is rather confusing. I would suggest to expand the $p^2_t$ variables and then to rearrange similar to\n$$r_t = -0.5\\cdot \\frac{\n (f_{t}^{\\star} - f_{t-1}^{\\star}) \\cdot ( f_{0}^{\\star} - f_{t}^{\\star} + f_{0}^{\\star} -  f_{t-1}^{\\star})\n}{\n(f_0^{\\star})^{2}\n}$$\nwhere $f_{t}^{\\star}$ denotes the best observed function value up til time-step $t$ (thus, $f_{0}^{\\star} = f(s_0)$).\nI think this highlights better that the reward function only reflects changes in the incumbent values. Also, to me, it better highlights how improvements over the default function value are rewarded (and how the reward is scaled).\nIn regards to that, I do not understand how Figure 1 should be interpreted. Since the reward is solely based on changes in incumbent, the reward is more sparse than Fiugre 1 suggests. While in the beginning it is reasonable to expect rapid improvements in incumbents, I believe in later stages improvements will be more infrequent, thus often leading to rewards of 0. Thus, overall I would expect more plateaus in Figure 1.\n\n-----\n\n### Claim about instance sets\nIt is stated that \"[...] instances should have similar properties, otherwise the good generalization ability is hardly achievable [9]\" (lines 156&157). That claim is false and the citation (to DACBench) does not seem correct. Biedenkapp et al. 2020 (cited as [2] in the paper) proposed artificial benchmarks on which one can control the heterogeneity of the problem instances. They showed that by using context information it is possible to learn policies even across very heterogeneous problem instances. Further, Shala et al. 2020 (cited as [29] in the paper) showed that DAC policies for step-size adaptation of CMA-ES are capable of generalization to even dissimilar functions. This is possible as context information enables RL agents to learn to distinguish between problem instances and learn about how they relate to each other.\nThus I believe the claim is wrong.\n\n-----\n\n### Experiments\nI believe the experiments would be strengthened by showing results on the artificial benchmarks of Biedenkapp et al. 2020 as they used a DQN to control multiple hyperapameters (of the same type) the same way the DQN baseline in this work was used. This could highlight the importance of a multi-agent approach over a single-agent approach already for larger (but homogeneous) action spaces. I believe this would be a particularly informative experiment as the benchmarks allow to arbitrarily scale the number of parameters and number of choices per parameter. This could neatly highlight the advantage of a multi-agent approach.\n\nOn the evaluated benchmarks, the DQN baseline actually does not make too much sense to me. On the real benchmark, the action space is 128. Biedenkapp et al. showed that DQN does not scale well with the number of parameters, even on the artificial benchmarks. In a more recent work by [Biedenkapp et al. 2022](https://arxiv.org/abs/2202.03259) showed that on more stochastic DAC problems, DQN already struggled to scale with the number of hyperparameter value choices of a single hyperparameter. In their experiments, the DQN already struggled to perform well with only 10 values for the single hyperparameter.\nI would suggest to change this baseline to a DQN that only changes the most important parameter (according to the ablation analysis from the paper). This would give the DQN at most an action space of size 4, which is much easier to learn. I think that would better highlight the importance of tuning all hyperparameters and provide a likely more competitive baseline.\n\nSome of the experimental setup is not immediately clear without the appendix. It would be beneficial for the reader to move parts of the appendix, that describe the experimental setup in more detail, to the main paper. To make room for this I would suggest to move some of the details of Tables 3 or 4 to the appendix instead and only report average ranks and the \"$+$/$-$/$\\approx$\" row. \n\n-----\n-----\n\n## Overall\nI am very enthusiastic about the presented work. I believe it provides an interesting extension to the DAC framework which will be interesting to researchers of AutoML/DAC but also for researchers of multi-agent RL. I believe the weaknesses I listed above should be addressed. (I think they can be addressed without too much effort). For now I vote for weak acceptance but I am willing to change my score based on the discussion with the authors and other reviewers.\n\n-----\n-----\n\n\n## References\n* [Kirk et al. 2021](https://arxiv.org/abs/2111.09794);  \"A Survey of Generalisation in Deep Reinforcement Learning\"; https://arxiv.org/abs/2111.09794\n* [Lindauer and Biedenkapp 2020](https://www.automl.org/wp-content/uploads/2020/08/AC_Tutorial_2020.pdf); \"Algorithm Configuration\nChallenges, Methods and Perspectives\"; Tutorial at PPSN 2020 and IJCAI 2020, https://www.automl.org/wp-content/uploads/2020/08/AC_Tutorial_2020.pdf\n* [Biedenkapp et al. 2022](https://arxiv.org/abs/2202.03259); Theory-inspired Parameter Control Benchmarks for Dynamic Algorithm Configuration; https://arxiv.org/abs/2202.03259", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Pl53Vjo5qwd", "reviewer": "Reviewer_fvNP", "summary": "This paper proposes to apply MARL to the dynamic algorithm conﬁguration (DAC) problem and instantiates it on a multi-objective optimization task. The method is mostly well presented and the experiments are illustrated clearly.", "questions": "1. Line 135-138: “higher rewards do not always imply better performance…”, I don’t understand what it means. How and why does your proposed reward function (equation 2) work?\n\n2. Figure 1 and its corresponding texts are also very confusing. What do you mean by “repeated rewards”? Why switching from good to bad states can still get good rewards?\n\n3. I cannot see any theoretical reason for using MARL on this problem. Specifically, I don’t know why DQN cannot work better than or at least as good as VDN?\n\n4. How do you set the episode length? \n", "limitations": "1. The theoretical motivation of applying MARL on DAC problems is not convincing, even though the authors argue it by the heterogeneity of configuration hyperparameters. However, that is just a dimension problem, which doesn't mean single-agent RL cannot handle it. ", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The researched question, i.e. applying MARL to DAC problems is interesting;\n2. The experiments are well conducted;\n\nWeaknesses:\n1. The theoretical motivation to use MARL rather than single RL is not clearly analyzed;\n2. The theoretical analysis for the different performance of the experiments is lacked;", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "FJQWNnw4tmD", "reviewer": "Reviewer_K9Qu", "summary": "This work introduces the multi-agent dynamic algorithm configuration problem which models the hyperparameter search of an arbitrary iterative algorithm as a contextual multi-agent MDP. Each agent in the environment is tasked to determine the value of a specific hyperparameter with its actions corresponding to hyperparameter values, states indicating information about the tuned algorithm and rewards indicating performance. An instantiation of this problem is presented based on the MOEA/D evolutionary algorithm as the MaMo environment and an algorithm based on value decomposition networks (VDN) is trained and evaluated in this problem, and compared to using evolutionary and alternative (MA)RL algorithms.", "questions": "1. For experiments towards RQ2, it is unclear in which exact problems the policy was trained. L. 293 “... the policy is trained on the problems with 3, 5, and 7 objectives.” - does this include all problems with the respective number of objectives?\n2. In line 302 it is stated that “Among all the policies, MA-DAC always takes the first or second place ...” - do you refer to MA-DAC (M) for this experiment?\n", "limitations": "The proposed problem of MA-DAC appears to have underlying assumptions which limit its applicability to new algorithms (outside of MOEA/D and similar iterative, evolutionary algorithms) and are insufficiently discussed.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\nThe primary contribution of this work is the formulation of the MA-DAC problem which models a hyperparameter search over multiple, heterogeneous hyperparameters as a multi-agent decision making problem. While the idea of dynamic algorithm configuration (DAC) is not novel, the framing as a multi-agent problem to the best of my knowledge is novel, interesting and significant.\n\n### Weaknesses\nHowever, the proposed framework lacks generality, the multi-agent algorithm presented alongside does not appear to contain novelty (outside of the modelled problem), and the evaluation is insufficient.\n\n**Major:**\n1. Lack of generality of MA-DAC problem modelling: The modelling of the hyperparameter search as a multi-agent decision making problem requires several properties from the underlying to-be-tuned algorithm which appear quite restrictive and are not sufficiently discussed.\n\n    1.1. In particular, the tuned algorithm needs to be iterative with some metric that for each iteration gives a reasonable indication of its future performance. This is not true for e.g. most (MA)RL algorithms where future performance often only becomes visible after thousands if not millions of steps. One could model the entire training of the underlying algorithm as a single transition but in that case each transition would be too costly for this to be computationally feasible.\n\n    1.2. Also, the modelling appears to assume that the tuned hyperparameters can be changed at each iteration of the algorithm. For some hyperparameters and algorithms, this is not possible (e.g. model size in DL).\n\n2. The set of baselines in the experiments seems insufficient. The proposed approach based on the VDN MARL algorithm is only compared to a multi-agent version of UCB, a single-agent centralised controller trained with DQN and the evolutionary approach of MOEA/D. Given its similarity to other MARL algorithms, comparisons to independent learning algorithms such as IDQN, IPPO, and other CTDE learning algorithms like MAPPO, MADDPG and QMIX would be essential to understand the relevance and challenge of the proposed algorithm.\n\n**Minor:**\n\n3. The MA-DAC (M) algorithm is compared to in Table 4 but not included as a column - I strongly suggest to include this as a baseline in the table.\n4. The authors suggest that “the importance of (tuned) hyperparameters varies” (l. 315f) because the setting of specific hyperparameters to “reasonable” default settings (l. 310) impacts performance differently. However, it is not clear whether these performances are obtained as a result of the relevance of the hyperparameters or the quality of the choice of the default setting for the different hyperparameters.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4CJUJHh52Pr", "reviewer": "Reviewer_55wR", "summary": "This paper addresses the problem of adjusting multiple heterogeneous hyper-parameters of a complex algorithm dynamically, and proposes a general framework, Multi-Agent Dynamic Algorithm Configuration (MA-DAC). The authors gave a concrete application of tuning the hyper-parameters of a popular evolutionary algorithm MOEA/D for solving multi-objective optimisation problems, and the experimental results on benchmark multi-objective problems show the effectiveness of MA-DAC.", "questions": "- I did not see any discussion on the scalability of MA-DAC. The application of MaMo involves four heterogeneous hyper-parameters. If there are more hyper-parameters, how about the runtime complexity of MA-DAC?\n\n- For RQ2 in the experiments, “MA-DAC (M) … trained on the problems with all 3, 5, and 7 objectives” What are the training problems? Still DTLZ2, WFG4 and WFG6?\n\n- If I have not misunderstood, MA-DAC (M) in Table 3 is just MA-DAC in Table 2. If so, why are the results slightly different?\n\n- The states in Table S1 defined for MOEA/D seem to be reasonable and comprehensive. Can they be directly applied when tuning the hyper-parameters of other MOEAs (e.g., NSGA-II and SPEA)? I’d like to see some discussion. \n\n- lines 109-110 in the appendix, what does $x$ mean in OP3 and OP4?\n\n- In the experiments, you used the popular benchmark multi-objective problems, DTLZ and WFG. I understand these are standard test problems. But if you can add some real-world problems, the contribution will be more significant.\n", "limitations": "YES", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "DAC (Biedenkapp et al., ECAI’20) has been an effective framework for dynamically adjusting a single type of hyper-parameters of algorithms, while this paper tries to address the more challenging problem, that is adjusting multiple heterogeneous hyper-parameters simultaneously. The authors modeled this problem as a contextual multi-agent MDP where one agent corresponds to one type of hyper-parameter, and applied multi-agent RL algorithms to solve it. \n\nAs far as I can tell, this is the first work to address the dynamic tuning of multiple heterogeneous types of hyper-parameters. The idea of multi-agent MDP modeling is natural, and also reasonable. The framework named MA-DAC is general. I like the authors gave guidance on the setting of elements of multi-agent MDP, which makes the MA-DAC framework easier to be applied. The triangle-shaped reward function encouraging long-term rewards is novel, the effectiveness of which the authors have also empirically validated. Very interesting work!\n\nTo show the effectiveness of MA-DAC, the authors considered the MOEA/D (Zhang & Li, TEvC’07), which is a popular evolutionary algorithm for solving multi-objective optimisation problems. They applied MA-DAC to dynamically adjust the hyper-parameters of MOEA/D, including weights, neighborhood size, types and parameters of crossover operator. This is a meaningful and practical application, and also constitutes a difficult benchmark (named MaMo in the paper) for MARL algorithms, which can be independent of interest for the MARL community.\n\nIn the experiments, the authors selected the DTLZ and WFG benchmark problems with various number of objectives, three of which are used for training and the others are for testing. They compared MA-DAC with DAC (using one agent to adjust the combination of the four heterogeneous hyper-parameters together), MA-UCB (using four UCB agents to adjust the four heterogeneous hyper-parameters, respectively) and two SOTA adaptive MOEA/D algorithms. Their experimental results clearly show the superiority of MA-DAC (MA-DAC is significantly better in most cases). Especially, the generalization ability of MA-DAC is impressive to me. MA-DAC trained using problems with a single number of objectives can even be better than the compared algorithms trained using problems with various number of objectives.\n\nI have read the appendix as well. The experiments are described in details, and I believe they are reproducible. The codes are also provided.\n\nIn my opinion, this is a very good work. The authors studied the challenging problem of dynamically tuning multiple heterogeneous types of hyper-parameters for the first time, proposed a simple, yet novel and general multi-agent RL solution, gave a practical application of tuning the hyper-parameters of MOEA/D, and showed the excellent performance by comprehensive experiments.\n\nI believe MA-DAC would be a supplementary for DAC, as well as a good resource to the AutoML community, and will bring some influence.\n\nThe paper is also well written and easy to follow. Minor issues:\n\n- line 17, “problems classes” -> “problem classes”\n- line 31, “the tuning learning rate” -> “the learning rate tuning”\n- line 72, “the the” -> “the”\n- Tables 3, 4, S3, S4, S5, S6, caption, “The best mean value of each problem” -> “The best mean value on each problem”\n- Table 4, caption, it would be better to clearly say “MA-DAC (M) in Table 3”, which is easier to be understood.\n- line 401, “nature” -> “Nature”\n- line 40 in the appendix, “lines 4-9” -> “lines 4-10”\n- line 43 in the appendix, “reproduction” -> “crossover”\n- line 136 in the appendix, “at step $t$” -> “by step $t$”?\n- Table S3, caption, “the MA-DAC” -> “MA-DAC”\n\nFor my major concerns, please see the questions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept