Sequential Multi-Agent Dynamic Algorithm Configuration
Authors: Chen Lu
Ke Xue
Lei Yuan
Yao Wang
Yaoyuan Wang
Fu Sheng
Chao Qian
Venue: NeurIPS 2025 poster ()
Keywords: Dynamic algorithm configuration
automl
reinforcement leraning
multi-agent reinforcement learning
Primary area: reinforcement_learning

Abstract: The performance of an algorithm often critically depends on its hyperparameter configuration. Dynamic algorithm configuration (DAC) is a recent trend in automated machine learning, which can dynamically adjust the algorithm’s configuration during the execution process and relieve users from tedious trial-and-error tuning tasks. Recently, multi-agent reinforcement learning (MARL) approaches have improved the configuration of multiple heterogeneous hyperparameters, making various parameter configurations for complex algorithms possible. However, many complex algorithms have inherent inter-dependencies among multiple parameters (e.g., determining the operator type first and then the operator's parameter), which are, however, not considered in previous approaches, thus leading to sub-optimal results. In this paper, we propose the sequential multi-agent DAC (Seq-MADAC) framework to address this issue by considering the inherent inter-dependencies of multiple parameters. Specifically, we propose a sequential advantage decomposition network, which can leverage action-order information through sequential advantage decomposition. Experiments from synthetic functions to the configuration of multi-objective optimization algorithms demonstrate Seq-MADAC's superior performance over state-of-the-art MARL methods and show strong generalization across problem classes. Seq-MADAC establishes a new paradigm for the widespread dependency-aware automated algorithm configuration. Our code is available at https://github.com/lamda-bbo/seq-madac.

Sequential Multi-Agent Dynamic Algorithm Configuration
Chen Lu Ke X Lei Yuan Yao Wang", Yaoyuan Wang'. Sheng Fu", Chao Qian
1.2
National Key Laboratory for Novel Software Technology, Nanjing University, China School of Artificial Intelligence, Nanjing University, China Advanced Computing and Storage Lab, Huawei Technologies Co., Ltd. Shenzhen, China {xuek, qianc} @lamda. nju. edu. edu..edu.cm cn
Abstract
The performance of an algorithm often critically depends its hyperparameter figuration. Dynamic algorithm configuration (DAC) is a recent trend in automated machine learning, which can dynamically adjust the algorithm's configuration during the execution process and relieve users from tedious trial-and-error tuning tasks. Recently, multi-agent reinforcement learning (MARL) approaches have improved the configuration of multiple heterogencous hyperparameters, making various parameter configurations for complex algorithms possible. However, many complex algorithms have inherent inter-dependencies among multiple parameters (e.g., determining the operator type first and then the operator's 's parameter), which are, however, not considered in previous approaches, thus leading to sub-optimal results. this paper, we propose the sequential multi-agent DAC -MDAC) framework to address this issue by considering the inherent inter-dependencies of multiple parameters. Specifically, we propose a sequential advantage decompo- sition network, which can leverage action-order information through sequential advantage decomposition. Experiments from synthetic functions to the config- uration of multi-objective optimization algorithms demonstrate Seq-MADAC's superior performance over state-of-the- MARL methods and show strong gen- eralization across problem classes. Seq-MADAC establishes
paradigm for the widespread dependency-nnne aware automated algorithm configuration. Our code is available /https://git thub. www/lamdaaaaaaooaoo bbo abo/seq-madac
Introduction
Identifying proper configurations of hyperparameters is critical for many learning and optimization algorithms [15 (26) Algorithm configuration (AC) [14 233 has emerged alleviate the user's burden of manual trial-and-error tuning. However, the static configuration policies obtained by AC may not achieve optimal performance, because algorithms may require different configurations at different execution stages (37). Dynamic AC (DAC) 12 II is a prevalent paradigm that enables dynamic adaptation of the algorithm's 's configuration during the execution process, which more flexible compared to AC. Specifically, DAC formulates the configuration process as a contextual Markoy decision process (MDP) and then leverages reinforcement learning (RL) 32 to learn dynamic configuration policies. DAC has been shown to outperform static AC methods on many tasks, such as learning rate tuning of deep neural networks 7 and step-size control of evolution strategies (35).
Due to the increasing complexity of real-world problem modeling, the performance of an algorithm usually relies on multiple types of hyperparameters. Consider MOEA/D (48), widely adopted
"Conesponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

evolutionary algorithm [4 for multi objeetiveeeeeee optimization problems, as case in point. It involves four categories of configuration hyperparameters: weights, neighborhood size, reproduction operator type, and parameters associated with the utilized reproduction operator. Each of these hyperparameters critical and exerts significant impact on the algorithm's performance [1]] For this challenging task of tuning multiple hyperparameters, traditional DAC methods tend to be ineffective [5] 42) These
methods typically tune one hyperparameter type while freezing the rest, limitation that prevents them from capturing the piler-relationshiss and dependencies across different hyperparameter types. Recently, cooperative multi-agent RL (MARL) [47] approaches have been proposed to enhance the dynamic configuration problem of multiple heterogeneous hyperparameters by modeling it as a contextual multi-agent MDP (MMDP) [44], broadening DAC's application scope and enabling various hyperparameter configurations for complex algorithms.
In addition to multiple complex hyperparameters, another important barrier of real-world DAC problems is that these hyperparameters are not completely decoupled, which may have inherent inter-dependencies. For instance, when configuring MOEA/D for multi-objective optimization, once the reproduction operator type is selected, the parameters associated with the utilized reproduction operator can subsequently be determined. Unfortunately, previous approaches do not consider this inherent property. Besides, current advanced MARL algorithms mainly focus on issues such as
coordination and adaptation mechanisms in non-stationary environment [47] 31 46 17 22), which not specifically designed to handle the inherent inter-dependencies among agents.
In this paper, we propose a sequential multi ageti DAC (Seq-MADAC) framework to address this issue by considering the inherent inter-dependencies of multiple parameters. Specifically, we formu- late this task as a contextual sequential MMDP and propose a sequential advantage decomposition network (SADN), which can leverage action-order information through sequential advantage de- composition. We also theoretically demonstrate the rationality of SADN based on the individual global max (IGM) principle [36). Leveraging this beneficial property, SADN achieves effective sequential decomposition of the demanding multiple-hyperparamete DAC task for joint action opti- mization, significantly easing the issues of combinatorial explosion of the action space and complex inter-dependencies. Empirically, we compare SADN with other methods that consider sequential actions, including ACE [19] and SAQL (3. as well as range of general advanced MARL algorithmss, including VDN 39], QMIX (II, MAPPO 46), HAPPO [17, and HASAC [22]. On controllable synthetic functions, we demonstrate that in nighes-dimensional problems and noisy scenarios, SADN shows a more pronounced advantage in optimization efficiency and robustness. On more complex and challenging multi-objective optimization problems, SADN shows superior performance over state-of-the-an MARL methods and demonstrates strong generalization across problem classes.
Our contributions include three perspectives:
1. Formulating a sequential decision socision-making framework to model the DAC task with multi- ple hyperparameters and inter-dependencies as contextual sequential MMDP, enabling sequential action ordering in dynamic hyperparameter configuration.
2. Introducing SADN with sequential advantage decomposition to exploit action-order infor- mation for improved coordination and better credit assignment.
3. Conducting extensive experiments on both simple synthetic and complex multi-objective op- timization problems to validate SADN's effectiveness in superior optimization performance and robust generalization capability.
2 Background
2.1 Dynamic Algorithm Configuration
Compared the static configuration scheme of AC, DAC focuses on dynamically adjusting algorithm configuration throughout the optimization process, which can be formulated as contextual MDP Mz := M 2 and can be addressed by leveraging RL techniques. Here, I represents the problem instance space, and each M; == (S. A. T., ri) corresponds to one target problem instance i E 2 9. Such a context notion I enables studying policy generalization [16]. For a target algorithm A with configuration hyperparameter space O a DAC policy E II takes the state E S as input and outputs an action a E A. Here, the state s S typically represents the historical performance changes of algorithm A, while the action A corresponds to a hyperparameter

---

configuration A E θ of algorithm A. DAC aims at improving the performance of algorithm A on set of instances (e.g.. optimization functions). Given a probability distribution p over the instance space I, the objective of DAC is to identify an optimal policy x²:
/ p(i)c(TT, i)di,
π E arg min
Jsez
Tenl
where E I is an instance to be optimized, and ((TT,) E IR is the cost function of the target algorithm with policy on the instance i. Previous works have shown the superiority of DAC compared to the static policies in many scenarios, e.g., learning rate adaptation in SGD [7], step-size adaptation in
CMA-ES B5 and heuristic selection in planning (37.. However, both traditional DAC methods and these applications only involve a single type of hyperparameter. Dynamic configurations of complex algorithms with multiple types of hyperparameters have been found to be difficult [.
MADAC 44 aims to simultaneously configure multiple hyperparameters of different types, in order to cope with the increasing complexity of algorithm structure and the increasing number of hyperparameters. To address this issue, MADAC models it as a cooperative multi- problem with one agent configuring one parameter, to take the interactions of different parameters into account. Another advantage of the cooperative multi-agent modeling framework is its capacity to address the combinatorial explosion challenge associated with joint action spaces. However, in practice, parameters may have inherent inter- dependencies on each other (e.g., determining the operator type first and then the operator parameters (12),, which is not considered in MADAC. It is quite important to take these inherent inter- deeeddeeendesees into consideration, since we can explore the configuration space more effectively without exploring illegal combinations, which may bring better results.
Recently, coupled action-dimensions with importance differences (CANDID) DAC 3 considers the interaction effects among hyperparameters exhibiting different levels of importance, assuming that the important parameters should be exccuted first. However, in practice, parameters exhibit complex inherent inter-dependencies. Consequently, important parameters should not be prioritized for adjustment if they depend on prior parameters. For example, while the operator parameter holds greater importance, optimized performance is attained by first confirming the operator type before adjusting the operator parameter. Moreover, CANDID DAC tries to address this issue by extending independent Q-learning (IQL) [40] to sequential agent Q-learning (SAQL). i.e., adding the actions made by prior agents to the state of subsequent agents. However, this may suffer from similar disadvantages to IQI.: The team reward is used directly in the individual agent's 's training and does not consider the credit assignment among agents, thus it cannot reveal the interactions between the agents. What's worse, it may lead to the non-convergence issue (31) because each agent's learning is interfered with by the learning process of others, and the team reward cannot provide the corresponding guidance. Thus, how to propose a framework that can efficiently leverage the internal relationships of the problem, including inherent parameter inter-dependencies and interactions among agents, remains an ongoing challenge.
2.2 Multi-Agent Reinforcement Learning
fully observable cooperative multi-agent system (45 can be formulated as an MMDP [5). defined as M := (N, {A, 1/= T,r). r). where . represents agents, S is the state space, and A, is agent j's action space. At each time-step, agent j E N acquires E S and then chooses an action (m)) leads to next state s' ~ T(- s, a) with a shared reward r(s, a). The goal of an MMDP is to find a joint policy that maps the states to probability distributions over joint actions, क : + (AAi x A x ... x An), where A (AA, x Ag x ... x An. stands for the distribution over joint actions, with the goal of maximizing the global value function: QTT a)= = Ex IE 7' at) so = s, ag = a).
a()) E Ay. The joint action a (a())
Many algorithms have been proposed to solve the cooperative MARL 10), which can be basically divided into two categories: value-based methods and policy gradient- value-based methods, VDN (9 addresses the problem of multi-agen collaboration through value function decomposition. It makes main assumption that the team Q-value function can be additively decomposed into the simple sum of the local Q-values of each agent: Q a) = > Qi a,). where ((s, (ss a) is the joint action value function for the whole team, 8 the shared state, a is the joint action, is the number of agents, Qi a;) is the agent i's local individual action value function, and a, is the action taken by agent i. One critical principle the value decomposition methods need to satisfy is the Individual Global Max (IGM) principle defined in Definition I
methods. Among the
3

---

Definition 1 (Individual Global Max [36).. For a joint action-value function Q S A there exist individual action-value functions (Q, S A, R such that the the following holds:
arg max Q1)s,a)) arg max Q2(8.02)
arg max ((s,a)
a
arg max Qn an))
then, we say that [Qa] satisfy IGM for Q, which means the decomposition satisfies the IGM principle.
This critical principle guarantees the optimal efficient decentralized execution, each agent only needs to choose own optimal action regardless other agents' actions. Obviously, VDN's s additive decomposition satisfies the IGM principle, but is restricted to its additive form, failing to model the complex situations in many cooperative MARL scenarios. QMIX [31] uses a mixing network with non-negative weights to learn monotonic combination of local individual action value functions as the joint action value function for the whole team, which extends the forms of factorization.
Another type of cooperative MARL algorithm is the policy gradient-based method. MAPPO 46 extends the popular aigete RL algorithm proximal policy optimization (PPO) 34 to the multi- agent RL setting and achieves surprising effectiveness. However, MAPPO is mainly designed for homogeneous agents with shared action space and policy parameters. HAPPO [17] further addresses heterogeneous agents with theoretical property of monotonic improvement guarantee. HASAC [22 obtains the maximum entropy objective for MARL from probabilistic graphical models to escape from converging to suboptimal Nash Equilibrium. A2PO [30] is agent-by-agent sequential update algorithm with a theoretical guarantee of the monotonic policy improvement that accelcrates optimization with semi-greed, agent selection rule. However, these recent works do not fit the needs of DAC well. Their execution is fully decentralized, with each agent unaware of others' current actions, which ignores the inherent inter-dependencies between the agents' actions.
ACE 19 models the multi-agent decision-makinn problem into sequential decision -making problem with the agents taking action one by one, formulating this as sequentially expanded MDP (SE-MDP). At each timestep, the agents make decisions based on the actions taken by their prior agents the current timestep, and the action value function of each individual agent is updated according to the performance of their subsequent agents, which make decisions based on the taken action. The update scheme of action value functions as follows:
Qi(s, a1.-1: a;) + max.n1 Qi+1(s, a1si a1t1),
if < n if i = n
r maxa; Q1('', o(1)
The sequential modeling relieves the non-stationary training problem in MARL because the agents are aware of their prior agents' actions, which also fits the need of DAC for being able capture the inherent inter- deter-dependences between the agents' actions. However, the update scheme of ACE can suffer from fragile training due to its long sequence of action value function update. That if some agents fail or fall into local optima within the long action value function update chain, the learning of the whole chain will be affected and even damaged.
Recently, similar ideas of sequential action choosing have been applied to offline RL in the centralized setting. Q-Transformer (6) and Q-Mamba (24 have been proposed for effective offline RL training in centralized manner, where advanced neural network architectures like transformer and Mamba integrated with the sequential Q function update scheme similar ACE.
3 Method
3.1
Contextual Sequential Multi-Agent MDP
Previous works have paid attention to sequential modeling to cope with the high-dimensional action spaces (Sequential MDP (27)) and non-stationary training (SE-MDP [9 in MARL. The main idea is to break the global state transition into sequence intermediate decision-making steps and select the current timestep's 's action one dimension by one dimension. That is, the original transition (s, a, 8') is transformed into n intermediate decision making steps (8, a1); (8, 01 02),....
.. (8, 01 02 an; 8').
where is the number of action dimension, and when selecting the i-th action a for dimension

---

Reward
Global advantage estimation Á(s,
Execution rollout
Execution rollout
Compute loss and update
Execution rollout
Mortie Dort 1ODA
Action
baido podale (P")
Global advantage function A(s, a)
Dynamic Configurations
fhrm oill Amad
Dituse ne effecing acialig to chsh Pia
A2(5,02.02)
4,
mit
Agent
Agent
Agent
Algorithm Environment
R,
44:2
A1:3
State Encoder
State
Figure I: Workflow of the proposed sequential advantage decomposition network (SADN).
i, the previous selected actions 01:1-1 for dimension to i - are available for decision making. Prior work shows that under the CANDID properties [3]. sequential policies coordinate action selection between dimensions while avoiding the combinatorial explosion of the action space. Their results encourage extended study of sequential policies for DAC.
However, the previous sequential modeling is typically a single-agent MDP, with one agent selecting actions one dimension by one dimension, which fails to model the complex cooperation and com- munication among heterogeneous agents. Moreover, the different contributions of different action dimensions to the global reward neglected in the sequential single-agent MDP modeling, which harms the learning efficiency and may lead to sub-optimal results.
In the DAC task, there are often different heterogeneous parameters (i.e., they are of different types and have different effects on the final performance of the target algorithm) with complex inherent detrrddependencis, which call for cooperation and communication among the tuning of different parameters. What's more, as the heterogeneous parameters have different effects on the target algorithm's 's performance, the tuning of them should be guided with differentiated credit assignment according to their contribution to the improvement of the target algorithm's performance.
Therefore, we formulate the Seq-MADAC task as contextual sequential MMDP as My := Mi-z, where represents the space of problem instances, and M; == (W, S, [Ai] Tr, r.) is each sequential MMDP corresponding to one target problem instance E I. In a sequential MMDP, at the timestep t, the agents are allowed sequentially take their actions and communicate with each other to make better decisions. That is, the available information for the i-th agent at timestep is (st, a a!....,a! a{__) = (st, -) and the i-th agent selects its action af accordingly. When all the agents finish selecting their actions, the state transforms the next state $441 T(- st, a²) based on the joint action a" (a) ... a')) an and all agents get a shared global reward r(s,, a²).
3.2 Sequential Advantage Decomposition Network
Recent sequential update schemes in MARL suffer from interference from other agents' learning (i.e.. SAQL B) or the fragility of the long action value function update chain (i.e., ACE (19]), while the Meq-MADAC task requires MMDP modeling with differentiated credit assignment. Thus, we propose decomposition--asse method called Sequential Advantage Decomposition Network (SADN) to address these issues, whose decomposition does not need any assumption, with the IGM principle satisfied, The workflow of our SADN is shown in Figure]
5

---

Firstly, we give the definition of the action value function and the advantage function of each agent in the sequential setting. We define the action value function for the previous agents state s as QT (8, 01:k), ), and the k-th agent's advantage function based on state and the actions di.k taken by its prior agents as AT (s, 01: (x) in the sequential form in Definition 2 following 17].
Definition (177). Let n. be the total number of agents, 01k denote the actions laken by agent to k at the current timestep, and Q (s. a) denote the global action value function. The corresponding multi-agent state-action value function is defined
Q a1a)) Eag im Ii (QT 0.1kks as:1:m)]
and the advantage function of agent la with the given state and the actions 0i:k-1 taken by its prior agents is
A (s, ai:- ax) = (T, a1sk)
Following Definition we give the multi-agent advantage decomposition lemma in the sequential form in Lemmal 1 For the proof, please see Appendix A.3
Lemma (Multi-Agent Advantage Decomposition 17).. In any cooperative Markov game, given joint policy æ, the global advantage function A (s, a) and n agents in total, for any state the following equations hold:
(s, a) = Σ (8. a1)
Inspired by the action value function decomposition and the implicit reward assignment by using a decomposition network proposed in [39], as well the multi-agent advantage decomposition lemma in [17), we decompose the advantage function sequentially. In specific, the i-th agent maintains network that models the i-th advantage function A (s, 01:4-1, az) and selects the action that maximizes it, and we compute the global advantage function by the sum of all individual advantage functions, using Lemma [ The i-th advantage function is updated implicitly when the global advantage function is updated by backpropagating gradients. To update the global advantage function, we maintain a global value network, like the critic the actor-critic framework, and compute the target global advantage function by Generalized Advantage Estimation (GAE, (33) with À = 0 (i.c., one-step temporal difference), which corresponds to the update scheme of action value function in Q-learning (the detailed proof see Appendix A.I. The global advantage function update scheme is as follows:
A(s. + A(s + VV(')) Vss) A(s, a)};
where A(s, a) is the global advantage function, & is the current state, a is the joint action, s' is the next state, is the steplength, is the discount factor, and (s)) is the global value function.
Our method also satisfies the IGM principle in the sequential setting, which stated in Theorem
Theorem 1. Selecting the best joint action for state maximize the global action value function equivalent sequentially selecting the best action 0; for state maximize the agent i's advantage function. That is,
arg max (8, a1) arg max.22(s arg maxA, A1 (s, a1).
arg magx(((sss))
02
"1
arg max (n. (s, arg max A a1). arg ma arg ma (s, a2)..... an)
61
og
"1
With this beneficial property, SADN provides efficient and effective sequential decomposition of the challenging sigh-dimensional task of joint action optimization, where the individual agent only needs to focus optimizing its own individual advantage function by taking its own action. It is of great benefit because the IGM principle guarantees the consistency between the joint action optimality and the individual action optimality, which significantly alleviates the problem of combinatorial explosion of the action space and complex inter-dependencies.. Detailed proofs are provided in Appendix A.2 due to space limitation.
The advantage decomposition the SADN allows each individual advantage net to updated independently and simultancously, reducing their mutual influence and compounding errors. In

---

contrast, ACE sequentially updates the Q nets based on the output of the previous agent (i.e., more centralized), which leads compounding errors along the update chain, especially worsening in the long sequence updates. Moreover, SADN decomposes the global advantage function sequen- tially, where the individual advantages are learned relatively independently, and the action sequence information is also captured in the update order of the advantage functions.
4 Experiment
4.1 Experimental Settings
To comprehensively evaluate the performance of SADN, we compare it with other RL methods that consider sequential actions, including ACE [9]] and SAQI. (3), as well as a range general advanced MARL algorithms, including value-based methods like VDN [39] and QMIX [31], and methods like MAPPO [46), HAPPO [17] and HASAC [22]. We also compare
policy gradient-
it with an extension of MAPPO, MAPPOar ΤлЛ which considers sequential actions by simply adding the prior actions to the state of the subsequent agents. We also investigate HAPPO's ability of sequential modeling by adjusting the permutation of updating agents.
We consider the following environments for comparing these methods, where the details of these environments can be found in Appendix B
Sigmoid. The Sigmoid task 2 is basically an approximation task with hyperparameters changing over time. For sampled instance i and the h-th hyperparameter, the target sigmoid function to approximate is defined as sig(t, St,h, St.h.
1+c 1+e"* (t and the team reward at timestep
on instance i is defined as r = Io (1 |sig(t, (t. & Pt.)) an,r)), where H is the number of hyperparameters, and aj,; is the configuration for the h-th hyperparameter at timestep t.
Seq-Sigmoid. Despite considering multiple hyperparameters, the original Sigmoid lacks hyperpa- rameters with inherent inter-dependencies, making it unable to reflect the actual situation and the behavior of complex algorithms. Therefore, we modify the original Sigmoid into Seq- sigmoid to bring in the inherent dependeeeessss among hyperparameters. The main modification that at timestep f, the former hyperparameter's value will determine a scaling factor ai,r, which controls the latter hyperparameter's s slope 8t,.- In this way, the former hyperparameter's configuration will have a strong influence on the configuration of the latter hyperparameter. Moreover, in order to avoid that having this dependency affects the configuration of the former hyperparameter itself, we use the formula ming]sig(t, an,t 54,6, P..A) ah,a |1 sig(t, a's,t" 5i,Ai Pi.A) ag,z) to measure the approximation, which returns the same value when two values of an,t are symmetrical about 0.5. The pseudo-code of the -igmodd benchmark is provided in Algorithm I
Algorithm 1 Benchmark Outline: Dig-Sigmoid
1: Benchmark Parameters: number H of hyperparameters, number C₁ of choice for each hyper-
parameter ht episode length T;
2: (((1((11000 100, 100, H);
3: P: ~ (N(T/2,T/44 H);
4: for t E {0, (0.1,...,17 T) do
5: The state at step t: state; = 34 Up U (t):
6: actions: ag,t E
Ch o ...., G-1
for all
H and 0 osts
C2
10 if a 2 0.5
7: 0o.t = 1,0 =
0 < h H;
0.1 if ff-111 < 0.5
8:
r/ + I h= (1 (nin(ssig(t, an, 8i,h, Pi, h) - and sig(1, ah, Si,,, an,)))
9:
end for
Seg-Sigmoid-Mask. This benchmark masks 8i,A and Pi.b (i.e., the agents can only observe state; = (()) and sets S... = to avoid too much randomness in Seq-Sigmoid, which makes the task single instance task. That is, we denote the instance as ig, and the goal to approximate to sig(t, 1, Pio.h)) but with randomness (i.c., Pio.h (T/2 T/ H)). This is to simulate the highly-random execution
7

---

Seq-Sigmoid
Seq-Sigmoid
Beq-Sigmoid-Mer
10 Seq-Sigmoid-Mass
K M
Return
2
0.0
0.4
0.6
1.0
0.7
0.6
0.8
0.0
04
0.0
06 Episodes
Episodes
Episodes
Episodes
SADN
ACE
SAQL
VON
QMIX
MAPPO
HAPPO
HASAC
Figure 2: Training curves of return value obtained by the compared methods on four Seq-Sigmoid variant tasks, where the results averaged over 6 runs.
of evolutionary algorithms, serving testbed for evaluating the DAC algorithms' ability when coping with randomness.
Seq-Sigmoid-Robee (n). This benchmark gives n hyperparameters completely random configura- tions (regardless of how agents tune these hyperparameters), while all other hyperparameters configured based on the actions. This is to simulate the complex scenarios where some agents can not learn the proper actions, serving as testbed for evaluating the robustness of DAC algorithms.
is a challenging task 44 based the well-known multi-objective evolutionary algorithm MOEA/D with heterogeneous hyperparameters, which is strong stochastic benchmark due to the randomness of the evolutionary optimization process. The action space of the MOEA/D environment includes four hyperparameters of MOEA/D: weights, neighborhood size, types of the reproduction operator, and the corresponding hyperparameters of the reproduction operators. The state includes the features of the problem instance, the optimization process, and the evolution statistics of the current population. For the reward, we use the triangle-based reward function proposed in 44 which has been proven effective in the MOEA/D environment.
MOEA/D. MOEA/D 44 48
4.2
RQ1: the sequential information useful?
On the original Sigmoid benchmark, we traditionally train the agents with set of given instances (i.e., a set 8i, and ps.a) and test them on other sampled instances. order to better reflect generalization ability of the algorithmss, every time one episode is done and the environment reset. Therefore, the agents are truly trained over the distribution of the problem instances, and every point the training curve can reflect the generalization ability of the learned policy at the exact training phase.
resample the instance (i.c., resample a new st, and pi.))
Figure 2 demonstrates the smoothed training curves of the return values of our method and the compared methods on -igmoodd and Soqg-iigoii-Maa with 5/10 dimensions (i.e., 5/10 hy- perparameters). As mentioned in the previous paragraph, we resample the instance every episode, and thereby the training curves reflect the policies' performance over the instance distribution at a certain training phase. It can be observed that our method, SADN, outperforms other methods on every benchmark, with faster convergence, less variance, and better final performance. Basically, the methods exploiting the sequential information (i.e., SADN, ACE, and SAQL) succeed in capturing the inherent inter-dependencics between the hyperparameters to some extent and achieve better performance, especially on the Beq--Sigmidd-Mi benchmark, where the traditional MARL methods fail to cope with the complex inter-dependencies and randomness. However, ACE shows a clear performance decay on the -igmoodd benchmark. This phenomenon may stem from the disruption of the long-chain action-value function update scheme ACE by diverse problem instances, where certain action-value functions in the chain fail achieve adaptation novel instances.
The comparison of the sequential methods based on the correct order and the reverse order given in Appendix C.2 due to space limitation. Basically, the reverse order underperforms the correct order

---

Seq- Sigmold Robust (1)
Seq-Sigmoid-Robern Robbust
10 Seq-Sigmoid-Roben
Sea- (2)
30
Return
3.5
2.0
155
20
0
Episodes
Episodes
Episodes
Episodes
SADN
ACE
SAQL
Figure 3: Training curves of return value obtained by the sequential methods on four Seq- -ig-Sigmoid- Robust tasks, where the results are averaged over 6 runs.
and produces a larger variance due to the wrong capture of the inherent inter-dependencier between the hyperparameters, except for HAPPO, which only considers the update order of the agents rather than the order of taking actions. The comparison of using the correct and the reverse order provides evidence for the effectiveness of modeling the inherent inter-dependencies correctly.
We also provide experimental results of our method and compared methods on the original Sigmoid in Appendix [.I The results show that the effectiveness of our method even without strong inter- dependencies between the hyperparameters.
4.3
RQ2: How robust is SADN?
In complex dynamic parameter configuration problems, some agents often struggle to learn effectively, e.g., the configuration range of certain parameter is not suitable, causing the corresponding agent to struggle to learn effectively. Especially in sequential scenarios, this issue can be more pronounced because those hindering agents give misleading information through communication other agents. Therefore, we test the methods designed for sequential Decision-making on Seq- Sigmoid- Robust examine their ability cope with this issue.
Figure plots the training curves of SADN (ours), ACE and SAQL on Seq-Sigmoid-Rond Robust with 5/10 dimensions and 1/2 random agents. SADN shows better performance and robustness when coping with this complex situation, while ACE shows apparent performance decay because its fragile long-chain update scheme suffers from the instability of learning caused by the random agents, and SAQL sticks in poor performance due to the interference of the random agents.
4.4 RQ3: How does SADN perform on tuning real complex algorithms like MOEA/D?
In the MOEA/D environment, we set the proper parameter order as: determining the neighbor- hood size to decide whether to explore or exploit; 2) choosing the operator type; 3) choosing the corresponding operator's parameters; 4) deciding whether to update the weights to bring in new subproblems. We compare our SADN with different state-of-the-art MARL methods, RL methods that consider sequential actions. We use the Inverted Generational Distance (IGD) 4 as the metric to measure the performance of the algorithms, which is the smaller the better.
well as the
As demonstrated in Table the top three problems are the problems for training, and the bottom five problems are for testing. Our proposed SADN achieves significantly superior performance on most problem instances, and averagely ranks the best on all problem dimension settings. There are only occasional cases where SADN underperforms the baseline, and all of them occur in the testing phase. This may be owing the very different function landscape that was not included in the training phase, and the leamned RL policy fails to generalize. This issue is always of concern regarding the generalization ability of RL, and we believe good state and reward designs that capture commonalities among various problem sets can further enhance generalization, datasets. In general, our SADN shows strong generalization abilities across problem classes. The comparison results of using the correct and the reverse order, which are provided in Appendix C.3 due
well as more diverse training

---

Table 1: IGD values obtained by SADN and the compared methods on different problems. Each result consists of the mean of 10 runs. The best mean value on each problem highlighted in bold. The symbols '+', and 'z' indicate that the result is significantly superior to, inferior to, and almost equivalent to our method SADN, respectively, according to the Wilcoxon rank-sum test with
significance level 0.05.
Problem
Dim
MOEAD [8]
SADN
ACE [1]
SAQL S
VDN 39
QMIX 31
MAPPO [6]
HAPPO [b]
HASAC 23
4.593e-02
3.804-022
3851e-02
3.950e-02
3.905-022
3.906e-12
3.790e412
3.838--122
4 4.945e412
DMTL22
4.598e-10
3.875-002
4.576-02
4.337e002
4.00e-022
4.120e-02
3.907e-02
0.095e-02
4.103e-02
12
4,599--02
3.874e-02
4.063-022
4.698c-02
3.958-002
4.161c-12
3.970-422
4.075e-422
4,1400:42
6
02
4,537e-02
4.626e-02
4.817e-02
4 4.904-022
4.827-002
4.639c-02
4.9 4.956e-022
S. 5.703--422
5.736c-02
5.190e-02
0.853e-02
5.365c-02
5.800-022
5,511-022
5.341c-02
6.8e-202
6.169.-02
12
0.
5,2133-00
5.751e-02
6.241c-02
5.687e-02
5,835c-02
5,340c-02
6.071e-02
6.187e-02
6
5.855e-02
3.908e-02
3.962e-02
3.964e-02
4.080e-02
4.214e-02
3.937e-02
4.022a-02
4.5366-42
WFG6
6.641c-02
4884c-12
5.111c022
4.384c-02
5.55%--02
7/466-422
6.194c-12
5921e-02
4879
6.8644-022
48376-02
5.490e-022
5.676-402
5.444c-02
7.999-402
6.067e-022
0.218e-02
4.993e-02
Traäning: +/-/u
0/9/0
0/8/1
1/8/0
0/90
0/9/0
1/8/0
0/9/0
0/8/1
6
5.455e-02
3.895-022
4.221=-02
0.119.422
3.966-022
4.331e402
3.909--22
3 9877e-02
4.410ee412
6.229e-02
7.319e-02
4.492-002
6.895e-02
4.361e-02
7.108e-02
6.168e-02
4.645e-02
4.678e-02
12
6.790e 02
7.852e-02
4.650 02
8.589e-02
4.647e-02
5.532e-02
5.794e-02
5.107e-02
4.825e-02
6
6. 0Z
4,749-002
4.752-002
4.3900-022
4.761c-02
4.786e-02
4.719,-02
4 4.746e-12
4, 4.744p-12
6.351c-02
4.773c-02
0.7922-02
4.788c002
1.784e-02
4,770x-02
4.762:-012
1.793.002
4.780c-02
12
6.38
4.793c-02
4.791.-02
4.954e-02
4.780c002
4.778e-02
4.789.-02
4.7996-02
4.785e-02
6
5.803e 02
3.893e-02
0.942e
3.958e-02
4.08
4.207e-02
3.918e-02
3.9800 02
4. 4.372e-02
WFG7
9
5.81 02
4.053e-02
4.470a-02
4.257e-02
4.566c-02
4.407e-02
4.119c-02
4.607e-02 02
4.632e-02
12
5.814e-02
4.060-022
4.4476-02
4.975e-02
4.432e-02
4.445e-02
4.132e-02
4.5976-02
4.689e-02
6
7.875e-02
1018c-01
L009c-01
1.054c001
1.169-01
1.04c-01
1.037c001
1.166-001
1,252e-01
WFGS
9
02
7.853-022
9.787e-02
9.716c-02
8.720-002
8.180--02
7,880-422
9 9.964c-02
9.991e-41
8.710e-02
1.891c-02
8.387e-02
9.023e-02
8.552e-02
8.279c-02
0.923e-02
8.636e-02
8.73
6
5.600c 02
3.987e-02
4.012.-02
4.010e-02
4.156e-02
4.266e-02
3,986e-02
4.041e 02
4,395e-02
WPG9
9
5. 3.748e-10 02
4.341e-02
02522-02
4.281c-02
7,965e-02
5,274e-02
4.573e-02
5.079e-02
5,4 5,429e-01
5.827e-02 12
4,249c-02
4.477e-02
7.291e-02
8.023e-02
5,246e-02
4.495e-02
6 6.556e-02 02
8.189e-02
Testing:
-/-/
1/12/2
2/9/
0/114
2/12/1
2/11/2
4/516
2/11/2
2/1033
6
8
1.75
3.625
5.5
5.75
6.25
2
4.875
7.25
7,375
2.5
4.75
4.375
4.75
5,375
3.125
6.25
6,5
7.25
2.5
3,875
8.125
4.875
3.625
5.75
to space limitation, also demonstrate the effectiveness of correctly considering the parameter order on the real-scenario multi-objective evolutionary algorithm. Generally, the correct order produces better results than the reverse order within the same sequential methods.
Moreover, we also provide a discussion of the generalization ability across different dimensions in Appendix C.4, Overall, the results demonstrate that our proposed method exhibits good generalization abilities across different dimensions. However, cross- doos--immesional generalization remains a significant challenge and is worth further investigation, such as through multi milti-dimennional mixed training and employing multi-head models.
Conclusion
This paper considers the inherent dependeeensss among hyperparameters in DAC, which are natural and common in practice. We propose contextual sequential MMDP formulation to capture the inherent interdeependencces among hyperparameters and also propose a sequential advantage decomposition network to solve it. Experiments from synthetic white-box tasks with apparent inter- dependencies to complex real-scenario multi-objective evolutionary algorithm demonstrate the effectiveness of our proposed method, as well as its strong generalization ability. One important future work is to modularize the algorithms that support more flexible reconstruction of different algorithm uob-modules to design diverse algorithm structures [2].
One limitation that the hyperparameter order requires to be given in advance. In practice, there is an intuitive and natural way to determine the order: setting it according to the sequence in which the hyperparameters take effect during the execution of the algorithm. Specifically, in the code implementation of the target algorithm, all hyperparameters take effect in a specific order, which is the order we have set. This strategy aligns with our intuition and performs well the experiments of this paper. In the future, one could apply large language models to set the proper order of the hyperparameter configuration, as well as causal models to automatically learn the hyperparameter's causal structure from the data (25).
10

---

Acknowledgment
The authors thank anonymous reviewers for their insightful and valuable comments. This work was supported by the National Science and Technology Major Project (20222D01 16600), the National Science Foundation of China (62276124, 624B2069, 62506159) 159), the Fundamental Research Funds for the Central Universities (14380020), and the Young Elite Scientists Sponsorship Program by CAST for PhD Students. The authors want to acknowledge support from the Huawei Technology Cooperation Project.
References
[1] S. Adriaensen, A. Biedenkapp. G. Shala, N. Awad, T. Eimer, M. Lindauer, and F. Hutter. Auto- mated dynamic algorithm configuration. Journal of Artificial Intelligence Research, 75:1633- 1699, 2022.
[2] A. Biedenkapp. H. F. Bozkurt, T. Eimer, Hutter, and M. Lindauer. Dynamic algorithm configuration: Foundation of new algoritggo framework. In Proceedings of the 24th European Conference on Artificial Intelligence, Santiago de Compostela, Spain, 2020.
[3] Bordne, M. A. Hasan, E. Bergman, N. Awad, and A. Biedenkapp. Candid dac: Leveraging coupled action dimensions with importance differences in DAC. arXiv:2407 2007092407,05789, 2024.
[4] A. N. Bosman and Thierens. The balance between proximity and diversity in multiobjective evolutionary algorithms. IEEE Transactions Evolutionary Computation, 7(2): 17--188888 2003.
[5] C. Boutilier. Planning, learning and coordination in multiagent decision processes. In Proceed- ings of the 6th Conference on Theoretical Aspects of Rationality and Knowledge, De Zeeuwse Stromen, The Netherlands, 1996.
[6] Y. Chebotar, Q. Vuong, K. Hausman, F. Xia, Y. Lu, A. Irpan, A. Kumar, T. Yu, A. Herzog, K. Pertsch, et al. Q-transformer: Scalable offline reinforcement learning via autoregressive q q-functions. In Proceedings of 7th Conference on Robot Learning, Atlanta, Georgia, 2023.
[7] C. Daniel, J. Taylor, and S. Nowozin. Learning step size controllers for robust neural network training. In Proceedings of the 30th AAAI Conference on Artificial Intelligence, Phoenix, AZ, 2016.
[8] K. Deb, L. Thiele, M. Laumanns, and E. Zitzler. Scalable multi-objective optimization test problems. In Proceedings of the 9th Congress on Evolutionary Computation, Honolulu, HI, 2002.
[9] Eimer, A. Biedenkapp, M. Reimer, Adriaensen, Hutter, and M. Lindauer. DACBench: benchmark library for dynamic algorithm configuration. In Proceedings of the 30th International Joint Conference on Artificial Intelligence, Montreal, Canada, 2021.
[10] Z. Feng, R. Xue, L. Yuan, Y. Yu, N. Ding, M. Liu, B. Gao, J. Sun, X. Zheng, and G. Wang. Multi-agent embodied AI: Advances and future directions. arXiv:2505.051 2025.
[11] W. Fu, C. Yu, Z. Xu, Yang, and Y. Wu. Revisiting some common practices in cooperative multi-agent reinforcement learning. In Proceedings of the 39th International Conference Machine Learning, Baltimore, Maryland, 2022.
[12] H. Guo, Z. Ma, J. Chen, Y. Ma, Z. Cao, X. Zhang, and Y.-J. Gong. ConfigX: Modular configuration for evolutionary algorithms via multitask reinforcement learning. Proceedings of the 39th AAAI Conference on Artificial Intelligence, Philadelphia, Pennsylvania, 2025.
[13] S. Huband, P. Hingston, I.. Barone, and L. While. A review of multiobjective test problems and scalable test problem toolkit. IEEE Transactions on Evolutionary Computation, 10(5):477-506, 2006.
[14] Hutter, H. Hoos, K. Broton and Stützle. Paramils: An automatic algorithm configuration framework. Journal of Artificial Intelligence Research, 36:267-306, 2009.
11

---

[15] F. Hutter, L. Kotthoff, and J. Vanschoren. Automated Machine Learning: Methods, Systems, Challenges. Springer, 2019.
[16] R. Kirk, A. Zhang, E. Grefenstette, and T. Rocktäschel. A survey of generalisation in deep reinforcement learning. arXiv:2111.09999) 2021.
[17] J. G. Kuba, R. Chen, M. Wen, Y. Wen, F. Sun, J. Wang, and Y. Yang. Trust region policy optimisation multi-agent reinforcement learning. arXiv:2109. 12511, 2021.
[18] S. Kukkonen and K. Deb. A fast and effective method for pruning of non-dominated solutions in many-objective problems. In Proceedings of the 9th International Conference on Parallel Problem Solving from Nature, Reykjavik, Iceland, 2006.
[19) C. Li, J. Liu, Y. Zhang, Y. Wei, Y. Niu, Y. Yang. Y. Liu, and W. Ouyang. Ace: Cooperative multi-agent q-learning with bidirectional action-dependency. In Proceedings of the 37th AAAI Conference Artificial Intelligence, Washington, DC, 2023.
[20] K. Li. Decomposition multi-objective evolutionary optimization: From ttate-of-the-at to future opportunitics. arXiv:2108. 09588, 2021.
[21] K. Li, A. Fialho, S. Kwong, and Q. Zhang, Adaptive operator selection with bandits for a multi- objective evolutionary algorithm based decomposition. IEEE Transactions on Evolutionary Computation, 18(1): 14-130, 2013.
[22] J. Liu, Y. Zhong, S. Hu, H. Fu, Q. Fu, X. Chang, and Y. Yang. Maximum entropy heterogeneous- agent reinforcement learning. Proceeding of the 12th International Conference on Learning Representations, Vienna, Austria, 2024.
[23] M. López-Ibáñez, J. Dubois-Lacoste, L. Pérez Cáceres, M. Birattari, and T. Stützle. The irace package: Iterated racing for automatic algorithm configuration. Operations Research Perspectives, 3:43-58, 2016.
[24] Z. Ma, Z. Cao, Z. Jiang. H. Guo, and Y.-J Gong. Beaa-black-hox-oppiiizzie through offline q-function learning. In Proceedings of 42th International Conference on Machine Learning, Vancouver, Canada, 2025.
[25] 7. Ma, Chen, H. Guo, and Y.JJ Gong. Neural exploratory landscape analysis for meta- In Proceeding of the 13th International Conference on Learning Representations, Singapore, 2025.
[26] Z. Ma, H. Guo, Y.-J. Gong, J. Zhang, and K. C. Tan. Toward automated algorithm design: A survey and practical guide to meta-black- bop.omiiizzooiiiinnn IEEE Transactions Evolutionary Computation, in press, 2025.
[27] L. Metz, J. Ibarz, N. Jaitly, and J. Davidson. Discrete sequential prediction of continuous actions for deep RL. ToXX:v:7705.00000 2017.
[28] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis. Human-level control through deep reinforcement learning. Nature, 518(7540):5295333 2015.
[29] Y. Qi, X. Ma, F. Liu, L. Jiao, J. Sun, and J. Wu. MOEA/D with adaptive weight adjustment. IEEE Transactions on Evolutionary Computation, 22( 2224,, 2014.
[30] Y. Qing, S. Liu, J. Cong, K. Chen, Y. Zhou, and M. Song. A2PO: Towards effective offline reinforcement learning from an advantage-awer perspective. In Advances in Neural Information Processing Systems 37, Vancouver, Canada, 2024.
[31] T. Rashid, M. Samvelyan, C. Schroeder, G. Farquhar, J. Foerster, and S. Whiteson. Qmix: Mono- tonic value function factorisation for deep multi-agent reinforcement learning. In Proceedings of the 35th International Conference on Machine Learning, Stockholm, Sweden, 2018
[32] S. R. S. and A. G. Barto. Reinforcement Learning: An Introduction. MIT Press, 2018.
12

---

[33] J. Schulman, P. Moritz, S. Levine, M. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv: 02438, 2015.
[34] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv:1707.06444 2017.
[35] G. Shala, A. Biedenkapp. N. H. Awad, S. Adriaensen, M. Lindauer, and F. Hutter. Learning step-size adaptation in CMA-ES. In Proceedings of the 16th International Conference on Parallel Problem Solving from Nature, Leiden, The Netherlands, 2020.
[36] K. Son, D. Kim, W. J. Kang, D. Hostallero, and Y. Yi. QTRAN: Learning to factorize with transformation for cooperative multi-agent reinforcement learning. In Proceedings of the 36th International Conference on Machine Learning, Long Beach, CA, 2019.
[37] D. Speck, A. Biedenkapp. F. Hutter, R. Mattmüller, and M. Lindauer. Learning heuristic selection with dynamic algorithm configuration. In Proceedings of the 31th International Conference on Automated Planning and Scheduling, Guangzhou, China, 2021.
[38] J. Sun, X. Liu, T. Bäck, and Z. Xu. Learning adaptive differential evolution algorithm from optimization experiences by policy gradient. IEEE Transactions on Evolutionary Computation, 25(4):666 (664),6666600000 2021.
[39] P. Sunehag, G. Lever, A. Gruslys, W. M. Czarnecki, V. F. Zambaldi, M. Jaderberg, M. Lanctot, N. Sonnerat, J. Z. Leibo, K. Tuyls, and T. Graepel. Value-decomposition networks for coop- erative multi-agent learning based on team reward. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, Stockholm, Sweden, 2018
[40] A. Tampuu, T. Matüsenn D. Kodelja, I. Kuzovkin, K. Korjus, J. Aru, J. Aru, and R. Vi- cente. Multiagent cooperation and competition with deep reinforcement learning. PloS one, 12(4))e0172395, 2017.
[41] A. Trivedi, D. Srinivasan, K. Sanyal, and A. Ghosh. A survey of multiobjective evolution- ary algorithms based decomposition. IEEE Transactions on Evolutionary Computation, 21(3):440-455 462, 2017.
[42] D. Vermetten, S. Rijn, T. Bäck, and C. Doerr. Online selection of CMA-ES variants. In A. Auger and T. Stützle, editors, Proceedings of the 21 Genetic and Evolutionary Computation Conference, pages 951-959, Prague, Czech Republic, 2019.
[43] Z. Wang. Q. Zhang, A. Zhou, M. Gong, and L. Jiao. Adaptive replacement strategies for MOEA/D. IEEE Transactions on Cybernetics, 46(2):474 486, 2016.
[44] K. Xue, J. Xu, L. Yuan, M. Li, C. Qian, Z. Zhang, and Y. Yu. Multi-agent dynamic algorithm configuration. In Advances in Neural Information Processing Systems 35, New Orleans, LA, 2022.
[45] Y. Yang and J. Wang. An overview of multi-agent reinforcement learning from game theoretical perspective. arXiv:2011.00582. 2020.
[46] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness of ppo in cooperative multi-agent games. In Advances in Neural Information Processing Systems 35. New Orleans, LA, 2022.
[47] L. Yuan, Zhang, L. Li, C. Guan, and Yu. A survey progress on cooperative multi-agent reinforcement learning in open environment. arXiv:2312.01005, 2023.
[48] Q. Zhang and H. Li. MOEA/D: A multiobjectivee evolutionary algorithm based on decomposition. IEEE Transactions on Evolutionary Computation, 11(6):712-73.) 2007.
[49] Z. Zhou, Y. Yu, and C. Qian. Evolutionary Learning: Advances in Theories and Algorithms. Springer, 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims are that sequential modeling is more effective in the DAC task and experiments in section 4 proves it.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In section we discuss the limitations of our work as requiring experts providing parameter order in advance.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer: [Yes]
Justification: In Appendix A, we provide the proof the advantage update scheme, our method's IGM principle satisfaction, and the lemma proposed in [17]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In Appendix B, we propose the detailed information about the environment setting parameters and the algorithms hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
In the Abstract, we provide anonymized URL source code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ccippi.cc guwdes/ddeeeee CooeeSumeeieeeeeeeeeee ssi onyylicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: In Appendix B, we propose the detailed information about the environment setting parameters and the algorithms' hyperparameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: In section 4 our experiments are all conducted on various different seeds and use the mean value as the results. We use the Wilcoxon rank-sum test with significance level 0.05 to measure whether we get significantly better performance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
16

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: All the experiments are easy to carry out, which has little requirement for computer resources, and the detailed information of the computer resources we use see Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics htttp:://euurpge: httpp//cccccppccccccecccciiee
Answer: [Yes]
Justification; This paper does not involve human subjects participants, and only uses open-source benchmarks. This paper focuses on algorithm configuration, which has little societal impact and potential harmful consequences.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This paper focuses on algorithm configuration, which has little societal impact and potential harmful consequences.
17

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not involve released models and datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper of all the open-source code and benchmarks use in our paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
18

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, papprsaipphaeeeooccaaapppaapppse thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide the documentation alongside our code in an anonymized URL.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper focuses on algorithm configuration, which does not involve crowd- sourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper focuses on algorithm configuration, which does not involve crowd- sourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
19

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
Every components of our method does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
20

---

A Proof
A.1 Deriving the global advantage update scheme from the Q-learning update scheme
In this appendix, we derive our global advantage update scheme (i.c., the GAE with À = 0): A(s. a) + A(s,a) + afr + > - -
from the Q-learning update scheme
Q(s,a) Q(s, + + Q(s ((s a)]
follows:
((s,a) ((,,a) + p Q(5 a') Q(s
A(s, a) V(s) A(s, + V alr + max + V A(s, V(s)]
a
A(s, a) + A(s, + air + max A(s d') - >V((') A(s. V(s)
A(s, a) + A(s,a) + air + >V((s) V(s) A(s,a)
Note that in Q-learning, we choose the action greedily with respect to the learned Q-function by (s): arg aa a(s,a), and V(s) E 12 t 7'rr|s0 = = max.E. Q(s, a). Therefore,
maxa A(d, a²) = maxa 2(s',at - Vss)) O.
A.2 Proof of Theorem 1
Theorem Selecting the best joint action a for state s to maximize the global action value function is equivalent to sequentially selecting the best action for state s to maximize the agent i's advantage function. That is,
arg max.i (8, a1) arg maxA1(s, arg maxA1 a1).
arg maxa(sssa)
02
arg Ln arg (s 01), arg max arg maxAi(s, . an) 4n
o2
"
Proof. Let a' denote the optimal action that maximizes the Q-function (that is, a" arg max, (s, a)), and for the agent i, its policy based on the actions of its prior agents be de- noted as (s, a1 == arg max,, Ai(s, a1s-1, a.) and the joint policy π == ( .., Tn).
By induction, we prove that for the agent i, if its prior agents have selected the optimal actions ai it can select its optimal action of by maximizing its advantage function (, (1)))))))
For agent n:
a arg max Q(s, aim--1, an)
arg max (Q(x a (1 NTT., [Q(s, ain- an)l]
arg max A, (s, a -. an)
where the second equation holds because Ea, [Q(s, - an) is a constant, and the third equation refers to the definition of A, aim- an) in Definition
1,
We now assume that for the agent i + to this property holds, and for the agent i:
a arg max Q(s, ai ag arg max (s, 114, a.t1:n) a; 1:m
a;
arg max E a im [Q(s, a 0; 1;4)
a;
arg max E. in 7m (Q a 0s [.41m))] E Ba..sT [Q(s, a am,)]]
4;
arg max (, a a;)
4,
21

---

where the second equation holds because the inductive hypothesis implies that if the i-th agent selects the optimal of I ai;)) ai+im. and thus we have
Va,, a++ lan Ba, 1o 1.m ais-; ai [m)]
= Q(, a/ (1
2 Q(s, aiu- a1+1m))
which means
Vag, Ba mis (Q,s, a 0,, B Ba:t - (Q aia-1: 0z, 0i+1:a)]
Therefore, aj = arg max Ea, il lnn
QQ( a
11 uz a:+ 1mm) and the second equation holds.
The third equation holds because E Pain -, (1 ai Oi: is constant, and the forth equation refers to the definition of Ar(s, ai ai) Definition
Therefore, by induction, we conclude:
Vi, a = arg max A1 ai))
R,
Notably, when i 1, oj arg max,, A1(s, and for subsequent agents, the agent 2 will select a. arg max A2(s, af (2), and so on, with the agent n selecting a arg max,,. An (8, (in- an).
Thus, we have completed the proof of Theorem I
A.3 Proof of Lemma
Lemma 1 (Multi-agent Advantage Decomposition (17). any cooperative Markov game, given
a). and agents in total, for any state the joint policy the global advantage function A ( as. following equations hold:
A a) Σ (s, a1=-1,a)
Proof. Similar to the proof in 17 according to the Definition we have
A (s. a) (5 (T)
(T, and) - -T. a1a-1)
Σ (8,
which finishes the proof.
B
Detailed settings and information of the environments
In this section, we introduce our detailed experimental settings, including network architectures, experimental configurations, and hyperparameter settings.
B.1 Seq-Sigmoid
In this subsection, we introduce the original Sigmoid benchmark, the Seq-Sigmoid benchmark, and its variants in detail.
Sigmoid (2)
The pseudo-code of the Sigmoid benchmark is provided in Algorithm 2 The Sigmoid task basically an approximation task with parameters changing over time, with the approximation target as sig(t, $4,bs P),)), the state provided in line 5, and the reward at t calculated in line 7.
22

---

Algorithm 2 Benchmark Outline: Sigmoid
1: Benchmark Parameters: number H of hyperparameters, number Ch, of choice for each hyper-
parameter h, episode length T;
2: 8; ((((0(000)) 100, H);
3: P; ~ T// H);
4: for L [0,1,-... T} do
5: The state at step 1: state, = 8, U
6:
Select actions: an,
Ch Ch ..... Ca
C--1
for all 0 S h < H and $ TT
7:
ri +
- (1 [sig(t, Si,hi Pi,h) an,: l):
8:
end for
Seq-Sigmoid. The pseudo-code of the Seq-Sigmoit benchmark is provided in Algorithm The main modification (i.e., in lines 7 and 8) that, at timestep t, the former parameter's value will determine scaling factor Ot,,t, which controls the latter parameter's s slope 8;,6- In this way, the former parameter configuration will have a strong influence on the configuration of the latter parameter. Moreover, avoid having this dependency affect the configuration of the former parameter itself, use the formula min( (t, & Pi.A) - an,z [1 sig(t, et,, Ps,h) - an,a)) to measure the approximation, which returns the same value when two an symmetrical about 0.5.
The pseudo-code of the Logg-Sigmii--Mee -ask benchmark is provided in Algo- rithm 3 The main modification compared to the Seq-Sigmoid benchmark is that mask s and Pi. (I.c., the state, E (t} in line 4) and set 8;,4 = 1 (i.e., line to avoid too much randomness.
Seq-
Algorithm 3 Benchmark Outline: Seq-Sigmoid-Mann
Benchmark Parameters: number H of hyperparameters, number C₁2 of choice for each hyper-
parameter h, episode length T;
2: p; //, H);
3: for E 1.. T) do
4: The state step state, = {t);
5: Select actions: Ones
C--1 for all 0 h H and SIST,
Ch Ch ..... Cs
10 if 11 2 0.5
6:
00. = 1.,:1,,,m 1h, =
0.1 if a < 0.5 0 < < H;
7: r + I (1 min([sig(t, an, 1,ps,)) .n.e sig(t, an, an,
1, - an,))))
8:
end for
Seq-Sigmoid-Robust (n). The pseudo-code of the Sog-Sigmoid-Robute (n) benchmark is provided in Algorithm 4 The benchmark gives random configurations for the parameters, matter how the corresponding agents tune them in Seq-Sigmoid (i.e., in line 7), with other parameters configured properly, simulate the ineffective learning of some agents in complex dynamic algorithm configuration scenarios. In particular, we give random configurations for the 11/22j-th parameter in Scq-Sigmmid-Rolee Robusst (1) and the [n/2] -th and the [n/2] 1-th parameters Seq-Sigmoid Sigg-Simmid-Robuu Robusst (2).
B.2 MOEA/D
MOEA/D is the proposed DAC benchmark 44 based on real-scenario multi-objective evolutionary algorithm MOEA/D (88) with heterogeneous parameters, which is a strong stochastic benchmark due to the randomness of the evolution optimization process.
MOEA/D algorithm The multi-objective evolutionary algorithm MOEA/D is originally designed to solve the Multi-objective Optimization Problems (MOPs), which can be defined as
min F(x) =
(x)) s.t. x E 2
where x (Ii,..., pD) solution, Ω - m constitutes m objective functions, Ω lo UID C RD0 is the solution space, and 1 is the objective space. MOP aims find set of
23

---

Algorithm 4 Benchmark Outline: Sog--SigmiddRRbbuu (n)
1: Benchmark Parameters: number H of hyperparameters, number C₁, of choice for each hyper-
parameter h, episode length T, n random hyperparameter indexes h1, .u.....,hu.
2: 8; (0(((((00)) 100, 100, H);
3: P; ~ ( H);
4: for L E [0,1,...,T]] do
5: The state at step 1: state, 8; p; (4:
6: Actions: on,s E
0
Cx-1 for all S H and S tt ST;
Ch
Ck
Ca
7:
Reselect ar,, i
[h1,
h2,
hn] random;
10
if a 2 0.5
8:
0,, = 0h, 0.1 if a < 0.5
, < H:
9: ri - I (1 min([sig((, (h, Pi..) an,e sig(t, 18, an,;)))
H-1
10: end for
solutions that represent the best possible trade-offs between competing objectives. The objective vectors of these solutions are collectively called the Pareto front (PF), defined as follows:
Definition 3. solution x" is Pareto-optimal with respect to Eq. (.2)) if Ba E Q such that Vi : fi(z) S fi(x") and 3i fi(x)) < f((x"). The set of all Pareto- oprimal solutions is called Pareto-optimal Set (PS). The set of the corresponding objective vectors of PS, i.e. (F(x) x E PS), is called Pareto Front (PF).
MOEA/D consists of two main processes, decomposition and collaboration 48 41 20
In decompo-
sition, MOEA/D solves a number of sub-problems through a number of weights and an aggregation function approximate the PF. Several aggregation functions have been proposed for MOEA/D. Here, we introduce the common Tchebycheff approach (TCH) used our paper as follows:
Given weight vector (wi...., Wm) where wy 0, {1....,m m} and E wy 1. the sub-problem by TCH is formulated as
min #ef
g( unn 2²) max {w, [f((z)) - 2 15
where =² (=ī, (2 2m) is the ideal point consisting of the best objective values obtained far.
The fundamental intuition behind collaboration is that adjacent sub-problems tend to share similar properties. For instance, they may have similar objective functions and/or optimal solutions (20). In particular, the neighborhood of a sub-problem is controlled by the Euclidean distance of its corresponding weight vector with respect the others, well as the hyperparameter neighborhood size: two sub-problems are neighbors of each other if the distance between them is smaller than the neighborhood size. In the selection process for sub- boobbrrobem,, parent solutions randomly selected from the corresponding neighborhood of each sub-problem. The sub- babb-roblem solutions within the same neighborhood then replaced with the newly generated offspring solution if it is better than the current one.
The pseudo-code of MOEA/D is shown in Algorithm 5
Action The action space MOEA/D four dimensions, corresponding to the four heterogeneous MOEA/D parameters:
1. Weights. In MOEA/D, weights used to transform MOP into multiple singlc-objective sub-problems. Inspired by MOEA/D-AWA 22), we set the action space for weights by adjusting (T) not adjusting (N) the weights. If the action is T, the weights will be updated; otherwise, the weights will remain the same. The weights adaptation mechanism is as follows.
The sparsity level of each solution (i) is calculated based on vicinity distance [1]
SL ( (x pp
III [( .)). 3=1
where ((x(0) is the Euclidean distance between (1) and its j-th nearest neighbor in the population { pp= In the calculation, the m closest neighbors in the population are used, where denotes

---

Algorithm 5 MOEA/D
Input: Population size N. number T of iterations
Output: A set of Pareto-optimal solutions
I: Initialize population (a (1) L.V. of solutions, and a corresponding set W ( of weight vectors
2: 1-0
3: while t
T do
4: for = 1: N do
Randomly select parent solutions from the neighborhood of w denoted as Ow(())
5:
6:
Use crossover and mutation operators to generate offspring solution x Evaluate the offspring solution to obtain Fxx"()) Update the ideal point z": for j E (1,2.-. .... m do
7:
8:
9:
10:
if (a((()) < 2j then
2 (z(ll)
12:
end if end for
Update the corresponding solution of each sub sub-problem within Ow()) by x(();
14:
for w Uw(()) do
16:
if g(wll)) (w(,)x* x*) < g(x(() ww() x²) then
17:
a - (()
18:
end if
19:
end for end for
21:
t t + 1
22:
end while
the number of objectives. Sub-problems corresponding to the solutions with sparsity levels in the bottom 5%, namely the overcrowded solutions, are subsequently removed.
To ensure that the total number of the sub-problems is still N, 0.05N new sub- sub-problems and their corresponding solutions should be added, which come from an elite population that keeps all historical non-dominated solutions, with maximum capacity of 1.5N. When the size of the elite population surpasses this capacity, the solutions with the lowest sparsity level are eliminated. For every solution x' in the elite population, its sparsity level is calculated with respect to the current population (i.c., SL Pop), where Pop represents the set of the remaining 0.95.V solutions). Subsequently, the solution with the highest sparsity level with respect to the current population is selected from the elite population and added to the current population. This selection and addition procedure carried out for 0.05.N times. For each newly added solution, the corresponding sub- sub-problem (i.e., weight vector) generated in a specific way as Algorithm
29
2. Neighborhood size. The neighborhood size is used to control the maximum distance between solutions and their neighbors. A small size aims exploit the local area, while large size aims to explore a wide objective space [43]. The action space is of four dimensions, that is, 15, 20, 25 and 30, where 20 is the default value.
Types of reproduction operators. We consider four types of DE operators with different search abilities introduced in (21). For reproducing an offspring solution for the i-th sub brobler, let (+) and (+)) denote its current solution and the generated offspring solution, respectively, and the equations for four types of DE operators are shown as follows:
OP1: (1)
x
+ F (2(11) xx(r)))
OP2: (1) x! (1)
+ F (2(11) (r(a)) + F (z(13) x((4)),
OP3: (1) x (1)
+ K x (i) -x(ni)) + F x (2())) x(r))) F x (x((x) x(1)))
OP4: z(()) x + K x (2(0) -((r)) + F x (2 x1a)))
25

---

Table 2: State at step in MOEA/D.
Index
Parts state
Feature
Notes
1/m
Number of objectives
1/D
D: Number of variables
2
T/T
Computational budget that has been used Stagnant count ratio
3
2
Nasg/T
3
HV, NDRatio, Dista
Hypervolume value
Ratio non-dominated solutions
3
Average distance
3
HV. HV,-1
Change of HV between steps I and
8
3
NDRatio: NDRatior-1
Change of NDRatio between steps / and Change of Dist between steps and t
3
Dist, Dist,-1
10
3
Mean(List(HVV..
Mean of HV in the last 5 steps
11
Mean(List (NDRRati,, 5))
Mean of NDRatio in the last 5 steps
12
Mean([.ist(Distt, (.))
Mean of Dist the last steps
13
3
Std(List(HV,,,
Standard deviation of HV in the last steps Standard deviation of NDRatio in the last steps
14
3
Std(List(NDRatto, t, ())
15
3
Sid(LisstDDist, 5))
Standard deviation of Dist in the last 5 steps
16
3
Mean(Tist(HVV.
Mcan of HV in all the steps so far Mean of NDRatio all the steps so far
17
Mean([ist(NDRRatio, f))
Mean(List(Dist,, (,))
Mean of Dist all the steps so far
19
3
Std(List(HV, ()))
Standard deviation of HV in all the steps so far Standard deviation of NDRatio in all the steps fär
20
3
Std(List(NDRatio,, ())
21
3
Std(List((Dsst
Standard deviation of Dist in all the steps so far
Here, ((11), (1(2), ((a)), ((44), and x (rs) are different parent solutions randomly selected from the neighborhood (6) The scaling factor F > controls the impact of the vector differences on the mutant vector, and K E [0, 1] functions similarly F.
Parameters of reproduction operators. The parameters (particularly the scaling factor) of the reproduction operators in MOEA/D have a significant effect on the algorithm's performance (38). We the scaling factor K to fixed value of 0.5 as recommended (2I) and dynamically adjust the scaling factor F with four discrete dimensions, i.e., 0.4, 0.5, 0.6 and 0.7, with 0.5 serving as the default value.
The state of MOEA/D includes three parts: 1. The feature of the problem instance (c.g., the numbers of objectives and variables); 2. The feature of the optimization process (c.g., the budget has been used); 3. Information about the evolution situation of the population (e.g., the hypervolume value and the average distance of the current population). The detailed state feature is demonstrated in Table
State
Transition One step of transition takes place one generation change in the evolutionary process of MOEA/D.
Reward A positive reward of the MOEA/D environment assigned to the agent team when better solution is found compared to the current best solution. Since optimization becomes harder when the current best solution is more optimal with time, we should assign more reward to better solutions found in the latter stage. We use the triangle- based reward function proposed in 44 defined as follows, which has been proved effective the MOEA/D environment.
At step 1
(1/2) (p - pf) if f( < 0
otherwise
where
f(xa) f(x(41) [(80]
f((41) fi
P1+1 =
Pt
otherwise
26

---

and ft is the minimum metric value found until step t.
Problem instance The problem instances of MOEA/D include the well-known multi-objective optimization problems (MOP) benchmarks DTLZ g and WFG [3] with variable numbers of objectives and problem dimensions, which cover different difficulty levels of MOPs.
B.3
Hyperparameters of the algorithms compared in the experiments
For all the compared MARL algorithms, we use their default suggested hyperparameter settings in EPYMARI (i.e., VDN (3)] QMIX 31. MAPPO [46)) their official implementation (i.e., HASAC (22).. For a fair comparison, we use the same hyperparameters for algorithms of the same type (i.e., use the same hyperparameters for the value-basee methods: SADN, ACE [1)] SAQL (3), VDN [39] and QMIX [3]], as well as the same hyperparameters for the policy gradient-based PPO methods: MAPPO 46 and HAPPO (17)). The detailed hyperparameters in the experiments are given in Table 3
For a fair comparison, we use one single 64-dimensional hidden layer for all the value networks and the policy networks.
Table 3:
The hyperparameters of the compared algorithms in the MOEA/D environment, and means the certain algorithm does not have that hyperparameter.
Hyperparameter
SADN ACE SAQL
VDN
QMIX
MAPPO
HAPPO HASAC
Hidden layer size
64
64
64
64
64
64
64
Learning rate
le-4
lc-4
lc-4
lc-4
lc-4
3e-4
3e-4
5e-4
Batch size
32
32
32
32
32
10
10
10
Discount
0.99
0.99
0.99
0.99
0.99
0.99
0.99
0.99
Target update interval
200
200
200
200
200
200
200
50
Number of steps to look ahead
1
5
5
20
Entropy coef
0.01
0.01
Grad norm clip
10
10
10
10
10
10
10
More detailed information can be found in our code available
htttp://gitthut com/
lamda- blla-bbbbbbee--aaa
C Additional results
C.1 Results on the original Sigmoid benchmark
We compare our method with the famous decomposition value-based method VDN [39 as well the widely used based single-agent RL algorithm DQN 28 on the original Sigmoid benchmark. As shown in Figure 4 our method can achieve competitive and even better results compared to the effective decomposition-basee VDN when there are no strong nnter-dependenciss among the parameters. Moreover, the single-agent agent RL algorithm DQN fails to learn effectively because of the combinatorial explosion the action space on the 5D Sigmoid, and even fails to train on the 1OD Sigmoid, which is also observed in 44 emphasizing the importance of the multi-agent modeling.
C.2 Comparison of the correct order and the reserve order on Seq-Sigmoid and Seq-Sigmoid-Wask
The comparison of the sequential methods based on the correct order and the reverse order is shown in Figure Basically, the reverse order produces lower performance and larger variance than the correct order due to the wrong capture of the inherent dependeepesssc between the parameters, which misleads the learning of the whole team. There is an exception, HAPPO, which only considers the update order of the agents rather than the order of taking actions. HAPPO avoids suffering from
hattps://gite http:///gtihuu..oooooooooooee/e/ hwg.thhwb.oouuuuo-ggggaa//aaaaaa
27

---

5 Sigmoid
10 Sigmoid
Return
4-
1.5
1.0
0.5
0.0
0.0
0.2
0.4
08
0.8
1.0
0.0
0.2
0.4
0.8
0.8
1.0
Episodes
165
Episodes
1e5
SADN
VDN
DQN
Figure 4: Training curves of the return value obtained by the compared methods on the original Sigmoid benchmark.
the misleading of the reverse order, but fails to exploit the sequential information of the correct order well. The comparison of using the correct and the reverse order provides evidence for the effectiveness of modeling the inherent inter-dependencies correctly, which also leads to more efficient training.
C.3
Comparison of the correct order and the reserve order on MOEA/D
The results in Table 4 demonstrate the effectiveness of correctly considering the parameter order on the real-scenario multi-objective evolutionary algorithm. Generally, the correct order produces better results than the reverse order within the same sequential methods, which evidently shows that the sequential information of the correct order can boost better performance, and the improvement of the performance comes from the correct order, rather than simply making the actions selected by the previous agents available to the subsequent agents. There are two exceptions: for MAPPO and SAQL, the rank using the reverse order is better than that using the correct order. This is because these two methods cannot efficiently make full use of the action sequence information, causing the performance of the correct order occasionally be far from optimal, and sometimes even worse than the reverse order. For MAPPO, it not originally designed to capture the sequence information, and we basically add the previous agent's 's action the next agent's 's state, according to [T For SAQL, as an extension to IQL [40], lacks reward allocation and fails to reveal the interactions among agents, which leads weak ability to make use of the action sequence information.
C.4 Discussion generalization ability with different dimensions
We test the cross-dimension results of SADN on the MOEA/D benchmark as shown in Table 5 Overall, the results demonstrate that our proposed method exhibits good generalization ability across different dimensions. However, we acknowledge that cross cooss-dimensional generalization remains a significant challenge, and will further investigate and improve its generalization capability in the future, such as through multi-dimensional mixed training and employing multi-head models.
C.5 Runtime analysis of different methods
We test the training time of the compared methods the 10D Seq- Reg-Siimoid benchmark for 1,050,000 steps and the MOEA/D benchmark for 405,000 steps. The results are shown Table 6
Due to the independent and simultaneous update of the individual advantage nets, SADN demonstrates good efficiency terms of training and inference time, noting that we test the learned policies at fixed intervals during the training process. In summary, to achieve the best average rank, the running time
28

---

5 Seo Sigmold
Retum
Sigmoid
Retum
D Sissood-Maskk
Deturn
10 San-Signoidd-Man
Return
Episodes
Episodes
Episodes
Episodes
Episodes
SADN
ACE
SAQL
MAPPOpr
HAPPO
SADN-reverse_com order
ACE-reverse order
SAQL
MAPPOar-reversee prder
HAPPO-reverse, arder
Figure 5: Training curves of return value obtained by correct order and reverse order on four Seq- Sigmoid variant tasks, where the results averaged over runs.
of our SADN is only slower than two value decomposition- methods, i.e., VDN and QMIX, while faster than SAQL, ACE, well the policy gradient-based MAPPO, HAPPO, and HASAC.
29

---

Table 4: IGD values of the compared methods with the correct order and reverse order (denoted as -T) on different problems. Each result consists the mean problem is highlighted in bold. The symbols '+', '-' and "x' indicate that the result is significantly superior to, inferior to, and almost equivalent to our method SADN, respectively, according to the Wilcoxon rank-sum test with significance level 0.05. The top three problems are the problems for training, and the bottom five problems are for testing.
runs. The best mean value on each
Problem
Dim
MORAD
SADN
SADN-r
ACE 19
ACE.
SAQI
SAQL
MAPPOar
MAPPOar-r
4.593e-02
3.809--02
3.819-022
3.851e-02
4.615e-02
3.950e022
4.061e-02
3.804e
3.786e-02
DTLZ2
02598e402
3.875e412
4.158c-02
4.057e-02
4.619e-02
4.337e-02
4.354e402
4.308e-02
4.167e-02
12
4.599e-02
3.874e-02
4.176e-02
4.063e-02
4.634e-02
4.698--02
4.734e-02
3.886e-02
4.284c-02
4.537e-02
6.282e-02
4.747e.02
WFG4
9
5,736e 02
5,190e-02
6.026e-02
5.853e-02
6.725e-02
5.365e-02
5,375e-02
5.471e-02
5,356e-02
5.751c-02
5.751e-02
6.9836-02
5.351e402
6
5.855e-02
3,908e-02
4.027e-02
3,962c-02
5.81
3.964c-02
3,993c402
3,944e-02
3.921c-02
WFG6
6.641e-02
4.884e-02
7.757e-02
5.111e-02
5.992e-02
4.384e-02
4.432e-02
6.093e-02
5,949e-02
12
6.864e-02
4.837e-02
7.674e-02
5.490e-02
7.908-022
5.676e-02
5.755e-02
5.786e-02
6.710c-02
Training: +/-fos
0/9/0
0//1
0//11
0//0
1/8/0
1/8/0
0/7/2
1/800
5.455e 02
3.895e412
3,980e
4.221e-02
5.598e 02
5.019e-02
4.856e
3.932e-02
4.008e 02
DTLZ2
9
229c-02
7.319c-02
5,3666-02
4.492e-02
5.765e-02
6.89 6.895e-02
5.833c-02
6,252o-02
1.847--22
7.852e-02
4.650e-02
6.667e-02
8.589e-02
8.688e-02
6
6.307e-02 02
4.749e-02
4.748e002
4.752e-02
6.1455-02
5.3 5.390e
5.539e 02
4.713e-022
4.719e-02
WIGS
4.773x412
4.778c-02
4.78%-002
4.787e-02
4.798c-02
12
6.381e-02
4.793e-02
4.759e-02
4.791c-02
1696--22
4.954e002
4.983e-02
4.751e402
4.980e-22
5.803c-02
3.893--02
3,994c-02
5.819-402
3,958c-02
3.980c402
3.922--02
WFG7
5,812e-02
4.053e402
4.954c-22
4.470le02
5.878e-02
4.257e-02
4.235e-02
4.285e-02
4.154c-02
12
5.814e-02
4,060e-02
4,950-02
4,447c-02
5.936e-02
4.975c-02
4.986c-02
4.115c-02
4.437e-02
7,875e-02
1.018-001
L077c-01
1.029e001
1.177-01
1.054c-01
1.031c-01
1005c-01
1.066-001
WFG8
9
9.680e-02
7,853-022
9.191c-02
9,787c-02
1.053-001
9.716e-02
9.299c-02
7:9476-02
7.911c-02
9.330e-02
6
5.600c-02
3,987e-02
4.053e-02
4.012c-02
5-495-002
4.010c-02
4,219c-02
3,994c-02
3.975e-02
WPG9
5. 7.748c-02
4,341c-02
5.831c-02
4,552c-02
5,562-022
4.281c-02
4.168c-02
4.677c-02
4,446c-02
12
5.827e-02
4.249c-02
5.839e-02
4.477e-02
7.265e-02
7.291e-02
6.463e-02
5.299e-02
4.986e-02
Testing: 1/
1/12/2
3/1/2
2/9/4
1/13/1
M/1
2/11/2
3///
4/9/2
2
5.625
8.625
5.75
6.25
2.25
everage rank
7.375
2.5
5.5
4.75
7.5
4.375
3.625
5:625
3.75
30

---

Table 5:
test of SADN the MOEA/D benchmark. SADN-n refers to the model trained exclusively on problems with n dimensions.
dim
MOEA/D
SADN-6
SADN-9
SADN-12
6
4.593
3,809
3.854
3.810
DTLZ2
9
4.598
3.928
3.875
3.840
12
4.599
3,936
3,882
3,874
6
5.729
4.537
4.543
4.521
WFG4
9
5,736
4.973
5.190
4.922
12
5.751
5.260
5.322
5.213
6
5.855
3,908
3.919
3,913
WFG6
9
6.641
5.044
4.884
5.010
12
6.864
5.036
5.138
4.837
6
5,455
3,895
3,970
3.968
DTLZ4
9
6.229
5.700
7.319
5.587
12
6.790
7.593
8.176
7,852
6
6.303
4.749
4.764
4.759
WFG5
9
6.351
4.814
4.773
4.788
12
6.381
4.824
4,823
4.793
6
5,803
3.893
3.886
3,893
WFG7
9
5,812
3,980
4.053
3.990
12
5.814
4.176
4.159
4.060
6
7.875
10.18
10.56
10.67
WFG8
9
9.680
9.020
7.853
8.948
12
8.710
8.013
8.191
7.891
6
5.600
3.987
3.997
3.989
WFG9
9
5.748
4.387
4.341
4.370
12
5.872
4.274
4.423
4.249
average rank
3.708
2.083
2.5
1.708
Table 6: The training time the compared methods on the 10D Seq-Sigmoid benchmark for 1,050,000 steps and the MOEA/D benchmark for 405,000 steps
SADN
ACE
SAQL
VDN
QMIX
MAPPO
HАPPO
HASAC
10DSeq-Sigmoie MOEA/D
4h28'
5h01
6h34'
3h23'
3h29'
12h32'
16h39* 20h02'
21h17'
14h56'
16h20'
19h46'
14h48*
15h23'
17h44'
23h52'
31


Reviews:
[{"review_id": "2QEpVdnPXk", "reviewer": "Reviewer_HrJN", "summary": "This paper considers the problem of dynamically configuring multiple hyperparameters of an algorithm, where the hyperparameters have a sequential order to tune. The authors proposed a contextual sequential MMDP formulation and the corresponding method, the sequential advantage decomposition network. Extensive experiments on some artificial benchmarks and the real-world task of tuning hyperparameters of the popular multi-objective evolutionary algorithm MOEA/D show the effectiveness of the proposed method.", "questions": "- Why can the proposed SADN avoid the long sequence update issue of ACE?\n- How to determine the right order of hyperparameters in practice?\n- For Seq-Sigmoid, it’s not clear to me why the change of the formulation (as shown in line 249) can avoid the influence on the former hyperparameter.\n- line 254, What do you mean by $\\mathcal{N}(T/2,T/4,H)$?\n- line 255, clearly mention $n < H$?\n- Though introduced in the appendix, I think the sequential order of the four hyperparameters of MOEA/D should be introduced in the main paper, which can ease the understanding of the readers.\n- Figures 2 & 3, are three runs sufficient?\n- Table 1, are training and testing on different benchmarks with the same number of objectives? If so, how about the performance with a different number of objectives, e.g., training on 6 objectives while testing on 9 objectives? Such a test can further examine the generalization ability.\n- I can understand the IGD metric used in the experiments of MOEA/D, but I suggest that a brief introduction is necessary, as the readers may not be familiar with multi-objective optimization. \n- line 890-891, how to calculate the sparsity of each solution when the size of the elite population exceeds the capacity?\n- line 937-941, you used the default hyperparameter settings for VDN, QMNIX, and MAPPO. Then, how to make them the same for fair comparison?\n- Table 4, for SAQL and MAPPO with 9 objectives, the rank using the reverse order is better than that using the correct order. This seems to be strange. Can you give some explanation?", "limitations": "Yes, the authors mentioned the difficulty of determining the right order of adjusting the hyperparameters in the conclusion part.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n1. The studied problem of dynamic algorithm configuration, where multiple hyperparameters have a sequential order that widely exists in practice, which is challenging but has been explored very little. Thus, the topic of this paper is significant. \n2. The paper proposes an effective method: Contextual sequential MMDP formulation and sequential advantage decomposition, where the action-order information can be utilized.\n3. The theoretical analysis of the IGM property of the proposed sequential advantage decomposition is interesting. The proof is accomplished by a standard inductive hypothesis. I have checked the proofs and believe they are correct.\n4. The experiments are generally good. The authors compared their proposed method with a wide range of methods, including existing sequential RL and advanced multi-agent RL methods. The empirical results on the artificial benchmark Seq-Sigmoid and its variants show the usefulness of considering the sequential order of adjusting hyperparameters and the robustness of the proposed method against the failure of adjusting some hyper-parameters. The results on tuning the four hyperparameters of the popular algorithm MOEA/D show the value of the proposed method in practice.\n\nWeaknesses\n1. Some experiment settings (e.g., the method of calculating the sparsity) and mathematical symbols (e.g, $\\mathcal{N}(T/2,T/4,H)$) need to be further clarified.\n2. Some experimental results need to be further analyzed, e.g., the rank using the reverse order is better than that using the correct order for SAQL and MAPPO with 9 objectives in Table 4.\n3. The generalization ability of MOEA/D can be enhanced by training and testing on different numbers of objectives. \n4. No guidance to determine the correct adjustment order of the hyperparameters of an algorithm.", "was_revised": true, "final_justification": "Thanks for the detailed response. My questions have been addressed well. I am happy to increase my score by 1.", "rebuttal": "Thanks for your valuable and encouraging comments! Below please find our response. \n\n## Q1 Why can SADN avoid the long sequence issue\n\nThanks for your comments. We apologize for not making this clear. The advantage decomposition in our SADN allows each individual advantage net to be updated independently and simultaneously, reducing their mutual influence and compounding errors. In contrast, ACE sequentially updates the Q nets based on the output of the previous agent (i.e., more centralized), which leads to compounding errors along the update chain, especially worsening in the long sequence updates. This phenomenon is also observed as \"error amplification\" [1-2], which often happens due to inter-agent interference [3]. One way to alleviate this phenomenon in the MARL community is to decompose the team value function into agent-wise value functions so that the individual agent can learn relatively independently [4]. As demonstrated in [4], the centralized approach consistently fails on relatively simple cooperative problems in practice due to the mutual influence of policy learning and the lack of reward allocation. Furthermore, [5] shows that the more centralized methods tend to suffer from scalability. To address this issue, our SADN decomposes the global advantage function sequentially, where the individual advantages are learned relatively independently, and the action sequence information is also captured. Experimental results in Figures 2-3 show that SADN significantly outperforms ACE. We will revise the paper to provide more analyses and clarify this point. Thank you very much!\n\n[1] Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning. NeurIPS, 2024.\n\n[2] An Outlook on the Opportunities and Challenges of Multi-Agent AI Systems. arXiv, 2025.\n\n[3] Model-Based Diagnosis of Multi-Agent Systems: A Survey. AAAI, 2022.\n\n[4] Value-Decomposition Networks for Cooperative Multi-Agent Learning. AAMAS, 2018.\n\n[5] QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement learning. ICML, 2019.\n\n\n## Q2 How to determine the right order of hyperparameters in practice?\n\nThank you for the insightful and constructive question! In practice, there is an intuitive and natural way to determine the correct order: setting it according to the sequence in which the hyperparameters take effect during the execution of the algorithm. Specifically, in the code implementation of the target algorithm, all hyperparameters take effect in a specific order, which is the order we have set. For example, in MOEA/D, after initializing the weights, we first determine the neighborhood subproblems (agent 1: neighborhood size), then optimize the subproblems using certain operators (agent 2: types of reproduction operators and agent 3: parameters of reproduction operators), and finally decide whether to adjust the weights (agent 4: weights) based on the results of this generation. \n\nThis strategy aligns with our intuition and performs well in the experiments. However, we acknowledge that although it can be a good choice, this order may not always be the optimal one for training efficiency and effectiveness, as we stated in the conclusion. Determining the best order for the dynamic configuration of hyperparameters in general algorithms can be challenging, and we leave it as our future work. This may involve applying large language models and leveraging their knowledge to set the proper order, as well as using causal models to automatically learn the hyperparameters' causal structure from the data. We will revise to make this part clear. Thank you.\n\n## Q3 Why can the symmetrical formulation in the Seq-Sigmoid avoid the influence on the former hyperparameter?\n\nThank you for your valuable comment. In our proposed Seq-Sigmoid, the previous agent determines the scaling factor of the next agent by choosing an action greater or less than 0.5. To eliminate the impact caused by different choices of scaling factors, we design the reward value to be symmetric around 0.5. This symmetry ensures that, regardless of the scaling factor selected, there is always an optimal reward value for the current agent. As a result, the agent does not need to sacrifice its own optimal reward to accommodate the optimization of subsequent agents—for instance, it no longer has to choose a suboptimal value in order to adjust the scaling factor. To clarify this further, we will add specific examples and reward landscape figures in the revised paper. Thank you again!\n\n## Q4 Explanation for $\\mathcal{N}(T/2, T/4, H)$ and the suggestion to clearly mention $n<H$.\nThank you for the question and suggestion. $\\mathcal{N}(T/2, T/4, H)$ means we have $H$ parameters which all follow a Gaussian distribution with a mean of $T/2$ and a variance of $T/4$. In our setting, we always have $n<H$. We will revise to make it clear. Thank you.\n\n## Q5 Concerns about the sufficiency of the three runs in Figure 2 and 3.\n\nThank you for the valuable question. We believe that three runs are sufficient to support our conclusion for the following reasons:\n\n- In our experimental setup, we resample the instance in each episode, meaning that every point on the training curve reflects the generalization ability of the learned policy at that specific training phase. We believe this setting significantly reduces the impact of randomness across different runs.\n\n- As shown in Figures 2 and 3, the variance of our method is very small, as is that of most well-performing methods, and our method clearly exhibits better performance and robustness on more challenging tasks. \n\nTo further address your concern, we have conducted additional runs. However, we could not include the detailed results due to space limitation. The relative performance of the algorithms and the experimental conclusions remain unchanged. We will include these results into our revised paper. Thank you.\n\n\n## Q6 Further examining the generalization ability.\n\nThank you for the insightful question! Yes, we maintain the dimensions consistent in one series of experiments' training and testing. We agree with you that testing with different dimensions can further examine the generalization ability. The results are shown below:\n\n| | MOEA/D | SADN-6 | SADN-9 |SADN-12|SADN|\n|---|---|---|---|---|---|  \n|average rank|3.708|2.083|2.5|1.708|1.542|\n\nDue to space limitations, we only report the average rank of several key methods. SADN-n refers to the model trained exclusively on problems with n dimensions. Overall, the results demonstrate that our proposed method exhibits good generalization ability across different dimensions. However, we acknowledge that cross-dimensional generalization remains a significant challenge, and we will further investigate and improve its generalization capability in the future, such as through multi-dimensional mixed training and employing multi-head models. We will incorporate these discussions into our paper. Thank you very much!\n\n\n## Q7 Introduce the IGD metric and the parameter order of MOEA/D in the main paper; other revision suggestions.\n\nThank you for the valuable suggestions. We will revise the paper carefully according to your suggestions to include these contents. \n\n## Q8 How to calculate the sparsity of each solution when the size of the elite population exceeds the capacity?\n\nThank you for the question. As we mentioned in line 892 and 893, every solution in the elite population has its sparsity level calculated with respect to the current population, and when a new solution is about to be added to the elite population, the solution with the lowest sparsity level in the elite population will be removed immediately and the size of the elite population remains the same. Therefore, we will not calculate the sparsity of each solution when the size of the elite population exceeds the capacity.\n\n## Q9 Concerns about fair comparisons between VDN, QMIX, and MAPPO with respect to the default hyperparameter settings.\n\nThank you for your question. For a fair comparison, we try our best to maintain hyperparameter consistency among the algorithms of the same type. For example, for the value-based methods like SADN, ACE, SAQL, VDN, and QMIX, we use the same hidden layer size, learning rate, batch size, etc. The detailed settings can be found in Table 3 of Appendix B.3. Since different types of algorithms have different components (e.g., value-based methods like VDN vs. policy gradient-based methods like MAPPO), we follow the suggested hyperparameter settings in EPyMARL, which is one of the most popular open-source frameworks for MARL. Therefore, we believe that our hyperparameter settings are fair.\n\n  \n## Q10 Explain why for SAQL and MAPPO, the rank using the reverse order is better than that using the correct order?\n\nThank you for your insightful question. This is because these two methods cannot efficiently make full use of the action sequence information, causing the performance of the correct order to occasionally be far from optimal, and sometimes even worse than the reverse order.\n\n- For MAPPO, it is not originally designed to capture the sequence information, and we basically add the previous agent's action to the next agent's state, according to [4]. \n\n- For SAQL, as an extension to IQL [5], it lacks reward allocation and fails to reveal the interactions among the agents, which leads to a weak ability to make use of the action sequence information.  \n\n[4] Revisiting some common practices in cooperative multi-agent reinforcement learning. ICML, 2022.\n\n[5] Multiagent cooperation and competition with deep reinforcement learning. PLoS ONE.\n\n## Q11 Paper formatting concerns and typos\n\nThank you very much for pointing these out! We will carefully revise them according to your comments.\n\n---\n\n**We hope that our response has addressed your concerns, but if we missed anything, please let us know.**"}, {"review_id": "ljLdfOj7p6", "reviewer": "Reviewer_FPTa", "summary": "This paper introduces an improved version of multi-agent DAC framework, where the authors make in-depth exploration on the sequential relationship among the hyper-parameters within a relatively complex algorithm configuration space. To this end, the authors decompose the per-step parameter control as sequential steps, where each step is made by a separate RL agent considering both the state information and the decisions of previous agents. The authors give a basic proof of effectiveness of such decomposition modelling and apply this scheme to multi-agent RL framework, which is proposed by the authors and termed as SDAN (generally a VDN structure). The effectiveness of SDAN is validated on multiple DAC tasks including a complex one: configuring MOEA/D, a state-of-the-are multi-objective optimization algorithm. The results demonstrate that SDAN can be generally utilized for different DAC tasks, its learning effectiveness is compared with many MARL baselines.", "questions": "See Strengths And Weaknesses.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. This paper provides a new perspective to rethink the MDP modelling for automated algorithm design, especially, a sub-task: algorithm configuration. Given the learning for algorithm design gets popular in recent ML researches, the significance of this paper is for sure.\n2. The motivation of this paper is clear and reasonable, since for complex algorithm design task, the submodule in the algorithms, together with the related hyper-parameter values, are somewhat correlated with with each other, hence modelling them with inter-relationship is very crucial to learn some effective configuration policies.\n3. The experiment is solid. I especially appreciate the authors to compare with multiple MARL baselines.\n\nWeaknesses:\n\n1. The writing and presentation of this paper has a large room to improve. For example, there are too many symbols and equations and lengthy elaboration of MARL, including both the related works and methodology, which can be easily organized into appendix and put more important content into the main body to enhance the motivation, the core design and the philosophy of this paper. Besides, the figures, especially the figure 1, is in poor quality (instance set part). Re-draw this figure to make it more clear the correspondence between the MDP and the algorithm solving instance.\n2. For the background (related works), I suggest the authors to enrich the Section 2.1, to include more recent works. For example, a very important and similar work, QMamba (https://openreview.net/forum?id=7qCkf5OT3x), which also decomposes the configuration space to address the inter-relationship of operators and parameter values. This paper should be at least cited and discussed. For Section 2.2, I suggest move some MARL basic knowledge to appendix, shorten this section and leave the most important part, that is, what is MARL and the limitation of naive MARL on complex DAC tasks. Besides, there are many sequential MDP approaches while this paper reviews few of them (such as Qtransformer , SDQN, etc).  \n3. I think the most confusing part in the decomposition scheme is the order of the actions. How to ensure the proposed SADN is robust to the different orders of hyper-parameters within the same algorithm? I can not find any special design to address this, only an experiment in appendix to show the effectiveness of correct and reverse order. Besides, the second confusing part is the scalability of SADN, since different algorithms may hold various number of hyper-parameters, I guess SADN can not be trained simultaneously on diverse algorithm structures and zero-shot to unseen algorithms. This is somehow inefficient for realistic world. A very promising way to resolve this, I think, is based on your references [10]. this work configx provides a modular algorithm space, which you can use multi-agent system to learn and control.\n4. Please include error bars into Table 1.\n5. As the learn to optimize domain develops, a more important thing is to analysis why a learning-based approach generalize well on some cases and not well on the others. For example, in Table 1, I observe that SADN underperforms baselines in problem set WFG5, why is that? Providing an analysis on this would significantly help the future works, even though it can not be addressed now. \n6. The citation format is wrong, correct it please.", "was_revised": true, "final_justification": "This paper presents an interesting idea supported by promising experimental results. The authors have also effectively addressed my initial concerns in the rebuttal. I recommend acceptance.", "rebuttal": "Thanks for your valuable and encouraging comments! Below please find our response. \n\n## Q1 and Q2 Suggestions for the writing and presentation improvement. \n\nThank you very much for your valuable suggestions. We will carefully revise the paper according to your suggestions. \n- We will move the background information of MARL to the appendix and put core theorem proofs into the main body of the paper. \n- We will also re-draw the figures you mentioned to make a clearer demonstration of the detailed information of our method (e.g., the MDP process and the problem instance). \n- We will enrich our related works with more recent and constructive works such as Q-Mamba [1] and Q-Transformer [2]. In the future works, we expect to incorporate more advanced architecture like these works into our algorithm, which may boost a stronger ability to capture action sequence information.\n\n[1] Meta-Black-Box-Optimization through Offline Q-function Learning. ICML, 2025.\n\n[2] Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions. CoRL, 2023.\n\n## Q3 How to ensure the proposed SADN is robust to the different orders of hyper-parameters within the same algorithm? Setting of the order?\n\nThank you for your insightful comment! In our setting, the number and order of hyperparameters must be determined before training and remain fixed during the algorithm's execution. Therefore, the setting you mentioned regarding different orders of hyperparameters within the same algorithm is beyond the scope of our framework. However, we fully acknowledge the importance of this setting, as it represents the next step in the DAC field and can be regarded as multi-algorithm configuration or universal algorithm configuration. Leveraging the ConfigX framework you mentioned may help achieve this, and we will address this discussion specifically in our response to Q4.\n\nAs for the setting of hyperparameter order, there is an intuitive and natural way to determine it: setting it according to the sequence in which the hyperparameters take effect during the execution of the algorithm. Specifically, in the code implementation of the target algorithm, all hyperparameters take effect in a specific order, which is the order we have set. For example, in the implementation of MOEA/D, after initializing the weights, we first determine the neighborhood subproblems (agent 1: neighborhood size), then optimize the subproblems with certain operators (agent 2: types of reproduction operators and agent 3: parameters of reproduction operators), and finally decide whether to adjust the weights (agent 4: whether to update weights) based on the results of this generation. Therefore, it is relatively easy for us to get an order with acceptably good performance.\n\nWe will revise to make it clearer. Thank you again for your valuable comment!\n\n\n## Q4 Concerns about scalability and zero-shot ability of SADN.\n\nThank you very much for your constructive and insightful question! We believe that modularization is a promising trend towards universal algorithm configuration, and the exploration of modular system in ConfigX [3] is inspiring. ConfigX exhibits obvious inter-dependencies in the action sequence (e.g., module type and module configuration). We believe it is interesting to incorporate our SADN into the ConfigX framework, using the multi-agent modeling to learn and control the module generation and module configuration, which may boost a more universal and general automated algorithm design and configuration. The Q2 of Reviewer aCAk holds similar ideas: the number of parameter changes when we choose different modules. We believe it is a promising future work and will add this discussion to our revised paper. Thank you very much!\n\n[3] ConfigX: Modular Configuration for Evolutionary Algorithms via Multitask Reinforcement Learning. AAAI, 2025.\n\n## Q5 Suggestions on including error bars into Table 1 and correcting the citation format.\n\nThank you for your comments. We will carefully revise the paper according to your suggestions.\n\n## Q6 The issue of SADN underperforming the baseline in one case.\n\nThank you for the question. As shown in Table 1, there are occasional cases where SADN underperforms the baseline, and all of them occur in the testing phase. This may be owing to the very different function landscape that was not included in the training phase, and the learned RL policy fails to generalize. This issue is always concerned regarding the generalization ability of RL, and we believe good state and reward designs that capture commonalities among various problem sets can further enhance generalization, as well as more diverse training datasets. In general, our SADN shows strong generalization abilities across problem classes.\n\n---\n\n**We hope that our response has addressed your concerns, but if we missed anything, please let us know.**"}, {"review_id": "zRHc8AWIlf", "reviewer": "Reviewer_aCAk", "summary": "The paper presents a new method to deal with inherent inter-dependencies between parameters in Dynamic Algorithm Configuration (DAC). First, the authors propose a sequential multi-agent formulation for the DAC problem, where each agent acts based on what all previous agents’ decisions at the same timestep. Then, they propose a sequential advantage decomposition network, where each agent models its own advantage function and show that sequentially selecting the actions that maximize the individual agents’ advantages is equivalent to selecting the best global action. The individual advantages are then combined to find the global advantage, which is used together with a learned global value function to jointly train all the agents. The presented framework is tested on several white-box environments, as well as several real-scenarios DAC benchmarks, showing very good performances against state-of-the-art MARL and sequential methods used in DAC, especially when the number of dimensions (configurable parameters) increases.", "questions": "1. In 4.2 you say that during training the instance is resampled every time an episode ends. Is the set of testing instance still disjunct from the training one?\n\n2. As I understand, every parameter to configure corresponds to a different agent. In the MOEA/D environments, do all the operators have the same number of parameters? \n  If so, how is this handled (e.g. are agents correspondingly enabled and disabled or are their actions ignored?); \n  If not, is this not a limitation, as it can often happen that the number of subsequent decision depend on the previous ones?\n\n3. Can you please discuss the performance of the different methods in terms of training and evaluation time?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n•\tThe paper is well written and easy to follow.\n\n•\tThe empirical experiments cover a fair number of white-box and more realistic and complex environments and show convincing performances against several state-of-the-art approaches.\n\n•\tWhile the selected application domain allows the authors to clearly explain the problem of decision inter-dependency, the proposed method can be easily applied to many other multi-agent problems.\n\nWeaknesses:\n\n•\tThe proposed method is mostly a combination of multi-agent advantage decomposition and value decomposition network.\n\n•\tIt is not very clear how the proposed Seq-MADAC differs from the SEMDP, besides from the addition of the contextual component.\n\n•\tIt is not clear whether the method can deal with environments where the number of parameters to configure at each timestep changes depends on decisions taken early in the sequence.", "was_revised": true, "final_justification": "The rebuttal has clarified the questions I had.", "rebuttal": "Thanks for your valuable and encouraging comments! Below please find our response. \n\n\n## Q1 The difference between the proposed Seq-MADAC and the SEMDP, besides from the addition of the contextual component.\n\nThank you for the question. We formulate the Seq-MADAC task as a contextual sequential MMDP. Besides from the addition of the contextual component, the main difference between the proposed contextual sequential MMDP and the SE-MDP is that we formulate this task as a multi-agent MDP, while the SE-MDP is a single-agent MDP. We have given the definition of MMDP in line 114. Moreover, we allow communication between the agents. The agents basically execute in a decentralized manner, with only the currently taken action being mutually notified.\n\n## Q2 Can the method deal with environments with the changing number of parameters?\n\nThank you for the insightful and constructive question! Although we did not address this issue in our paper, we believe it can be left as a promising future work just as discussed in the Q4 of Reviewer FPTa. ConfigX [1] proposes a novel modularization system that supports flexible reconstruction of different algorithm sub-modules. Inspired by the exploration of modular system in ConfigX, we can modularize the algorithm, and when we choose different modules, the corresponding number of parameters varies. We can have one agent responsible for the parameter configuration of one module, with one leader agent to schedule the sub-modules. Moreover, this modeling also exhibits obvious inter-dependencies in the action sequence (e.g., module type and module configuration), where our method may achieve good performance. Besides, we may enable or disable some agents to deal with the changing number of parameters. For example, when extra parameters are activated due to the previous configurations, the corresponding agents are enabled. We will add this discussion into our revised paper. Thank you very much.\n\n[1] ConfigX: Modular Configuration for Evolutionary Algorithms via Multitask Reinforcement Learning. AAAI, 2024.\n\n## Q3 Is the set of testing instance still disjunct from the training one?\n\nYes, we resample the testing instance from the problem instance distribution, once one episode is done and the environment is reset.\n\n## Q4 In the MOEA/D environments, do all the operators have the same number of parameters? \n\nIn the MOEA/D environment, we have 4 agents configuring 4 types of hyperparameters: \n1. Agent 1: neighborhood size, which controls the maximum distance between solutions and their neighbors; \n\n2. Agent 2: types of reproduction operators shown as follows; \n\n3. Agent 3: parameters of reproduction operator, the scaling factor F in specific; \n\n4. Agent 4: whether to update the weights to bring in new subproblems. \n\nThe operators have a different number of parameters, which are shown below:\n\n- OP1: \n    $$\\boldsymbol {x}'^{(i)}=\\boldsymbol{x}^{(i)}+F \\times\\left(\\boldsymbol{x}^{(r_{1})}-\\boldsymbol{x}^{(r_{2})}\\right),$$\n    \n- OP2:\n    $$\n    \\boldsymbol {x}'^{(i)}=\\boldsymbol{x}^{(i)}+F \\times\\left(\\boldsymbol{x}^{(r_{1})}-\\boldsymbol{x}^{(r_{2})}\\right)+F \\times\\left(\\boldsymbol{x}^{(r_{3})}-\\boldsymbol{x}^{(r_{4})}\\right),\n    $$\n- OP3:\n    $$\n    \\boldsymbol {x}'^{(i)}=\\boldsymbol{x}^{(i)}+K \\times\\left(\\boldsymbol{x}^{(i)}-\\boldsymbol{x}^{(r_{1})}\\right)+F \\times\\left(\\boldsymbol{x}^{(r_{2})}-\\boldsymbol{x}^{(r_{3})}\\right)+F \\times\\left(\\boldsymbol{x}^{(r_{4})}-\\boldsymbol{x}^{(r_{5})}\\right),\n    $$\n- OP4:\n    $$\n    \\boldsymbol {x}'^{(i)}=\\boldsymbol{x}^{(i)}+K \\times\\left(\\boldsymbol{x}^{(i)}-\\boldsymbol{x}^{(r_{1})}\\right)+F \\times\\left(\\boldsymbol{x}^{(r_{2})}-\\boldsymbol{x}^{(r_{3})}\\right).\n    $$\n\nOP1 and OP2 have only one operator parameter, which is $F4$, while OP3 and OP4 have two operator parameters, namely $K$ and $F$. In our setting, agent 3 configures the operator $F$, a common parameter shared by all operators, with an action space of {0.4, 0.5, 0.6, 0.7}, while keeping factor $K$ fixed at 0.5. More details can be found in Appendix B.2. We will revise the text to make this clearer. Thank you.\n\n## Q5 Runtime analysis of different methods.\n\nThank you for the valuable question. We agree that adding runtime analysis will further strengthen our results. We tested the training time of the compared methods on the 10D Seq-Sigmoid benchmark for 1,050,000 steps and the MOED/D benchmark for 405,000 steps. The results are shown below:\n\n| |SADN|ACE|SAQL|VDN|QMIX|MAPPO|HAPPO|HASAC|\n|---|---|---|---|---|---|---|---|---|\n|10DSeq-Sigmoid|4h28'|5h01'|6h34'|3h23'|3h29'|12h32'|16h39'|21h17'|\nMOEA/D|14h56'|16h20'|19h46'|14h48'|15h23'|17h44'|20h02'|23h52'|\n\nDue to the independent and simultaneous update of the individual advantage nets, SADN demonstrates good efficiency in terms of inference time. In summary, to achieve the best average rank, the running time of our SADN is only slower than two value decomposition-based methods, i.e., VDN and QMIX, while faster than SAQL, ACE, as well as the policy gradient-based MAPPO, HAPPO, and HASAC. We will include these results into our main paper. Thank you very much!\n\n---\n\n**We hope that our response has addressed your concerns, but if we missed anything, please let us know.**"}, {"review_id": "r3HC2YOOBS", "reviewer": "Reviewer_xzDh", "summary": "The work proposes an improvement for multi-agent dynamic algorithm configuration by making hyperparameter interdependence explicit to the agent. The work evaluates the effectiveness of this improvement on a variety of whitebox benchmarks, with controllable interdependence as well as on real optimization scenarios from the dynamic algorithm configuration community. This evaluation highlights that explicitly taking hyperparameter interdependence into account can significantly improve algorithm performance and lead to faster learning of successful configuration policies.", "questions": "* How does the proposed approach handle continuous hyperparameters?\n* How difficult do you think it will be for users to apply your approach to novel DAC problems?\n* How large is the action space per agent in the considered scenarios?", "limitations": "The impact of the optimization order is not discussed well in the main paper (see also my comments in the strengths/weaknesses section)", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I very much like the work and particularly appreciate that whitebox benchmarks are used to evaluate a variety of challenges that could be faced in real world optimization problems. Further, I commend that the experiment evaluating the influence of action ordering is also performed on the real world scenario and not only on white-box benchmarks. Thus, I overall am very positive about this work and the empirical evaluation.\n\nThe biggest weakness in my opinion is the brevity of the discussion about the influence of hyperparameter ordering in the setup. While the appendix provides additional insights about the influence of the ordering, it is barely discussed in the main paper. The multi-agent DAC setting, as presented in this paper, requires quite some prior knowledge of a user to properly setup the configuration space and overall scenario. This is a significant limitation for the proposed method to be practical to real-world use cases and thus should be addressed in more detail in the main paper and not just in the appendix.\n\nThe description of the white-box benchmarks in the appendix shows that there is a parameter C that determines the action space size of an agent. I could not find how this parameter was set in the presented experiments.\n\nI believe there +/-/approx column in the Tables might not be correct or at least difficult to read. For example in Table 4 SAQL/QMIX have 0/2 marked in the + part but 2/1 values highlighted in bold face, indicating the best performance. Either the counts are off or the elements in the table should be highlighted (e.g. italicized if they are in the + or underlined if they are in the approx category) differently to make it easier to match the results.", "was_revised": true, "final_justification": "The work proposes an interesting approach to dynamic algorithm configuration. The authors answered all my questions and, with minor revisions to incorporate the overall reviewers feedback, I believe that the work can provide interesting discussions at NeurIPS.", "rebuttal": "Thanks for your valuable and encouraging comments! Below please find our response. \n\n## Q1 The brevity of the discussion about the influence of hyperparameter ordering in the setup.\n\nThank you for providing such constructive insights. In practice, it is indeed a significant problem to determine a proper order of the hyperparameters, and here we provide a simple and natural way to determine a right order, that is, setting the order in which the hyperparameters take effect during the execution of the algorithm. Since we will know when certain hyperparameters take effect in the target algorithm if we want to dynamically configure it, this order can be relatively easy to obtain and treated as an order with acceptable performance. However, we have to point out that such an order may not always be optimal with respect to training efficiency and effectiveness. We believe that determining the best order for the hyperparameters of general algorithms can be left as challenging and promising future works, which may include applying large language models to leverage their knowledge to set the proper order, as well as causal models to automatically learn the hyperparameters' causal structure from the data and further determine the proper order using the topological order of the learned causal graph.\n\n## Q2 How the parameter C is set?\n\nWe are sorry for not making it clear. In the Sigmoid-series benchmarks, we have the agents' action space defined as follows:\n$$\na_{h,t} \\in \\left \\\\{ \\frac{0}{C_h}, \\frac{1}{C_h}, \\dots, \\frac{C_h-1}{C_h} \\right\\\\}. \n$$\nWe set $C_h$ as 10, which is a relatively large action space for the agents to choose actions from. We will revise to make it clear. Thank you for pointing it out.\n\n## Q3 Some concerns about the table notations.\n\nThank you for the valuable suggestions. However, there are some misunderstandings that need to be clarified. \n\nWe separately summarize the Wilcoxon rank-sum test results in the training problem instances and the testing problem instances. One result highlighted in bold denotes that it is the best result, but not necessarily significantly superior to our method SADN (marked with \"+\"), according to the Wilcoxon rank-sum test. \n\nFor example, as you mentioned (actually in Table 1, we believe), SAQL has two bold results, one in the training part and the other in the testing part. In the testing part, SAQL is marked as 0/11/4 because its only bold result in the testing part is almost equivalent to our method SADN according to the Wilcoxon rank-sum test. While QMIX is marked as 2/11/2 in the testing part, because it has two results significantly superior to our method SADN (i.e., DTLZ4 with dim 9 and 12). These two results of QMIX are not highlighted in bold because they are not the best (VDN achieves the best results on DTLZ4 with dim 9 and 12).\n\nWe gladly take the advice of highlighting them differently, e.g., make the values in the table italic or underlined if they are in different categories of significant test. We will revise it accordingly to make the tables easier to read. Thank you very much.\n\n## Q4 How does the proposed approach handle continuous hyperparameters?\n\nThank you for the question. In the MOEA/D environment, we have 4 agents configuring 4 types of hyperparameters: \n1. Agent 1 is responsible for adjusting the neighborhood size, which controls the maximum distance between solutions and their neighbors. Agent 1 can choose the neighborhood size from the action space {15, 20, 25, 30}; \n\n2. Agent 2 is responsible for determining the types of reproduction operators with action space {0, 1, 2, 3}, denoting the different operator types shown as follows; \n\n3. Agent 3 is responsible for determining the parameters of reproduction operator, scaling factor F specifically, with action space {0.4, 0.5, 0.6, 0.7}; \n\n4. Agent 4 is responsible for determining whether to update the weights to bring in new subproblems, with action space {0, 1}, 1 for updating and 0 for not updating the weights.\n\nThe detailed information about the operators is shown below:\n- OP1: \n    $$\\boldsymbol {x}'^{(i)}=\\boldsymbol{x}^{(i)}+F \\times\\left(\\boldsymbol{x}^{(r_{1})}-\\boldsymbol{x}^{(r_{2})}\\right),$$\n    \n- OP2:\n    $$\n    \\boldsymbol {x}'^{(i)}=\\boldsymbol{x}^{(i)}+F \\times\\left(\\boldsymbol{x}^{(r_{1})}-\\boldsymbol{x}^{(r_{2})}\\right)+F \\times\\left(\\boldsymbol{x}^{(r_{3})}-\\boldsymbol{x}^{(r_{4})}\\right),\n    $$\n- OP3:\n    $$\n    \\boldsymbol {x}'^{(i)}=\\boldsymbol{x}^{(i)}+K \\times\\left(\\boldsymbol{x}^{(i)}-\\boldsymbol{x}^{(r_{1})}\\right)+F \\times\\left(\\boldsymbol{x}^{(r_{2})}-\\boldsymbol{x}^{(r_{3})}\\right)+F \\times\\left(\\boldsymbol{x}^{(r_{4})}-\\boldsymbol{x}^{(r_{5})}\\right),\n    $$\n- OP4:\n    $$\n    \\boldsymbol {x}'^{(i)}=\\boldsymbol{x}^{(i)}+K \\times\\left(\\boldsymbol{x}^{(i)}-\\boldsymbol{x}^{(r_{1})}\\right)+F \\times\\left(\\boldsymbol{x}^{(r_{2})}-\\boldsymbol{x}^{(r_{3})}\\right).\n    $$\n\nWe handle continuous hyperparameters by discretizing the continuous action space into a discrete action space, which is a common practice when dealing with the continuous action space. Empirically, discretization has achieved good performance. In the future works, we could apply other techniques to directly handle continuous action, e.g., determine the value by the probability distribution output of the actor network.\n\n## Q5 How difficult do you think it will be for users to apply your approach to novel DAC problems?\n\nThank you for the constructive question. We believe it is relatively easy to apply our approach to real-scenario DAC problems. \n\nFirstly, the users need to follow some common procedures to address the DAC problems:\n\n- Construct their target algorithm as an environment that receives multiple hyperparameters and executes step by step. \n\n- Set the configuration space of the multiple hyperparameters in their target algorithm.\n\nSecondly, the users need to determine the hyperparameter order. They can either use our recommended order described in Q1 or set the order according to their own expert knowledge.\n\nMoreover, our method has a relatively smaller amount of hyperparameters compared to policy gradient-based MARL algorithms. Empirically, we find that our method does not require delicate hyperparameter tuning to achieve good performance. The default setting in our code (config/alg/sadn_ns.yaml) can boost good performance in many cases. You can also conveniently and simply modify it based on your experience.\n\nIn conclusion, we believe that our method is user-friendly and has the potential to be widely applied to solve the DAC problems.\n\n## Q6 How large is the action space per agent in the considered scenarios?\n\nThank you for the question. In the white-box benchmarks (the Seq-Sigmoid series), the dimension of the action space per agent is 10. In the MOEA/D, the dimension of the action space for Agents 1 to 3 is 4, and the dimension of the action space for Agent 4 is 2.\n\n---\n\n**We hope that our response has addressed your concerns, but if we missed anything, please let us know.**"}]

Author rebuttal:

Meta-review: Three reviewers recommended accepting the paper, with one weakly accepting it. The reviewers found the topic of the paper to be relevant, the method to be effective and the experiments to be convincing. I recommend to accept the paper.
Decision: Accept (poster)