Meta-Black-Box-Optimization through Offline Q-function Learning
Authors: Zeyuan Ma
Zhiguang Cao
Zhou Jiang
Hongshu Guo
Yue-Jiao Gong
Venue: ICML 2025 poster (2025)
Keywords: Black-Box Optimization
Dynamic Algorithm Configuration
Learning to Optimize
Offline Reinforcement Learning
Primary area: optimization->zeroorder_and_blackbox_optimization
TL;DR: A novel offline learning-based MetaBBO framework tailored for Dynamic Algorithm Configuration in evolutionary algorithms.

Abstract: Recent progress in Meta-Black-Box-Optimization (MetaBBO) has demonstrated that using RL to learn a meta-level policy for dynamic algorithm configuration (DAC) over an optimization task distribution could significantly enhance the performance of the low-level BBO algorithm. However, the online learning paradigms in existing works makes the efficiency of MetaBBO problematic.
To address this, we propose an offline learning-based MetaBBO framework in this paper, termed Q-Mamba, to attain both effectiveness and efficiency in MetaBBO. Specifically, we first transform DAC task into long-sequence decision process. This allows us further introduce an effective Q-function decomposition mechanism to reduce the learning difficulty within the intricate algorithm configuration space. Under this setting, we propose three novel designs to meta-learn DAC policy from offline data: we first propose a novel collection strategy for constructing offline DAC experiences dataset with balanced exploration and exploitation. We then establish a decomposition-based Q-loss that incorporates conservative Q-learning to promote stable offline learning from the offline dataset. To further improve the offline learning efficiency, we equip our work with a Mamba architecture which helps long-sequence learning effectiveness and efficiency by selective state model and hardware-aware parallel scan respectively. Through extensive benchmarking, we observe that Q-Mamba achieves competitive or even superior performance to prior online/offline baselines, while significantly improving the training efficiency of existing online baselines. We provide sourcecodes of Q-Mamba \href{https://github.com/MetaEvo/Q-Mamba}{online}.

Mette-Blak-Box:OOOiiiiztooo
through Offline Q-function Learning
Zeyuan Ma Zhiguang Cao? 2 Zhou Jiang'
Hongshu Guo
Yue-Jiao Gong'
Abstract
1. Introduction
Black-Box- Optiimizatiiin (BBO) problem is challenging due to agnostic problem formulation, requiring effective BBO algorithms such as Evolutionary Computation (EC) to ad- dress (Zhan et al., 2022). For decades, various EC methods have been crafted by experts in optimization domain to solve diverse BBO problems (Slowik & Kwasnicka, 2020; Guo et al., 2024c). However, one particular technical bottleneck of human-crafted BBO algorithms that they hardly gener- alize across different problems (Fiben & Smit, 2011), 1). Deep expertise is required adapt an existing BBO algorithm for novel problems, which impedes EC's further spread.
Recent progress in Meta-Blacks Bae---llk--B---ppp Box-Opiimizztiinn (MetaBBO) has demonstrated that using RL to learn a meta-level policy for dynamic algorithm configuration (DAC) over an optimization task distribution could significantly enhance the perfor- mance of the low-level BBO algorithm. However, the online learning paradigms in existing works makes the efficiency of MetaBBO problematic. To address this, we propose an offline learning- based MetaBBO framework in this paper, termed Q-Mamba, to attain both effectiveness and effi ciency in MetaBBO. Specifically, we first trans- form DAC task into long-sequence decision pro- cess. This allows us further introduce an effective Q-function decomposition mechanism to reduce the learning difficulty within the intricate algo- rithm configuration space. Under this setting, we propose three novel designs to meta-learn DAC policy from offline data: we first propose a novel collection strategy for constructing offline DAC experiences dataset with balanced exploration and exploitation. We then establish decomposition- based Q-loss that incorporates conservative Q- learning to promote stable offline learning from the offline dataset. To further improve the of- fline learning efficiency. we equip our work with a Mamba architecture which helps long-sequence learning effectiveness and efficiency by selective state model and hardware-aware aware parallel sean re- spectively. Through extensive benchmarking, we observe that Q-Mamba achieves competitive even superior performance to prior online/offline haselines, while significantly improving the train- ing efficiency of existing online baselines. We provide sourcecodes of M-Mamba online.
Recent Meta- Black-lackBBBo Beee-Box---aae -otox-Optiizztiionn (MetaBBO) works address the aforementioned generalization gap by introduc- ing a bi-level learning to optimize paradigm (Ma et al., 2024d; Yang et al., 2025), where a neural network-based policy (e.g., RL (Sutton, 2018)) maintained at the meta level and meta-trained serve as experts for configuring the low-level BBO algorithm to attain maximal performance gain on problem distribution. Though promising, exist- ing MetaBBO methods suffer from two technical bottle- necks: a) Learning Effectiveness: some advanced BBO algorithms hold massive configuration spaces with various apper-parameters which is challenging for RL to learn effec- tive DAC policy. Training Efficiency: Existing MetaBBO methods employ online RL paradigm, showing inefficiency for sampling training trajectories.
Given a such dilemma in-between the effectiveness and efficiency, we this paper propose an offline MetaBBO framework, termed Q-Mamba, to meta-learn effective DAC policy through an offline RL pipeline. Specifically, to re- duce the difficulty of searching for optimal policy from the entire configuration space of BBO algorithms, we first transform DAC task into a long-sequence decision process and then introduce a Q-function decomposition scheme to represent each hyper-parameter parameter in the BBO algorithm as a single action step in the decision process. This allows us to learn Q-policy for each hyper- pareme- in an autoregressive manner. We propose three core designs to support offline RL under such setting: a) Offline data collection strategy: We collect offline DAC experience trajectories from both strong MetaBBO baselines and random policy to provide exploitation and exploration data used for robust training.
'South China University of Technology, China 'Singapore Management University, Singapore. Correspondence to: Yuc-Jiao Gong <gongyuejiao@ gmail.com
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).

---

Meta-Black- Bone-llacc-BooOOtiiiiaioinn through Offline Q-function Learning
b) Conservative Q-learning: we proposed compositional Q-loss that integrates conservative loss term (Kumar al., 2020) to address endemic distributional shift issue (Wang et al., 2021) in offline RL. <) Mamba-based RL agent: the Q-function decomposition scheme would make the DAC decision process in Q-Mamba much longer than those in existing MetaBBO. We hence design a Mamba-based neural network architecture as the RL agent, which shows stronger long-sequence learning capability through its selective state model, and appealing training efficiency through its par- allel scan on whole trajectory sample. Accordingly, we summarize our contributions in three-folds:
configuration (DAC) during the low-level optimization can he regarded Markov Decision Process (MDP), where the state reflects the status of the low-level optimization pro- cess, action denotes the configuration space of the low-level algorithm and a reward function is designed to provide feed- back to the meta-level control policy. Existing MetaBBO methods differ with each other in the configuration space. In general, the configuration space of the low-level algorithm involves the operator selection and/or the tuning. For the operator selection, initial works such as DE-DDQN (Sharma et al., 2019) and DE-DQN (Tan & Li, 2021) facilitate Deep Q-network (DQN) (Mnih, 2013) as the meta-level policy and dynamically suggest one of the pre- pared mutation operators for the low-level Differential Evo- lution (DE) (Storn & Price, 1997) algorithm. Following such paradigm, PG-DE (Zhang et al., 2024) and RL-DAS (Guo et al., 2024a) further explore the possibility of using Pol- icy Gradient (PG) (Schulman et al., 2017) methods for the operator selection and demonstrate PG methods are more ef- fective than DQN methods. Besides, RLEMMO (Lian et al., 2024) and MRL-MOEA (Wang et al., 2024) extend the tar- get optimization problem domain from single-ohjective op- timization to multi-modal optimization and multi-objective optimization respectively. Unlike the operator selection, the action space in hyper-parameter tuning is not merely dis- crete since typically the hyper-paramene parameters of an algorithm are continuous with feasible ranges. In such continuous setting, the action space is infinite and can be handled either by discretizing the continuous value range to reduce this space (Liu et al., 2019; Xu & Pi, 2020; Hong et al., 2024; Yu et al., 2024) or directly using PG methods for continuous control (Yin et al., 2021; Sun et al., 2021; Wu & Wang, 2022; Ma et al., 2024b).
i) Our main contribution is Q-Mamba, the first offline MetaBBO framework which shows superior learning ef- fectiveness and efficiency prior MetaBBO baselines.
ii) To ensure offline learning effectiienesss, Q-function decomposition scheme is embedded into the DAC deci- sion process of BBO algorithm which facilitates separate Q-function learning for each action dimension. Besides, a novel data collection strategy constructs demonstration dataset with diversified behaviours, which can be effectively learned by Q-Mamba through compositional Q-loss which enhances the offline learning by removing distributional shift. To further improve the training efficiency, we design a Mamba- based RL agent which seamlessly aligns with the Q-function decomposition scheme and introduces desirable training acceleration compared to Transformer structures, through parallel scan.
iii) Experimental results show that our Q-Mamba effectively achieves competitive or even superior optimization perfor- mance to prior online/offline learning baselines, while con- suming at most half training budget of the online baselines. The learned meta-level policy can also be readily applied to enhance the performance of the low-level BBO algorithm on unseen realistic scenarios, e.g., Neuroevolution (Such et al., 2017) on continuous control tasks.
While simply doing operator selection or hyper-paramen parameter tuning for part of an algorithm has shown certain perfor- mance boost, recent MetaBBO researches indicate that con- trolling both sides gains more (Xue et al., 2022: Zhao et al., 2024). In particular, an up-to-date work termed as Con- figX (Guo et al., 2024b) constructs a massive algorithm space and has shown possibility of meta-learning a universal configuration agent for diverse algorithm structures. How- ever, the massive action space in such setting and the online RL process in these MetaBBO methods make it challenging to balance the training effectiveness and the efficiency.
2. Related Works
2.1. Meca-Black-BBxxOppiiinaalle
Meta-Black-Bar See-BoxxOOpiiizatinn (MetaBBO) aims to learn the optimal policy that boosts the optimization performances of the low-level BBO algorithm over an optimization prob- lems distribution (Ma et al., 2024d). Although several works facilitate supervised learning (Chen et al., 2017; Song et al., 2024; Li et al., 2024b;e; Wu et al., 2023), Neuroevo- lution (Lange et al., 2023b;a; Ma et al., 2024a) or even LLMS (Ma et al., 2024c; Liu et al., 2024) to meta-learn the policy, the majority of current MetaBBO methods adopt reinforcement learning for the policy optimization to strike a balance between effectiveness and efficiency (Li et al., 2024a; Ma et al., 2023). Specifically, the dynamic algorithm
2.2. Offline Reinforcement Learning
Offline RL (Levine et al., 2020) aims at learning the opti- mal control policy from pre-collected demonstration set, without the direct interaction with the environment. This is appealing for real-world complex control tasks, where on-policy data collection is extremely time-consuming (i.c., the dynamic algorithm configuration for black-box opti- mization discussed in this paper). A critical challenge in
2

---

through Offline Q-function Learning
offline RL is the distribution shift (Fujimoto et al., 2019): learning from offline data distribution might mislead the policy optimization for out-of-distribution transitions hence degrades the overall performance. Common practices in offline RI. to relieve the distribution shift include a) learn- ing policy model (e.g.. Q-value function) by sufficiently exploiting the Bellman backups of the transition data in the demonstration set and constraining the value functions for out-of-distribution ones (Haarnoja et al., 2018; Kumar et al., 2020). b) conditional imitation learning (Chen et al., 2021; Janner et al., 2021; Dai et al., 2024b) which turns the MDP into sequence modeling problem and uses se- quence models (e.g., recurrent neural network, Transformer or Mamba) to imitate state-action-reaare sequences in the demonstration data. Although the conditional imitation learning methods have been used successfully in control domain, they have stitching issue: they do not provide any mechanism to improve the demonstrated behaviour as those policy model learning methods. To address this, QDT (Ya- magata et al., 2023) and QT (Hu et al., 2024) additionally train a value network to relabel the return-to-ge in offline dataset, so as to attain stitching capability. Differently, Q- Transformer (Chebotar al., 2023) combines the strength of both lines of works by first decomposing the Q-value function for the entire high-dimensional action space into separate dimeeiinnsoon Q-value functions, and then leverag- ing transformer architecture for sequential Bellman backups learning. Q- Q-Transformer allows policy improvement during the sequence-tor saaeence-to-sqqueee tos--uueceeeeeeeeeeeeeeeeeeeeeee learning hence achieves superior performance to the prior works.
dependent Q-function representations for each action di- mension to escape the curse of dimensionality, For the i-th action dimension, the decomposed Q-function is written as:
max
ai Q di.),
if i<k
Qaalss)) + R(ss..fi.. + + max Q(a (
af"
if i i= K
(2)
Such a decomposition allows using sequence modeling tech- niques to learn the optimal policy effectively, while holding the learning consistency with the Bellman backup in Eq. (1). We provide a brief proof in Appendix A.
3.2. State Space Model and Mamba
For an input sequence x E RxD with time horizon and D-dimensional signal channels at each time step, State Space Model (SSM) (Gu et al., 2022) processes it by the following first-order differential equation, which maps the imput signal xtt) E R2 to the time- umeedeeendent output y(t) E RD through implicit latent state h(t) as follows:
h(t)= = Ah(t - 1) + Br((), v(t) = Ch(t).
(3)
Here, A. B and C' are learnable parameters, A and B are obtained by applying zero-order hold (ZOH) discretization rule. An important property of SSM is linear time invariance. That is, the dynamic parameters (c.g., A, B and C) are fixed for all time steps. Such models hold limitations for sequence modeling problem where the dynamic is time-dependent. To address this bottleneck, Mamba (Gu & Dao, 2023) lets the parameters B and C' be functions of the input xxt). Therefore, the system now supports time-varying sequence modeling. In the rest of this paper, we use mamba_ bblock()bo to denote a Mamba computation block described in Eq. (3).
3. Preliminaries
3.1. Decomposed Q-function Representation
(A1,.., Ak),, R, T, 7h where the action space is associated by a series of K ac- tion dimensions, S, R((S,A) T(S y denote the state, reward function, transition dynamic and discount factor, respectively. Value-based RI methods such as Q-learning (Watkins & Dayan, 1992) learn a Q-function Q(s', af.x) ) as the prediction of the accumulated return from the time step by applying af. at s The Q-function can be iteratively approximated by Bellman backup:
Suppose we have a MDP A A
4. Q-Mamba
4.1. Problem Formulation
A MetaBBO task typically involves three key ingredients: neural network-based meta-level policy ma a BBO algo- rithm A and a BBO problem distribution P to be solved.
(aaixxl9 + R(s*, af:x)) + max 4-1
((aft)) (1)
Optimizer A. BBO algorithms such as Evolutionary Algorithms (EAs) have been discussed and developed over decades. Initial EAs such as Differential Evo- lution (DE) (Storn & Price, 1997) holds few hyper- parameters (only two, F and Cr for balaneing the mutation and crossover strength). Modern variants of DE integrate various algorithmic components to enhance the optimiza- tion performance. Taking the recent winner DE algorithm in IEEE CEC Numerical Optimization Competition (Mohamed et al., 2021), MadDE (Biswas al., 2021) as an example, it
However, suppose there are at least M action bins for each of the K action dimensions, the Bellman backup above would be problematic since the associated action space contains M feasible actions. Such dimensional curse challenges the learning effectiveness of the value-based RL methods. Recent works such as SDQN (Metz et al., 2017) and Q-Transformer (Chebotar et al., 2023) propose decomposing the associated Q-fuuctions into series of time-
3

---

HetaaBllack-oo--Oimmiziin through Offline Q-function Learning
has more than ten hyper-parameters, which take either con- tinuous discrete values. Hence, the configuration space of MadDE is exponentially larger than original DE. In this paper, we use A {A1. A2, } to represent an algo- rithm with K' parameters. We use additional az to represent the taken value of A,
tribution P, we pre-train a series of up-to-date MetaBBO
methods (e.g., RLPSO (Wu & Wang. 2022), LDE (Sun et al., 2021), GLEET (Ma et al., 2024b)) which control hyper- parammteasssrs of A to optimize the problems in P. Then we rollout the pre-trained MetaBBO methods on problem instances in P to collect μ D complete trajectories. We then use the random strategy to randomly control the hyper- parameters of A to optimize the problems in P and collect (1 11) D trajectories. By combining the exploitation ex- perience in the trajectories of MetaBBO methods and the exploration experience in the random trajectories, Q-Mamba learns robust and high-performance meta-level policy. In this paper, we set μ = 0.5 to strike a good balance.
Problem distribution P. By leveraging the generaliza- tion advantage of meta-learning, MetaBBO trains we over problem distribution P. A common choice of P in exist- ing MetaBBO works is the CoCo BBOB Testsuites (Hansen et al., 2021), which contains 24 basic synthetic functions, each can be extended to numerous problem instances by ran- domly rotating and shifting the decision variables. Training on all problem instances in P is impractical. We instead sample a collection of N instances {fi, f2....,''' from P as the training set. For the j-th problem fj. we use fj to represent its optimal objective value, and ʃg (1)) as the objective value at solution point x.
4.3. Conservative Q-learning Loss
Online learning is widely adopted in existing works, which is especially inefficient under MetaBBO setting, where the low-level optimization typically involves hundreds of op- timization steps hence extremely consuminn,, In this paper we propose learning the decomposed sequential Q- function through offline RL to improve the training effi- ciency of MetaBBO. Concretely, we consider trajectory = {s) (a]..., ok),, r ....," s" (aT,.... ..... a²k), r"), which is previously sampled by an offline policy â. Here, of de- notes the action bin selected for A, at time step 1. The training objective of Q-Mamba is a synergy of Bellman backup update (Eq. (2)) and conservative regularization
For an algorithm A and a problem instance fj. suppose we have a control policy Ti at hand and we use A to optimize f3 for T time steps (generations). At the t-th generation, denote the solution population as X An optimization state s¹ is first computed to reflect the optimization status information of the current solution population X' and the corresponding objective values fx')). Then the control policy dictates a desired configuration for A: af.x = To('') A optimizes Xt by of and obtains an offispring popula- tion X A feedback reward computed as a measurement of the performance improve- ment between (X*) and f (X++)). The meta-objective of MetaBBO is to search the optimal policy Te- that maximizes the expectation of accumulated performance improvement over all problem instances in the training set:
can then be
T K M
(= = Σ M
10
1=1
2 (0iss - max Di ) if i< K,j = of
1 Oii (1" + 7 maa 01t1)
N
T
0" = arg max 0
Ris (o)),
(4)
if i= K.j at is j
N
200 - -)2,
t=
(5)
where such a meta-objective can be regarded as MDP. An effective policy search technique for solving MDP is RL, which is widely adopted in existing MetaBBO methods. In this paper, we focus on particular type of RL: Q-learning, which performs prediction on the Q-function in dynamic programming way, as described in Eq. (1).
where 0f,, is the Q-value of the j-th bin in Q. which is outputted by our Mamba-based Q-Learner Te. with [s token(a[_1)] as input. The first two branches in Eq. (5) are TD errors following the Bellman backup for decom- posed Q-function (as described in Eq. (2)). We additionally add weight 3 (we set B = 10 in this paper) on the last action dimension to reinforce the learning on this dimension. As described in Eq. (2), the other action dimensions are up- dated by the inverse maximization operation, so ensuring the accuracy of the Q-value in the last action dimension helps secure the accuracy of the other dimensions. The last branch in Eq. (5) is the conservative regularization intro- duced in representative offline RL method CQL (Kumar et al., 2020), which is used to relieve the over-estimation due to the distribution shift. Here, the Q-values of action bins that are not selected in the trajectory T (j # at) are
4.2. Offline E&E Dataset Collection
The trajectory samples play a key role in offline RL appli- cations (Ball et al., 2023). On the one hand, good quality data helps the training converges. On the other hand, ran- domly generated data help RI. explore and learn more robust model. In Q-Mamba, we collect a trajectory dataset C of size D = 10K which combines the good quality data and randomly generated data. Concretely, for a low-level BBO algorithm with K hyper- param-pprrmeters and a problem dis-
4

---

Meta-Black--al Booplkk-ooo-Otiizaationn through Offline Q-function Learning
regularized to O. which is the lower bound of the Q-values in optimization. This would accelraaaee the learning of the TD enror. We set X x= 1 in this paper to strike a good balance.
outputs the Q-function values Q for each A, in A.
Mamba block. To obtain Qi the first step is to prepare the input as the concatenation of the optimization state s and the previously selected action bin token token(a;_)) Then, we apply a Mamba block with the computation described in Section 3.2. It receives the hidden state h and the embedding feature E and outputs the decision information Of and hidden state hf. Of is used to parse Q-function Q and M is used for next decision step follows:
4.4. Mamba-based Q-Learner
Existing MetaBBO works primarily struggle in learning meta-level policy with massive joint-action space, which is the configuration space A: A2.... J associated by K hyper-parameters of the low-level algorithm A. To relieve this learning difficulty, we introduce Q-function de- composition strategy as described in Section 1. For each syper-parameter A; in A. we represent its Q-function as discretized value function Q; {Qi,,; 2..,,-.., Qi.M); where M is a pre-defined number of action bins for all A; in A (M = 16 in this paper). For any A, which takes values from a continuous range, we uniformly discretize the value range into M bins to make universal representation across all A, By doing this, we turn the MDP in MetaBBO into a sequence prediction problem: we regard predicting each Q; as single decision step, then at time step f of the low- level optimization, the complex associated configuration K of A can be sequentially decided. We further design Mamba-based Q-Learner model to assist sequence model- ing of decomposed Q-functions. The overall workflow of the Mamba- Q-Learner is illustrated in Figure 1. We next elaborate elements in the figure with their motivations.
Of, M = baaambabblock[( (ookemaa. hi [W mamba))
(6)
where Wmamba denotes all learnable parameters in Mamba, which includes the state transition parameters A, B and C. the parameters of discretization step matrix, and time- varying mapping parameters for the state transition param- eters. In this paper we use the mamba-block in Mamba repo', with default settings. To obtain 04, the last hid- den state of time step - 1, h's is used. The motivation of using Mamba is that: a) MetaBBO task features long- sequence process that involves thousands of decision steps since there are hundreds of optimization steps and K hyper- parameters to be decided per optimization step. Mamba is adopted since it parameterizes the dynamic parameters as functions of input state token, which facilitate flexible learning of long-term and short-term dependencies from his- torical state sequence (Ota, 2024). b) Mamba equips itself with hardware-awarr aware I/O computation and a fast parallel scan algorithm: PrefixSum (Blelloch, 1990), which allows Mamba has the same memory efficiency as highly optimized FlashAttention (Dao et al., 2022).
Optimization state s". In MetaBBO, optimization state s' profiles two types of information: the properties of the optimization problem to be solved and the low-level op- timization progress. In Q-Mamba, we construct the opti- mization state s similar with latest MetaBBO methods (Ma et al., 2024b; Chen et al., 2024: Li et al., 2024b). Concretely, at each time step in the lew-level optimization, an opti- mization state 8² E R is obtained by calling a function statistical features
Q-value head. The Q-value head parses the decision infor- mation Of into the decomposed Q Q-function Qf through a linear mapping layer.
cal state(). The first 6 dimensions about the population distribution, objective value distribu- tion, etc., which provide the problem property information. The last 3 dimensions are temporal features describing the low-level optimization progress. We leave the calculation details of s' in Appendix B.
O = @Linearr (O:) Whead, bhead))
Here, a is Leaky ReLU activation function, Whead E I 16616 and bhead 1 are weights and bias. When we ob- tain Qi we select the action bin with the maximum value for hyper-parametts A; of arg 3 Qi.j. and use (a!)
Tokenization of action bins. We represent the M 16 action bins of each hyper-parameters A, in A by 5-bit binary coding: 00000 ~ 01111. Besides, since we sequentially predict the Q-function for A to Ak, we additionally use 11111 as a start token to activate the sequence prediction. We have to note that for an algorithm A, some of its discrete syper-parameters might hold less than M action bins. For this case, we only use the first several tokens to represent the action bins in these hyper-parameet paper, we use loken(a)) (af) to denote the binary coding of the action bin selected for A; at time step t of the low-level op- timization. The Mamba- Mamba-based Qeaamrer regr-seggesyvvely
for inferring the decomposed Q-function Qi+1 of next de- cision step. Once the action bins of all hyper-parameters A1 Ak have been decided, the algorithm A parse all selected action bins to concrete hyper-parameter values and then use them to optimize the problem for one step and obtains the optimization state s/41 from the updated solu- tion population (detailed in Appendix C). To summarize, in Q-Mamba, the meta-level policy Te is the Mamba-base Q-Learner, of which the learnable parameters / includes W mambaa W Wead, bhead). To meta-train the Q-Mamba, we use AdamW with learning rate 5e 3 minimize the ex-
In the rest of this
hittpa///gtthbb com/ state- swwcssaacssseeeeee spaces/mamba /mamba
5

---

Meta-Black-Band Boalck-Box-OOtiiiaaaionn through Offline Q-function Learning
Environment Transition
execute
MI Q-value Head
MI.P Q-value Head
MLP Q-value Head
Rg
hg
Kk-3
hk
Mamba Block
Mamba Block
Mamba Block
Mamba Block
start
Start
Time Step1
Time Step L+1
Figure 1. The workflow of the Mamba- Mamba-based Q-Learner. The forward process of the neural network is similar with the Recurrent Neural Network. At each time step, the Q-function each decomposed action dimension is outputted by conditioning the current state and selected action bins of the previous action dimensions. The environment transition is executed once all action dimensions are outputted.
pectation training objective (r))) After the training, the learned Te can be directly used control A for unseen problems: either those within the same problem distribution P or totally out-of-distribttion ones.
RLPSO (Wu & Wang. 2022) that uses simple MLP architec- ture for controlling low-level algorithms. LDE (Sun et al., 2021) that facilitates LSTM architecture for sequential con- trolling low-level algorithms using temporal optimization information. GLEET (Ma et al., 2024b) that uses Trans- former architecture for mining the exploration-exploitation tradeoff during the low-level optimization. These three baselines are all trained to output associated configuration without decomposition as our Q-Mamba. Since there is offlin MO baseline yet, we examine the learning effectiveness of Q-Mamba by comparing it with series of offline RL baselines: DT (Chen et al., 2021). DeMa (Dai et al., 2024a), QDT (Yamagata et al., 2023) and QT (Hu et al., 2024) are four baselines that apply conditional imita- tion learning on the E&E dataset, where the state, actions and reward in E&E dataset are transformed into RTG to- kens, state tokens associated action tokens for supervised suuuence-to-eqqeene learning. The differences are: DT and DeMa follow naive paradigm with Transformer and Mamba architecture respectively. QDT and QT train a sep- arate Q-value predictor during the grquencc-to-sequemee learning, which relabels the RTG signal to attain policy im- provement. Q- Q-Transformer (Chebotar et al., 2023) shows similar Q-value decomposition scheme as our Q-Mamba, while the neural network architecture is Transformer. The settings of all baselines follow their original papers, except that the training data is the prepared three E&E datasets. To ensure the fairness of the comparison, all baselines are trained for 300 epochs with batch size 64.
5. Experimental Results
In the experiments, we aim to answer the following ques- tions: a) How Q-Mamba performs compared with the other online/offline baselines? b) Can Q-Mamba be zero-shot to more challenging realistic optimization scenario? c) How important are the key designs in Q-Mamba?
5.1. Experiment Sctup
Training dataset. We first sample 3 low-level BBO al- gorithms from the algorithm space constructed in Con- figX (Guo et al., 2024b), which are three evolutionary algo- rithms including 3, 10 and 16 hyper-parameters, showing different difficulty-levels for MetaBBO methods. We in- troduce their algorithm structures in Appendix D.1. The problem distribution selected for the training the CoCo BBOB Testsuites (Hansen et al., 2021), which contains 24 basic synthetic functions with diverse properties. We denote it as Pooor. We divide it into 16 problem instances for train- ing and 8 problem instances for testing. These functions range from ~ 50-dimensional, with random shift and ro- tation on decision variables. More details are provided in Appendix D.2. Based on the 16 training functions, we cre- ate a E&E Datasets for each BBO algorithm following the procedure described in Section 4.2. For online MetaBBO baselines, we train them on each low-level algorithm to op- timize the training functions. For offline baselines including our Q-Mamba, we train them on each E&E Dataset. Note that the total optimization steps for the low-level optimiza- tion is set as T 500.
Performance metric. We adopt the accumulated perfor- mance improvement Per Per I' f(TT) for measuring the opti- mization performance of the compared basclines and our Q- Mamba. Given MetaBBO baseline no, the corresponding low-level algorithm A and an optimization problem instance f. the accumulated performance improvement is calculated as the sum of reward feedback at each optimization step t: fer (A, fl = Σr= r'. The reward feedback is calcu- lated as the relative performance improvement between two consecutive optimization steps: rf p-0 f-0-f"
We compare wide range of baselines to obtain comprehensive and significant experimental observations. Concretely, we compare three online MetaBBO baselines:
Baselines.
where
6

---

Meta-Black-Box Blie-Bllccc-Boo-Oppiiiattiono through Offline Q-function Learning
Table 1. Performance comparison between Q-Mamba and other onlinc/offline baselines. All baselines are tested on unseen problem
instances within the training distribution Pubab. We additionally present the averaged training/inferring time of all baselines in the last row.
Online
Offine
RLPSO
LDE
GLEET
DT
DeMa
QDT
QT
Q Transfommer
Q Q-Mamba
(MLP)
(LSTM)
(Trasformer)
Algu
9.855E-01
9.563E-01
9.616E-01
9,325E-01
9,4926-01
9.6838-01
9.729E-01
9.6661-01
9.889E-01
K 33
+9.0388-03
+18830B-02
13.100
+2,680E-02
=2,467E-02
=2.1318-02
+1,934E-02
=1.975E-02
+7.779E-03
Alg1
9.953E-01
9.877E-01
9.938E-01
6,764E-01
9.015E-01
9.917E-01
9,955E-01
9.951E-01
9.973E-01
K=
+3.322F-03
18.118
+2.834E-03
+1.1938-01
=1,688F-02
+5.454F-03
+3.1155-03
+3.4877-03
+2,441E-03
Aly2
9.914E-01
9:904E-01
9.910E-01
8.706E-01
9.159E-01
9,919E-01
9:926E-01
9.8936-01
9,950E-01
K= 16
24.497E-03
26,306B-03
=5.846E-03
+3,951E-02
=2,015E-02
#7.456E-03
+6.8775-03
#6,754E-03
+9,981E-03
Avg Time
28h 1Is
28b/122 12s
25h/ 13s
13b 10s
12h/ 10s
20h / 12s
20h 12s
16h / 11s
13h 10s
f is the objective value of the best found solution until time step L, ʃ* is the optimum of f. The maximal accu- mulated performance improvement is when the optimum of f is found. Note that f* is unknown for training prob- lem instances, we instead use a surrogate optimum for it, which can be easily obtained by running an advanced BBQ algorithm on the training problems for multiple runs.
imitation-learning based offline RL. We note that this ob- servation is limited within MetaBBO domain in this paper, further validation tests are expected to examine Q-Mamha other RL tasks, which we leave as future works.
iii) Q-Mamba v.s. QDT&QT. Comparing QDT&QT with DT, a tradeoff between the learning effectiveness and train- ing efficiency. QDT&QT both propose additional Q-value predictor, which is subsequently used to relabel the RTG tokens in offline dataset. Although relabeling RTG with learned Q-value allows for policy improvement during the conditional imitation learning. additional training resource is introduced (20h v.s. 13h). Nevertheless, QD&&QT. as the online MetaBBO baselines, searches DAC policy from the massive associated action space. Compared with them, our Q-Mamba achieves not only superior optimization per- formance since we learn easier decomposed Q-function for each typer-parameter in BBO algorithm, but also similar training efficiency with DT since the fardware-friendly com- putation and parallel scan algorithm in Mamba.
5.2. In-distribution Generalization
After training, we compare the generalization performance of our Q-Mamba and other baselines on the 8 problem instances in Pobos which have not been used for training. Specifically, for each baseline and each low-level algorithm, report in Table 1 the average value and error bar of the accumulated performance improvement f-r across the 8 tested problems and 19 independent runs We additionally present the average training time and inferring time (time consumed to complete a DAC process for BBO algorithm A on a given optimization problem) for each baseline in the last row. The results show following key observations:
iv) Q-Mamba v.s. Q- Q-Transformer. While our Q-Mamba shares the Q-function decomposition scheme with Q- Transformer, a major novelty we introduced is the Mamba architecture and the corresponding weighted Q-loss func- tion. The superior performance of Q-Mamba to the Q- Transformer possibly roots from the linear time invari- ance (LTT) of Transformer, which presents fundamental limitation in selectively utilizing short-term or long-term temporal dependencies in the long Q-function sequence. In contrast, Mamba architecture holds certain advantages: it allows Q-Mamba selectively remembers forgets his- torical states based on current token. Mamba architecture achieves this through parameterizing the dynamic parame- ters in Eq. (3) as functions of input state tokens.
i) Q-Mamba v.s. Online MetaBBO. Surprisingly, Q- Mamba achieves comparable/superion optimization perfor- mance to online baselines RLPSO, LDE and GLEET, while showing clear advantage in training efficiency. The per- formance superiority might origins from the Q-function decomposition scheme in Q-Mamba, which avoids search- ing DAC policy from the massive associated configuration space as these online baselines. The improved training efficiency validates the core motivation of this work. By learning from the offline F&F dataset, Q-Mamba reduces the training budget to less than half of those online base- lines. his is especially appealing for BBO scenarios where the simulation is expensive or time-consuning.
ii) Q-Mamba v.s. Dea. We observe that DT and DeMa hold similar training efficiency with our M-maba. The difference between them and Q-Mamba is that they generally imitates the trajectories in E&E dataset by predict- ing the tokens autoregressively. Results in the table show the performances of DT and DeMa are quite unstable (with large variance). In opposite, our Q-Mamba allows policy im- provement during the sequence learning, which shows better learning convergence and effectiveness than the conditional
5.3. nut-of-distribution Generalization
We have to note that the core motivation of MetaBBO is generalizing the meta-level policy trained on simple and eco- nomic BBO problems towards complex realistic BBO sce- narios. We hence examine the generalization performance of Q-Mamba and three online MetaBBO baselines on a chal- lenging scenario: neuroevolution (Such al., 2017) tasks. In a neuroevolution task, BBO algorithm is used to evolve
7

---

Meta-Black Beae--Blackkoo Boaeck--ooOOtimzzatinn through Offline Q-function Learning
HallChhettub.vt x44
Porformanee
500
Table 2. Importance analysis on and & in compositional Q-loss function.
5000-
250
= 0
A= 10
4000
9.756B.01
9.828E:01
9.855E
8 1
+1.570P-02
+1.203F-02
+1.192F-02
3000
250
8 10
9.8336-01
9.889-001
9.857E-01
10
20
10
50
#1.424E-02
+7.780E-03
+11344-02
Pusher-v4
Ant-v4
Performance
-300
-600
1100
Table 3. Performance of Q-Mamba under different proportion of exploitation data with good quality.
-500
1000
0.25
0.5
0.75
9.832E-01
9.874E-01
9.889E-01
9.793E-01
98834-01
10
20
30
50
0
10
20
40
50
Perf.
#1,264E-02 16.489E-03
27.780E-03 #1.614E-02 19.692E-03
Optimization Steps
Optimization Steps
Q-Mamba
LDE
GLEET
RLPSO
Figure 2. Zero shot performance of Q-Mamba and online MetaBBO baselines on neuroevolution tasks.
Data ratio in E&E dataset. Another key design in Q- Mamba is the construction of E&E dataset. When collecting DAC trajectories to construct it, we set a data mixing ratio I which controls the proportion of exploitation data and exploration data. When μ = 0, all trajectories come from random configuration policy, which provides exploratory experiences with relatively low quality. When μ 1, all trajectories come from the well-performing MetaBBO base- lines, which provides at least sub-optimal DAC experiences with high quality. The results in Table reveal that mixing these two types of data equally (μ 0.5) might enhance Q-Mamba's 's learning effectiveness by leveraging the rich his- torical experiences from both exploration and exploitation. This actually follows a common sense that increasing data diversity could reduce the training bias in offline learning.
a population of neural networks according to their perfor- mance on a specific machine learning task, i.e., classifica- tion, robotic control (Galván & Mooney, 2021). Specifically, we consider continuous control tasks in Mujoco (Todorov al., 2012). We optimize 2-layer MLP policy for each task by Q-Mamba and other baselines trained for controlling Algo on Pobos. To align with the challenging condition in realistic BBO tasks, we only allow the low-level optimiza- tion involves small population (10 solutions) and 7 = 50 optimization steps. We present the average optimization curves across 10 independent runs in Figure 2. The results underscore the potential positive impact of Q-Mamba for MetaBBO domain: a) while only trained on synthetic prob- lems with at most 50 dimensions, our Q-Mamba is capable of optimizing the MLP polices which hold thousands of parameters in neuroevolution tasks. b) compared to online MetaBBO baselines, Q-Mamba is capable of learning ef- fective policy with comparable generalization performance, while only consuming less than half training resources.
6. Conclusion
In this paper, we propose Q-Mamba as novel offline learning-based MetaBBO framework which improves both the effectiveness and the training efficiency of existing on- line leaning- based MetaBBO methods. To achieve this, Q-Mamba decomposes the associated Q-function for the massive configuration space into sequential Q-functions for each configuration. We further propose a Mamba-based Q-Learner for effective sequence learning tailored for such Q-function decomposition mechanism. By incorporating with a large scale offline dataset which includes both the ex- ploration and exploitation trajectories, Q-Mamba consumes less than half training time of existing online baselines, while achieving strong control power across various BBO algorithms and diverse BBO problems. Our framework does have certain limitation. Q-Mamba is trained for a given BBO algorithm and requires re-training for other algorithms. An effective algorithm feature extraction mechanism may en- hance Q- M-Mabb''s 's co-training on various algorithms. We mark this as an important future work. At last, we hope Q-Mamba could serve as a meaningful preliminary study, providing first-hand experiences on integrating efficient of- fline learning pipeline into MetaBBO systems.
5.4. Ablation Study
Coefficients in Q-loss. In Q-Mamba, a key design that ensures the learning effectiveness is the proposed com- positional Q-loss in Eq. (5), which calculates bellman backup the decomposed Q-function sequence first and applies conservative regularization on out-o--distribution action bins. As shown in Table 2, when x = O, the training objective in Eq. (5) turns into the Bellman backup without conservative regularization. The performance degradation under this setting reveals the importance of the conservative term for relieving the distribution shift caused by offline leaning. When 3 1, the training objective would not focus on the Q-value prediction of the last action dimen- sion, which in turn interferes the prediction of other action dimensions through the inverse maximization operation in Eq. (2). A setting with À x 1 and 3 = 10 generally ensures the overall learning effectiveness.
8

---

Meta-Black- Bhie-BllacckBBooOopmiiaiiiono through Offline Q-function Learning
Acknowledgements
cision transformer: modeling. Advances in neural information processing systems, 2021.
Reinforcement learning via sequence
This work is supported part by Guangdong Provincial Natural Science Foundation for Outstanding Youth Team Project (Grant No. Natural Science Foundation of China (Grant No. 62276100), in part by Guangzhou Science and Technology Elite Talent Leading Program for Basic and Applied Basic Research (Grant No. SL20224040013611, This research is also sup- ported by National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: AISG3-RP- 2022-031).
in part by National
Chen, Y., Hoffman, M. W., Colmenarejo, S. G., Denil, M., Lillicrap, T. P., Botvinick, M., and Freitas, N. Learning to learn without gradient descent by gradient descent. In International Conference on Machine Learning, 2017.
Dai, Y., Ma, O., Zhang. L., Liang, X., Hu, S., Wang, M., Ji, S., Huang, J., and Shen, L. Is mamba compatible with trajectory optimization in offline reinforcement learning? arXiv preprint arXiv:2405. /204,, 2024a.
Impact Statement
Dai, Y., Ma, O., Zhang. L., Liang, X., Hu, S., Wang, M., Ji, s., Huang, J., and Shen, I.. mamba compatible with trajectory optimization in offline reinforcement learning? arXiv preprint arXiv:2405. /2094, 2024b.
This paper presents work whose goal is to advance the field of Evolutionary Computation and Black-Box -oatiioo-- In particular, explores the emerging topic Learning to Op- timize where meta-learning for automated algorithm design has been widely discussed and studied. A potential impact of this work highly aligns with the impact of automated algo- rithm design for human society, that is, reducing the design bias introduced by human experts in existing human-crafien BBO algorithms to unleash their optimization performance over industrial-level application and important scientific discovery process. Besides, since this paper provides a pio- neering exploration on effectiveness of offline learning in learning-based automated algorithm design, it potentially impacts existing online learning paradigms, hence poten- tially accelerating the development in this area.
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Process- ing Systems, 2022.
Deb, K., Agrawal, R. B., et al. Simulated binary crossover for continuous search space. Complex systems, 1995.
Dobnikar, A., Steele, N. C., Pearson, D. W., Albrecht, F,, Deb, K., and Agrawal, S. niched-penalty approach for constraint handling in genetic algorithms. In Artificial Neural Nets and Genetic Algorithms: Proceedings of the International Conference in Portorož, Slovenia, 1999, 1999.
References
Ball, P. J.. Smith, L., Kostrikov, I., and Levine, S. Effi- cient online reinforcement learning with offline data. In International Conference on Machine Learning, 2023.
Eiben, A. E. and Smit, S. K. Parameter tuning for configur- ing and analyzing evolutionary algorithms. Swarm and Evolutionary Computation, 2011.
Biswas, S., Saha, D., De, S., Cobb, A. D., Das, S., and Jalaian, B. A. Improving differential evolution through bayesian hyperparameter optimization. In Proceedings of the IEEE Congress of Evolutionary Computation, 2021.
Fujimoto, S., Meger, D., and Precup, Off-policy deep reinforcement learning without exploration. In Interna- tional Conference Machine Learning, 2019.
Galván, E. and Mooney, P. Neuroevolution in deep neural networks: Current trends and future challenges. IEEE Transactions on Artificial Intelligence, 2021.
Blelloch, G. E. Prefix sums and their applications. 1990.
Chebotar, Y., Vuong, Q. Hausman, K., Xia, F., Lu, Y., Irpan, A. Kumar, A., Yu, T., Herzog, A., Pertsch, K., et al. Q-transformer: Scalable offline reinforcement learning via autoregressive q- functions. In Conference on Robot Learning, 2023.
Goldherg, D. F. and Deb, K. comparative analysis of
selection schemes used genetic algorithms. In Founda- tions of genetic algorithms. 1991.
Chen, J., Ma, Z., Guo, H., Ma, Y., Zhang, J., and Gong, Y.- J. SYMBOL: Generating flexible black-box optimizers through symbolic equation learning. In International Conference on Learning Representations, 2024.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 2023.
Gu, A., Goel, K., and Re, C. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De-

---

Meta-Black-Boxxx BoxpoiiiaoooOOtiiiaatiinn through Offline Q-function Learning
Guo, H., Ma, Y., Ma, Z., Chen, J., Zhang, X., Cao, Z., Zhang, J., and Gong, Y.-J. Deep reinforcement learning for dynamic algorithm selection: A proof-of-principte study on differential evolution. IEEE Transactions on Systems, Man, and Cyberneties: Systems, 2024a.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative q-learning for offiine reinforcement learning. In Advances in Neural Information Processing Systems, 2020.
Lange, R., Schaul, T., Chen, Y., Lu, C., Zahavy, T., Dal- ibard, V., and Flennerhag, S. Discovering attention-basel genetic algorithms via meta-black-bo optimization. In Proceedings of the Genetic and Evolutionary Computa- tion Conference, 2023a.
Guo, H., Ma, Z.,, Chen, J., Ma, Y., Cao, Z., Zhang, X., and Gong, Y.-J. Configx: Modular configuration for evolu- tionary algorithms via multitask reinforcement learning. arXiv preprint ar arXiv:2412.07500 2024b.
Guo, Q., Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G.. Bian, J., and Yang, Y. Connecting large language mod- els with evolutionary algorithms yields powerful prompt optimizers. In International Conference on Learning Representations, 2024c.
Lange, R. T., Schaul, T., Chen, Y., Zahavy, T., Dalibard, V., Lu, C.,, Singh, S., and Flennerhag. S. Discovering evolution strategies via meta-black-box optimization. In International Conference on Learning Representations, 2023b.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-poliqg maximum entropy deep reinforce- ment learning with stochastic actor. In International Conference on Machine Learning, 2018.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint 200v:2005.01443 2020.
Li, P., Hao, J., Tang, H., Fu, X., Zhen, Y., and Tang, K. Bridging evolutionary algorithms and reinforcement learning: A comprehensive survey on hybrid algorithms. IEEE Transactions on Evolutionary Computation, 2024a.
Halton, J. H. On the efficiency of certain quasi-randon sequences of points in evaluating multi-dimensional inte- grals. Numerische Mathematik, 1960.
Li, X., Wu, K., Li, Y. B., Zhang, X., Wang, H., and Liu, J. Pretrained optimization model for zero-shot black box optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b.
Hansen, N., Auger, A., Ros, R., Mersmann, O., Tušar, T., and Brockhoff, D. Coco: A platform for comparing con- tinuous optimizers in a black-box setting. Optimization Methods and Software, 2021.
Li, X., Wu, K., Zhang, X., and Wang, H. B2opt: Learning to optimize black-box optimization with little budget. In The 39th Annual AAAI Conference on Artificial Intelligence, 2024c.
Holland, H. Adaptation in natural and artificial systems: an introductory analysis with applications to biology, con- trol, and artificial intelligence. 1992.
Hong, J., Shen, B., and Pan, A. reinforcement learning- based neighborhood search operator for multi- op- timization and its applications. Expert Systems with Ap- plications, 2024.
Lian, H., Ma, Z., Guo, H., Huang, T., and Gong, Y.-.. Rlemmo: Evolutionary multimodal optimization assisted by deep reinforcement learning. In Proceedings the Ge- netic and Evolutionary Computation Conference, 2024.
Hu, S., Fan, Z., Huang. C., Shen, L., Zhang, Y., Wang, Y., and Tao, D. Q-value regularized transformer for offline reinforcement learning. arXiv preprint 170iv: 2024.
Liu, S., Gao, C., and Li, Y. Large language model agent for hyper param--ar optimization. arXiv preprint arXiv:2402.01 088 2024.
Liu, Y., Lu, H., Cheng, S., and Shi, Y. An adaptive online parameter control algorithm for particle swarm optimiza- tion based reinforcement learning. In IEEE Congress on Evolutionary Computation, 2019.
Janner, M., Li, Q., and Levine, S. Offline reinforcement learning as one big sequence modeling problem. In Ad- vances Neural Information Processing Systems, 2021.
Kadavy, T., Viktorin, A., Kazikova, A., Pluhacek, M., and Senkerik, R. Impact of boundary control methods on bound-constrained optimization benchmarking. Pro- ceedings of the Companion Conference on Genetic and Evolutionary Computation, 2023.
Ma, Z., Guo, H., Chen, J., Li, Z., Peng, G., Gong, Y.-J., Ma, Y., and Cao, Z. Metabox: A benchmark platform for meta-black-box optimization with reinforcement learning. In Advances in Neural Information Processing Systems, 2023.
Kennedy, and Eberhart, R. Particle swarm optimization. In Proceedings of ICNNN'9-Interraatii Internaatiiilll Conference on Neural Networks, 1995.
Ma, Z., Chen, J., Guo, H., and Gong, Y.J. Neu- ral exploratory landscape analysis. arXiv preprint arXiv:2408. 10672, 2024a.
10

---

Meta-Black--al Baank--Bx-Optimization through Offline Q-function Learning
Ma, Z.., Chen, J., Guo, H., Ma, Y., and Gong, Y.-J Auto- tradeoff in evolution-
Sun, J., Liu, X., Bäck, T., and Xu, Z. Learning adaptive differential evolution algorithm from optimization experi- ences by policy gradient. IEEE Transactions Evolu- tionary Computation, 2021.
configuring exploration-explootaieee ary computation via deep reinforcement learning. In Pro- ceedings of the Genetic and Evolutionary Computation Conference, 2024b.
Sutton, R. S. Reinforcement learning: An introduction. A Bradford Book, 2018.
Ma, Z., Guo, H., Chen, J., Peng, G., Cao, Z., Ma, Y., and Gong, Y.-J. Llamoco: Instruction tuning of large language models for optimization code generation. arXiv preprint wrXiv:2403.01333 131, 2024c.
Tan, Z. and Li, K. Differential evolution with mixed muta- tion strategy based on deep reinforcement learning. Ap- plied Soft Computing. 2021.
Ma, 7., Guo, H., Gong, Y.J.,, Zhang, J., and Tan, K. C. To- ward automated algorithm design: A survey and practical guide to meta-black black-box-oppeee -pttmiiiizztiin. arXiv preprint arXiv:24/1 1.04/1100625.. 2024d.
Tanabe, R. and Fukunaga, A. S. Improving the search per- formance of shade using linear population size reduction. In 2014 IEEE Congress on Evolutionary Computation, 2014.
Metz, L., Ibarz, J., Jaitly, N., and Davidson, J. Discrete sequential prediction of continuous actions for deep rl. arXiv preprint arXiv:
2017.
Todorov, E.. Erez, T., and Tassa, Y. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ in- ternational conference on intelligent robots and systems, 2012.
Mnih, V. Playing atari with deep reinforcement learning- arXiv preprint arXiv: 1312.5602, 2013.
Mohamed, A. W. Hadi, A. A., Mohamed, A. K., Agrawal, P., Kumar, A., and Suganthan, P. N. Problem definitions and evaluation criteria for the cec 2021 on single objective bound constrained numerical optimization. In Proceed- ings of the IEEE Congress of Evolutionary Computation, 2021.
Wang. J., Zheng, Y., Zhang. Z., Peng, H., and Wang, H. A novel multi-state reinforcement learning-basse based multi- objective evolutionary algorithm. Information Sciences, 2024.
Wang, R., Wu, Y., Salakhutdinov, R., and Kakade, S. Insta- bilities of offline rl with pre-trained neural representation. In International Conference on Machine Learning. 2021.
Ota. T. Decision mamba: Reinforcement learning via sequence modeling with selective state spaces. arXiv preprint 1XXv:2403.119255 2024.
Watkins, C. and Dayan, Q-learning. Machine learning. 1992.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint 1707.0664774 2017.
Wu, D. and Wang, G. G. Employing reinforcement learning enhance particle swarm optimization methods. Engi- neering Optimization, 2022.
Sharma, M., Komninos, A., López--däiez, M., and Kazakov, D. Deep reinforcement learning based parameter control in differential evolution. In Proceedings of the Genetic and Evolutionary Computation Conference, 2019.
Wu, K., X., Liu, P., and Liu, Deen: Evolution inspired deep convolution network for black-box optimization. arXiv preprint arXiv:2304.099:
Slowik, A. and Kwasnicka, H. Evolutionary algorithms and their applications to engineering problems. Neural Computing and Applications, 2020.
2023.
Xu, Y. and Pi, D. A reinforcement learning- based commu- nication topology in particle swarm optimization. Neural Computing and Applications, 2020.
Song, L., Gao, C., Xue, K., Wu, C., Li, D., Hao, J., Zhang, Z., and Qian, C. Reinforced in-context black- box opti- mization. arXiv preprint arXiv:2402. 17423, 2024.
Xue, K., Xu, J., Yuan, L., Li, M., Qian, C., Zhang, Z., and Yu, Y. Multi-agent dynamic algorithm configuration. In Advances in Neural Information Processing Systems, 2022.
Storn. and Price, K. Differential evolution-a simple and efficient heuristic for global optimization over continuous spaces. Journal of Global Optimization, 1 11:341, 1997.
Such, F. P., Madhavan, V., Conti, E., Lehman, J., Stanley, K. O., and Clune, J. Deep neuroevolution: Genetic algo- rithms are a competitive alternative for training deep neu- ral networks for reinforcement learning. arXiv preprint arXiv: /71. 2017.
Yamagata, T., Khalil, A., and Santos-Rodriguez, R. Q- learning decision transformer: Leveraging dynamic pro- gramming for conditional sequence modelling in offline rl. In International Conference on Machine Learning, 2023.

---

Meta- Meta-Black--oogg Box-otiiiziiiiiiin through Offline Q-function Learning
Yang, X., Wang, R., Li, K., and Ishibuchi, H. Meta-black- box optimization for evolutionary algorithms: Review and perspective. Swarm and Evolutionary Computation, 2025.
Yin, S., Liu, Y., Gong, G., Lu, H., and Li, W. Rlepso: Reinforcement learning based ensemble partiele swarm optimizer. In International Conference Algorithms, Computing and Artificial Intelligence, 2021.
Yu, X., Xu, P., Wang, F., and Wang, X. Reinforcement learning-based differential evolution algorithm for con- strained multi-objeccivve optimization problems. Engi- neering Applications of Artificial Intelligence, 2024.
Zhan, Z.-H., Shi, L., Tan, K. C., and Zhang, J. survey on evolutionary computation for complex continuous opti- mization. Artificial Intelligence Review, 2022.
Zhang, H., Sun, J., Bäck, T., and Xu, Z. Learning to select the recombination operator for derivative-frre optimiza- tion. Science China Mathematics, 2024.
Zhao, Q. Liu, T., Yan, B., Duan, Q. Yang. J., and Shi, Y. Automated metaheuristic algorithm design with au- toregressive learning. arXiv preprint arXiv:2405.0344 2024.
12

---

Meta-Black-Box-DD Blloiiicckooooooiiiitttiioon through Offline Q-function Learning
A. Proof of Q-function Decomposition
To show that transforming MDP into a per-action-dimension form still ensures optimization the original MDP, we show that optimizing the Q-function for each action dimension equivalent optimizing the Q-function for the full action. We omit the time step superscript & for the ease of presentation.
If we consider apply full action d1:K at the current state to transit to the next step state 8'. The Bellman update of the optimal Q-function could be written as:
max ((ai:kss) = max
R(s, a1.) + max ((axx)ss)
m
01K
= R(s, R(ss,aj.)) + max Q(a1x(ss))
(8)
where R(-) is the reward we get after executing the full action a1.K. Under the Q-function decompostion, the Bellman update of the optimal Q-function for each action dimension ag is:
max Q(ai)aaaaaaa... max a.
max (ast11ss,aju)
a:-1
max 214442250000111 oct2
max
max or-1
d,
R(s, + y maxxxxxxe Q(ails') 21
R(s, + y max max Q(az)s .1)
R(s,ai.)) + max (an.k(ss) ovk
Here the first two lines are the inverse maximization operation as described in Section 3.1, the fourth line is the Bellman update for the last action dimension. The last three lines also follow the inverse maximization operation. By comparing Eq. (8) and Eq. (9) we prove that optimizing the decomposed Q-function consistently optimizes the original full MDP.
B. Optimization State Design
The formulation of the optimization state features is described in Table 4. States 8/1~6) optimization problem property features which collectively represent the distributional features and the statistics of the objective values of the current candidate population. Specifically, state 81 represents the average distance between each pair of candidate solutions, indicating the overall dispersion level. State 82 represents the average distance between the best candidate solution the current population and the remaining solutions, providing insights into the convergence situation. State 83 represents the average distance between the best solution found so far and the remaining solutions, indicating the exploration-exploitaaio stage. State $4 represents the average difference between the best objective value found in the current population and the remaining solutions, and $5 represents the average difference when compared with the best objective value found far. State $6 represents the standard deviation of the objective values of the current candidates. Then, states (1,,8.9 collectively represent the time-stamp features of the current optimization progress. Among them, state denotes the current process, which can inform the framework about when to adopt appropriate strategies. States sg and sy are measures for the stagnation situation.
13

---

Meta-Black-Boxxe Booppiiaooo-piiiizaatinn through Offline Q-function Learning
Table Formulations of features.
States
Notes
Average distance between any pair of individuals in current population.
mean || - xylle 2i EX1
Average distance between each individual and the best individual in t-th generation.
$2
mean | - -x*1112 mE
Problem Property
Average distance between each individual and the best-so-far solution.
mean ||0 - x²112 2:EX:
Average objective value gap between each individual and the best-so-far solution.
mean (f(x)) f(x")) 21EX
Average objective value gap between each individual and the best individual in t-th generation.
mean(f(x:) f((x))))
mEXx
Standard deviation of the objective values of population in f-th generation, value cquals 0 denotes converged.
8
std (f(z)) x,
The potion of remaining generations, T denotes maximum generations for one run.
(T -)/T
Optimization Progress
st denotes how many generations the algorithm stagnates improving.
st/T
s/g
if f(x*.)) < f(x*)
Whether the algorithm finds better individual than the best-so-far solution.
otherwise
C. Action Discretization and Reconstruction
Given the M = 16 bins Q values Q for the i-th action, if the i-th hyper-parameter A; of the low-level algorithm is in continuous space, we first uniformly discretize the space into M bins: Á; = (At1, A12, , } where A₁, and ArM are the lower and upper bounds of the space. Then we use the action at obtained by af = arg max Qi, as an index and
assign the value of the i-th hyper- paper-parameter A; with A, A[aaf] If the hyper--pramene papammerr is in discrete space À with m, S M candidate choices, the action of is obtained by af arg max Qi and the value of the i-th hyper-parameterr is Ala,].
jE[L.m. After the value of all paramettsssssers are decided, the algorithm A takes a step of optimization with the and return the next state from the updated population.
D. Experiment Setup
D.1. Backend Algorithm Generalization
In this paper, randomly sample 3 algorithms with action space dimensions 10 and 16 from the algorithm construction space proposed in ConfigX (Guo et al., 2024b), which contains various operators with controllable parameters such as the mutation and crossover operators from DE (Storn & Price, 1997), PSO update rules (Kennedy & Eberhart, 1995), crossover and mutation operators from GA (Holland, 1992). Operators without controllable parameters such as selection and population reduction operators are also included. Then, to get an algorithm with n controllable actions, we keep randomly sampling algorithms from the algorithm construction space and eliminating the algorithms that are not meeting
14

---

Meta- BlootiiiaooooOtiiiiattioooo through Offline Q-function Learning
Algorithm 1 Pseudo code of AlgO
1: Input: Optimization problem f. optimization horizon T, Meta-level agent
2: Output: Optimal solution x² arg min f(x)..
3: Uniformly initialize a population X1 with shape N1 100 and evaluate it with problem fi
4: for t= to T' do
5: Receive the 3 action values a, = {F1, F2, Cr} from the agent .,;
6: Generate X by using DELcurrent-to-rand// (Eq. (10)) on Xx;
7: Apply Exponential crossover (Eq. (11)) X, and X; to get X7";
Clip the values beyond the search range in X"':
9: Calculate {(X();;
10: Compare f'(X.) and [(X""), select the better solutions to generate X.41;
11: end for
the requirement, until the algorithm with n controllable actions is obtained. The uncontrollable hypee-parameters of the algorithm such as the initial population sizes are randomly determined.
Alg0 (as shown in Algorithm 1) towrrenttttorand//ae I/exooental (Storn Price. 1997) with Linear Population Size Reduction (LPSR) (Tanabe & Fukunaga, 2014). The mutation operator DEJcurrent-to-rand/11 is formulated as:
2 = 24 + F1 - ..) + F2 - xy3)
(10)
where h-- are randomly chosen solutions and F1, F2 E [0, 1] are controllable parameters. The Exponential crossover operator is formulated as:
Z/.y if randyy Cr and k L <jsL+H
,..,.... Dim Dim
(11)
Fijj otherwise
where Dim is the solution dimension, I. (1,..., Dim} is random length, rand E [0, 1] mm is random vector, x²f is the trail solution generated by mutation operator and Cr E [0, 1] is controllable parameter. At the beginning, population X with size 100 is uniformly sampled and evaluated. In each optimization generation, given the parameters F1, F2, Cr from the meta-level agent, algorithm applies DE/current-te toourrentttt-radd/ mutation and Exponential crossover operator on the population to generate the trail solution population X²" An comparison is conducted between population X, and X where the better solutions are selected for the next generation population X Finally the worst solutions are removed from X in the LPSR process.
The second algorithm Alg1 (as shown in Algorithm 2) is hybrid algorithm comprising two sub-populations optimized by GA and DE respectively. The population is sampled in Halton sampling (Halton, 1960) and then divided into two cub-populations with sizes 50 and 200. The first GA sub-population uses the Multi-Point Crossover (MPX) (Holland, 1992) and Gaussian mutation (Holland, 1992) accompanying with the Roulette selection (Holland, 1992). MPX crossover is formulated as:
Frass if randy Cri T.ey otherwise
1/ =
.J 1, Dim
(12)
where randy E [0, 1] are random numbers, Cr₁ is a controllable parameter and æ,1 is a random solution. The sample method of 2r1 is also a controllable action Xrmpa which can be uniform sampling or sampling with fitness based ranking. The Gaussian mutation is written as:
x =N= - (ub (b))
(13)
where ub and lb are the upper and lower bounds of the search space and E [0, 1] is a controllable parameter. The mutated solution then bound controlled using a composite bound controlling operator which contains 5 bound controlling methods: "clip", "rand", "periodic", "reflect" and "halving" (Kadavy et al., 2023), the selection of the bounding methods a controllable parameter be, E [0, 4). Besides, the GA sub-population adopts the L.SSR technique from initial population size 50 the final size 10.
In the second DE sub- -oppuloul,,ion, DE/best/2 (Stom& & Price, 1997) mutation and binomial (Storn & Price, 1997) crossover are used. DE/best/2 is formulated as:
x = x + F1 (xr1 Xr2) + F2 (x+3 Tr4)
(14)
15

---

Meta-Black Bllomiiikkoooooiiiiitttiinon through Offline Q-function Learning
Algorithm 2 Pseudo code of Alg1
1: Input: Optimization problem f. optimization horizon T, Meta-level agent
2: Output: Optimal solution x² arg min f(x).
=gX
3: Initialize 2 sub- sub-populations {X1,1} } and { ) using Halton sampling with sizes 50 and 200;
4: Evaluate the sub-populations with problem f:
5: for ( t= to T do
6: Receive the 10 action values ar from the agent कς;
7: Generate X1,4+11 using MPX (Eq. (12)), Gaussian mutation (Eq. (13)) and Roulette selection on X1,6
8: Generate X2,441 using DE/best/2 mutation (Eq. (14)) and binomial crossover (Eq. (15));
9: for i = to 2 do
10: Replace the worst solution in X1,++ by the best solution in x+m,
11: end for
12: Apply LPSR on sub-population X1,441;
13:
end for
where zy. are randomly selected solutions, x" is the best solution, F1, F2 [0, 1] are controllable parameters.
The Binomial crossover uses similar process MPX but introduces randomly selected index jrand {1, Dim} to ensure the difference between the generated solution and the parent solution:
T if rand, Cra or j = jrand T otherwise
.j 1, Dim
(15)
where rand, are random numbers and Cr² E [0, 1] is the controllable parameter. The DE sub-population also contains the composite bound control method with controllable parameter be2-
Besides, both the two uub-populations employ the information sharing methods which will replace the worst solution in current sub-population X; with the best solution from Xem,, where i {1, 2} in this algorithm. The parameters em1, cm2 E {1 are two controllable parameters for the sharing operator in the two sub- uub-population, respectively. If the action decides to share with itself (cm; = i), the sharing is stopped.
In summary, the action space of Alg1 is {Cr1, X Xmmpz: o. be,, cmn, F1, Cr2, be2, cm2) with the shape of 10.
For Alg2 (as shown in Algorithm 3), the population sampled in Halton sampling (Halton, 1960) is divided into four sub-populations. The first sub- aub-population uses GA operators MPX (Holland, 1992) crossover formulated in Eq. (12) and Polynomial mutation (Dobnikar et al., 1999) accompanying with the Roulette selection (Holland, 1992). The Polynomial mutation is as follow:
1 ((uu) Fm - () - 1b), if S 0.5;
(16)
x + (1 (2 2u) Tf (ub x1)) if >> 0.5.
where 7/m E (1, a controllable parameter, E [0, 1] a random number, ub and lb are the upper and lower bound of the search range.
The second pobulaatttti uses SBX crossover (Deb al., 1995), Gaussian mutation (Holland, 1992) and Tournament selection (Goldberg & Deb, 1991):
(2u) Time 1, 0.5;
/ = 0.5 [[1
F 9)x; + (1 + ()),,), where /
(17)
IT if > 0.5.
where E {1,2,3} is controllable parameter and u E [0. is random number. Similar to MPX, SBX also uses an action Xrsta to select parent solutions T,1- The Gaussian mutation operator formulated in Eq. (13) has controllable parameter E [0, 1].
The third sub-population is DF/rrand/2/expooentia (Storn Price, 1997) where the DF/rand/2 mutation operator is:
x =x1
I,1 + Fla(xr2 - xra) + F22(2r4 -x5)
(18)
where x,. are randomly selected solutions and Fl3, F23 E [0, 1] are controllable parameters for the third sub- sub-population. The Exponential crossover formulated as Eq. (11) used in this sub-population with parameter Cra [0, 1].
16

---

Meta- Beta-Blacc-Box--O0 Blaock---oo-Opimiizaiiiin through Offline Q-function Learning
Algorithm 3 Pseudo code of Alg2
1: Input: Optimization problem f. optimization horizon T, Meta-level agent
2: Output: Optimal solution x² arg min f(x).
=Ex
3: Initialize 4 sub- sub-populations { 2,,,,,,,4 using Halton sampling with sizes f200, 100, 100, 100].
4: Evaluate the sub-populations with problem f:
5: for t= to T do
6: Receive the 16 action values ar from the agent ικ;
Generate X1,0+11 using MPX (Eq. (12)), Polynomial mutation (Eq. (16)) and Roulette selection on X1.
8: Generate X2,41 using SBX (Eq. (17)), Gaussian mutation (Eq. (13)) and Tournament selection on X2,1
9:
Generate X3.++1 using DE/rand/2 mutation (Eg. (18)), Exponential crossover (Eq. (11)) on Xa,e:
10:
Generate X4.++1 using DElcurrent-to-bes// mutation (Eq. (19)), Binomial crossover (Eq. (15)) on X4.65
11:
i to 4 do
12: Replace the worst solution in X1,441 by the best solution in Xem,,
13:
end for
14:
end for
The last sub-population is DE//urrenttto-bbs////iinnn (Storn & Price, 1997). The mutation operator with parameter
F14, F2, E [0, 1] is formulated as:
2 = + F F - -2y2)
-
(19)
where x² is the best performing solution in the sub-population. The Binomial crossover formulated in Eq. (15) contains controllable parameter Cr4 E [0. 1].
Besides, Alg2 conducts the controllable information sharing among the sub-populations where the worst solution in current sub-population .g replaced by the best solution from the target sub tobb-ooulatiin Xem.g. cm(1.23344) E (1, 4] are four actions indicating the target sub-population.
Given the 16 actions
Tm, Te, Xr ebee O, F13, Cr3, F',,, F21, CrA, em1, cm2,
cm3, cm4). Alg2 uses these parameters to configure the mutation and crossover operators and applies them on the 4 sub-populations. Then the information sharing is activated for better exploration. Finally, the next generation population is obtained through the population reduction processes.
17

---

MccaaBlack-Box-Oppiiiiaa through Offline Q-function Learning
D.2. Train-test split of BBOB Problems
As shown in Table 5, the BBOB testsuite (Hansen et al., 2021) contains 24 different optimization problems with diverse characteristics such unimodal multi-modal, separable or non-separable, high conditioning low conditioning. To maximize the problem diversity of the training problem set and hence empower the agent better generalization ability, choose the most diverse 16 problem instance for training, whose fitness landscapes in 2D scenario are shown in Figure 3. The rest instances are used as testing set whose 2D landscapes are shown in Figure 4. The dimensions of each problem instances in both training and testing set are randomly chosen from (5, 10, 503.
Table
Overview the BBOB testsuites.
Problem
Functions
Dimensions
fi
Sphere Function
50
f2
Ellipsoidal Function
Separable functions
f3
Rastrigin Function
5
f4
Rastee-asttiggn Function
10
55
Linear Slope
50
f6
Attractive Sector Function
5
Functions with low or moderate conditioning
f7
Step Ellipsoidal Function
20
88
Rosenbrock Function, original
10
fg
Rosenbrock Function, rotated
10
f10
Ellipsoidal Function
10
Functions with high conditioning and unimodal
f11
Discus Function
5
f12
Bent Cigar Function
50
f13
Sharp Ridge Function
10
f14
Different Powers Function
20
f15
Rastrigin Function (non nep-raeparable counterpart F3)
Multi-modas
f16
Weierstrass Function
20
functions
fi7
Schaffers F7 Function
50
with adequate
f18
Schaffers F7 Function, moderately ill-conditioned
50
global structure
f19
Composite
Function F8F2
10
120
Schwefel Function
20
Multi-modal functions with weak
f21
Gallagher's Gaussian 101-me Peaks Function
20
122
Gallagher's Gaussian 21-hi Peaks Function
10
f23
Katsuura Function
20
global structure
J21
Lunacek bi Rastrigin Function
20
Default search range: [-5, 5)
18

---

Meta-Black-Bar Baack--oxOOptmizattin through Offline Q-function Learning
(a)
(b) f4
(c) fn
(d) f12
(c) f1a
( fia
(g) fis
(h) fie
① J17
(i) fia
(k) fio
(l) f20
(m) fz1
(n) J22
(o) f23
(p) Jz
Figure 3. Fitness landscapes of functions in BBOB train set when dimension is set to
(a) fz
(b) fa
(c) fs
(d) fo
(e) fr
(1) fs
(g) fo
(h) fio
Figure 4. Fitness landscapes of functions in BBOB test set when dimension is set to 2.
19

---

Meta-Black-Bie Boalck-Box-Opiiiizatin through Offline Q-function Learning
E. Additional Experimental Results
E.1. Impact of Action Bin Numbers
As we described in Section 4.4, when the hyper-parameter to be controlled is continuous, the Q-function decomposition scheme have to discretize the continuous space into discrete action bins. The number of action bins determines the control grain. If
large number of action bins, the parameter controlling is fine-grainet while the action space is increased. If the action bin number small, the control grain is coarse but the network scale is smaller. In this section we investigate the impact of the action bin numbers on the performance. Concretely, we implement Q-Mamba agents with 16, 32, 64, 128, 256 and 512 bins. Their binary coding of the actions are represented in 10-bits, and the output dimensions of the Q-value head in these baselines are set to 16~5112 accordingly. We train these agents for controlling Algo on the 16 training BBOB problem instances and then test them on the 8 instance BBOB testing set. The boxplots the performance values of these baselines over 19 independent rums are presented Figure The results show that Q-Mamba is compatible with large action bins and fine- mne-grained controlling. However, increasing action bin numbers may not always lead better performance due to main reasons: the increased network scales and training difficulty. b) for BBO algorithm such evolutionary algorithms in this paper, their hyper-parameten often show low sensitivity on slight value changes. In this case, increasing number of action bins makes little effect on the target BBO algorithm. In conclusion, the setting of 16 action bins is an ideal choice balancing the control grain and training efficiency, which is used in all trainings of Q-Mamba in our experiments.
1.00
0.99
Rewards
0.98
0.97
16
32
64
128
256
5i2
Action Bins
Figure 5. The performances of Q-Mamba trained with different action bin granularities.
20


Reviews:
[{"review_id": "Xu8TXSpj0e", "reviewer": "Reviewer_yyGg", "summary": "The paper introduces Q-Mamba, an offline reinforcement learning framework for Meta-Black-Box Optimization, aimed at efficiently learning Dynamic Algorithm Configuration without online training. It decomposes the Q-function into sequential decisions, applies Conservative Q-Learning to address distribution shift, and uses the Mamba architecture for long-sequence learning. Results show that Q-Mamba performs comparably or better than existing MetaBBO methods.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate reviewer #yyGg for the thorough review and valuable comments. For the concerns raised above, we provide following point-to-point responses to address them.\n\n**[Performance improvement significance]** We would like to first clarify that **the seemingly small relative performance improvement** of Q-Mamba against the online baselines (RLPSO, LDE and GLEET) it learns from **is mainly due to the normalized metrics** we used for better presentation (as explained in the end of page 6, Section 5.1). Based on the above elaboration, the seemingly small relative performance improvement in Table **actually represents significant absolute performance improvement**. To demonstrate this, we additionally provide the absolute optimization performance comparison results in https://anonymous.4open.science/r/QMamba_review-C0CF/Numerical_results.md, where we provide 3 tables ($Alg0 \\sim Alg2$), each for the tested 8 problem instances in CoCo-BBOB testsuites, showing the average best found objective value and error bar across the same 19 independent runs. We hope this could clear the reviewer’s concern. We would add these results into the appendix if the paper was accepted and remind the readers in main text to check them. \n\n**[Decomposition necessity]** We provide comparison results on two additional BBO algorithm sampled from the same ConfigX (https://arxiv.org/abs/2412.07507) modular algorithm space, which hold 22 and 37 hyper-parameters respectively. The offline dataset preparation and the settings follow those in our experiments. Due to the 5000 characters limitation in rebuttal, we provide the results in https://anonymous.4open.science/r/QMamba_review-C0CF/Decomposition_necessity.md where the effectiveness of the decomposition scheme is further validated by the consistent performance superiority of Q-Mamba to both the non-decomposed online methods (RLPSO, LDE and GLEET) and offline methods. We would add these results into the paper if it was accepted and respectifully request the reviewer to check them.\n\n**[Order of action dimension]** We would like to clarify Q-Mamba does not have to address the order bias since we use the pre-order traversal on a legal algorithm structure (represent the algorithm structure as a workflow tree and traversal it in a depth-first way), which can naturally reduce the permutations of the operators in a BBO algorithm to a definitive ordered permutation. On the one hand, doing so could apparently reduce the training difficulty. On the other hand,  the auto-regressive learning in Q-Mamba could implicitly learns the semantic context of the algorithm structure, hence improving the potential generalization performance across various algorithm structures. We thanks the reviewers for this insightful comments and would add this elaboration in our paper to improve the clarity of the methodology.  \n\n**[Relation with CQL]** Let us make a quick explanation of the relation between the training objective in Eq. (5) and CQL. First, we adopt the original definition of CQL in Eq. (1) of  its paper (https://arxiv.org/pdf/2006.04779). We also align our training objective with the implemented version in Eq. (2) of the q-Transformer paper (https://arxiv.org/abs/2309.10150). In our decomposition setting, the former two cases in Eq. (5) apply OOD sampling for the policy evaluation. Specifically, we sample maximum q values of the next action step $\\max \\limits_{j} Q_{i+1,j}^t$ for the action dimension within the same time step yet before the last one, and $\\max \\limits_{j} Q_{1,j}^{t+1}$ for the last one dimension. Due to the potential distribution shift, such maximum sampling possibly varies with the demonstration we have collected, hence we use the third case in Eq. (5) to regularize the OOD actions, reducing their q-values to the minimal accumulated reward that could be achieved in the optimization process, which is 0 in this case. We respectifully request the reviewer to check the above explanation and hope this could address your concern.\n\n**[Random seeds]** We use three random seeds (1, 333, 9485) for the training of Q-Mamba and other baselines. We use 19 random seeds (100, 200,…,1900) for testing baselines on the CoCo-BBOB problem isntances. We use 10 random seeds (100,200,…,1000) for testing baselines on Neuroevolution problems.", "claims_and_evidence": "- Q-Mamba achieves competitive or superior performance to online MetaBBO methods while reducing training costs: \nQ-Mamba achieves comparable or slightly superior performance to online MetaBBO baselines such as RLPSO, LDE, and GLEET while reducing training costs. However, since the offline dataset is collected using these same methods, it is expected that Q-Mamba should at least match their performance. In offline RL the goal is to outperform the logged policies used for training. Looking at Table 1, the reported differences in performance metrics between Q-Mamba and the baselines are relatively small, raising the question of how meaningful these improvements are. It is unclear whether these small gains translate into a real advantage.\n\n- Q-function decomposition reduces learning complexity in high-dimensional configuration spaces: \nThe paper argues that decomposing the Q-function simplifies learning in high-dimensional configuration spaces. However, the experiments use action spaces with only 3, 10, and 16 dimensions, which are not particularly large compared to standard RL environments like Humanoid (17-dimensional) or Ant (8-dimensional). Given this, it is unclear whether decomposition is necessary in these cases.\nAdditionally, Q-Mamba predicts the action values sequentially, which raises concerns about whether the ordering of action dimensions affects learning and performance. If the sequence matters, it could introduce biases that the paper does not address. A more thorough evaluation would be needed to justify whether decomposition provides a real advantage, particularly in larger action spaces.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria seem to align with the Meta-Black-Box Optimization (MetaBBO) problem; however, I do not have specific expertise or background in MetaBBO to fully assess their appropriateness. The use of offline reinforcement learning makes sense given the inefficiency of online MetaBBO methods, and the CoCo BBOB benchmark suite appears to be a reasonable choice for evaluating optimization performance. The inclusion of both online (RLPSO, LDE, GLEET) and offline baselines (DT, DeMa, QDT, QT, Q-Transformer) provides a comprehensive comparison.\n\nHowever, one potential concern is that Q-Mamba is trained on data generated by RLPSO, LDE, and GLEET, yet it is later compared against them. The reported improvements are relatively small, raising questions about the significance of these gains.", "theoretical_claims": "This work makes no theoretical claims.", "experimental_designs_or_analyses": "I did. Please check previous sections.", "relation_to_broader_scientific_literature": "The paper's contributions relate to prior work in Meta-Black-Box Optimization (MetaBBO), offline reinforcement learning (RL), and sequence modeling. It builds on existing MetaBBO methods like RLPSO and GLEET, by introducing an offline learning framework to improve efficiency. In offline RL, it applies Conservative Q-Learning (CQL) but modifies the regularization term to constrain Q-values of unseen actions. The Q-function decomposition approach aligns with Q-Transformer’s autoregressive Q-learning. Finally, it adopts Mamba, a state-space model, as an alternative to Transformer-based architectures for long-sequence learning, though without direct empirical comparison to validate its advantage.", "essential_references_not_discussed": "I think the relevant literature is discussed, but I don’t have extensive knowledge about the field of MetaBBO.", "other_strengths_and_weaknesses": "Strengths:\n- The authors share the code.", "other_comments_or_suggestions": "None.", "questions_for_authors": "- Please check the previous concerns.\n- The conservatism in Equation 5 is not clear. How is it related to CQL? In CQL, out-of-distribution (OOD) actions are sampled, and their estimated Q-values are explicitly decreased to ensure that the learned policy avoids selecting them.\n- How many seeds were used?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cbyAljvsP5", "reviewer": "Reviewer_xiEG", "summary": "This paper provides an exploration on effectiveness of offline reinforcement learning in Meta-Black-Box-Optimization to address the training efficiency problem of the online learning paradigms in existing works. The authors transform the DAC task into long sequence decision process and apply a Q-function decomposition scheme with a conservative Q loss. They further use a Mamba-based neural network architecture as the RL agent for long sequence learning capability and efficient training. They conduct both in-distribution and out-of-distribution experiments to show the training efficiency and effectiveness on both in-distribution and out-of-distribution problem instances of their method.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the thorough and insightful review. We provide following point-to-point responses to address the concerns in your valuable comments.\n\n**[Performance metric]** We would like to first note that such nomalized metric has been widely used in recent works, e.g., SYMBOL (https://openreview.net/forum?id=vLJcd43U7a , ICLR 24), GLHF (https://openreview.net/forum?id=fWQhXdeuSG , NeurIPS 24) and ConfigX (https://arxiv.org/abs/2412.07507 , AAAI 25). In the rollout process, for each of the 8 tested problem instances, we apply each baseline including our Q-Mamba for controlling $Alg0 \\sim Alg2$ to optimize the given problem instances, across 19 independent runs. We would like to clarify that testing a baseline by different initial population is a common practice in BBO testing to measure the performance robustness. As we fix these 19 random seeds for different baselines, these baselines are tested under the same 19 initial conditions hence the fairness of our performance metric is ensured. Besides the normalization in the single step reward provides convenience for presenting average performance across different problem instances for our readers, since various problems show various objective scales. We hope the above explanation could address your concern.\n\n**[Numerical results]** Following your suggestion, we have provided the numerical results of in-distribution testing in https://anonymous.4open.science/r/QMamba_review-C0CF/Numerical_results.md. We provide 3 tables ($Alg0 \\sim Alg2$) for each of the 8 tested problems respectively, this is because if we do not normalize them as we did in the paper, these results can not be averaged into one table. We respectifully request the reviewr to check these results, which show that the actual performance superiorty of Q-Mamba is even more larger.\n\n**[Evidence of distribution shift]** We thank the reviewer for this valuable suggestion! Indeed, we should provide more direct evidence to validate there is indeed distribution shift in offline MetaBBO. To this end, we have additionally trained Q-Mamba models under no-CQL setting. In specific, we modify the training objective in Eq. (5) of our paper by removing the q-value regularization term on OOD actions (the thrid case). We provide the performance results in https://anonymous.4open.science/r/QMamba_review-C0CF/no-CQL_ablation.md. The results provide a clear evidence that, if we remove the q-value regularization, the distribution shift could significantly downgrade the performance of Q-Mamba.\n\n**[SOTA offline RLs in OOD]** Following the suggestion, we zero-shot two offline RL baselines (QDT ICML 23, QT ICML 24) to the four neuroevolution tasks we have tested and provide comprehensive comparison results in https://anonymous.4open.science/r/QMamba_review-C0CF/neuroevolution_zero_shot.md. The results there consistently reveal that offline RLs generally underperform online MetaBBO methods, which further underscores the significance of our Q-Mamba. We hope this could address your concern.\n\n**[Novelty]** We would like to argue that the only similarity of our work with Q-transformer is the q-function decomposition scheme. Q-Mamba differentiate with Q-transformer in the following aspects:\n\n1. **Target tasks**: we have to note that Q-Mamba represents pioneering efforts to explore the possibility of offline RL in MetaBBO tasks, which outperforms online baselines with significantly less training cost . In contrast, Q-transformer, as well as many other offline RLs are developed and examined for classic control tasks. \n2. **Special designs**: We have explored many customized design choices in this paper to make offline-RL adaptable for MetaBBO:\n    - **Neural network design**: we have combined strength of newly propsoed Mamba architecture into the q- decomposition scheme to boost MetaBBO long sequence tasks (the comparison of Q-transofrmer and Q-Mamba in Table 1 provides a validation).\n    - **Offline dataset construction**: we have proposed a novel dataset construction scheme (Section 4.2) which could collect rigorous exploration and exploitation experiences from diverse baselines. Further ablation study on the data ratio $\\mu$ (Section 5.2, Table 3) provide valuable insight of how to make good datasets for offline MetaBBO task.\n    - **Training objective redesign**: compared to the training objective in original CQL and Q-transformer, we additionally add a weight $\\beta$ (second case in Eq. (5)) to enhance the q-value learning in the last decomposed action dimensions since the q-value update of the other action dimentions before depends on the accuracy of the last one. The ablation study in Section 5.2 Table 2 demonstrate the effectiveness of such design.\n\nWe sincerely request the reviewer to review the above elaboration on the novelty of our work. We appreciate your efforts in reviewing our paper. Please feel free to discuss with us if any concern remains in the author-reviewer discussion period.", "claims_and_evidence": "1. The performance metric the authors use in the in-distribution experiment can be misleading, for it depicts the improvement ratio of the best objective value found in the last run to the best objective value in the initial population. If the initial population is not set the same, the performance metric cannot depict how good the final found best objective value is. \n\n2. Although the authors claim their contribution of applying conservative Q loss to address the distributional shift issue in offline RL (which is already applied in Q-transformer[1]), they do not provide any evidence of offline reinforcement learning in Meta Black-Box-Optimization suffering from this distributional shift issue.\n\n3. Although the authors conduct the out-of-distribution experiment to show the generalization performance of their method, they do not compare their method to the state-of-the-art offline RL methods. Moreover, being only compared to zero-shot performance of online Meta-BBO baseline cannot provide solid evidence of the claimed generalization capability of the proposed method, since it is possible that all the methods do poorly in generalization.\n\n[1] Chebotar, Yevgen et al. “Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions.” Conference on Robot Learning (2023).", "methods_and_evaluation_criteria": "Please see the above comments.", "theoretical_claims": "No theory", "experimental_designs_or_analyses": "Yes, I have checked the experiments.", "relation_to_broader_scientific_literature": "This paper provides an exploration on effectiveness of offline reinforcement learning in Meta-Black-Box-Optimization to address the training efficiency problem of the online learning paradigms in existing works. The proposed method is basically derived from Q-transformer[1], with the transformer architecture replaced by the proposed RNN-like Mamba architecture.", "essential_references_not_discussed": "Generally good", "other_strengths_and_weaknesses": "1. This paper provides an exploration on effectiveness of offline reinforcement learning in Meta-Black-Box-Optimization to address the training efficiency problem, which provides a foundation for future works.\n\n2. The authors make clear and detailed writing.\n\n3. There are thorough analyses on the experiment results, with their method compared to every single baseline.\n\n4. This paper does not show clear novelty in their method designing: They basically replace the transformer architecture in Q-transformer[1] with the RNN-like Mamba architecture (the Q-function decomposition scheme and the conservative Q loss are almost the same as those in Q-transformer[1]), with no significant improvement on performance and inferring time.", "other_comments_or_suggestions": "none", "questions_for_authors": "1. Can you provide more detailed information about the performance metric you use in the in-distribution experiments (eg how you set the initial population) and the reason you use it?\n\n2. Can you provide numerical results of the best objective function found finally in the in-distribution experiments?\n\n3. Can you provide SOTA offline RL method baselines in the out-of-distribution experiments?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nNhWtbIz9t", "reviewer": "Reviewer_6EuB", "summary": "This paper proposes a Mamba architecture-based meta-black-box optimization framework, Q-Mamba. By conducting offline reinforcement learning on demonstration dataset with diversified behaviours, Q-Mamba achieves competitive or superior performance and efficiency in dynamically configuring BBO algorithms for black-box optimization and Neuroevolution tasks.\n\n## Update after rebuttal\nI have carefully read the authors' rebuttal and the feedback has well addressed my questions and concerns. Thus, I would like to insist on the score of accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for such comprehensive review and insightful comments. We provide following point-to-point responses to address the concerns in “Experimental Designs Or Analyses”, “Other Strengths And Weaknesses”, “Other Comments Or Suggestions” and “Questions For Authors”.\n\n**[Train-test split]** We agree with the reviewer that a more uniform train-test split can further demonstrate the learning effectiveness of Q-Mamba. Following this suggestion, we have tried a uniform train-test split to compare Q-Mamba and other baselines. Due to the space limitation, we provide the results in https://anonymous.4open.science/r/QMamba_review-C0CF/train_test_split.md. The results there consistently validate the superior perforamence of Q-Mamba.\n\n**[Comparing Q-Mamba with the pre-trained MetaBBO method]** We would like to remind the reviewer that the online baselines we compared in Table 1 are exactly the ones we used to construct our E&E dataset.\n\n**[Problem dimension in OOD testing]** We used two-layer MLP structure (#state_dim, #hidden_dim=64, #action_dim) as the optimizees in these neuroevolution tasks. Since the #state_dim and #action_dim vary across the four Gym enviroments we used, the problem dimension for “InvertedDoublePendulum-v4” is 833, for “HalfCheetah-v4” is 1542, for “Pusher-v4” is 1991 and for “Ant-v4” is 2312. We would add these details into the paper if it was accepted.\n\n**[Impact of E&E dataset size]** Following the suggestion, we have randomly extracted 1K, 3K, 5K trajectories from our previously constructed E&E dataset (10K) to train and test Q-Mamba on $Alg0 \\sim Alg2$ respectively. We report the same performance metric as Table 1 of these Q-Mamba variants in https://anonymous.4open.science/r/QMamba_review-C0CF/Dataset_size_impact.md. We observe that, generally, if the dataset includes very narrow experiences data, the performance of Q-Mamba might suffers. A large scale pre-training could ensure the overall learning effectiveness.\n\n**[Training difficulty/cost introduced by the Q-decomposition]** We would like to explain for the reviewer that the training difficulty is significantly reduced by the Q-decomposition. As we elaborated in Section 4.4, line 237-253, applying policy learning on the massive associated configuration space of EAs is challenging, while decomposing this space makes simple q-learning possible, which is usually effective than policy gradient method. The training cost of Q-Mamba is further reduced by our proposed offline learning paradigm. \n\n**[Reason of faster inference]** We note that as we have decribed in Section 4.4, right col, line 248-265, we only used a single default mamba block, with input_dim as 14 (#state_dim=9 + #action_token_dim=5) and output_dim as 16 (the chosen discretization granularity in our paper). With this setting, the total learnable parameters are 5124. For online baselines, the MLP in RLPSO holds 6496 learnable parameters and the LSTM in LDE holds 6560 learnable parameters. This indicates Q-Mamba remains the same complexity. The faster inference comes from the hardware-aware design in Mamba, where it arranges the expanded states and its parameters in GPU SRAM rather than GPU HBM, such “on the fly” computation ensure an acceptable inference efficiency although the sequence is much longer. \n\n**[Numerical scale of state feature]** In Q-Mamba, when we compute state feature, we first min-max normalize the population positions by the searching range of the given problem, then min-max normalize the obejctive values within that population. This facilitate the generalization across various problems.\n\n**[Ablation of online setting Mamba architecture]** Following the suggestion, we have conducted two additional ablation studies:\n\n- First, we remove the thrid case in the training objective (Eq. (5), the CQL regularization in offline setting) and train Q-Mamba in online paradigm. we provide the results in https://anonymous.4open.science/r/QMamba_review-C0CF/Online_Mamba_ablation.md, which indicate that major difference between the online and offline learning of Q-Mamba is the training efficiency. This is due to that in offline setting, we can load the entire trajectory into GPU to facilitate efficient parallel scan of Mamba, hence significantly reducing the training cost.\n- Then we remove the mamba_block in Q-Mamba, leaving only the MLP q-value head, train a Q-Mamba variants with the same training setting. We provide the comparison results in https://anonymous.4open.science/r/QMamba_review-C0CF/Mamba_ablation.md. The results show that Mamba architecture contributes to the performance of Q-Mamba significantly. We would like to remind the reviewer that, in Table 1, the comparison of our Q-Mamba and Q-transformer also provide evidence that Mamba architecture is more suitable for long sequence tasks such as MetaBBO.", "claims_and_evidence": "This paper claims that decomposing Q-functions and introducing Mamba architecture would be more efficiency and effective. The comparison results on optimization performance and training/inferring time with online and offline baselines validate the claims on the framework. However, the effectiveness of introducing Mamba architecture and offline RL is not rigorously demonstrated.", "methods_and_evaluation_criteria": "This paper decomposes Q-functions for each action dimensions and introduces Mamba architecture for efficiency training. The offline RL training on diverse demonstration dataset ensures the effectiveness of the proposed framework. The accumulated rewards in the MDP act as the evaluation criteria, which makes sense.", "theoretical_claims": "In Appendix A, the authors show the proof of Q-function decomposition, which shows that optimizing the Q-function for each action dimension is equivalent to optimizing the Q-function for the full action.", "experimental_designs_or_analyses": "In Section 5.1. Experiment Setup, the authors split the CoCo BBOB Testsuites into 16 training problem instances and 8 testing problem instances.However, as shown in Appendix D.2, the complexity and difficulty of the training and testing functions are unbalanced. The testing set mostly contains unimodal and low conditioning functions, which may not evaluate well.\n\nIn Section 5.2 In-distribution Generalization, comparing Q-Mamba with the pre-trained MetaBBO methods constructing the E&E dataset may further validate the effectiveness of Q-Mamba.\n\nIn Section 5.3 Out-of-distribution Generalization, the problem dimensions of the Neuroevolution tasks are not claimed.\n\nIn Section 5.4 Ablation Study, the authors investigate the coefficient settings in Q-loss and the data ratio in E&E dataset. Besides these two, the impact of the size of the E&E dataset may also be worth exploring.", "relation_to_broader_scientific_literature": "1. Mamba Architecture: This paper adopts the Mamba architecture for decomposed Q-function decisions.\n\n2. Meta-Black-Box Optimization: This paper proposes a novel MetaBBO framework.\n\n3. Offline Reinforcement Learning: This paper integrates an offline RL method for model training.", "essential_references_not_discussed": "The discussion on related methods is sound, I didn’t see any essential references not discussed.", "other_strengths_and_weaknesses": "Weakness: \n\nDecomposing Q-functions and making decisions for each action dimensions leads to fine-grained algorithm configuration and prospective better performance, however, it also extends the trajectory lengths and may leads to higher training difficulty and time cost.", "other_comments_or_suggestions": "Comments: \n\nThe comparison with the pre-trained MetaBBO methods constructing the E&E dataset and the impact of the E&E dataset size may be worth exploring.", "questions_for_authors": "Q1: In Table 1 Q-Mamba demonstrates faster inference than MLP and LSTM based methods, since the architecture of Mamba may be more complex than MLP and LSTM, what is the reason of the faster inference?\n\nQ2: The optimization state design includes the distances between solutions and objective values, however, the search range and objective value scales of different functions may vary, how to deal with the scale variance in the states?\n\nQ3: It seems that Q-Mamba can also be used for online setting, how it performs on online setting?\n\nQ4: Can authors provide an additional ablation study that validate the effectiveness of the offline-learning and Mamba-architecture?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zkDWrH4Eo2", "reviewer": "Reviewer_TQqs", "summary": "This paper introduces Q-Mamba, a meta-black-box optimization framework integrates offline reinforcement learning and Mamba architecture to achieve effectiveness and efficiency. Q-Mamba is trained on 16 black-box optimization tasks to meta-learn an optimal algorithm configuration, demonstrating comparable or superior performance on black-box optimization tasks and neuroevolution tasks.\n\n## update after rebuttal\nAll my comments have been addressed. I am also leaning towards acceptance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for your valuable comments. We provide reponses as below to address your concerns.\n\n**[Advantages of Q-Mamba]** We would like to first clarify that the core motivation of Q-Mamba is providing an offline learning paradigm for MetaBBO domain, with at least comparable performance with online MetaBBO approaches and significantly reduced training cost. From Table 1 (in-distribution test) and Figure 2 (OOD test), we have validated such aspects. We would like to also argue that by using Mamba architecture to facilitate efficient parallel scan during the training, Q-Mamba actually improve the efficiency of Q-transformer by approximate 20% (13h vs 16h).  \n\n**[Train-test split]** We would like to first explain that the train-test split in experiments follows up-to-date MetaBBO methods, where the motivation of such split is to make the meta-level policy learn comprehensively across as many complex problem landscape as possible, hence ensuring a good generalization performance. Meanwhile, we agree with the reviewer that a more uniform train-test split can further demonstrate the learning effectiveness of Q-Mamba. Following this suggestion, we have tried a uniform train-test split (move f16, f17 and f21 from train set to test set, move f6, f8 and f9 from test set to train set) to compare Q-Mamba and other baselines. Due to the space limitation, we provide the results in https://anonymous.4open.science/r/QMamba_review-C0CF/train_test_split.md (due to the narrow rebuttal window, we only provide results on $Alg0$). The results there consistently validate the superior perforamence of Q-Mamba. \n\n**[Ablation on BBO algorithm without Q-Mamba pre-train]** We agree with the reviewer that the learning effectiveness could be further validated by such ablation. Following the suggestion, we have used the same 19 random seeds and $Alg0 \\sim Alg2$ to optimize the 8 tested problems. We provide the comparison results in https://anonymous.4open.science/r/QMamba_review-C0CF/Without_pre-train_ablation.md. The results above clearly demonstrate the learning effectiveness of Q-Mamba, which could boost the optimization performance of the backend BBO optimizer.\n\n**[Essential References]** For reference [1][2] the reviewer suggests, we would like to explain that the reason why we only list them as related works rather than compare them as basleines is that Q-Mamba aims at DAC tasks in BBO, while [1][2] explore representing BBO optimizer by NN. We hence chose RLPSO, GLEET and LDE, which are tailored for DAC tasks as baselines to construct the offline data and compare.  \n\n**[Self-adaption of Q-Mamba]** We would like to clarify that Q-Mamba inherits the adaption ability of MetaBBO methods. Specifically, as we illustrated in page 5, Figure 1, in each decision step of Q-Mamba, we let the model knows the current optimization status $s^t$, which informs it the optimziation problem and landascape information so that Q-Mamba could adaptovely adjust the hyper-parameters. We hope this could address your concern.\n\n**[Results on f16, f21 and f22]** Since the f16, f21 and f22 are now located at the training dataset in the paper, we have not tested them in in-distribution test in Section 5.2, Table 1. However, we could provide the testing results of the trained Q-Mamba on these training functions, which you can access at https://anonymous.4open.science/r/QMamba_review-C0CF/Performance_f16_f21_f22.md. The results there show that Q-Mamba could significantly boost $Alg0 \\sim Alg2$ on these three problem isntances with complex properties. \n\n**[The low-level BBO algorithm optimizes for only one generation on the problem per DAC process…]** This is a very interesting question and we admit that, currently, once Q-Mamba decides a configuration for the low-level optimizer, this configuration is only used for that generation. Q-Mamba need to decide again for the next generation. However, we would like to clarify that this “one config one step” paradigm align with traditional adaptive EAs where in each generation the hyper-parameters are adjusted. We denote this issue as an interesting future work.", "claims_and_evidence": "The authors demonstrate the efficiency of Q-Mamba by comparing its training/inferring time cost with other baselines. But this demonstration doesn’t rigorously show the efficiency comes from the Mamba architecture, even when compared to the structurally similar Q-Transformer.", "methods_and_evaluation_criteria": "The overall method of Q-Mamba framework is adopting offline reinforcement learning and Mamba architecture to ensure training efficiency, and Q-function decomposition scheme for learning effectiveness, which dose make sense.", "theoretical_claims": "The only proof in this submission is the proof of Q-function decomposition in Appendix A, which proved the consistency between optimizing the Q-function for each action dimension and optimizing the Q-function for the full action. I didn’t see any issues.", "experimental_designs_or_analyses": "1)\tExperiment Setup\nThe authors split the CoCo BBOB Testsuites into 16 training functions and 8 testing functions. According to Appendix D.2, the training functions are mostly multi-modal with high conditioning, while the testing functions are mostly unimodal with low conditioning, under such split, the performance of Q-Mamba may not be well evaluated.\n2)\tAblation Study\nThe performance differences between low-level BBO algorithms pre-trained with Q-Mamba and the same algorithms without such pre-training to show the effectiveness of Q-Mamba.", "relation_to_broader_scientific_literature": "1)\tMeta-Black-Box Optimization: The proposed Q-Mamba framework is an offline MetaBBO method.\n2)\tOffline Reinforcement Learning: Q-Mamba employs an offline reinforcement learning method at the meta-level.\n3)\tOver-Estimation Relieving: The conservative Q-learning loss is applied in Q-Mamba to mitigate the overestimation issue caused by distribution shift.\n4)\tQ-Function Decomposition: Q-Mamba adopts a Q-function decomposition strategy to enhance its learning effectiveness.\n5)\tMamba Architecture: Q-Mamba employs the Mamba architecture for long sequence learning.", "essential_references_not_discussed": "There are 2 online MetaBBO methods, both of which are trained/tested on CoCo BBOB Testsuites:\n[1] Lange, R. T., Schaul, T., Chen, Y., Zahavy, T., Dalibard, V., Lu, C., Singh, S., and Flennerhag, S. Discovering evolution strategies via meta-black-box optimization. In International Conference on Learning Representations, 2023b. \n[2] Li, X., Wu, K., Li, Y. B., Zhang, X., Wang, H., and Liu, J. Pretrained optimization model for zero-shot black box optimization. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b.", "other_strengths_and_weaknesses": "Weakness: Q-Mamba appears to lack the capability for hyperparameter self-adaptation, it cannot adaptively adjust the hyperparameters of the low-level BBO algorithm based on the optimization problem at hand.", "other_comments_or_suggestions": "It might be worth exploring other ways of train-test splitting on CoCo BBOB Testsuites.", "questions_for_authors": "Q1: Did the authors test Q-Mamba on black-box functions as complex as  f16, f21, f22 in CoCo BBOB Testsuites? If so, what were the results?\nQ2: According to paragraph Q-value head, subsection 4.4, in training phase, the low-level BBO algorithm optimizes for only one generation on the problem per DAC process. Is this sufficient to evaluate the current hyperparameter configuration?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a meta-black-box optimization algorithm based on off policy RL that combines several advancements, namely Q-value decomposition and Mamba. Experiments indicate that the algorithm achieves competitive results with the best on-policy approach with less computation. One weakness is that the experiments do not sufficiently show the value of each component of the algorithm through ablations, which I think can be addressed for the camera ready. Therefore, I would recommend weak accept.
Decision: Accept (poster)