Ensemble Mashups: A Simple Recipe For Better Bayesian Optimization

Venue: NeurIPS_cc_2024_Workshop_BDU
Published: 2024-09-04 21:03:52

Abstract: Bayesian optimization (BO) is a popular approach to optimizing costly, black-box functions that rely on a statistical surrogate model of the function, typically a Gaussian process (GP), and the so-called acquisition function (AF). Although the choices of the GP kernel and the AF can strongly affect the results, there does not exist an automatic way of selecting them. 
Ensembling, namely, using several, different kernels (Multi-Model) or AFs (Multi-AF) is one possibility for deriving a BO algorithm that is robust and safer.  These ideas have been considered separately in the past. In this work, we consider ensembles of both kernels and AFs (Multi-Model Multi-AF) and perform an empirical comparison to show their superiority with respect to single-ensemble algorithms.

Main paper: 
 introduction: 
 Bayesian optimization (BO) has become an important tool for black-box optimization and is widely used in fields such as machine learning, automated design, and complex system optimization [1; 2; 3]. It predicts the behavior of the objective function by positing a proxy model, usually a Gaussian process (GP) [4] and uses these predictions to guide the optimization process, greatly improving the efficiency of the algorithm. However, in BO, the choice of kernel function and acquisition function (AF) is crucial to the final optimization result. The choice of the kernel affects the smoothness and complexity of the model. The AF guides how to select new points for evaluation. If the AF cannot effectively explore-exploit the search space, it may fall into a local optimum or waste computing resources. Different choices will directly determine the performance and efficacy of the optimization process. Therefore, how to select and optimize these components has become an important issue in current research [2; 1] .Ensembling is a popular strategy in machine learning for leveraging multiple learning algorithms and improving the performance [5] . Indeed, the advantages of ensembling in BO are well-recognized, appearing in the majority of the best-performing algorithms that took part in the recent competition [6] . Ensembling is part of a recent federated BO [7] , which can be interpreted as using a mixture of GP models. Ensemble models have also gained popularity in transfer learning extensions to BO, where information from base tasks are combined via weighted rankings of individual predictions or via weighted combination of AFs [8; 9; 10] . Still, as noted in [6] , more studies are needed to understand ensembling in BO.In this work, we focus on ensembles of two important components in GP-based BO: kernels and AFs. The ensembling strategy that we consider consists of sampling one representative from a pool of candidates. Existing works have been limited to either different kernel functions or different AFs. For example, the GP-hedge algorithm [11] improves optimization performance by integrating several acquisition functions, but still uses a single pre-selected kernel. This algorithm is an instance of a Multi-AF (MA) method. At the same time, Bayesian Model Averaging (BMA) improves optimization results by integrating different kernel functions [12] , but does not adopt an ensembling strategy for AFs. This algorithm is an instance of a Multi-Model (MM) method. Although these methods have proved to be useful, their interaction has not been explored in the literature. To address these issues, we propose an MMMA (Multi-Model-Multi-AF) approach to BO. This approach fills the gaps in existing methods by integrating kernel functions and AFs at the same time. We conducted experiments on 16 benchmarks and showed empirical evidence that MMMA-BO provides a more robust approach to BO. Related Work. Some previous works that have considered using ensembles of AFs are [2; 11; 13; 14; 15; 16; 17] . "Ensembling" is a very broad term that can interpreted as (i) individually optimizing a portfolio of AFs and choosing one of the maximizers according to some criteria [11; 13] ; (ii) selecting and optimizing one AF from the portfolio [14; 15] ; (iii) jointly optimizing several AFs in a multi-objective fashion [16; 17] ; or (iv) integrating out the AF using an approximate fully Bayesian GP [2] . The use of an ensemble of models (kernels) can be also interpreted differently [2; 12; 18] .The fully Bayesian treatment of GP hyperparameters naturally leads to an ensemble of kernels, each one with a hyperparameter sample, drawn approximately from the posterior; this model in conjunction with, e.g., the expected improvement (EI) AF, leads to the "integrated" AF mentioned above [2] . On the other hand, a series of different kernels can be weighted according to their marginal likelihoods (i.e., the probability of the data given the model/kernel) using BMA weights; then, at each iteration, a kernel is selected according to its BMA weight [12; 18] . 
multi-model multi-acquisition function bayesian optimization: 
 Background. We aim to solve max x f (x) by sequentially acquiring a datasetD t = {X t , y t } = {(x i , y i )} t i=1 , where y i = f (x i ) + u i , u i ∼ N (0, σ 2 ). The function f (x) is assumed to be sampled from a GP, f (x) ∼ GP(µ(x), κ(x, x ′ )), where µ(x) = 0 and κ(x, x ′ ) is the kernel function. At each iteration, the function is queried atx t+1 = arg max x α(x; D t ), collecting y t+1 = f (x t+1 )+u t and updating D t+1 = D t ∪ {(x t , y t )}. The criterion α(x; D t ) is the AF which is computed using the posterior GP, f |D t ∼ GP(m t (x), s 2 t (x)), where m t (x) and s 2 t (x) are the posterior predictive mean and variance of f (x) at x [4] . Although the particular choices of κ and α determine the performance of the algorithm, they are usually kept fixed throughout the algorithm.Motivation. Consider a scenario where we want to implement a BO algorithm but have limited prior knowledge about the target function and insufficient expertise in selecting a suitable AF. The standard recommendations for the kernel κ and AF α are often the Matérn 5/2 kernel and Expected Improvement (EI), respectively. However, these defaults may not be "optimal" for our specific problem. Ideally, if we could experiment with different combinations of (κ, α), we would identify the best pair in hindsight. Furthermore, when applying BO across various problems, different combinations may yield better results. Unless one combination is clearly dominant, selecting the "best" pair remains challenging-even in hindsight. For example, Figure 1 illustrates the regret for several combinations of κ ∈ {Matérn-5/2, RBF} and α ∈ {EI, PI, UCB} for two different test functions.For either function, the simple regret after 100 iterations varies by roughly one order of magnitude depending on the choice of (κ, α). Notably, the best-performing configuration for the Rastrigin function, (RBF, UCB), is not among the top two configurations for the DixonPrice function.Let us consider M different kernels {κ m } M m=1 and N different AFs {α n } N n=1 . Our goal is to ensemble both kernels and AFs using two sets of normalized weights {w κ m } M m=1 and {w α n } N n=1 such that w κ m ≥ 0 and w α n ≥ 0 represent respectively the probability of choosing kernel κ m and AF α n at each iteration. By using multiple kernels and AFs simultaneously, we avoid making a potentially suboptimal choice at the outset. The MMMA-BO algorithm in 1 is essentially the same as standard BO but it involves sampling a pair-one kernel and one AF-at each iteration, rather than committing to a single combination throughout the optimization process.The sampling probabilities for each kernel and AF can either be fixed in advance or adaptively adjusted based on performance during the optimization process. A straightforward and effective fixed choice is to assign uniform weights, w κ m = 1 M and w α n = 1 N , corresponding to the random selection of a different BO algorithm at each iteration. However, a performance-guided approach offers a more dynamic alternative, where "rewards" are assigned to kernels and AFs based on their contribution to the optimization process. For kernels, setting the weights with BMA [12] , w κ m ∝ p(y t |κ m ), where p(y t |κ m ) is the GP marginal likelihood with optimized hyperparameters, makes sense from a theoretical perspective. This method allows for kernel selection based on their fit to the observed data. Nevertheless, in low-data regimes-typical in BO-this approach may not always be appropriate. For AFs, the w α n can be adjusted based on rewards reflecting each α n 's contribution to the optimization. For instance, we reward α n if it identifies a candidate with a function value greater than the current best [15] . A particular case of this strategy is the Hedge algorithm [11] ("Bandit" in the experiments). (i) Sample a kernel κ * from {κ m } M m=1 using the weights {w κ m } M m=1 6:(ii) Fit the Gaussian Process (GP) with kernel κ * to the dataset D end if 13: end for 14: Return: The best found point x bestQuestion: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: See Appendix B Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. will be specifically instructed to not penalize honesty concerning limitations. 
experiments: 
 The importance of ensembling can be demonstrated by contrasting MMMA against algorithms of varying degrees of ensembling at different levels (kernel or AF) and different strategies of ensembling. In this study, we considered a pool of kernels including RBF, Matérn 5/2, and Matérn 3/2, along with the set of AFs, EI, PI and UCB. These design choices were guided by their popularity and general applicability. For more details, see Table 2 in the appendix. The algorithms were evaluated on 16 test functions from Botorch's suite [19] (see Table 1 in the appendix). We ran 25 independent simulations where, at each simulation, we ran BO for 100 iterations.We consider 3 metrics to evaluate the algorithms: (1) the gap metric, (2) simple regret, and (3) cumulative regret. Each metric focuses on a different aspect of the optimizer's behavior. The gap metric measures improvement relative to the starting condition, simple regret quantifies the distance between the best-observed value and the global optimum at each iteration, and cumulative regret quantifies the rate at which regret accumulates over time.The gap metric. Figure 2-(a ) presents boxplots of the final gap metric, illustrating that increased ensembling leads to better results. This improvement is due to the optimizer having access to a broader selection of kernels and AFs. Notably, the baseline performs similarly to kernel-level ensembling, which can be attributed to the similarity among the kernels used in our study. However, when ensembling is extended to include AFs, there is a significant improvement due to the greater diversity among the AFs. Importantly, the results of MMMA are more concentrated to the right, indicating less variability and greater robustness in performance compared to other methods.Regret growth. Given a function class F , a common objective is to find an optimizer π which achieves sublinear rate of regret for all functions f ∈ F . Formally, ∀f ∈ F , the objective is to havelim t→∞ Rt(π,f ) t = 0, where R t is the cumulative regret at iteration t. We can make this objective stricter by imposing a functional form on the rate of regret. For example, ∀f ∈ F , R t (π, f ) ≤ Ct b , C > 0 and b < 1. A good optimizer has b ≪ 1. To determine the value of b, we fit a polynomial to the median cumulative regret using non-linear least squares. It is again interesting to note that ensembling at the AF level yields significant improvement, while ensembling at the kernel level provides marginal improvement.Random v/s Adaptive weights. We observe that random weights (in both AF and kernel space ) tend to perform better than adaptive weight selection. When the adaptive strategy is employed, the "best" performing AF/ kernel will dominate the other candidates and cause the ensemble to collapse to a single choice (see Figs. 3 4 ). Indeed, with these strategies, the ensemble collapses to an exploitative choice: in BMA, the rewards are assigned to kernels that fit better the current data; in bandit, an AF that suggests points with large function values will be preferred. Hence, these strategies do not reward exploration.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: The paper does not include any theoretical proofs.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
conclusions: 
 In this work, we described MMMA-BO, a framework for combining multiple kernels and acquisition functions in BO. The MMMA-BO algorithm involves sampling one kernel and one acquisition function at each iteration, rather than committing to a single combination throughout the optimization process. We conducted experiments on 16 benchmark functions and systematically compared them with existing baseline methods. The experimental results confirm that the MMMA approach provides more robust performance. In future work, we aim to explore better procedures for determining the ensemble weights and deriving theoretical bounds on the cumulative regret of MMMA-BO. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes]Justification: See Section 3 and Appendix section A Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
open access to data and code: 
 Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: Anonymous repository is linked in Appendix A Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: All experimental design choices and implementations are described in Section 3 and Appendix Section A. Exact details about execution are available in a README file inside the repository.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: All figures relating to the main results have results displayed alongside statistical significance. For example, Figures 1 shows the median simple regret in log scale along with the 25% and 75% quantiles.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: See Appendix A Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: The paper has no societal impact.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risk Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.• For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Design choices like AF and kernel candidates, the weighing parameter of UCB, number of restarts or raw samples of the optimizer, the test functions, are standard limitations of a BO problem. Additionally, as noted by [11] , when employing a bandit (or BMA) strategy, the selection often converges eventually to an exploitative strategy. This convergence behaviour can be seen in Figures 3 and 4 .There might be scenarios where converging to exploitative strategies is not beneficial, especially if majority of the function's landscape is not explored and the optimizer is stuck in a local minima.Furthermore, the current MMMA setup only support single-objective test functions and analytical AFs. Future work will primarily be geared towards exploring better selection strategies and degree of ensembling. We will also include a larger array of test functions, support for multi-objective test functions and Monte-Carlo base AFs. Providing theoretical bounds for the gap metric and rate of regret will also be explored. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: The research was conducted in accordance with the Code of Ethics Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 



Reviews: review: title: title: Review
main_review: main_review: The authors investigate Bayesian optimization where data are acquired using a mixture of acquisition functions applied to a mixture of kernels.

The experimental evaluation in the paper is quite thorough, with many benchmarks. The thoroughness of this evaluation shows that there is no supreme combination of kernels and acquisition functions that works on every problem.

The setup seems a bit unprincipled, especially for Bayesian optimization. A mixture of GPs with different kernels is not itself a Bayesian model average, since the components in a Bayesian model average must themselves be weighted based on a posterior belief on the choice of model. Similarly, acquisition functions are generally derived through a principled formulation of Bayesian decision theory, and a mixture of acquisition functions doesn't have such a formal basis.

The authors also do not elaborate on optimizing an ensemble of acquisition functions. While certain acquisition functions like EI/PI/UCB have simple closed-form expressions amenable to gradient-based optimization, other acquisition functions like KG or Thompson sampling have more complicated procedures for optimization. Can the proposed method handle these other acquisition functions?

With all that said, I think this work could benefit from discussion at the workshop and so I lean towards acceptance.
title: title: Ensemble Mashups: A Simple Recipe For Better Bayesian Optimization
main_review: main_review: ## Summary and Contributions

The paper "Ensemble Mashups: A Simple Recipe For Better Bayesian Optimization" introduces MMMA-BO (Multi-Model-Multi-Acquisition Function Bayesian Optimization), a novel framework that ensembles multiple Gaussian Process (GP) kernels and acquisition functions (AFs) to enhance the robustness and performance of Bayesian Optimization (BO). Traditional BO relies on a single kernel and AF, whose selection critically influences optimization outcomes. MMMA-BO addresses the lack of an automatic selection mechanism by sampling pairs of kernels and AFs from predefined pools at each iteration, thereby mitigating the risk of suboptimal choices. The authors conduct extensive experiments on 16 test functions from Botorch's suite, demonstrating that MMMA-BO outperforms single-level ensemble methods in terms of gap metrics, simple regret and cumulative regret. Additionally, the paper provides a thorough analysis of different ensemble strategies, including fixed uniform weights and performance-guided adaptive weights.


## Strengths

1. **Innovative Ensemble Approach**: The integration of both multiple kernels and acquisition functions in a unified ensemble framework is a significant advancement. By considering the interaction between kernels and AFs, MMMA-BO offers a more comprehensive solution compared to previous methods that only ensemble one component.
2. **Comprehensive Experimental Evaluation**: The authors evaluate MMMA-BO across a diverse set of 16 test functions, providing robust empirical evidence of its effectiveness. Running 25 independent simulations for each function adds credibility to the results.
3. **Clear Algorithmic Description**: Algorithm diagrams provides a clear and concise description of the MMMA-BO process, facilitating understanding and potential replication by other researchers.


## Weaknesses

1. **Limited Theoretical Analysis**: While the empirical results are promising, the paper lacks a theoretical foundation explaining why ensembling both kernels and AFs leads to improved performance. Providing theoretical guarantees or insights into the convergence properties of MMMA-BO would strengthen the contribution.
2. **Scalability Concerns**: Although the experiments are thorough, the paper does not address how MMMA-BO scales with an increasing number of kernels and AFs. Discussing computational complexity and scalability would provide a more comprehensive understanding of the method's applicability to high-dimensional or large-scale optimization problems.
3. **Ablation Studies Missing**: To isolate the impact of ensembling kernels versus AFs, ablation studies that systematically evaluate the contribution of each component would be beneficial. This would help in understanding whether the combined ensemble offers synergistic benefits or if most gains are attributable to one of the two components.