Title: MOBO-OSD: Batch Multi-Objective Bayesian Optimization via Orthogonal Search Directions
Venue: NeurIPS 2025
Accepted: yes
Keywords: Bayesian optimization

multi-objective optimization

Gaussian process
Abstract: Bayesian Optimization (BO) is a powerful tool for optimizing expensive black-box objective functions. While extensive research has been conducted on the single-objective optimization problem, the multi-objective optimization problem remains challenging. In this paper, we propose MOBO-OSD, a multi-objective Bayesian Optimization algorithm designed to generate a diverse set of Pareto optimal solutions by solving multiple constrained optimization problems, referred to as MOBO-OSD subproblems, along orthogonal search directions (OSDs) defined with respect to an approximated convex hull of individual objective minima. By employing a well-distributed set of OSDs, MOBO-OSD ensures broad coverage of the objective space, enhancing both solution diversity and hypervolume performance. To further improve the density of the set of the Pareto optimal candidate solutions without requiring an excessive number of subproblems, we leverage a Pareto Front Estimation technique to generate additional solutions in the neighborhood of existing solutions. Additionally, MOBO-OSD supports batch optimization, enabling parallel function evaluations to accelerate the optimization process when resources are available. Through extensive experiments and analysis on a variety of synthetic and real-world benchmark functions with two to six objectives, we demonstrate that MOBO-OSD consistently outperform the state-of-the-art algorithms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose “Batch Multi-Objective Bayesian Optimization via Orthogonal Search Directions” (MOBO-OSD), a novel Bayesian optimization method for expensive, multi-objective black-box optimization problems. MOBO-OSD solves multiple constrained optimization problems along orthogonal search directions (OSDs) in order to obtain a final diverse set of Pareto optimal solutions. These multiple constrained problems are called MOBO-OSD subproblems. Three key strategies are used by MOBO-OSD to obtain good performance. First, to maintain a broad search region, MOBO-OSD approximates the Convex Hull of Individual Objective Minima (CHIM) using the best (ideal) and worst (nadir) observed points. Second, MOBO-OSD generates a well-distributed set of OSDs using the Riesz s-Energy method. Third, to further enhance density and diversity of solutions without requiring an excessive number of subproblems, MOBO-OSD uses Pareto Front Estimation (PFE) for local exploration around candidate solutions. The authors also extend MOBO-OSD to the batch optimization setting and use the Kriging Believer method of considering distinct exploration regions in order to enhance diversity in the selected batch. The authors provide extensive experiments on various synthetic and real-world benchmark tasks with between 2 and 6 objectives. Results demonstrate MOBO-OSD consistently outperforms SOTA baseline methods in both the sequential and batch optimization settings.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: \nThe work is high quality. Figures and text are well polished. Ablation studies are extensive and provide useful insight. \n\nClarity: \nThe work is very well written and the author's method and motivation are clearly explained. \n\nSignificance: \nThis work is significant largely due to the strength of the empirical results. The proposed MOBO-OSD clearly outperforms SOTA baselines in the literature across tasks. This is further strengthened by the inclusion of the 4 real-world benchmark problems. Since multi-objective black-box optimization is particularly relevant and applicable to many areas of the broader research community, this work will have significant impact. The primary weakness of the method is that it cannot be readily applied to settings with noisy observations and this is the case in many real-world settings. However, the authors acknowledge this limitation and I think that this is a very minor weakness of the work overall. \n\nOriginality: \nThis work is clearly novel since it prevents a novel strategy to handle the problem of finding a diverse set of pareto-optimal solutions for multi-objective black-box optimization problems.\n\nQUESTIONS:\nQuestion 1: \nMOBO-OSD approximates the CHIM with an ideal point and a nadir point to define boundries. Can the authors elaborate on the choice of using the nadir point (the worst observed value for each objective)? How does this specific choice encourage a \"broader search region\"? \n\n\nQuestion 2: \nThe authors provide an ablation study showing that MOBO-OSD is robust to the number of OSDs, largely due to the PFE component. What are the practical trade-offs a practitioner should consider when applying MOBO-OSD in real-world problem settings. In particular, when deciding between using PFE with fewer OSDs, versus using more OSDs possibly without PFE, how might this impact computational cost and diversity/performance of the final solutions obtained? \n\n\nQuestion 3: \nThe authors cite their focus on settings with noiseless observations as a limitation of MOBO-OSD. What do you foresee as the primary challenges in adapting the proposed MOBO-OSD to handle noisy observations effectively?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTheir method is based on NBI, which computes the CHIM (convex hull of the individual minima) and works well when there is plenty of budget. Since MOBO problems have limited budget, they propose a technique to efficiently approximate the CHIM. Starting from this approximated CHIM, they generate orthogonal search directions (OSDs) evenly distributed, defined as constrained single-objective subproblems in which they look for the intersection points (candidates) with the Pareto front generated by the GPs. After solving these subproblems, they employ the Pareto Front Estimation (PFE) technique to increase the number of candidates (instead of solving more subproblems). Then, for each candidate they calculate its HVI, and if batch > 1 they use Kriging Believer to choose the rest of the batch candidates one by one, ensuring diversity among candidates. Finally, they claim that their method performs as well as or better than the state-of-the-art and that it can scale to any number of objectives in both sequential and batch evaluations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n - Their explanatory figures help to understand their method.\n\n - They have run several benchmarks from the literature to test their method in both sequential and batch scenarios.\n\n - Their method outperforms all other methods on all the benchmarks they present.\n\nWeaknesses:\n\n - Their method only handles problems without noise.\n\n - They mention JES [1] and PFES [2] in the \"Related Work\" section but they do not include them in the experiments. While these methods can be somewhat costly, I think it is important to include them for M <= 4 in the sequential, non-batch, problems.\n\n- The greedy selection for the batch of points to evaluate is not the best option. Although the Hypervolume Subset Selection Problem is #P-hard, there are better alternatives [3].\n\n[1] Ben Tu, Axel Gandy, Nikolas Kantas, and Behrang Shafei. Joint entropy search for multi- objective bayesian optimization. Advances in Neural Information Processing Systems, 35: 9922–9938, 2022.\n\n[2] Shinya Suzuki, Shion Takeno, Tomoyuki Tamura, Kazuki Shitara, and Masayuki Karasuyama. Multi-objective bayesian optimization using pareto-frontier entropy. In International conference on machine learning, pages 9279–9288. PMLR, 2020.\n\n[3] Guerreiro, Andreia P., Fonseca, Carlos M., & Paquete, Luís (2021). The Hypervolume Indicator: Computational Problems and Algorithms. ACM Computing Surveys (CSUR), 54(6), 1–42.\n\nQUESTIONS:\nHave you considered using IGD [4], IGD^+ [5], and the ε-indicator [6] alongside HVI for batch selection? HVI maximizes dominated volume, IGD/IGD^+ ensure uniform coverage, and the ε-indicator guarantees worst-case bounds. What are your thoughts?\n\n[4] Coello, C. A. C., & Lamont, G. B. (2004). Applications of multi-objective evolutionary algorithms (Vol. 1). World Scientific.\n\n[5] Ishibuchi, H., Masuda, H., Tanigaki, Y., & Nojima, Y. (2015). Modified distance calculation in generational distance and inverted generational distance. In Evolutionary Multi-Criterion Optimization: 8th International Conference, EMO 2015, Guimarães, Portugal, March 29--April 1, 2015. Proceedings, Part II 8 (pp. 110-125). Springer International Publishing.\n\n[6] Zitzler, E., Thiele, L., Laumanns, M., Fonseca, C. M., & Da Fonseca, V. G. (2003). Performance assessment of multiobjective optimizers: An analysis and review. IEEE Transactions on evolutionary computation, 7(2), 117-132.\n\n--\n\nThese sentences sound strange:\n\n - Many existing approaches rely on scalarization techniques [5, 8, 36], which often fail to capture a diverse Pareto front, leading to suboptimal **performance in performance** such as hypervolume.\n\n- In order to generate a **well-distributed** set of OSDs, it is essential to construct a **well-distributed** set of U(β) points. This is equivalent to generate a **well-distributed** set of {β} over a M -dimensional unit simplex, as U(β), by definition, is the linear transformation of β from a unit simplex to the approximated CHIM.\n\nLIMITATIONS:\nI have described the limitations I found in the Weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel algorithm for multi-objective Bayesian optimization (MOBO) that also supports batched evaluations. MOBO is the problem setting in which one aims to optimize multiple, potentially competing objectives. Therefore, instead of searching for solutions that minimize all objectives, one aims to find Pareto optimal solutions, that is, solutions where one cannot find a better solution for an objective without worsening another objective.\n\nSeveral algorithms for MOBO have been proposed, but they are often either not scalable in the number of objectives or suffer from high computational cost. \n\nThis paper aims to fill this gap by proposing an algorithm that is more scalable in the number of objectives and does not exhibit high computational cost. To this end, it approximates the convex hull of individual object minima (CHIM) to define search directions orthogonal to the CHIM. These orthogonal search directions are then used to find Pareto-optimal points, each of which is further augmented to find a dense set of points close to the Pareto front. \n\nThese points then serve as candidates for the next Bayesian optimization iteration. Batch BO is realized by conditioning the GP on previously selected points, using the posterior mean as a hallucinated outcome (Kriging believer).\n\nThe proposed method is evaluated on a wide range of benchmarks and shows promising performance.\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths__\n\n* The paper is excellently written, and the presentation of the relatively complex algorithm is clear.\n* The algorithm outperforms the competitors throughout the bank.\n* The paper has high-quality figures that help understand the algorithm.\n\n__Weaknesses__\n\n* The methodology for batching seems disconnected from the paper and is orthogonal to the multi-objective design.\n* Not all runs in the empirical evaluation have converged or are close to convergence, so it's not always clear if MOBO-OSD actually beats the other methods in the end.\n\n__Minor Comments__\n\n* Typo in the Figure in Section 4.1 ('Inidividual')\n* The small Figures in 4.1 and 4.2 also deserve a number and a caption.\n\nQUESTIONS:\n* How many objectives can MOBO-OSD handle? Can you construct (synthetic) benchmarks with arbitrarily many objectives?\n* What's the significance of the batching? Can't the Kriging believer be combined with other MOBO AFs as well?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MOBO-OSD, a novel algorithm for batch multi-objective Bayesian optimization (MOBO). The core idea is to adapt the Normal Boundary Intersection (NBI) method to the limited-budget BO setting. The algorithm works by first defining an approximated convex hull of individual objective minima (CHIM). It then generates a set of well-distributed Orthogonal Search Directions (OSDs) and solves a series of constrained single-objective subproblems along these directions to identify candidate solutions on the Pareto front. To enhance solution density, the method incorporates a Pareto Front Estimation (PFE) technique to explore the local neighborhood of these candidates. Finally, a batch selection strategy using the Hypervolume Improvement acquisition function, Kriging Believer, and a diversity-promoting mechanism is employed to select points for parallel evaluation. Extensive experiments show that MOBO-OSD consistently outperforms state-of-the-art baselines on a variety of problems.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- Novel Geometric Approach for Diversity: The paper's main strength is its novel adaptation of the NBI method to MOBO. By defining search directions geometrically (the OSDs), the algorithm has a structured and principled mechanism for generating a diverse set of candidate solutions, directly addressing a key challenge in MOBO.\n- Effective Combination of Techniques: MOBO-OSD intelligently combines multiple powerful techniques. The use of Riesz s-Energy to generate well-distributed OSDs, the PFE technique to enrich the candidate pool without solving an excessive number of subproblems, and a thoughtful batch selection strategy all contribute to a robust and effective algorithm.\n- Strong and Comprehensive Empirical Validation: The authors conduct extensive experiments on nine benchmark problems (with two to six objectives) against seven state-of-the-art baselines in both sequential and batch settings. The results consistently demonstrate the superiority of MOBO-OSD in terms of hypervolume difference, which is a strong piece of evidence for the method's effectiveness.\n- Designed for Batch Optimization: The algorithm is designed from the ground up to support batch optimization. The batch selection strategy, which combines Kriging Believer with a mechanism to ensure candidates are drawn from different exploration zones, is a practical and well-reasoned approach to accelerate the optimization process with parallel resources.\n\n**Weakness**\n-  Incomplete Contextualization within the MOBO Landscape: The paper's positioning could be improved with a broader discussion of the existing MOBO literature.\n      \n      - Conceptual Overlap with Scalarization: The core mechanism involves solving multiple single-objective subproblems. While the generation of these subproblems is geometrically motivated by OSDs, the paper could provide a deeper discussion on how this fundamentally differs from advanced scalarization techniques that also generate diverse weight vectors for batch optimization.\n      -  Omission of related MOBO baseline: The literature review focuses entirely on methods that approximate the full Pareto front but omits the following work: Zuluaga, Marcela, Andreas Krause, and Markus Püschel. \"e-pal: An active learning approach to the multi-objective optimization problem.\" Journal of Machine Learning Research 17, no. 104 (2016): 1-32.\n\n- Limited Discussion of Related Single-Objective Methods: The paper misses an opportunity to connect its directional search approach to similar ideas in high-dimensional single-objective BO. Methods like those by Kirschner et al. (2019) and Ngo et al. (2025) also leverage 1D line searches to make complex problems tractable. Discussing these parallels would better situate MOBO-OSD within the broader BO literature.\n- Algorithmic Complexity and Hyperparameters: The complete MOBO-OSD algorithm is a multi-stage pipeline with several components and associated hyperparameters . While an ablation study on $n_\\beta$ is provided, a more comprehensive analysis of the method's complexity and sensitivity to these different moving parts would be beneficial.\n\nQUESTIONS:\n1. Could you further elaborate on the fundamental advantages of the OSD-based subproblem formulation compared to a well-designed scalarization method that uses a similarly diverse set of weight vectors for batch selection? While empirically superior, the conceptual distinction could be clarified.\n\n2. Your approach of searching along 1D Orthogonal Search Directions in the objective space is conceptually similar to methods that use 1D subspaces for high-dimensional single-objective BO (e.g., Kirschner et al., 2019; BOIDS). Have you considered this connection, and do you see any potential for cross-pollination of ideas between these domains?\n\n**References**\n- Ngo, Lam, Huong Ha, Jeffrey Chan, and Hongyu Zhang. \"BOIDS: High-Dimensional Bayesian Optimization via Incumbent-Guided Direction Lines and Subspace Embeddings.\" In Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 18, pp. 19659-19667. 2025.\n- Kirschner, Johannes, Mojmir Mutny, Nicole Hiller, Rasmus Ischebeck, and Andreas Krause. \"Adaptive and safe Bayesian optimization in high dimensions via one-dimensional subspaces.\" In International Conference on Machine Learning, pp. 3429-3438. PMLR, 2019.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}