Title: Enhancing Zero-Shot Black-Box Optimization via Pretrained Models with Efficient Population Modeling, Interaction, and Stable Gradient Approximation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Meta Black-box Optimization

Black-box optimization

Pretrained Optimization Model

Neural Combinatorial Optimization
Abstract: Zero-shot optimization aims to achieve both generalization and performance gains on solving previously unseen black-box optimization problems over SOTA methods without task-specific tuning. Pre-trained optimization models (POMs) address this challenge by learning a general mapping from task features to optimization strategies, enabling direct deployment on new tasks.

In this paper, we identify three essential components that determine the effectiveness of POMs: (1) task feature modeling, which captures structural properties of optimization problems; (2) optimization strategy representation, which defines how new candidate solutions are generated; and (3) the feature-to-strategy mapping mechanism learned during pre-training. However, existing POMs often suffer from weak feature representations, rigid strategy modeling, and unstable training.

To address these limitations, we propose EPOM, an enhanced framework for pre-trained optimization. EPOM enriches task representations using a cross-attention-based tokenizer, improves strategy diversity through deformable attention, and stabilizes training by replacing non-differentiable operations with a differentiable crossover mechanism. Together, these enhancements yield better generalization, faster convergence, and more reliable performance in zero-shot black-box optimization.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes EPOM (Efficient Pretrained Optimization Models), a new architecture for zero-shot black-box optimization. It improves on previous POMs by introducing (1) richer population tokenization (LMM/LCM Tokenizers), (2) deformable attention for better adaptive interaction between individuals (Efficient LMM), and (3) Differentiable LCM for more stable training gradients. The method is extensively benchmarked on BBOB test functions and an RL locomotion task, achieving improved performance and convergence over state-of-the-art population-based optimization methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n- Addresses an important and challenging problem: efficient zero-shot black-box optimization.\n\n- Proposes a technically sound architecture with improvements over existing POM methods.\n\n- Uses attention mechanisms and improved tokenization to enhance performance.\n\n- Shows strong experimental results on BBOB benchmarks (d=30 and d=100).\n\n- Provides good ablation studies and analysis of module contributions.\n\nWeaknesses\n\n- The innovations are mostly architectural — no fundamentally new learning paradigm.\n\n- Experiments are limited to BBOB and a simple RL control task — lacks validation on more realistic scientific or engineering problems.\n\n- No comparison to recent LLM-based optimizers (though the authors argue they are weaker for this problem class).\n\nQUESTIONS:\n- Can you provide more justification for not comparing EPOM to LLM-based optimization approaches? Are there settings where such methods might be competitive?\n\n- Have you tested EPOM on real-world scientific or engineering problems (e.g., materials discovery, control design), beyond synthetic BBOB tasks?\n\n- What is the computational cost of EPOM during training and inference, compared to POM or other methods? Is the method practical for problems with extremely large dimensions?\n\n- Does EPOM require large-scale pretraining datasets? How would it perform in low-data regimes?\n\nLIMITATIONS:\nYes, included in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work presented in this paper, titled \"Enhancing Zero-Shot Black-Box Optimization...\", introduces a framework referred to as EPOM. This framework is designed to improve upon a prior method, Pre-trained Optimization Models (POMs), for the task of zero-shot black-box optimization. The authors identify three primary weaknesses in the existing POM framework—namely, insufficient population modeling, inefficient information exchange, and unstable gradient approximation—and propose a corresponding set of architectural solutions. These solutions involve: (1) a new tokenizer module to create richer embeddings of the solution population; (2) a mutation operator, which they call an \"Efficient LMM,\" based on deformable attention to facilitate adaptive interactions; and (3) a crossover operator, which they call a \"Differentiable LCM,\" that utilizes a weighted-sum mechanism to improve the stability of the meta-training process. The authors provide an empirical evaluation on the BBOB benchmark suite, where their proposed EPOM model demonstrates superior performance compared to the baseline POM and other established optimization algorithms.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe primary strength of this work lies in its empirical results. The experimental evaluation is extensive and appears to be robust, showcasing that the proposed EPOM system consistently outperforms a range of strong baselines, including the original POM architecture and classical methods like CMA-ES, on a challenging benchmark suite. The engineering effort is clearly substantial, and the final model is demonstrably effective at its stated goal.\n\nWeaknesses:\nDespite the strong empirical results, the paper is beset by significant weaknesses in its presentation, clarity, and conceptual positioning, which collectively form a significant barrier to its understanding and potential impact.\n\nClarity and Terminology: The writing, particularly in the Abstract and Introduction, is convoluted and appears almost intentionally gatekeeping. It is laden with bespoke jargon and acronyms that are not immediately defined, making the paper exceptionally difficult to penetrate for any reader not already deeply familiar with the specific baseline paper [7]. This lack of clarity is a major flaw.\n\nMisleading Acronyms: The use of the acronyms 'LMM' and 'LCM' for \"Learned Mutation Module\" and \"Learned Crossover Module\" is exceptionally poor practice and must be addressed. In the broader machine learning community, LMM almost universally stands for Large Multimodal Model, while LCM has been recently associated with Large Context Models or Latent Consistency Models. Defining these overloaded acronyms deep within the paper (Section 3.1.1) is insufficient. This choice is not merely confusing; it is actively misleading and suggests a lack of awareness of the wider field.\n\nConceptual Positioning: The work is presented as \"zero-shot optimization\" via \"Pre-trained Optimization Models,\" creating an artificial distinction from the well-established field of meta-learning. A reader well-versed in meta-learning is left to wonder why this new terminology is necessary when the described process—training a model on a distribution of tasks to generalize to new tasks—is the very definition of meta-learning. This failure to connect to, or even acknowledge, the foundational concepts of the field it operates in is a critical conceptual weakness.\n\nIn summary, while the engineering is sound, the poor writing, confusing terminology, and flawed conceptual framing make the work hard to digest and will severely limit its potential impact and its ability to be integrated into the broader discourse on meta-learning.\n\nQUESTIONS:\nMy current rating is 'Borderline Reject' due to the severe issues with clarity and conceptual framing. For the paper to reach an acceptance status, the authors must undertake a major revision to address the following points. A successful revision that thoroughly resolves these issues would be necessary to reconsider my evaluation.\n\nRewrite for Clarity and Accessibility: The paper must be substantially rewritten. All bespoke acronyms, especially the misleading 'LMM' and 'LCM', must be clearly defined upon their first use in the Abstract and Introduction, or preferably, replaced with less ambiguous terms. The convoluted sentence structures should be simplified to make the core ideas accessible to a general machine learning audience.\n\nExplicitly Position the Work within Meta-Learning: The paper must situate itself correctly within the meta-learning literature. The introduction and related work sections need to explicitly discuss how \"Pre-trained Optimization Models\" relate to and build upon decades of research in meta-learning. The authors should either justify their novel terminology or adopt the standard language of the field.\n\nClarify Architectural Contributions: Given the complexity of the system, a clearer ablation is needed. What is the single most impactful component? Is the performance gain primarily from the deformable attention mechanism, the tokenizer, or the differentiable crossover? A more fine-grained analysis would help distill the core engineering takeaway of the paper.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Efficient Pretrained Optimization Models (EPOM), a method for solving black-box optimization problems without task-specific training. EPOM builds off of prior Pretrained Optimization Models (POM) in a few ways: The addition of an LMM/LCM tokenizer which embeds the population into fixed-length tokens to allow the model to capture more landscape cues, a deformable attention module for the LMM, and replacing the simple gumbel softmax LCM with a learned module that has stable gradients. Evaluation on a suite of black box problems show better performance and convergence compared to prior methods.\n\nSTRENGTHS AND WEAKNESSES:\nThe overall quality is good, the plots are well made with nice error bars and figure 1 is clear. Figure 4 is a bit confusing with small text and overlapping lines, I think a simple table would work better, an also a better explanation of which function is being optimized.\n\nThe clarity seems fair. The methods section is very clear with the description of each component and loss/bbox functions. There are a few things that seem unclear. First, there seems to be some discrepancies with figures 6-9 in the appendix and table 2. For example, f4 with d=100 shows better performance for POM in table 2, but has EPOM doing better in figure 8 (d). Additionally, section 4.2 claims EPOM is especially good with d=100, but table 2 shows that it's more often worse than POM for d=100 (POM is better 13/24 tasks). Finally, the crossover weight doesn't deviate much from 0.5 but the text doesn't justify if this is significant.\n\nOverall the significance also seems good. EPOM seems to converge much better in many problems which is promising, however it adds much more complexity and possibly compute. I have some concerns with the results mentioned previously that make me unsure if conclusions drawn are accurate. \n\nThe originality is also good. There are many novel adjustments made to POM and they are justified with ablations. The performance improvements would certainly be of interest to the larger community.\n\nQUESTIONS:\n1. Could you clarify the results in table 2?\n\nThe paper makes multiple claims about how EPOM is especially good with higher dimensions, d=100, but table 2 seems to suggest that it's worse in this case. In fact, it does worse than POM more often than not for d=100. Furthermore, there seems to be a miss match between the plots in figures 6-9 of the appendix and table 2 mentioned previously. I lean reject with these issues outstanding, but I'm willing to increase my recommendation based on the authors response.\n\n2. How much deviation from a crossover weight of 0.5 is significant?\n\nFigure 5(c)(d) shows very little deviation in the crossover weight (0.49-0.52), but the text claims that this is significant. I would imagine replacing this with a static 0.5 would lead to very similar performance. Looking at the appendix, there do seem to be cases where the crossover weight has much more variance, but these examples weren't selected for figure 5. Is there a pattern we can draw for when the crossover weight is close to 0.5 vs when it is not?\n\n3. How does the runtime of EPOM compare to POM?\n\nThe appendix discusses the compute requirements, but doesn't say how this compares to POM. Does EPOM require much more or less compute than POM?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Efficient Pre-trained Optimization Models which embeds landscape cues into population tokens for richer modeling and seamless feature fusion, and introduces the deformable attention module to enable adaptive, diversity-promoting interactions. Experimental results demonstrate superior performance over both traditional BBO methods and existing pre-trained algorithms.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed method efficiently embeds the optimization population and generates the next population using neural networks without handcrafted optimization strategies. This significantly reduces reliance on human effort and expert knowledge in algorithm design.\n2. The proposed loss function could balance the population diversity and convergence which is essential for effective optimization.\n3. EPOM demonstrates strong generalization ability across both synthetic tasks and neuroevolution tasks, as well as problems of varying dimensions.\n\nWeaknesses:\n1. The related work could be richer. Surveys for meta-learned BBO algorithms [1][2] could enrich the paper. Recent meta-learned algorithm ConfigX [3] and LLM for optimization method LLaMEA [4] are also worth mentioning. Besides, including NeurELA [5] which shares the similar idea of using Attention to embed population would also benefit the paper. \n2. The shapes of some network weights depend on population size n, hindering EPOM's generalization across different population sizes.\n3. In the Tokenizer, the 2-dimensional $H^t$ is concatenated with the $d_T \\times d_{1V}$-dimensional $E^t$ which has significantly higher dimension than $H^t$. It might dilute the contribution of $H^t$.\n\n\nOther issues:\n1. The reference for Q-Mamba links to MetaBox instead of its intended source, it might be a mistake.\n2. Section 3.1.1 2) LCM appears to lack an equation that utilizes introduced variables. Additionally, the shape of $W_{2V}$ in line 180 is incompatible for multiplication with $G_t$. In Equation (3) $K_{1M}$ might need a transpose.\n3. In Algorithm 1, line 9, $V^t$ should be a parameter of LCM().\n\nSuggested references:\n\n[1] Ma, Zeyuan, et al. \"Toward automated algorithm design: A survey and practical guide to meta-black-box-optimization.\" IEEE Transactions on Evolutionary Computation (2025).\n\n[2] Yang, Xu, et al. \"Meta-Black-Box optimization for evolutionary algorithms: Review and perspective.\" Swarm and Evolutionary Computation 93 (2025): 101838.\n\n[3] Guo, Hongshu, et al. \"Configx: Modular configuration for evolutionary algorithms via multitask reinforcement learning.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 25. 2025.\n\n[4] van Stein, Niki, and Thomas Bäck. \"Llamea: A large language model evolutionary algorithm for automatically generating metaheuristics.\" IEEE Transactions on Evolutionary Computation (2024).\n\n[5] Ma, Zeyuan, et al. \"Neural exploratory landscape analysis for meta-black-box-optimization.\" The Thirteenth International Conference on Learning Representations. 2025.\n\nQUESTIONS:\n1. What is the meaning and value of $\\bar{d}$ in Section 3.2.2? How to shape and reshape the solutions?\n2. In Section 3.2.3, what is the meaning of prob(x)? How to use the top-p sampling to obtain the $n \\times k$ indices in $N_t$?\n3. It seems that $m_i$ is different across $x_i$, how to determine the value of $m_i$ and why is they different?\n4. How to calculate $S^t$ using $N_t$?\n5. Since solutions are directly tokenized, can EPOM generalize across problems with different search spaces?\n6. The population size n and $d_{out}$ determine the shape of some of the network weights, they should be known values before training and testing. Why do lines 181-183 suggest uncertainty between $d_{out}$ and n?\n7. What is EPOM’s time efficiency compared to traditional BBO methods and pre-trained methods?\n\nLIMITATIONS:\n1. Though promising in the generalization across problem types and dimensions, EPOM might not be generalizable across population sizes.\n2. EPOM significantly reduces the required expert knowledge in BBO optimization, however, it might introduce more computational complexity nd slow down the optimization.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of zero-shot optimization by identifying three limitations in existing Pretrained Optimization Models (POMs): (1) limited population modeling, (2) static information exchange, and (3) noisy gradient estimates. To tackle these challenges, the authors propose a new framework called Efficient Pretrained Optimization Models, composed of three components: LMM/LCM Tokenizer, Efficient LMM, and Differentiable LCM. Experimental evaluations are conducted to assess the effectiveness of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is generally well-written and easy to read at the sentence level.\n2. The identified challenges (e.g., noise, representation, and information flow) are relevant and important for improving zero-shot optimization performance.\n\nWeakness:\n\n1. Unclear Problem Definition and Background. The Introduction section lacks a clear and precise formulation of the research question. Readers unfamiliar with pretrained optimization may struggle to understand what exactly is being addressed. A more structured introduction that defines the zero-shot optimization task, its applications, and the role of POMs would significantly improve clarity.\n2. The motivation is unclear and unconvincing. The authors outlined three limitations of existing works in Pre-trained Optimization Models. However, the motivation for addressing the three limitations is not well-grounded. The current descriptions use abstract and general terms (e.g., \"distribution characteristics,\" \"convergence,\" \"generalizability,\" \"approximate gradients\") without clearly explaining their practical meaning or impact. For instance, the authors mentioned that \"POMs often fail to adequately capture the distribution characteristics of the population, which undermines the convergence and generalizability of the derived optimization strategies.\" In their statements, they typically used broad terms, while did not illustrate them in detail. Specifically, what specific distribution characteristics are important in modeling the population, and how do they affect optimization performance? Why should we capture these characteristics? Besides, \"The reliance on approximate gradients during training can destabilize the model, hindering its overall effectiveness.\" Why does relying on approximate gradients lead to model instability? Are there theoretical or empirical studies supporting this claim? Providing concrete examples, intuitive explanations, or visual illustrations would help clarify the rationale behind the design choices.\n3. Limited Novelty and Coherence. The issues addressed in this work (population modeling, information exchange, gradient estimation) have been studied individually in prior work, as the authors themselves acknowledge. While integrating these three aspects into a single framework could be valuable, the paper does not present strong evidence of novelty in either methodology or insight. Furthermore, the relationship among the three components is not well-articulated. The method appears as a collection of loosely related techniques rather than a cohesive, unified solution. The authors should elaborate on how these components interact and collectively contribute to solving zero-shot optimization.\n4. The experiments are a little hard to follow. For example, it is difficult to understand the performance of various methods from Table 2. It also hard to identify the effectiveness of the proposed method compared with existing methods.\n\nQUESTIONS:\nPlease refer to Weakness.\n\nLIMITATIONS:\nPlease refer to Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-05"}