Title: Omnipresent Yet Overlooked: Heat Kernels in Combinatorial Bayesian Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Bayesian Optimization

Combinatorial Spaces

Heat Kernels
Abstract: Bayesian Optimization (BO) has the potential to solve various combinatorial tasks, ranging from materials science to neural architecture search. However, BO requires specialized kernels to effectively model combinatorial domains. Recent efforts have introduced several combinatorial kernels, but the relationships among them are not well understood. To bridge this gap, we develop a unifying framework based on heat kernels, which we derive in a systematic way and express as simple closed-form expressions. Using this framework, we prove that many successful combinatorial kernels are either related or equivalent to heat kernels, and validate this theoretical claim in our experiments. Moreover, our analysis confirms and extends the results presented in Bounce: certain algorithms' performance decreases substantially when the unknown optima of the function do not have a certain structure. In contrast, heat kernels are not sensitive to the location of the optima. Lastly, we show that a fast and simple pipeline, relying on heat kernels, is able to achieve state-of-the-art results, matching or even outperforming certain slow or complex algorithms.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper mainly presents an analysis to unify different kernels used commonly in categorical BO. The paper states that these common kernels, e.g. kernels in CASMOPOLITAN, COMBO, Bounce, CoCaBO, etc., are either heat kernels (diffusion kernels), or parts of a certain generalized class containing heat kernels. Notable claims include: (1) CASMOPOLITAN and COMBO are both fundamentally heat kernels, (2) CASMOPOLITAN and Bounce come from the class of Hamming-based heat kernel (3) COMBO comes from the class of graph-based heat kernel, (4) Hamming-based and graph-based kernels are equivalent under a non-trivial assumption, (5) heat kernels can be extended to relate to CoCaBO and RDUCB kernels. All claims are supported with theoretical analysis. Experiments demonstrate the equivalence of certain kernels, and show that a simple heat kernel pipeline can have equivalent performance to the state-of-the-art baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n-\tThe paper proposes a unifying framework for many common kernels in categorical BO, which is very interesting and important to understand this field of BO.\n-\tThe paper is well written and easy to follow.\n-\tThere are many theoretical results that clearly state the assumptions and drawbacks.\n-\tExperimental results provide interesting insights on the equivalence of many kernels.\n\nWeaknesses:\n\n-\tI think the authors could leverage the ideas and propose more effective kernels that can empirically outperform the state-of-the-art baselines. See my questions. \n-\tI do not think no-ordinal variables is a standard in combinatorial BO papers (line 350). In my opinion, many recent papers either treat ordinal variables as categorical or avoid addressing them due to the additional complexity involved, especially when their methods already demonstrate strong performance on categorical inputs. Hence the ordinal variables have received less attention. In fact, many real-world tasks require the ordinal variables, notably in HPO (number of trees or max depth in random forests, number of layers and units in neural networks, etc.). Examples in other fields include [1-3], which have all been used in BO works [4-5].\n-\tThe empirical performance is only on par with the state-of-the-art, slightly reducing the contribution of the work.\n\n[1] Kohira, Takehisa, et al. \"Proposal of benchmark problem based on real-world car structure design optimization.\" Proceedings of the Genetic and Evolutionary Computation Conference Companion. 2018.\n\n[2] Dreifuerst, Ryan M., et al. \"Optimizing coverage and capacity in cellular networks using machine learning.\" ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2021.\n\n[3] Wang, Boqian, et al. \"Harnessing a novel machine-learning-assisted evolutionary algorithm to co-optimize three characteristics of an electrospun oil sorbent.\" ACS Applied Materials & Interfaces 12.38 (2020): 42842-42849.\n\n[4] Daulton, Samuel, et al. \"Bayesian optimization over discrete and mixed spaces via probabilistic reparameterization.\" Advances in Neural Information Processing Systems 35 (2022): 12760-12774.\n\n[5] Daulton, Samuel, et al. \"Multi-objective bayesian optimization over high-dimensional search spaces.\" Uncertainty in Artificial Intelligence. PMLR, 2022.\n\nQUESTIONS:\n-\tGiven the simplicity of heat kernel pipeline, can the authors implement the discussed new ideas in Hamming-based approach in Sec. 3.4.1(via rational quadratic kernel), and similarly for Graph-based kernels in Sec. 3.4.2 (via graph Matern kernel and sum-of-inverse polynomial kernel)?\n-\tIs there any way to make heat kernel enjoy the same benefit of BODi and BOSS kernels such that the performance is also great in problems with special structure? It would be much better if a kernel can work well in both scenarios.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a unifying review of methods in combinatorial BO. Authors show that the kernels of popular CASMOPOLITAN and COMBO methods are mathematically equivalent and that they both belong to a wider class of heat kernels. This finding has profound implications. First of all, it means that the only differences between CASMOPOLITAN and COMBO we empirically observed in the past were not coming from differences in surrogate models, which authors empirically demonstrate by running both methods with exact same settings and showing their performance is indistinguishable. Secondly, it means one can perform the computation of the COMBO kernel much faster, which authors demonstrate achieving 2x speedup. Lastly, authors show that a basic form of the heat kernel with the best performing acquisition optimiser and trust region is sufficient to achieve state-of-the-art performance across a number of problems.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper unravels a vital connection between two well-known methods that the community was previously oblivious to.\n- The paper supports this surprising theoretical result with convincing and comprehensive experiments \n- The papers draws a number of important conclusions from their theoretical results that are incredibly useful to practitioners, for example, that given the same hyperparameters and acquisition optimiser, CASMOPOLITAN and COMBO will perform the same and that the computation of COMBO kernel can be massively sped up\n- The paper provides clear guidance to practitioners, proposing a simple, yet powerful baseline\n- The paper is accompanied by an extensive appendix, which discusses how to adopt the heat kernels to invariances or how to incorporate an additive structure into them\n\nWeaknesses:\n- I do not think there are any significant weaknesses, however, I believe the presentation of the paper could be slightly improved. A lot of important results and insights appear only in the appendix. Authors should think carefully about how to best use the additional page in camera-ready version and move the most important parts of the appendix there. It would also be nice to report the number of seeds used for each run in the main text.\n- (typo): I believe equation 4 should have a minus inside the exponent\n\nQUESTIONS:\nHow do you think one could incorporate continuous variables into the heat kernel framework in the Mixed-space BO problem setting?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

comments: SUMMARY:
Recent works on combinatorial BO have studied the impact of acquision function optimisation, the use of trust regions and the surrogate models. 
While GP-based approaches are SOTA in general combinatorial BO in the absense of prior information, there is still a gap in the understanding of the impact of the kernel choice 
on the BO performance. In this work the authors aims at bridging this gap, not only by empirically assessing the merits of the standard combinatorial kernels 
(proposed in COMBO, CASMOPOLITAN, ...) but by actually revealing that most of these standard kernels are somewhat equivalent and belong to a larger family of Hamming-based kernels.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper is clearly written and easy to follow.
- The unifying framework presenting Casmopolitan and COMBO kernels as heat kernels is theoretically sound, and offers a new perspective on the combinatorial kernel landscape.
- The experiments support the equivalence claim and the proposed pipeline is simple and fast.
- The proposed generalization to Hamming-based kernels open the way to new investigation.

Weakness:
- One could say the main results is simply derived given the results of Kondor and Lafferty on the diffusion kernel for the complete graph (though it is true that it was not exploited or noticed before)
- It could be interesting to consider more real-world black-boxes (such as the ones from the MCBO library); Please evaluate on those: 


Minor comments:
- line 229: it should be "by choosing k", not "\mathbb{k}"
- Have a line style distinct for heat-based and non-heat-based to ease the analysis

Extra question:
- Is there a way to see the kernel from BODi as a heat kernel variation in some sense?

QUESTIONS:
Please see above

LIMITATIONS:
ok
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 2
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses algorithms for combinatorial Bayesian optimization (CBO), a problem that occurs in many practical applications. Specifically, this paper focuses on heat or diffusion kernels that operate on a graph that models the search space and Matern-class kernels (including the RBF kernel) relying on the Hamming distance.\n\nThe authors show that these Matern-class kernels _are_ heat-kernels if one considers categorical problems where the graph is modeled as a Hamming graph. Furthermore, they show that any Hamming kernel is a $\\Phi$-kernel, which is a generalization of the Heat kernel.\n\nBased on these insights, the authors propose a faster implementation of the heat kernel based on the Hamming distance.\n\nThe authors benchmark their kernel against various baselines on different categorical benchmark problems. They show that their simplified heat kernel is competitive with the state of the art. They also use random permutations of the categorical variables as proposed by Papenmeier et al. (2023) and show that the SSK kernel is also sensitive to those permutations (Papenmeier et al. have shown this for BODi and COMBO).\n\nSTRENGTHS AND WEAKNESSES:\n__Strengths__\n\n* The paper is well written and follows a clear structure. The problem is relevant - categorical BO has various applications.\n* Showing that an RBF kernel with Hamming distance is equivalent to COMBO reduces the computational complexity of running COMBO drastically.\n* The authors show that yet another algorithm (BOSS) is sensitive to permutations of categories. This is highly relevant because categorical problems have no notion of 'order'. In most cases, it is arguably unexpected that randomly permuting the order of the categories affects performance, and the methods that are affected by this have not stated this implicit bias. Hence, it is arguably not by design.\n* The authors are open about the fact that the analysis only holds for categorical problems.\n\n__Weaknesses__\n\nI see two main weaknesses. 1) The analysis only holds for categorical problems. 2) The proposed algorithm method is very close to already existing methods. I am not particularly concerned about 2) since the empirical performance is not the focus of this paper, and acts more as a sanity check. 1) is a bigger limitation, even though the authors argue that ordinal problems might not be as prevalent in practice.\n\nQUESTIONS:\n* Why do we need Figures 2 and 4? Aren't they showing mostly the same thing?\n* Do you have concrete ideas on how one could extend the analysis to ordinal problems?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}