Hyperparameter Optimization via Interacting with Probabilistic Circuits

Authors: Jonas Seng, Fabrizio Ventola, Zhongjie Yu, Kristian Kersting
Primary area: optimization
Keywords: Hyperparameter Optimization, Probabilistic Circuits, Tractable Probabilistic Models

Abstract:
Despite the growing interest in designing truly interactive hyperparameter optimization (HPO) methods, to date, only a few allow to include human feedback. However, these methods add friction to the interactive process, rigidly requiring to fully specify the user input as prior distribution ex ante and often imposing additional constraints on the optimization framework. This hinders the flexible incorporation of expertise and valuable knowledge of domain experts, which might provide partial feedback at any time during optimization. To overcome these limitations, we introduce a novel Bayesian optimization approach leveraging tractable probabilistic models named probabilistic circuits (PCs) as surrogate model. PCs encode a tractable joint distribution over the hybrid hyperparameter space and evaluation scores, and enable exact conditional inference and sampling, allowing users to provide valuable insights interactively and generate configurations adhering to their feedback. We demonstrate the benefits of the resulting interactive HPO through an extensive empirical evaluation of diverse benchmarks, including the challenging setting of neural architecture search.

Decision: reject
Reviewer scores: 8, 3, 6, 5

Reviews and discussion:
<thread>
creation date:
1730481869721

thread_type:
official_review

<thread_object>
creator: reviewer_ntfs

<content>
summary:
The paper presents an algorithms for performing Bayesian optimization for hyperparameter tuning that is amenable to interaction with users. Users can indeed input their belief of promising hyperparameters at any iteration (compared to previously proposed approaches that only allow user to input at the beginning of the process) in the form of (possibly point mass) distributions. The method relies on previously proposed probability circuits and a sampling mechanism recalling Thompson sampling. 
The authors report various experiments showing how the proposed method IBO-HPC performs both without user inputs and with beneficial and detrimental user inputs, comparing to various baselines.

strengths:
- The motivation of the work is well presented and clear
- the application of probability circuits to HPO is novel to the best of my knowledge
- PCs the integration of user knowledge seems natural and justified

weaknesses:
- some unclear notation and not properly introduced variables (see questions)
- to me, the overall novelty value of this work is not immediately clear. On the one hand, it does look like to me that the work is a direct application of the general framework of BO with a surrogate model given by PC. The integration of user knowledge in the form of prior distribution seems fairly straightforward, but not necessarily unique to the proposed framework (although this work focuses on this aspect). The user knowledge decay seems a reasonable heuristics. Perhaps I am missing some novelty aspect, I'd appreciate the authors comments on this.
- It is unclear to me why other HPO methods "cannot take user feedback" at any time, especially if this is given in the form of "fixing some hyperparameter value" (aka point mass distribution). Even thinking about random search, at any time (i.e. after any number of draws/trials), one can replace any previously given distribution with an updated guess by the user and continue the process. This sounds to me a reasonable simple baseline that is not included in the paper
- similarly (although I am not familiar with the methods) I would assume that both BOPrO and muBO could be modified in a straightforward way to allow for anytime user belief update by warm-restarting (i.e. using all previously evaluated configurations as starting points). Is there any fundamental limitation that prevents this integration?

For me these last aspects pared by the low level of novelty I see at the moment make me lean toward rejection.

questions:
- notation:
  - what's the meaning of p(f|D) in 3.1? when I read this notation I think of a distribution over functions, but rather since f here is given, probably the authors mean a distribution over function values (given hyperparameters)? 
  - what is $\mathcal{S}$ and $\mathcal{K}$ (are they just generic collections)?
  - what's the measure of the integral in definition 3?
- if possible, I would like to see comparison with at least simple baselines such as RS with the same user belief integration used by IBO-HPC.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730592218931

thread_type:
official_review

<thread_object>
creator: reviewer_uxru

<content>
summary:
The paper leverages probabilistic circuits to enable user interaction in Bayesian optimization during hyperparameter optimization.

strengths:
The paper is well-written and the methodology seems to be original.

weaknesses:
Definition 3 would be clearer if it included an explanation of why the second condition is necessary and how it ensures the policy accurately represents the user's knowledge.

questions:
Please provide more reasoning for the second condition on Definition 3.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730616900354

thread_type:
official_review

<thread_object>
creator: reviewer_cr4s

<content>
summary:
The paper introduces a hyperparameter optimization approach by incorporating user feedback directly into the optimization process through a Bayesian framework using probabilistic circuits (PCs). The proposed method, Interactive Bayesian Optimization via Hyperparameter Probabilistic Circuits (IBO-HPC), aims to overcome limitations in existing HPO by allowing dynamic incorporation of expert knowledge at any point during the optimization. The method leverages PCs for tractable inference and sampling, enabling users to provide valuable insights interactively and generate configurations adhering to their feedback.

strengths:
- The paper addresses the important topic of interactive hyperparameter optimization.
- It presents a novel approach by using probabilistic circuits as a surrogate model.

weaknesses:
1. In comparison to prior methods that incorporate static priors, this paper introduces flexibility in the timing of knowledge integration. However, the study of human knowledge remains restricted to optimal solutions or their distributions. It would be beneficial if the introduction provided some application scenarios to underscore the importance of addressing this limitation. Additionally, the primary distinction of this method from prior work lies in its approach to knowledge-based sampling, yet the proposed sampling technique does not seem to offer substantial theoretical advancements, such as analyzing the impact of the sampling method and prior integration on the convergence bounds of the algorithm.

2. Lines 277-283: The sampling method first draws a set of candidate points based on a prior distribution, then further samples from these points. The paper does not discuss how this approach ensures accuracy in approximating Equation 2. Additionally, it is unclear why the two product terms in the equation are approximated through separate sampling steps. Could you provide a theoretical justification evidence for the accuracy of their approximation method and give an explanation for why the two-step sampling process was chosen over alternatives.

3. The paper does not clearly explain how sampling is conducted in Algorithm 1, line 13, or how the value of 𝑠 is calculated in line 14.

4. The method for constructing the PC model using data and how to perform sampling with this model are not well-explained. What are the advantages of the PC model compared to other models? Is there any drawback of it?

5. Existing prior-based methods can be applied to the application scenario in this work. Could you provide a more explicit comparison of the method to existing prior chased method (for example, BOPrO and πB) highlight the key differences and advantages. The paper does not discuss related work in interactive hyperparameter optimization. Provide a discussion of related work in interactive hyperparameter optimization would help contextualize the contribution.

6. The rationale behind the time and quality of knowledge used in the experimental setup is unclear. What real-world applications might benefit from this approach?

7. The test problems presented are limited, making it difficult to verify the method’s applicability across a diverse range of problems. Furthermore, comparisons with more recent prior-based HPO methods, such as Priorband [1] and PFNs4BO [2], as well as HPO methods like DPL [3] and Hyper-tune [4], are lacking. For the problems tested, you could refer to the test cases used in these methods.
[1] Mallik N, Bergman E, Hvarfner C, et al. Priorband: Practical hyperparameter optimization in the age of deep learning[J]. Advances in Neural Information Processing Systems, 2024, 36.
[2] Müller S, Feurer M, Hollmann N, et al. Pfns4bo: In-context learning for bayesian optimization[C]//International Conference on Machine Learning. PMLR, 2023: 25444-25470.
[3] Kadra A, Janowski M, Wistuba M, et al. Scaling laws for hyperparameter optimization[J]. Advances in Neural Information Processing Systems, 2024, 36.
[4] Li Y, Shen Y, Jiang H, et al. Hyper-tune: towards efficient hyper-parameter tuning at scale[J]. Proceedings of the VLDB Endowment, 2022, 15(6): 1256-1265.
8. The paper could be improved with clearer explanations. For example：there are many equations $p(\boldsymbol{\Theta}, st, K) ̸= p(\boldsymbol{\Theta}, st, \phi)$, $\int_{\mathcal{H} \backslash \hat{\mathcal{H}}} p\left(\boldsymbol{\Theta}, s_t, \mathcal{K}\right)=q(\hat{\mathcal{H}})$. After defining these equations, it is essential to explain why satisfying these conditions supports the corresponding conclusions.

questions:
1. The example in Figure 1 is confusing: "Users can guide the HPO algorithm with their knowledge (here, N = 16 and W = 3), thus considerably increasing convergence speed and the quality of the final solution by focusing the optimization on remaining hyperparameters." In what cases would users accurately know the precise value of certain hyperparameters during optimization? Why wouldn’t they fix these parameter values at the start of optimization? How does this work relate to preference-based optimization algorithms?

2. "Moreover, these approaches might not reflect user knowledge precisely as defined in the prior due to the non-linear integration of the priors in the acquisition function. For example, in Fig. 1 (Right), the configurations selected by both BOPrO (Souza et al., 2021) and πBO (Hvarfner et al., 2022) during the first 20 iterations of optimization remarkably deviate from the given user prior." How did the authors define "remarkably deviate"? If I understand correctly, these algorithms aim to move towards the region defined by the prior, rather than sampling exactly according to the prior distribution, since the user-provided prior might not be accurate. In this context, the example in Fig. 1 (Right) does not seem to support the authors’ claim.

3. What is the difference between "user knowledge" and "feedback" in Definitions 2 and 3?

4. There are several symbols in Definition 3, such as "an interactive policy $p$," "$p(\Theta, st, K)$," and "call $p$ feedback." I find these symbols confusing and difficult to interpret.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704851969

thread_type:
official_review

<thread_object>
creator: reviewer_knfb

<content>
summary:
This paper introduces a Bayesian Optimization (BIO) framework for interactive Hyperparameter Optimization (HPO).
This framework is termed Bayesian Optimization via Hyperparameter Probabilistic Circuits (IBO-HPC).
By interactive, they allow human experts to provide feedback during the HPO process through tractable Probabilistic Circuits (PCs).
Specifically, they use PCs, a tractable joint distribution over hyperparameters and evaluation scores, to allow exact conditional inference and sample.
They back up their method with extensive numerical experiments.

strengths:
This paper presents a refreshing and  interesting idea with solid execution.
**Clarity.** The writing of this paper is good. This paper is easy to follow. I especially appreciate the line-by-line walkthrough of algo1 in page 5.

**Originality.**  This paper utilizes PCs to perform exact Bayesian inference, especially conditional inference to include human experts’ feedback. While it makes sense as it sounds, I haven’t seen other solid executions of similar ideas.


**Significance.** I find IBO-HPC significant. Beyond its empirical effectiveness, unlike many human-in-the-loop HPO methods, it does not add computational burden.

**Solid Experiments.** Numerical validations are solid and extensive, making this paper more convincing.

weaknesses:
NA

questions:
How does IBO-HPC handle multiple human experts’ feedback? For example, consider a distributed or federated setting.

Can IBO-HPC handle online or life-long learning models with human experts monitoring in real time? If yes, can you sketch the response time of the model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733305028326

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Changes

comment:
We thank all the reviewers again for their time and feedback. Given the end of the discussion phase, we would like to briefly summarize all the points we addressed that have been mentioned in the reviews and the discussion phase.

We would like to remark that the clarifications and the additional empirical evidence provided during the rebuttal have only confirmed and further supported our previous claims based on the initial submitted results. Thus, the integrity, consistency, and validity of our method have not been compromised and, instead, further consolidated.

**Reviewer cr4S**

1. We clarified the rationale behind the time and quality of user knowledge provided to IBO-HPC and to all baselines allowing the integration of user priors (see [our rebuttal](https://openreview.net/forum?id=uC003NHlEi&noteId=btEfwOFOfm)).
2. We added a rigorous convergence analysis of IBO-HPC (see App. B.2)
3. We provided a theoretical justification showing the accuracy of our approximation in Eq. 2 (see App. B.4)
4. To improve clarity, we added more explanations to the equations employed to formally introduce our method (see Sec. 3 in our revision).
5. We added a brief discussion of LearnSPN, which is employed to learn the surrogate model during optimization (see App. C)
6. We better clarified the differences between PiBO, BOPrO, and IBO-HPC, focusing on how user knowledge is incorporated into the generation and selection process of configurations. We made this clearer (see [our rebuttal](https://openreview.net/forum?id=uC003NHlEi&noteId=vEokiw8MlF) and Sec. 1, 2, 3 in our revision).
7. As requested by the reviewer, we added Priorband as an additional multi-fidelity baseline to our evaluation (see Sec. 4 in our revision).
8. We discussed our experimental setup in detail and showed that we follow standard practices of the HPO/NAS literature by indicating several published and peer-reviewed works (see our answer to cr4S above). This discussion includes the qualitative choice and the number of selected benchmarks and tasks on which we evaluate our method IBO-HPC.
9. Besides the choice of benchmarks, we also discussed the choice of user knowledge employed in our experiments. Our paper provides a detailed description and justification of how user knowledge is specified. Moreover, we provided references to peer-reviewed published works that follow a similar specification of user knowledge (see our answer to cr4S above).
10. We provided a detailed explanation of how IBO-HPC balances exploration and exploitation. Next to the theoretical viewpoint, we also empirically verified our explanation by showing that the sampling variance decreases over time, indicating that IBO-HPC tends to explore in early iterations and to exploit in later iterations (see App. D.5; Fig. 14 and 15).
11. We added the one-sided Wilcoxon tests that further confirms our results and shows the statistical significance of our findings, i.e., IBO-HPC outperforms PiBO, BOPrO, and Priorband on a set of diverse tasks (see our [answer to cr4S](https://openreview.net/forum?id=uC003NHlEi&noteId=oQJa1jmjwO)).
12. We discussed and empirically showed the sensitivity of the parameters of IBO-HPC, referring to our ablation analysis in App. D.3 (Fig. 10-12). The empirical results demonstrate that IBO-HPC has low sensitivity with respect to the choice of these parameters.

**Reviewer nTFs**

1. As requested by the reviewer, we provided two additional baselines in our experiments (random search with early user interaction and random search with late user interaction, see Sec. 4 in our revision).
2. We pointed out the advantages of IBO-HPC's mechanism to incorporate user knowledge compared to other methods that allow the incorporation of user priors (see Introduction, Fig. 1 and App. A in our manuscript).
3. We have better highlighted the contributions we make and will incorporate appropriate changes in the camera-ready version of the paper (these changes were done at the time no revised PDF could be uploaded anymore; see our [response](https://openreview.net/forum?id=uC003NHlEi&noteId=UmJGTaJV34)).
4. We polished our notation for the camera-ready version to make it easier for the reader to follow (these changes were done at the time no revision could be uploaded anymore; see our [response](https://openreview.net/forum?id=uC003NHlEi&noteId=NE3s5JEWXj)).
5. In Sec. 3 and Sec. 4, we further highlighted the advantages of the novel idea of using PCs in BO to better justify our choice of PCs as a surrogate model in interactive BO (these changes were done at the time no revision could be uploaded anymore; see our response [response](https://openreview.net/forum?id=uC003NHlEi&noteId=DlFF2FVe9p)).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel iteractive Bayesian optimization framework using probabilistic circuits (PCs) as the surrogate model. It allows users to provide external knowledge and intervene the optimization process. It's evaluated on both HPO and neural architecture search benchmarks.

Pros:
- Novel integration of PCs in the interactive Bayesian optimization framework
- Flexible integration of user feedbacks.
- Tractable conditional inference that enables sampling from the exact conditional hyperparameter distribution

Cons:
- Unclear notation and presentation
- Significance compared to other methods could allow incorporating user knowledge
- Insufficient evaluation
- Unclear rational in the user intervention design

Reviewers have divergent opinions on the quality of the presentation, the design of user intervention and empirical evaluation. Authors feedback addressed some concerns (see the additinoal comments section), but do not change the negative reviewers' assessment on the main concerns. The reviewers with positive ratings provided fairly short and abstract reviews, and did not engage in the following discussion. There is no sufficient evidence to support acceptance of this paper.