SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Then, since  $\theta_{i}$  follows a uniform distribution, the overall probability for a point of zero sequence is given by
$$
\begin{array}{l} \mathbb {P} \left(\text {" z e r o s e q u e n c y "}\right) = 1 - \underbrace {\prod_ {i = 1} ^ {m} \int_ {0} ^ {1} \left(1 - \theta_ {i} ^ {D} - (1 - \theta_ {i}) ^ {D}\right) \underbrace {p \left(\theta_ {i}\right)} _ {= 1} d \theta_ {i}} _ {\text {p r o b . o f m t i m e s n o t z e r o s e q u e n c y}} \\ = 1 - \prod_ {i = 1} ^ {m} \left(\theta_ {i} - \frac {\theta_ {i} ^ {D + 1}}{D + 1} + \frac {(1 - \theta_ {i}) ^ {D + 1}}{D + 1}\right) \Bigg | _ {\theta_ {i} = 1} \\ = 1 - \left(1 - \frac {2}{D + 1}\right) ^ {m}, \\ \end{array}
$$
i.e., the probability of at least one dictionary element being of sequence zero is  $1 - \left(1 - \frac{2}{D + 1}\right)^m$ . The probability of BODi's dictionary to contain a zero-sequence point increases with the number of dictionary elements  $m$  and decreases with the function dimensionality  $D$  (see Figure 18).
For instance, for the 60-dimensional MaxSAT60 benchmark, the probability that at least one dictionary element is of sequence zero is  $1 - \left(1 - \frac{2}{60 + 1}\right)^{128} \approx 0.986$  (see Figure 18).
Note that at least one point  $z^{*}$  has a probability of  $\leq 1 / 2^{d}$  to be drawn. The probability of the dictionary containing that  $z^{*}$  is less than or equal to  $1 - \left(1 - 1 / 2^{d}\right)^{m}$  which is already less than
Figure 18: Probabilities of BODi to contain a zero-sequence solution for different choices of the dictionary size  $m$  and the function dimensionality  $D$ .
0.01 for  $d = 14$  and  $m = 128$ . In Section 4, we show that randomizing the optimal point structure leads to performance degradation for BODi. We hypothesize this is due to the reduced probability of the dictionary containing the optimal point after randomization.
Categorical problems. We calculate the probability that BODi contains a vector in its dictionary where all elements are the same. For categorical problems, BODi first samples a vector  $\theta$  from the  $\tau_{\mathrm{max}}$ -simplex  $\Delta^{\tau_{\mathrm{max}}}$  for each vector  $a_i$  in the dictionary, with  $\tau_{\mathrm{max}}$  being the maximum number of categories across all categorical variables of a problem. We assume that all variables have the same number of categories as is the case for the benchmarks in Deshwal et al. [21]. Let  $\tau$  be the number of categories of the variables. For each element in  $a_i$ , BODi draws a value from the categorical distribution with probabilities  $\theta$ . While line 7 in Algorithm 5 in Deshwal et al. [21] might suggest that the elements in  $\theta$  are shuffled for every element in  $a_i$ , we observe that  $\theta$  remains fixed based on the implementation provided by the authors<sup>7</sup>. The random resampling of elements from  $\theta$  is probably only used for benchmarks where the number of realizations differs between categorical variables.
Then, for a fixed  $\theta$ , the probability that all  $D$  elements in  $\pmb{a}_i$  for any  $i$  are equal to some fixed value  $t \in \{1, 2, \dots, \tau\}$  is given by  $\theta_t^d$ . The probability that, for any of the  $m$  dictionary elements, all  $D$  elements in  $\pmb{a}_i$  are equal to some fixed value  $t \in \{1, 2, \dots, \tau\}$  is given by
$$
\mathbb {P} \left(\text {" a l l o n e s p e c i f i c c a t e g o r y "}\right) = 1 - \prod_ {i = 1} ^ {m} \int \left(1 - \theta_ {t} ^ {D}\right) p \left(\theta_ {t}\right) d \theta_ {t}. \tag {2}
$$
We note that  $\theta$  follows a Dirichlet distribution with  $\alpha = 1$  [54]. Then,  $\theta_{t}$  is marginally  $\mathrm{Beta}(1,\tau -1)$ -distributed [54]. With that, Eq. (2) becomes
$$
\begin{array}{l} \mathbb {P} \left(\text {" a l l o n e s p e c i f i c c a t e g o r y "}\right) = 1 - \prod_ {i = 1} ^ {m} \mathbb {E} _ {\theta_ {t} \sim \operatorname {B e t a} (1, \tau - 1)} \left[ 1 - \theta_ {t} ^ {D} \right] \\ = 1 - \prod_ {i = 1} ^ {m} 1 - \mathbb {E} _ {\theta_ {t} \sim \operatorname {B e t a} (1, \tau - 1)} \left[ \theta_ {t} ^ {D} \right] \\ \end{array}
$$
now, by using the equality  $\mathbb{E}[x^D] = \prod_{r=0}^{D-1} \frac{\alpha + r}{\alpha + \beta + r}$  for the  $D$ -th raw moment of a  $\mathrm{Beta}(\alpha, \beta)$  distribution [54]
$$
\begin{array}{l} = 1 - \left(1 - \prod_ {r = 0} ^ {D - 1} \frac {1 + r}{\tau + r}\right) ^ {m} \\ = 1 - \left(1 - \frac {1}{\tau} \cdot \frac {2}{\tau + 1} \cdot \dots \cdot \frac {D}{\tau + D - 1}\right) ^ {m} \\ \end{array}
$$
$^7$ Sec https://github.com/aryandeshwal/bodi/blob/aa507d34a96407b647bf808375b5e162ddb10664/bodi/categorical_dictary_kernel.py#L18
Figure 19: Probabilities of BODi's dictionary to contain at least one categorical point where each category has the same value. The probability increases with the number of dictionary elements  $m$  but decreases with the number of categories  $\tau$  and the number of problem dimensions  $D$ .
$$
= 1 - \left(1 - \frac {D ! \tau !}{(\tau + D - 1) !}\right) ^ {m}
$$
We discussed in Section 4.3 that the PestControl benchmark obtains a good solution at  $\pmb{x} = \pmb{5}$ . One could assume that BODi performs well on this benchmark because its dictionary will likely contain this point. However, we observe that the probability is effectively zero for  $\tau = 5$ ,  $m = 128$ , and  $D = 25$  (see Figure 19), which are the choices for the PestControl benchmark in Deshwal et al. [21]. This raises the question of (i) whether our hypothesis is wrong and (ii) what the reason for BODi's performance degradation on the PestControl benchmark is.
We show that BODi's reference implementation differs from the algorithmic description in an important detail, causing BODi to be considerably more likely to sample category five on PestControl (or the "last" category for arbitrary benchmarks) than any other category.
Figure 20: Histograms over the number of dictionary element entries set to each category for 20,000 repetitions of the sampling of dictionary elements for the PestControl benchmark. For each of the five categories and each value on the  $x$ -axis, the figure shows how often the number of entries in a dictionary element equals the value on the  $x$ -axis for the given category. For example, the count for  $x = 0$  and category 5 is zero, indicating that all 20,000 dictionary points had at least one entry '5'. There is a considerably higher chance for a dictionary element entry to be set to category five than to any of the other categories.
Figure 20 shows five histograms over the number of dictionary elements set to each category. The values on the  $x$ -axis give the number of elements in a 25-dimensional categorical vector being set to a specific category. One would expect that the histograms have a similar shape regardless of the category. However, for category 5, we see that more elements are set to this category than for the other categories: The probability of  $k$  elements being set to category 5 is almost twice as high as the probability of being set to another category for  $k \geq 3$ . In contrast, the probability that no element in the vector belongs to category 5 is virtually zero. This behavior is beneficial for the PestControl benchmark, which obtained the best value found during our experiments for  $x^{*} = (5,5,\dots,5,1)$
(see Section 4). While we see that the probability of each dictionary entry being set to category 5 is very low, we assume that we sample sufficiently many dictionary elements within a small Hamming distance to the optimizer such that B0Di's GP can use this information to find the optimizer.
The reason for oversampling of the last category lies in a rounding issue in sampling dictionary elements. In particular, for a given dictionary element  $a_i$  and a corresponding vector  $\theta$  with  $|\theta| = \tau$ , for each  $i \in \{1, 2, \ldots, \tau - 1\}$ , Deshwal et al. [21] set  $\lfloor D\theta_i \rfloor$  elements to category  $i$ . The remaining  $D - \sum_{i=1}^{\tau-1} \lfloor D\theta_i \rfloor$  elements are then set to category  $\tau$ . This causes the last category to be overrepresented in the dictionary elements. For the choices of the PestControl benchmark,  $D = 25$  and  $\tau = 5$ , the first four categories had a probability of  $\approx 0.1805$ . In contrast, the last one had a probability of  $\approx 0.278$  for  $10^8$  simulations<sup>8</sup>. We assume that the higher probability of the last category is the reason for the performance difference between the modified and the unmodified version of the PestControl benchmark.
H.2 COMBO on categorical problems
On the categorical PestControl benchmark, COMBO [56] behaved similar to BODi [21]
Figure 21: The histogram shows how many element entries of a 25-dimensional dictionary element are set to each of the five categories. There is a considerably higher chance for a dictionary element entry to be set to category 1 or 5 than to one of the other categories.
The histogram in Figure 21 shows how many element entries of a 25-dimensional dictionary element are set to each of the five categories. We see that the first and the last categories are over-represented on both benchmark versions. As discussed in Section 4, this benchmark attains the best observed value for  $x^{*} = (5,5,\dots ,5,1)$ . We observe that on the original and modified version of the benchmark, where the categories are shuffled, COMBO sets disproportionately many elements to categories one and five. Note that for the modified version of the benchmark, all categories are equally likely in the optimal solution.
We argue that this behavior is at least partially caused by implementation error in the construction of the adjacency matrix and the Laplacian for categorical problems<sup>9</sup>. This error causes categorical variables to be modeled like ordinal variables. According to Oh et al. [56], categorical variables are modeled as a complete graph (see Figure 22).
The  $95\%$  confidence intervals for categories 1-5 are (0.1799, 0.1807), (0.1802, 0.1810), (0.1803, 0.1811), (0.1801, 0.1809), (0.2775, 0.2783). Pairwise Wilcoxon signed-rank tests between categories 1-4 and category 5 gives  $p$  values of 0 ( $W \approx 4.7 \cdot 10^{10}$  each).
<sup>9</sup>https://github.com/QUVA-Lab/COMBO/blob/9529eabb86365ce3a2ca44fff08291a09a853ca2/COMBO/experiments/test-functions/multiple_categorical.py#L137, last access: 2023-04-26
Figure 22: Following the description in the paper of Oh et al. [56], a categorical variable with five categories is modeled as a complete graph.
Looking into the source code of COMBO, we find the adjacency matrix for the first category of a categorical variable with five categories is constructed as
$$
\left( \begin{array}{c c c c c} 0 & 1 & 0 & 0 & 0 \\ 1 & 0 & 1 & 0 & 0 \\ 0 & 1 & 0 & 1 & 0 \\ 0 & 0 & 1 & 0 & 1 \\ 0 & 0 & 0 & 1 & 0 \end{array} \right),
$$
which is the adjacency matrix for a five-vertex path graph.