SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \mathbf {1} ^ {\top} \left(\boldsymbol {h} ^ {0} \odot \dots \odot \boldsymbol {h} ^ {2 d - 2} - \mathbf {1}\right) = 2 \mathbf {1} ^ {\top} \left(\boldsymbol {h} ^ {0} \odot \dots \odot \boldsymbol {h} ^ {d - 1} - \mathbf {1}\right) - \mathbf {1} ^ {\top} \left(\boldsymbol {h} ^ {0} - \mathbf {1}\right) \\ + \mathbf {1} ^ {\top} \left(\left(\boldsymbol {h} ^ {1} \odot \dots \odot \boldsymbol {h} ^ {d - 1} - \mathbf {1}\right) \odot \boldsymbol {h} ^ {0} \odot \left(\boldsymbol {h} ^ {- d - 1} \odot \dots \odot \boldsymbol {h} ^ {- 1} - \mathbf {1}\right)\right) \tag {103} \\ \end{array}
$$
Now, we are ready to prove our main Theorem 1.
Proof. From Lemma 3, we know that
$$
e ^ {2} \left(\mathcal {H} _ {k}; \mathcal {P}\right) = \frac {1}{n} \mathbf {1} ^ {\top} \left(\boldsymbol {h} ^ {0} \odot \dots \odot \boldsymbol {h} ^ {d - 1} - \mathbf {1}\right) \tag {104}
$$
$$
\begin{array}{l} \mathbf {1} ^ {\top} \left(\boldsymbol {h} ^ {0} \odot \dots \odot \boldsymbol {h} ^ {d - 1} - \mathbf {1}\right) \\ = \frac {1}{2} \mathbf {1} ^ {\top} \left(\boldsymbol {h} ^ {0} \odot \dots \odot \boldsymbol {h} ^ {2 d - 2} - \mathbf {1} - \left(\boldsymbol {h} ^ {1} \odot \dots \odot \boldsymbol {h} ^ {d - 1} - \mathbf {1}\right) \odot \boldsymbol {h} ^ {0} \odot \left(\boldsymbol {h} ^ {- 1} \odot \dots \odot \boldsymbol {h} ^ {- (d - 1)} - \mathbf {1}\right)\right) \\ + \frac {1}{2} \mathbf {1} ^ {\top} \left(\boldsymbol {h} ^ {0} - \mathbf {1}\right) \tag {105} \\ \end{array}
$$
Plug Eq.(105) into Eq.(104), we have that
$$
\begin{array}{l} e ^ {2} (\mathcal {H} _ {k}; \mathcal {P}) \\ = \frac {1}{2 n} \mathbf {1} ^ {\top} \left(\mathbf {h} ^ {0} \odot \dots \odot \mathbf {h} ^ {2 d - 2} - \mathbf {1} - \left(\mathbf {h} ^ {1} \odot \dots \odot \mathbf {h} ^ {d - 1} - \mathbf {1}\right) \odot \mathbf {h} ^ {0} \odot \left(\mathbf {h} ^ {- 1} \odot \dots \odot \mathbf {h} ^ {- (d - 1)} - \mathbf {1}\right)\right) \\ + \frac {1}{2 n} \mathbf {1} ^ {\top} \left(\mathbf {h} ^ {0} - \mathbf {1}\right) \tag {106} \\ \end{array}
$$
Note that  $\pmb{h}^0 = \pmb{F}\gamma$  and  $\pmb{F}$  denotes the discrete Fourier matrix, we have that
$$
\begin{array}{l} \mathbf {1} ^ {\top} \left(\boldsymbol {h} ^ {0} - \mathbf {1}\right) = \mathbf {1} ^ {\top} \boldsymbol {F} \boldsymbol {\gamma} - n (107) \\ = \boldsymbol {b} ^ {\top} \gamma - n (108) \\ \end{array}
$$
where  $\pmb {b} = [0,0,\dots ,0,n]^{\top}$
Note that the  $n^{th}$  element in  $\gamma$  is  $\gamma_{n} = 1 + \frac{2}{n^{\alpha}}\zeta (\alpha ,1)$ , where  $\zeta (\cdot ,\cdot)$  denotes the Hurwitz zeta function. It follows that
$$
\mathbf {1} ^ {\top} \left(\boldsymbol {h} ^ {0} - \mathbf {1}\right) = \boldsymbol {b} ^ {\top} \boldsymbol {\gamma} - n = n + n \frac {2}{n ^ {\alpha}} \zeta (\alpha , 1) - n = n \frac {2}{n ^ {\alpha}} \zeta (\alpha , 1) \tag {109}
$$
Plug Eq.(109) into Eq.(106), we achieve the result in Theorem 1
$$
\begin{array}{l} e ^ {2} (\mathcal {H} _ {k}; \mathcal {P}) \\ = \frac {1}{2 n} \mathbf {1} ^ {\top} \left(\boldsymbol {h} ^ {0} \odot \dots \odot \boldsymbol {h} ^ {2 d - 2} - \mathbf {1} - \left(\boldsymbol {h} ^ {1} \odot \dots \odot \boldsymbol {h} ^ {d - 1} - \mathbf {1}\right) \odot \boldsymbol {h} ^ {0} \odot \left(\boldsymbol {h} ^ {- 1} \odot \dots \odot \boldsymbol {h} ^ {- (d - 1)} - \mathbf {1}\right)\right) \\ + \frac {1}{n ^ {\alpha}} \zeta (\alpha , 1) \tag {110} \\ \end{array}
$$
B Benchmark Test Functions
The benchmark test functions employed in section 4.1 are listed in Table 2, which contains multi-mode functions and non-smooth functions that are challenging for optimization.
Table 2: Test functions
name	function
Rosenbrock	∑i=1d-1(100(xi+1-x2i)2+(1-xi)2)
Nesterov	1/4|x1-1| + ∑i=1d-1|xi+1-2|xi|+1|
Rastrigin	10d + ∑i=1d(xi2-10 cos (2πxi))
C Training Time and Fast Coordinate Search Time
(a) Training Time (seconds)
(b) Fast Coordinate Search Time (seconds)
Figure 5: Training Time and Fast Coordinate Search Time (seconds) v.s. the number of samples
We provide the training time of our rank-1 lattice GP and the time of our fast coordinate search for targeted sampling in Figure 5(a) and Figure 5(b), respectively. The dimension of the rank-1 lattice data is set to  $d = 50$ . The number of samples  $n$  is set to the parameter in  $\{1783, 5347, 10099, 51283, 100189, 501139, 1000099\}$ . The number of samples  $n$  is a prime number such that  $(2d - 1) \mid (n - 1)$  to achieve our closed-form rank-1 lattice construction. The number of epochs of training is set to 2000. The number of iterations of fast coordinate search is set to  $T = 50$ . All the experiments are performed in 50 runs on a single NVIDIA A40 Card.
We report the mean value  $\pm$  std in Figure 5. The standard deviation of the time is small. From Figure 5(a), we can see that it takes around 50 seconds for our rank-1 GP training with one million lattice data. Moreover, our fast coordinate search for targeted sampling takes around 1.5 seconds to optimize rank-1 lattice GP posterior prediction conditioned on one million lattice data.
D Additional Experiments of Black-box Prompt Fine-tuning
(b) SS2
(d) AG's News
Figure 6: Hinge loss v.s. the number of query evaluations on different black-box fine-tuning models.
(f) RTE
We provide additional experimental results of black-box prompt fine-tuning for large language models. We employ the deep model in [Sun et al., 2022a] as the backbone. It has 24 layers. For each layer, we set the dimension of the continuous prompt to 50. Thus, the total dimension is  $24 \times 50$ . We employ the hinge loss of training data as the black-box objective.
In all the experiments, we keep the number of batch samples and the initialization the same for RLTS, INGO and CMAES. We set the number of batch samples to 200. Our RLTS employs 199 rank-1 lattice QMC Gaussian samples and one sample from targeted sampling. INGO employs 199 rank-1 lattice QMC Gaussian samples and one Gaussian sample. CMAES employs 200 Gaussian samples. We initialize the  $\mu = 0$  for all the methods. For INGO and RLTS, we set the step-size parameter  $\beta = 0.2$  in all experiments. For RLTS, we set the parameter  $\eta = 1$  in all experiments. All the experiments are performed in five independent runs with seeds in  $\{1,2,3,4,5\}$ .
The experimental results of mean objective  $\pm$  std v.s. the number of queries are shown in Figure 6. From Figure 6, we can observe that RLTS decreases the loss consistently faster than INGO and CMAES on all benchmark datasets. More importantly, RLTS decreases the loss significantly faster than INGO. Note that RLTS employs INGO as the backbone algorithm, which shows that RLTS improves the query efficiency of INGO.