SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yueming LYU
Centre for Frontier AI Research (CFAR)
Institute of High Performance Computing (IHPC)
Agency for Science, Technology and Research (A*STAR)
1 Fusionopolis Way, #16-16 Connexis, Singapore 138632
Lyu_Yueming@cfar.a-star.edu.sg
Black-box optimization has gained great attention for its success in recent applications. However, scaling up to high-dimensional problems with good query efficiency remains challenging. This paper proposes a novel Rank-1 Lattice Targeted Sampling (RLTS) technique to address this issue. Our RLTS benefits from random rank-1 lattice Quasi-Monte Carlo, which enables us to perform fast local exact Gaussian processes (GP) training and inference with  $O(n \log n)$  complexity w.r.t.  $n$  batch samples. Furthermore, we developed a fast coordinate searching method with  $O(n \log n)$  time complexity for fast targeted sampling. The fast computation enables us to plug our RLTS into the sampling phase of stochastic optimization methods. This improves the query efficiency while scaling up to higher dimensional problems than Bayesian optimization. Moreover, to construct rank-1 lattices efficiently, we proposed a closed-form construction. Extensive experiments on challenging benchmark test functions and black-box prompt fine-tuning for large language models demonstrate the query efficiency of our RLTS technique.
Black-box optimization has gained great attention for its success in many recent applications, such as prompt fine-tuning for large language models [Sun et al., 2022b,a], policy search for robot control and reinforcement learning [Choromanski et al., 2019, Lizotte et al., 2007, Barsce et al., 2017, Salimans et al., 2017], automatic hyper-parameters tuning in machine learning problems [Snoek et al., 2012], black-box architecture search in engineering design [Wang and Shan, 2007], drug discovery [Negoescu et al., 2011] and accelerated simulation for scientific discovery [Maddox et al., 2021, Hernández-Lobato et al., 2017], etc. Many efforts have been made for black-box optimization in the literature, including Bayesian optimization (BO) methods [Srinivas et al., 2010, Gardner et al., 2017, Nayebi et al., 2019], stochastic optimization methods like evolution strategies (ES) [Back et al., 1991, Hansen, 2006, Wierstra et al., 2014b, Lyu and Tsang, 2021] and genetic algorithms [Srinivas and Patnaik, 1994, Mirjalili and Mirjalili, 2019].
Bayesian optimization usually builds a global (GP) model as a surrogate and provides queries by optimizing some acquisition functions [Snoek et al., 2012]. Although BO achieves good query efficiency for low-dimensional problems, it often fails to handle high-dimensional problems with large sample budgets [Eriksson et al., 2019]. The computation of GP with a large number of samples itself is expensive, and the internal optimization of the acquisition functions is challenging. Recently, Müller et al. [2021], Nguyen et al. [2022] builds a GP model for both the function value and the gradient and performs local Bayesian optimization. Although these methods improve the scalability of global BO, they usually cannot scale up to five hundred dimensional complex problems. This may
be because the learned gradient heavily depends on the accuracy of the GP model. However, achieving an accurate GP model is challenging for high-dimensional problems. A slightly misspecified GP model may lead to a wrong estimated gradient due to the highly nonlinear acquisition functions.
On the other line, stochastic optimization methods, e.g., ES [Rechenberg and Eigen, 1973, Nesterov and Spokoiny, 2017], natural evolution strategies (NES) [Wierstra et al., 2014b], CMAES [Hansen, 2006], and implicit natural gradient optimizer (INGO) [Lyu and Tsang, 2021], typically sampling form Gaussian distribution and approximate the (natural) gradient for the update of the Gaussian distribution parameters for continuous optimization. These methods can scale up to higher dimensional problems compared with BO. However, the gradient approximation may have a large variance, especially for high-dimensional problems. Thus, the update direction may not be toward the descent direction, leading to inferior query efficiency.
To address high-dimensional black-box problems with good query efficiency, we propose a novel Rank-1 Lattice Targeted Sampling (RLTS) technique. Our RLTS has a  $O(n\log n)$  time complexity, which is fast for plugging into the sampling phase of stochastic optimization methods. In this way, our methods can improve the query efficiency of stochastic optimization methods while addressing higher-dimensional problems than BO. Our contributions are summarized as follows:
- We propose a novel Rank-1 Lattice Targeted Sampling (RLTS) technique. Our RLTS builds a local GP with a random rank-1 lattice, which enables fast exact GP training and inference with  $O(n \log n)$  time complexity w.r.t.  $n$  batch samples. Furthermore, we develop a fast coordinate search that enables target sampling with  $O(n \log n)$  time complexity.
- We propose a closed-form subgroup rank-1 lattice by considering the dual lattice regarding the integral approximation error of functions in Korobov space. Our rank-1 lattice has a more regular pattern of approximation error terms. Moreover, our subgroup rank-1 lattice capitalizes on constructing a circulant kernel Gram matrix benefit from its group property. This enables efficient  $O(n \log n)$  computations in GP training/inference and fast candidate searching. In contrast, low-discrepancy QMC sequences, such as Sobol sequences or Halton sequences, lack these capabilities. In addition, our new closed-form rank-1 lattice may have potential applications in downstream tasks beyond black-box optimization.
- We plug our RLTS into the sampling phase at each step of stochastic optimization methods to improve query efficiency. In this way, during the optimization procedure, our RLTS sampling from an updated promising region instead of a fixed one at each step. This approach can scale up to address high-dimensional problems.
- Empirically, extensive experiments on high-dimensional challenging benchmark test functions and practical black-box prompt fine-tuning for large language models demonstrate the effectiveness of our RLTS technique.
2.1 Black-box Optimization
Given a proper function  $f(\pmb{x}) : \mathbb{R}^d \to \mathbb{R}$  such that  $f(\pmb{x}) > -\infty$ , black-box optimization is to minimize  $f(\pmb{x})$  by using function queries only. Black-box stochastic optimization methods typically employ a sampling distribution  $p(\pmb{x}; \pmb{\theta})$  and optimize the parameter of the distribution regarding the relaxed problem:  $J(\pmb{\theta}) := \mathbb{E}_{p(\pmb{x}; \pmb{\theta})}[f(\pmb{x})]$ .
Evolution Strategies (ES) [Rechenberg and Eigen, 1973, Nesterov and Spokoiny, 2017] employ a Gaussian distribution  $\mathcal{N}(\boldsymbol{\mu}, \sigma^2\boldsymbol{I})$  for sampling. The approximate gradient descent update is given as
$$
\boldsymbol {\mu} _ {t + 1} = \boldsymbol {\mu} _ {t} - \frac {\beta}{n \sigma} \sum_ {i = 1} ^ {n} \epsilon_ {i} f \left(\boldsymbol {\mu} _ {t} + \sigma \boldsymbol {\epsilon} _ {i}\right), \tag {1}
$$
where  $\epsilon_{i}\sim \mathcal{N}(\mathbf{0},\mathbf{I})$  and  $\beta$  denotes the step-size. The ES method performs the approximate first-order gradient descent update. As a result, the convergence of ES may be slow. Several second-order gradient descent methods have been proposed to improve convergence. Wierstra et al. [2014a] proposed the natural evolution strategies (NES), which perform the approximate natural gradient update. When a Gaussian distribution  $\mathcal{N}(\boldsymbol {\mu},\boldsymbol {\Sigma})$  is employed for sampling. The update rule of NES is
given in Eq.(2) and Eq.(3):
$$
\boldsymbol {\Sigma} _ {t + 1} = \boldsymbol {\Sigma} _ {t} - \frac {\beta}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\mu} _ {t} + \boldsymbol {\Sigma} _ {t} ^ {\frac {1}{2}} \boldsymbol {\epsilon} _ {i}\right) \left(\boldsymbol {\Sigma} _ {t} ^ {\frac {1}{2}} \boldsymbol {\epsilon} _ {i} \boldsymbol {\epsilon} _ {i} ^ {\top} \boldsymbol {\Sigma} _ {t} ^ {\frac {1}{2}} - \boldsymbol {\Sigma} _ {t}\right) \tag {2}
$$
$$
\boldsymbol {\mu} _ {t + 1} = \boldsymbol {\mu} _ {t} - \frac {\beta}{n} \sum_ {i = 1} ^ {n} f \left(\boldsymbol {\mu} _ {t} + \Sigma_ {t} ^ {\frac {1}{2}} \boldsymbol {\epsilon} _ {i}\right) \Sigma_ {t} ^ {\frac {1}{2}} \boldsymbol {\epsilon} _ {i}. \tag {3}
$$
where  $\epsilon_{i}\sim \mathcal{N}(\mathbf{0},\mathbf{I})$  and  $\Sigma^{\frac{1}{2}} = \Sigma^{\frac{1}{2}\top}$  and  $\Sigma^{\frac{1}{2}}\Sigma^{\frac{1}{2}} = \Sigma$ . The NES takes advantage of second-order gradient information, which improves the convergence of ES.
Lyu and Tsang [2021] proposed an implicit natural gradient optimizer (INGO) for black-box optimization, which provides an alternative way to compute the natural gradient update. The update rule of INGO is given as in Eq.(4) and Eq.(5):
$$
\boldsymbol {\Sigma} _ {t + 1} ^ {- 1} = \boldsymbol {\Sigma} _ {t} ^ {- 1} + \beta \sum_ {i = 1} ^ {n} \frac {f (\boldsymbol {x} _ {i}) - \widehat {\mu}}{n \widehat {\sigma}} \left(\boldsymbol {\Sigma} _ {t} ^ {- 1} \left(\boldsymbol {x} _ {i} - \boldsymbol {\mu} _ {t}\right) \left(\boldsymbol {x} _ {i} - \boldsymbol {\mu} _ {t}\right) ^ {\top} \boldsymbol {\Sigma} _ {t} ^ {- 1}\right) \tag {4}
$$
$$
\boldsymbol {\mu} _ {t + 1} = \boldsymbol {\mu} _ {t} - \beta \sum_ {i = 1} ^ {n} \frac {f (\boldsymbol {x} _ {i}) - \widehat {\mu}}{n \widehat {\sigma}} \left(\boldsymbol {x} _ {i} - \boldsymbol {\mu} _ {t}\right). \tag {5}
$$
where  $\pmb{x}_i\sim \mathcal{N}(\pmb{\mu}_t,\pmb{\Sigma}_t)$ ,  $\widehat{\mu} = \frac{\sum_{i = 1}^{n}f(\pmb{x}_i)}{n}$  and  $\widehat{\sigma}$  denotes the standard deviation of  $f(\pmb{x}_i)$ . The normalization  $\frac{f(\pmb{x}_i) - \widehat{\mu}}{\widehat{\sigma}}$  is employed to reduce the variance.
CMAES [Hansen, 2006] provides a more sophisticated update rule and performs well on a wide range of black-box optimization problems. All the above stochastic optimization methods rely on sampling. Thus, the sampling phase is vitally important. And a better sampling technique is promising to achieve further improvement.
2.2 Rank-1 Lattice
A rank-1 lattice is a particular case of the general lattice with a simple operation for point-set construction. It can be used as Quasi-Monte Carlo for integral approximation [Sloan, 2000, Dick et al., 2013]. A rank-1 lattice point set  $\mathcal{P} = \{\pmb{x}_1,\dots ,\pmb{x}_n\}$  can be constructed as Eq.(6):
$$
\boldsymbol {x} _ {i} := \frac {i z \bmod n}{n}, i \in \{1, \dots , n \}, \tag {6}
$$
where  $\mathbf{z} \in \mathbb{Z}^d$  is the so-called generating vector, and mod denotes the modulo operation.
Korobov [1960] proposes a rank-1 lattice with the generating vector having a particular form as Eq.(7)
$$
z := [ 1, k, \dots , k ^ {d - 1} ] \bmod n, \tag {7}
$$
where  $k$  is searching over  $\{1, \dots, n - 1\}$  to reduce approximation error.
Sloan and Reztsov [2002] further proposed a component-by-component searching method for the generating vector without assuming the Korobov form in Eq. (7). Recently, Lyu et al. [2020] proposed a simple closed-form subgroup-based rank-1 lattice by considering the Toroidal distance in the primal lattice space. The generating vector is given as Eq.(8)
$$
z = \left[ g ^ {0}, g ^ {\frac {n - 1}{2 d}}, g ^ {\frac {2 (n - 1)}{2 d}}, \dots , g ^ {\frac {(d - 1) (n - 1)}{2 d}} \right] \bmod n, \tag {8}
$$
where  $g$  denotes the primitive root modulo the prime number  $n$ . More details of the lattice rules for numerical integration can be found in the book [Dick et al., 2022].
In this paper, we proposed a closed-form subgroup rank-1 lattice by ensuring the approximation error terms of the dual lattice have a more regular pattern. In contrast, Lyu et al. [2020] construct the rank-1 lattice evenly spaced in the primal lattice space.
3 Fast Rank-1 Lattice Targeted Sampling
3.1 Random Rank-1 Lattice Quasi-Monte Carlo Gaussian Sampling
We first show how to construct random rank-1 lattice Quasi-Monte Carlo Gassuin samples. These samples enable us to perform the black-box stochastic optimization listed in section 2.1. More importantly, the nice property of the structure of these samples facilitates a fast targeted sampling.
(a) Our Closed-form Rank-1 Lattice Sampling
(b) i.i.d. Gaussian Sampling
(c) Our Closed-form Rank-1 Lattice Sampling
(d) i.i.d. Gaussian Sampling
Figure 1: Illustration of the our closed-form Rank-1 Lattice sampling and i.i.d. Gaussian sampling.
Given a rank-1 lattice point set  $\mathcal{P} = \{\pmb{x}_1,\dots ,\pmb{x}_n\}$ , we first construct a random shifted rank-1 lattice [Dick et al., 2013] as Eq. (9),
$$
\bar {x} _ {i} = x _ {i} + \Delta \mod 1 \forall i \in \{1, \dots , n \}, \tag {9}
$$
where  $\Delta \sim \text{Uniform}[0,1]^d$ , and the mod 1 operation denotes a modulo operation that takes the non-negative fractional part of the input number element-wise. Then, we can construct random QMC Gaussian samples as Eq. (10)
$$
\boldsymbol {\epsilon} _ {i} = \Phi^ {- 1} (\bar {\boldsymbol {x}} _ {i}) \forall i \in \{1, \dots , n \}, \tag {10}
$$
where  $\Phi^{-1}(\cdot)$  computes the inverse cumulative density function of the standard Gaussian distribution w.r.t. the input element-wise. Then, the samples for Gaussian  $\mathcal{N}(\boldsymbol{\mu},\boldsymbol{\Sigma})$  can be constructed as follows:
$$
\boldsymbol {\xi} _ {i} = \boldsymbol {\mu} + \boldsymbol {\Sigma} ^ {\frac {1}{2}} \boldsymbol {\epsilon} _ {i}. \tag {11}
$$
An illustration of the random QMC Gaussian samples constructed by our closed-form rank-1 lattice is shown in Figure 1. We can see that our rank-1 lattice QMC Gassuan samples are spaced more evenly w.r.t. the density.
3.2 Fast Exact GP Training and Inference with Rank-1 Lattice
This subsection will show how to perform fast exact GP training and inference using our rank-1 lattice samples with a  $O(n \log n)$  time complexity w.r.t  $n$  samples.
Let  $\pmb{K}_{\theta}$  denotes the kernel Gram matrix, i.e.,  $\pmb{K}_{\theta} = [k_{\theta}(\pmb{x}_i,\pmb{x}_j)]_{1\leq i,j\leq n}$ , the marginal log-likelihood of a GP model [Williams and Rasmussen, 2006] can be formulated as Eq. (12)
$$
\mathcal {L} (p (\boldsymbol {y} | \boldsymbol {X})) = - \frac {1}{2} \boldsymbol {y} ^ {\top} \left(\boldsymbol {K} _ {\theta} + \sigma^ {2} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {y} - \frac {1}{2} \log \left(\left| \boldsymbol {K} _ {\theta} + \sigma^ {2} \boldsymbol {I} \right|\right) - \frac {n}{2} \log 2 \pi . \tag {12}
$$
The standard GP model needs a  $O(n^{3})$  time complexity to compute the marginal log-likelihood, which is prohibitive for fast training as an inner step for stochastic optimization.
In this paper, we construct the random QMC samples based on rank-1 lattice, which enables us to perform fast GP training. Specifically, we build the GP model with the rank-1 lattice as the training data instead of the Gaussian samples. Define modulo kernel as Eq. (13):
$$
k \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) := k _ {\Delta} \left(\phi \left(\boldsymbol {x} _ {i} - \boldsymbol {x} _ {j}\right)\right), \tag {13}
$$
where  $k_{\Delta}(\cdot)$  is a shift-invariant kernel, and the function  $\phi (\pmb {x}_i - \pmb {x}_j)$  is given as Eq. (14)
$$
\phi \left(\boldsymbol {x} _ {i} - \boldsymbol {x} _ {j}\right) = \min  \left(\left(\boldsymbol {x} _ {i} - \boldsymbol {x} _ {j}\right) \bmod 1, \mathbf {1} - \left(\boldsymbol {x} _ {i} - \boldsymbol {x} _ {j}\right) \bmod 1\right), \tag {14}
$$
where operation  $\min (\cdot ,\cdot)$  outputs the minimum among its two inputs element-wise, and mod 1 output the positive fractional parts of its inputs element-wise. The nonnegative fractional part of a real number  $x$  is  $x - \lfloor x\rfloor$ , where  $\lfloor \cdot \rfloor$  denotes the floor function.
For a GP model with a modulo kernel defined in Eq.(13), the kernel Gram matrix is a circulant matrix thanks to the properties of rank-1 lattice. To be concrete, for rank-1 lattice data, we have Eq.(15)
$$
k \left(\boldsymbol {x} _ {i}, \boldsymbol {x} _ {j}\right) = k \left(\boldsymbol {x} _ {i + 1}, \boldsymbol {x} _ {j + 1}\right) = k _ {\Delta} \left(\min  \left(\frac {(i - j) z \bmod n}{n}, 1 - \frac {(i - j) z \bmod n}{n}\right)\right). \tag {15}
$$
Then the marginal log-likelihood  $\mathcal{L}(p(\boldsymbol{y}|\boldsymbol{X}))$  can be computed with a  $O(n\log n)$  time complexity by Fast Fourier Transform (FFT).
Specifically, note that the kernel Gram matrix  $\pmb{K}_{\theta} + \sigma^{2}\pmb{I}$  is a symmetric circulant matrix generated by vector  $\pmb{k}_{\Delta}$ , where  $\pmb{k}_{\Delta}$  is a vector with its  $i^{th}$  element given as Eq. (16).
$$
k _ {\Delta i} = k _ {\Delta} \left(\min  \left(\frac {(i - 1) z \bmod n}{n}, \mathbf {1} - \frac {(i - 1) z \bmod n}{n}\right)\right). \tag {16}
$$
We know that  $K_{\theta} + \sigma^{2}\mathbf{I}$  can be diagonalized as  $K_{\theta} + \sigma^{2}\mathbf{I} = \frac{1}{n} F^{*}\Lambda F$ , where the  $j^{th}$  row and  $k^{th}$  column element of  $F$  is  $F_{jk} = e^{-2\pi jki / n}$ . And the matrix  $\Lambda$  is the diagonal eigenvalue matrix that can be computed as  $\Lambda = \mathrm{diag}(Fk_{\Delta})$ . The matrix-vector product  $Fk_{\Delta}$  can be computed via FFT with  $O(n\log n)$  time complexity. And matrix-vector product  $\frac{1}{n} F^{*}\pmb{v}$  for a vector  $\pmb{v}$  can be computed via inverse FFT. More details about the properties of circulant matrices and fast computation via FFT can be found in [Gray et al., 2006].
Then, we achieve the fast computation of the terms in log-likelihood as Eq.(17) and Eq.(18):
$$
\boldsymbol {y} ^ {\top} \left(\boldsymbol {K} _ {\theta} + \sigma^ {2} \boldsymbol {I}\right) ^ {- 1} \boldsymbol {y} = \boldsymbol {y} ^ {\top} \operatorname {i f f t} (\operatorname {f f t} (\boldsymbol {y}) / \operatorname {f f t} \left(\boldsymbol {k} _ {\Delta}\right)) \tag {17}
$$
$$
\log \left(\left| \boldsymbol {K} _ {\theta} + \sigma^ {2} \boldsymbol {I} \right|\right) = \sum_ {i = 1} ^ {n} \log \left(\lambda_ {i} + \sigma^ {2}\right) = \mathbf {1} ^ {\top} \log \left(\operatorname {f f t} \left(\boldsymbol {k} _ {\Delta}\right)\right), \tag {18}
$$
where  $\mathrm{ifft}(\cdot)$ ,  $\mathrm{fft}(\cdot)$  denotes the inverse FFT and FFT operation, respectively, the operator / in Eq.(17) performs divide element-wise. And the  $\log (\cdot)$  is an element-wise operation. And  $\lambda_{i}$  in Eq.(18) denotes the eigenvalue of kernel Gram matrix  $K_{\theta}$ .
For inference, GP model has closed-form posterior mean and variance [Williams and Rasmussen, 2006] given as Eq.(19) and Eq.(20):
$$
\widehat {m} (\boldsymbol {x}) = \boldsymbol {k} _ {\theta} (\boldsymbol {x}) ^ {\top} \left(\boldsymbol {K} _ {\theta} + \sigma^ {2} I\right) ^ {- 1} \boldsymbol {y} \tag {19}
$$
$$
\hat {\sigma} ^ {2} (\boldsymbol {x}) = k _ {\theta} (\boldsymbol {x}, \boldsymbol {x}) - \boldsymbol {k} _ {\theta} (\boldsymbol {x}) ^ {\top} \left(\boldsymbol {K} _ {\theta} + \sigma^ {2} I\right) ^ {- 1} \boldsymbol {k} _ {\theta} (\boldsymbol {x}), \tag {20}
$$
where  $\pmb{k}_{\theta}(\pmb{x}) = [k_{\theta}(\pmb{x},\pmb{x}_1),\dots,k_{\theta}(\pmb{x},\pmb{x}_n)]^\top$
With rank-1 lattice input data, we can perform fast inference by Eq.(21) and Eq.(22):
$$
\widehat {m} (\boldsymbol {x}) = \boldsymbol {k} _ {\theta} (\boldsymbol {x}) ^ {\top} \operatorname {i f f t} (\operatorname {f f t} (\boldsymbol {y}) / \operatorname {f f t} (\boldsymbol {k} _ {\Delta})) \tag {21}
$$
$$
\widehat {\sigma} ^ {2} (\boldsymbol {x}) = k _ {\theta} (\boldsymbol {x}, \boldsymbol {x}) - \boldsymbol {k} _ {\theta} (\boldsymbol {x}) ^ {\top} \operatorname {i f f t} \left(\operatorname {f f t} \left(\boldsymbol {k} _ {\theta} (\boldsymbol {x})\right) / \operatorname {f f t} \left(\boldsymbol {k} _ {\Delta}\right)\right). \tag {22}
$$
Both the exact GP training and inference benefit from the structure of rank-1 lattice and FFT acceleration, which can be performed with a  $O(n\log n)$  time complexity. A deep learning toolbox, e.g., Pytorch, can be used to train the parameters of the kernel.
Algorithm 1 Fast Coordinate Search
Input: Number of iterations  $T$ , weight vector  $\mathbf{w}$ , and generating vector  $\mathbf{z} = [z_1, \dots, z_d]$  for rank-1 lattice  $\mathbf{X}$ .  
Initialization: Initialize  $\mathbf{x}^*$  by uniformly sampling from grids  $\{0, \frac{1}{n}, \dots, \frac{n - 1}{n}\}^d$ .  
for  $t = 1:T$  do  
    for  $q = 1:d$  do  
        Compute  $c^q = \mathrm{iff}(f t k_{\Delta}^{q}(0)) \odot f t k_{\Delta}^{q} \odot w)$  by Eq.(27).  
        Get the index  $i^*$  of the minimum elements in  $c^q$ , and set  $x_q^* = \frac{i^* z_q \bmod n}{n}$ .  
    end for  
end for  
Return:  $\mathbf{x}^*$
3.3 Fast Coordinate Search for Targeted Sampling
This subsection shows how to perform a fast coordinate search for targeted sampling. A rank-1 lattice with  $n$  points is contained in a grid  $\{0, \frac{1}{n}, \dots, \frac{n - 1}{n}\}^d$ . We thus perform a coordinate descent search from the index set  $\{0, 1, \dots, n - 1\}^d$  to minimize the GP posterior mean in Eq.(19).
Let  $k(\cdot, \cdot) = k_{\Delta}(\cdot)$  be a shift-invariant kernel with a decomposition structure as Eq. (23):
$$
k \left(\boldsymbol {x} ^ {*}, \boldsymbol {x}\right) = k _ {\Delta} \left(\phi \left(\boldsymbol {x} ^ {*} - \boldsymbol {x}\right)\right) = \Pi_ {q = 1} ^ {d} k _ {\Delta} \left(\phi \left(x _ {q} ^ {*} - x _ {q}\right)\right), \tag {23}
$$
where  $x_{q}^{*}, x_{q}$  denotes the  $q^{th}$  element in  $\boldsymbol{x}^{*}$ ,  $\boldsymbol{x}$ , respectively. We can perform a coordinate search by fixing all the components except the  $q^{th}$  one as the current working component for index searching. Formally, let  $\boldsymbol{w} = (\boldsymbol{K}_{\theta} + \sigma^{2}\boldsymbol{I})^{-1}\boldsymbol{y}$ . Then, we have the GP posterior mean function given as Eq. (24):
$$
\widehat {m} \left(\boldsymbol {x} ^ {*}\right) = \boldsymbol {k} _ {\Delta} ^ {q \top} \left(x _ {q} ^ {*}\right) \left(\widehat {\boldsymbol {k}} _ {\Delta} ^ {q} \odot \boldsymbol {w}\right), \tag {24}
$$
where  $\odot$  denotes the element-wise product, and  $k_{\Delta}^{q}(x_{q}^{*})$  denotes a vector with  $i^{th}$  element given as  $k_{\Delta i}^{q} = k_{\Delta}(\phi (x_{q}^{*} - X_{qi}))$ , and  $X_{qi}$  denotes the element in  $q^{th}$ -row and  $i^{th}$ -column of the rank-1 lattice matrix  $\mathbf{X} = [\pmb{x}_1,\dots ,\pmb{x}_n]$ . The vector  $\widehat{\pmb{k}}_{\Delta}^{q}$  denotes the remainder vector with its  $i^{th}$ -element given as Eq. (25):
$$
\widehat {\boldsymbol {k}} _ {\Delta i} ^ {q} = \frac {1}{k _ {\Delta} \left(\phi \left(x _ {q} ^ {*} - \boldsymbol {X} _ {q i}\right)\right)} \Pi_ {q = 1} ^ {d} k _ {\Delta} \left(\phi \left(x _ {q} ^ {*} - \boldsymbol {X} _ {q i}\right)\right). \tag {25}
$$
To optimize the  $q^{th}$  component  $x_{q}^{*}$  of  $\pmb{x}^{*}$ , we fix the other components of  $\pmb{x}^{*}$  and the corresponding vector  $\widehat{\pmb{k}}_{\Delta}^{q}$ . We find  $x_{q}^{*}$  by solving the subproblem given in Eq. (26)
$$
x _ {q} ^ {*} = \underset {x \in \{0, \dots , n - 1 \}} {\arg \min } \boldsymbol {k} _ {\Delta} ^ {q} (x) ^ {\top} \left(\widehat {\boldsymbol {k}} _ {\Delta} ^ {q} \odot \boldsymbol {w}\right). \tag {26}
$$
Directly enumerate computation of the problem (26) needs a  $O(n^{2})$  time complexity. In our paper, we can perform a fast computation with  $O(n\log n)$  time complexity thanks to the rank-1 lattice  $X$ . Specially, when  $X$  is a rank-1 lattice with the generating vector  $z = [z_{1},\dots ,z_{d}]$ , then the matrix  $K_{\Delta}^{q} = [k_{\Delta}^{q}(0),k_{\Delta}^{q}(\frac{1z_{q}\bmod{n}}{n}),\dots ,k_{\Delta}^{q}(\frac{(n - 1)z_{q}\bmod{n}}{n})]$  forms a circulant matrix, and the problem (26) can be accelerated via FFT by Eq. (27)
$$
\boldsymbol {c} ^ {q} = \boldsymbol {K} _ {\Delta} ^ {q \top} \left(\widehat {\boldsymbol {k}} _ {\Delta} ^ {q} \odot \boldsymbol {w}\right) = \operatorname {i f t} \left(\operatorname {f f t} \left(\boldsymbol {k} _ {\Delta} ^ {q} (0)\right) \odot \operatorname {f f t} \left(\widehat {\boldsymbol {k}} _ {\Delta} ^ {q} \odot \boldsymbol {w}\right)\right), \tag {27}
$$