SlowGuess/ACL_2020_OCR huggingface dataset:
Lin Zheng $^{1}$ , Qinliang Su $^{1,2*}$ , Dinghan Shen $^{3}$ , Changyou Chen $^{4}$
$^{1}$ School of Data and Computer Science, Sun Yat-sen University
$^{2}$ Guangdong Key Laboratory of Big Data Analysis and Processing, Guangzhou, China
<sup>3</sup>Microsoft Dynamics 365 AI
$^{4}$ CSE Department, SUNY at Buffalo
zhenglin6@mail2.sysu.edu.cn, suqliang@mail.sysu.edu.cn
dishen@microsoft.com, changyou@buffalo.edu
Generative semantic hashing is a promising technique for large-scale information retrieval thanks to its fast retrieval speed and small memory footprint. For the tractability of training, existing generative hashing methods mostly assume a factorized form for the posterior distribution, enforcing independence among the bits of hash codes. From the perspectives of both model representation and code space size, independence is always not the best assumption. In this paper, to introduce correlations among the bits of hash codes, we propose to employ the distribution of Boltzmann machine as the variational posterior. To address the intractability issue of training, we first develop an approximate method to reparameterize the distribution of a Boltzmann machine by augmenting it as a hierarchical concatenation of a Gaussian-like distribution and a Bernoulli distribution. Based on that, an asymptotically-exact lower bound is further derived for the evidence lower bound (ELBO). With these novel techniques, the entire model can be optimized efficiently. Extensive experimental results demonstrate that by effectively modeling correlations among different bits within a hash code, our model can achieve significant performance gains.
Similarity search, also known as nearest-neighbor search, aims to find items that are similar to a query from a large dataset. It plays an important role in modern information retrieval systems and has been used in various applications, ranging from plagiarism analysis (Stein et al., 2007) to content-based multimedia retrieval (Lew et al., 2006), etc. However, looking for nearest neighbors in the Euclidean space is often computationally
prohibitive for large-scale datasets (calculating cosine similarity with high-dimensional vectors is computationally-expensive). Semantic hashing circumvents this problem by representing semantically similar documents with compact and binary codes. Accordingly, similar documents can be retrieved by evaluating the hamming distances of their hash codes much more efficiently.
To obtain similarity-preserving hash codes, extensive efforts have been made to learn hash functions that can preserve the similarity information of original documents in the binary embedding space (Shen et al., 2015; Liu et al., 2016). Existing methods often require the availability of label information, which is often expensive to obtain in practice. To avoid the use of labels, generative semantic hashing methods have been developed. Specifically, the variational autoencoder (VAE) is first employed for semantic hashing in (Chaidaroon and Fang, 2017), and their model is termed VDSH. As a two-step process, the continuous document representations obtained from VAE are directly converted into binary hash codes. To resolve the two-step training problem, Bernoulli priors are leveraged as the prior distribution in NASH (Shen et al., 2018), replacing the continuous Gaussian prior in VDSH. By utilizing straight-through (ST) technique (Bengio et al., 2013), their model can be trained in an end-to-end manner, while keeping the merits of VDSH. Recently, to further improve the quality of hash codes, mixture priors are investigated in BMSH (Dong et al., 2019), while more accurate gradient estimators are studied in Doc2hash (Zhang and Zhu, 2019), both under a similar framework as NASH.
Due to the training-tractability issue, the aforementioned generative hashing methods all assume a factorized variational form for the posterior, e.g., independent Gaussian in VDSH and independent Bernoulli in NASH, BMSH and Doc2hash. This assumption prevents the models from capturing
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 777-788 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
dependencies among the bits of hash codes. Although uncorrelated bits are sometimes preferred in hashing, as reported in (Zhang and Li, 2014), this may not apply to generative semantic hashing. This is due to the fact that the independent assumption could severely limit a model's ability to yield meaningful representations and thereby produce high-quality hash codes. Moreover, as the code length increases (to e.g. 128 bits), the number of possible codes (or simply the code space) will be too large for a dataset with limited number of data points. As a result, we advocate that correlations among bits of a hash code should be considered properly to restrict the embedding space, and thus enable a model to work effectively under a broad range of code lengths.
To introduce correlations among bits of hash codes, we propose to adopt the Boltzmann-machine (BM) distribution (Ackley et al., 1985) as a variational posterior to capture various complex correlations. One issue with this setting, relative to existing efficient training methods, is the inefficiency brought in training. To address this issue, we first prove that the BM distribution can be augmented as a hierarchical concatenation of a Gaussian-like distribution and a Bernoulli distribution. Using this result, we then show that samples from BM distributions can be well reparameterized easily. To enable efficient learning, an asymptotically-exact lower bound of the standard evidence lower bound (ELBO) is further developed to deal with the notorious problem of the normalization term in Boltzmann machines. With the proposed reparameterization and the new lower bound, our model can be trained efficiently as the previous generative hashing models that preserve no bit correlations. Extensive experiments are conducted to evaluate the performance of the proposed model. It is observed that on all three public datasets considered, the proposed model achieves the best performance among all comparable models. In particular, thanks to the introduced correlations, we observe the performance of the proposed model does not deteriorate as the code length increases. This is surprising and somewhat contrary to what has been observed in other generative hashing models.
Generative Semantic Hashing In the context of generative semantic hashing, each document is represented by a sequence of words  $x =$
$\{w_{1},w_{2},\dots ,w_{|x|}\}$  , where  $w_{i}$  is the  $i$  -th word and is denoted by a  $|V|$  -dimensional one-hot vector;  $|x|$  and  $|V|$  denotes the document size (number of words) and the vocabulary size, respectively. Each document  $x$  is modeled by a joint probability:
$$
p _ {\theta} (x, s) = p _ {\theta} (x | s) p (s), \tag {1}
$$
where  $s$  is a latent variable representing the document's hash code. With the probability  $p_{\theta}(x,s)$  trained on a set of documents, the hash code for a document  $x$  can be derived directly from the posterior distribution  $p_{\theta}(s|x)$ . In existing works, the likelihood function, or the decoder takes a form  $p_{\theta}(x|s) = \prod_{i=1}^{|x|} p_{\theta}(w_i|s)$  with
$$
p _ {\theta} (w _ {i} | s) \triangleq \frac {\exp \left(s ^ {T} E w _ {i} + b _ {i}\right)}{\sum_ {j = 1} ^ {| V |} \exp \left(s ^ {T} E e _ {j} + b _ {j}\right)}, \tag {2}
$$
where  $E \in \mathbb{R}^{m \times |V|}$  is the matrix connecting the latent code  $s$  and the one-hot representation of words; and  $e_j$  is the one-hot vector with the only '1' locating at the  $i$ -th position. Documents could be modelled better by using more expressive likelihood functions, e.g., deep neural networks, but as explained in (Shen et al., 2018), they are more likely to destroy the crucial distance-keeping property for semantic hashing. Thus, the simple form of (2) is often preferred in generative hashing. As for the prior distribution  $p(s)$ , it is often chosen as the standard Gaussian distribution as in VDSH (Chaidaroon and Fang, 2017), or the Bernoulli distribution as in NASH and BMSH (Shen et al., 2018; Dong et al., 2019).
Inference Probabilistic models can be trained by maximizing the log-likelihood  $\log p_{\theta}(x)$  with  $p_{\theta}(x) = \int_s p_{\theta}(x,s)ds$ . However, due to the intractability of calculating  $p_{\theta}(x)$ , we instead optimize its evidence lower bound (ELBO), i.e.,
$$
\mathcal {L} = \mathbb {E} _ {q _ {\phi} (s | x)} \left[ \log \frac {p _ {\theta} (x | s) p (s)}{q _ {\phi} (s | x)} \right], \tag {3}
$$
where  $q_{\phi}(s|x)$  is the proposed variational posterior parameterized by  $\phi$ . It can be shown that  $\log p_{\theta}(x) \geq \mathcal{L}$  holds for any  $q_{\phi}(s|x)$ , and that if  $q_{\phi}(s|x)$  is closer to the true posterior  $p_{\theta}(s|x)$ , the bound  $\mathcal{L}$  will be tighter. Training then reduces to maximizing the lower bound  $\mathcal{L}$  w.r.t.  $\theta$  and  $\phi$ . In VDSH (Chaidaroon and Fang, 2017),  $q_{\phi}(s|x)$  takes the form of an independent Gaussian distribution
$$
q _ {\phi} (s | x) = \mathcal {N} \left(s | \mu_ {\phi} (x), \operatorname {d i a g} \left(\sigma_ {\phi} ^ {2} (x)\right)\right), \tag {4}
$$
where  $\mu_{\phi}(x)$  and  $\sigma_{\phi}(x)$  are two vector-valued functions parameterized by multi-layer perceptrons (MLP) with parameters  $\phi$ . Later, in NASH and BMSH (Shen et al., 2018; Dong et al., 2019),  $q_{\phi}(s|x)$  is defined as an independent Bernoulli distribution, i.e.,
$$
q _ {\phi} (s | x) = \operatorname {B e r n o u l l i} \left(g _ {\phi} (x)\right), \tag {5}
$$
where  $g_{\phi}(x)$  is also vector-valued function parameterized by a MLP. The value at each dimension represents the probability of being 1 at that position. The MLP used to parameterize the posterior  $q_{\phi}(s|x)$  is also referred to as the encoder network.
One key requirement for efficient end-to-end training of generative hashing method is the availability of reparameterization for the variational distribution  $q_{\phi}(s|x)$ . For example, when  $q_{\phi}(s|x)$  is a Gaussian distribution as in (4), a sample  $s$  from it can be efficiently reparameterized as
$$
s = \mu_ {\phi} (x) + \sigma_ {\phi} (x) \cdot \epsilon \tag {6}
$$
with  $\epsilon \sim \mathcal{N}(0,I)$ . When  $q_{\phi}(s|x)$  is a Bernoulli distribution as in (5), a sample from it can be reparameterized as
$$
s = \frac {\operatorname {s i g n} \left(g _ {\phi} (x) - \epsilon\right) + 1}{2} \tag {7}
$$
where  $\epsilon \in \mathbb{R}^m$  with elements  $\epsilon_{i}\sim \mathrm{uniform}(0,1)$ . With these reparameterization tricks, the lower bound in (3) can be estimated by the sample  $s$  as
$$
\mathcal {L} \approx \log \frac {p _ {\theta} (x | s _ {\phi}) p (s _ {\phi})}{q _ {\phi} (s _ {\phi} | x)}, \tag {8}
$$
where  $s$  has been denoted as  $s_{\phi}$  to explicitly indicate its dependence on  $\phi$ . To train these hashing models, the backpropagation algorithm can be employed to estimate the gradient of (8) w.r.t.  $\theta$  and  $\phi$  easily. However, it is worth noting that in order to use the reparameterization trick, all existing methods assumed a factorized form for the proposed posterior  $q_{\phi}(s|x)$ , as shown in (4) and (5). This suggests that the binary bits in hash codes are independent of each other, which is not the best setting in generative semantic hashing.
3 Correlation-Enhanced Generative Semantic Hashing
In this section, we present a scalable and efficient approach to introducing correlations into the bits of hash codes, by using a Boltzmann-machine distribution as the variational posterior with approximate reparameterization.
3.1 Boltzmann Machine as the Variational Posterior
Many probability distributions defined over binary variables  $s \in \{0,1\}^m$  are able to capture the dependencies. Among them, the most famous one should be the Boltzmann-machine distribution (Ackley et al., 1985), which takes the following form:
$$
b (s) = \frac {1}{Z} e ^ {\frac {1}{2} s ^ {T} \Sigma s + \mu^ {T} s}, \tag {9}
$$
where  $\Sigma \in \mathbb{R}^{m\times m}$  and  $\mu \in \mathbb{R}^m$  are the distribution parameters; and  $Z\triangleq \sum_{s}e^{\frac{1}{2}s^{T}\Sigma s + \mu^{T}s}$  is the normalization constant. The Boltzmann-machine distribution can be adopted to model correlations among the bits of a hash code. Specifically, by restricting the posterior to the Boltzmann form
$$
q _ {\phi} (s \mid x) = \frac {1}{Z _ {\phi}} e ^ {- E _ {\phi} (s)} \tag {10}
$$
and substituting it into the lower bound of (3), we can write the lower bound as:
$$
\mathcal {L} = \mathbb {E} _ {q _ {\phi} (s | x)} \left[ \log \frac {p _ {\theta} (x | s) p (s)}{e ^ {- E _ {\phi} (s)}} \right] + \log Z _ {\phi}, \tag {11}
$$
where  $E_{\phi}(s)\triangleq -\frac{1}{2} s^T\Sigma_{\phi}(x)s - \mu_{\phi}^T (x)s;$  and  $\Sigma_{\phi}(x)$  and  $\mu_{\phi}(x)$  are functions parameterized by the encoder network with parameters  $\phi$  and  $x$  as input. One problem with such modeling is that the expectation term  $\mathbb{E}_{q_{\phi}(s|x)}[\cdot ]$  in (11) cannot be expressed in a closed form due to the complexity of  $q_{\phi}(s|x)$ . Consequently, one cannot directly optimize the lower bound  $\mathcal{L}$  w.r.t.  $\theta$  and  $\phi$ .
3.2 Reparameterization
An alternative way is to approximate the expectation term by using the reparameterized form of a sample  $s$  from  $q_{\phi}(s|x)$ , as was done in the previous uncorrelated generative hashing models (see (6) and (7)). Compared to existing simple variational distributions, there is no existing work on how to reparameterize the complicated Boltzmann-machine distribution. To this end, we first show that the Boltzmann-machine distribution can be equivalently written as the composition of an approximate correlated Gaussian distribution and a Bernoulli distribution.
Proposition 1. A Boltzmann-machine distribution  $b(s) = \frac{1}{Z} e^{\frac{1}{2}s^{T}\Sigma s + \mu^{T}s}$  with  $\Sigma \succ 0$  can be equivalently expressed as the composition of two distributions, that is,
$$
b (s) = \int p (s | r) p (r) d r, \tag {12}
$$
where  $p(r) = \frac{1}{Z}\prod_{i=1}^{m}(e^{r_i} + 1)\cdot \mathcal{N}(r;\mu,\Sigma)$ ;  $p(s|r) = \prod_{i=1}^{m}p(s_i|r_i)$  with  $s_i$  and  $r_i$  denoting the  $i$ -th element of  $s$  and  $r$ ; and  $p(s_i|r_i) \triangleq \operatorname{Bernoulli}(\sigma(r_i))$  with  $\sigma(\cdot)$  being the sigmoid function.
Based on Proposition 1, we can see that a sample from the Boltzmann-machine distribution  $q_{\phi}(s|x)$  in (10) can be sampled hierarchically as
$$
r \sim q _ {\phi} (r | x) \quad \text {a n d} \quad s \sim \operatorname {B e r n o u l l i} (\sigma (r)), \tag {13}
$$
$$
q _ {\phi} (r | x) = \frac {1}{Z} \prod_ {i = 1} ^ {m} \left(e ^ {r _ {i}} + 1\right) \cdot \mathcal {N} (r; \mu_ {\phi} (x), \Sigma_ {\phi} (x)) \tag {14}
$$
and  $\sigma (\cdot)$  is applied to its argument element-wise. From the expression of  $q_{\phi}(r|x)$ , we can see that for small values of  $r_i$ , the influence of  $(e^{r_i} + 1)$  on the overall distribution is negligible, and thus  $q_{\phi}(r|x)$  can be well approximated by the Gaussian distribution  $\mathcal{N}(r;\mu_{\phi}(x),\Sigma_{\phi}(x))$ . For relatively large  $r_i$ , the term  $(e^{r_i} + 1)$  will only influence the distribution mean, roughly shifting the Gaussian distribution  $\mathcal{N}(r;\mu_{\phi}(x),\Sigma_{\phi}(x))$  by an amount approximately equal to its variance. For problems of interest in this paper, the variances of posterior distribution are often small, hence it is reasonable to approximate samples from  $q_{\phi}(r|x)$  by those from  $\mathcal{N}(r;\mu_{\phi}(x),\Sigma_{\phi}(x))$ .
With this approximation, we can now draw samples from Boltzmann-machine distribution  $q_{\phi}(s|x)$  in (10) approximately by the two steps below
$$
r \sim \mathcal {N} (r; \mu_ {\phi} (x), \Sigma_ {\phi} (x)), \tag {15}
$$
$$
s \sim \operatorname {B e r n o u l l i} (\sigma (r)). \tag {16}
$$
For the Gaussian sample  $r\sim \mathcal{N}(r;\mu_{\phi}(x),\Sigma_{\phi}(x))$  similar to (6), it can be reparameterized as
$$
r = \mu_ {\phi} (x) + L _ {\phi} (x) \cdot \epsilon , \tag {17}
$$
where  $L_{\phi}(x)$  is the Cholesky decomposition matrix of  $\Sigma_{\phi}(x)$  with  $\Sigma_{\phi}(x) = L_{\phi}(x)L_{\phi}^{T}(x)$ ; and  $\epsilon \in \mathbb{R}^m$  with  $\epsilon \sim \mathcal{N}(0, I)$ . It should be noted that in practice, we can define the function  $L_{\phi}(x)$  in advance and then obtain  $\Sigma_{\phi}(x)$  as  $\Sigma_{\phi}(x) = L_{\phi}(x)L_{\phi}^{T}(x)$ , thus the Cholesky decomposition is not needed.
Given the Gaussian sample  $r$ , similar to the reparameterization of Bernoulli variables in (7), we can reparameterize the Bernoulli sample  $s \sim$
Bernoulli  $(\sigma (r))$  as  $s = \frac{\mathrm{sign}(\sigma(r) - u) + 1}{2}$ , where  $u\in \mathbb{R}^m$  with each element  $u_{i}\sim$  uniform(0,1). By combining the above reparameterizations, a sample from the Boltzmann-machine distribution  $q_{\phi}(s|x)$  can then be approximately reparameterized as
$$
s _ {\phi} = \frac {\operatorname {s i g n} \left(\sigma \left(\mu_ {\phi} (x) + L _ {\phi} (x) \cdot \epsilon\right) - u\right) + 1}{2}, \tag {18}
$$
where the subscript  $\phi$  is to explicitly indicate that the sample  $s$  is expressed in terms of  $\phi$ .
With the reparameterization  $s_{\phi}$ , the expectation term in (11) can be approximated as  $\log \frac{p_{\theta}(x|s_{\phi})p(s_{\phi})}{e^{-E_{\phi}(s_{\phi})}}$ . Consequently, the gradients of this term w.r.t. both  $\theta$  and  $\phi$  can be evaluated efficiently by backpropagation, with the only difficulty lying at the non-differentiable function  $\mathrm{sign}(\cdot)$  of  $s_{\phi}$  in (18). Many works have been devoted to estimate the gradient involving discrete random variables (Bengio et al., 2013; Jang et al., 2017; Maddison et al., 2017; Tucker et al., 2017; Grathwohl et al., 2018; Yin and Zhou, 2019). Here, we adopt the simple straight-through (ST) technique (Bengio et al., 2013), which has been found performing well in many applications. By simply treating the hard threshold function  $\mathrm{sign}(\cdot)$  as the identity function, the ST technique estimates the gradient as
$$
\frac {\partial s _ {\phi}}{\partial \phi} \approx \frac {1}{2} \frac {\partial [ \sigma (\mu_ {\phi} (x) + L _ {\phi} (x) \epsilon) - u ]}{\partial \phi}. \tag {19}
$$
Then, the gradient of the first term in ELBO  $\mathcal{L}$  w.r.t.  $\phi$  can be computed efficiently by backpropagation.
3.3 An Asymptotically-Exact Lower Bound
To optimize the ELBO in (11), we still need to calculate the gradient of  $\log Z_{\phi}$ , which is known to be notoriously difficult. A common way is to estimate the gradient  $\frac{\partial \log Z_{\phi}}{\partial \phi}$  by MCMC methods (Tieleman, 2008; Desjardins et al., 2010; Su et al., 2017a,b), which are computationally expensive and often of high variance. By noticing a special form of the ELBO (11), we develop a lower bound for the ELBO  $\mathcal{L}$ , where the  $\log Z_{\phi}$  term can be conveniently cancelled out. Specifically, we introduce another probability distribution  $h(s)$  and lower bound the original ELBO:
$$
\widetilde {\mathcal {L}} = \mathcal {L} - \mathrm {K L} (h (s) | | q _ {\phi} (s | x)). \tag {20}
$$
Since  $\mathrm{KL}(\cdot)\geq 0$  we have  $\widetilde{\mathcal{L}} (\theta ,\phi)\leq \mathcal{L}$  holds for all  $h(s)$  , i.e.,  $\widetilde{\mathcal{L}}$  is a lower bound of  $\mathcal{L}$  , and equals to the ELBO  $\mathcal{L}$  when  $h(s) = q_{\phi}(s|x)$  . For the choice
of  $h(s)$ , it should be able to reduce the gap between  $\widetilde{\mathcal{L}}$  and  $\mathcal{L}$  as much as possible, while ensuring that the optimization is tractable. Balancing on the two sides, a mixture distribution is used
$$
h _ {k} (s) = \frac {1}{k} \sum_ {i = 1} ^ {k} p (s | r ^ {(i)}), \tag {21}
$$
where  $k$  denotes the number of components;  $p(s|r^{(i)})$  is the multivariate Bernoulli distribution and  $r^{(i)}$  is the  $i$ -th sample drawn from  $q_{\phi}(r|x)$  as defined in (14). By substituting  $h_k(s)$  into (20) and taking the expectation w.r.t.  $r^{(i)}$ , we have
$$
\widetilde {\mathcal {L}} _ {k} \triangleq \mathcal {L} - \mathbb {E} _ {q _ {\phi} (r ^ {(1 \dots k)} | x)} [ \mathrm {K L} (h _ {k} (s) | | q _ {\phi} (s | x)) ] \tag {22}
$$
where  $q_{\phi}(r^{(1\dots ,k)}|x) = \prod_{i = 1}^{k}q_{\phi}(r^{(i)}|x)$ . It can be proved that the bound  $\widetilde{\mathcal{L}}_k$  gradually approaches the ELBO  $\mathcal{L}$  as  $k$  increases, and finally equals to it as  $k\to \infty$ . Specifically, we have
Proposition 2. For any integer  $k$ , the lower bound  $\widetilde{\mathcal{L}}_k$  of the ELBO satisfies the conditions: 1)  $\widetilde{\mathcal{L}}_{k + 1}\geq$ $\widetilde{\mathcal{L}}_k$ ; 2)  $\lim_{k\to \infty}\widetilde{\mathcal{L}}_k = \mathcal{L}$ .
By substituting  $\mathcal{L}$  in (11) and  $h_k(s)$  in (21) into (22), the bound can be further written as
$$
\begin{array}{l} \widetilde {\mathcal {L}} _ {k} = \mathbb {E} _ {q _ {\phi} (s | x)} \left[ \log \frac {p _ {\theta} (x | s) p (s)}{e ^ {- E _ {\phi} (s)}} \right] \\ \left. - \mathbb {E} _ {q _ {\phi} (r ^ {(1 \dots k)} | x)} \left[ \mathbb {E} _ {h _ {k} (s)} \left[ \log \frac {h _ {k} (s)}{e ^ {- E _ {\phi} (s)}} \right] \right] \right., \tag {23} \\ \end{array}
$$
where the  $\log Z_{\phi}$  term is cancelled out since it appears in both terms but has opposite signs. For the first term in (23), as discussed at the end of Section 3.1, it can be approximated as  $\log \frac{p_{\theta}(x|s_{\phi})p(s_{\phi})}{e^{-E_{\phi}(s_{\phi})}}$ . For the second term, each sample  $r^{(i)}$  for  $i = 1,\dots ,k$  can be approximately reparameterized like that in (17). Given the  $r^{(i)}$  for  $i = 1,\dots ,k$ , samples from  $h_k(s)$  can also be reparameterized in a similar way as that for Bernoulli distributions in (7). Thus, samples drawn from  $r^{(1\dots k)}\sim q_{\phi}(r^{(1\dots k)}|x)$  and  $s\sim h_k(s)$  are also reparameterizable, as detailed in Appendix A.3. By denoting this reparametrized sample as  $\tilde{s}_{\phi}$ , we can approximate the second term in (23) as  $\log \frac{h_k(\tilde{s}_{\phi})}{e^{-E_{\phi}(\tilde{s}_{\phi})}}$ . Thus the lower bound (23) becomes
$$
\widetilde {\mathcal {L}} _ {k} \approx \log \frac {p _ {\theta} (x | s _ {\phi}) p (s _ {\phi})}{e ^ {- E _ {\phi} (s _ {\phi})}} - \log \frac {h _ {k} (\tilde {s} _ {\phi})}{e ^ {- E _ {\phi} (\tilde {s} _ {\phi})}}. (2 4)
$$
With the discrete gradient estimation techniques like the ST method, the gradient of  $\widetilde{\mathcal{L}}_k$  w.r.t.  $\theta$  and  $\phi$  can then be evaluated efficiently by backpropagation. Proposition 2 indicates that the exact  $\widetilde{\mathcal{L}}_k$  gets closer to the ELBO as  $k$  increases, so better bound can be expected for the approximated  $\widetilde{\mathcal{L}}_k$  as well when  $k$  increases. In practice, a moderate value of  $k$  is found to be sufficient to deliver a good performance.
3.4 Low-Rank Perturbation for the Covariance Matrix
In the reparameterization of a Gaussian sample,  $r_{\phi} = \mu_{\phi}(x) + L_{\phi}(x)\cdot \epsilon$  in (17), a  $m\times m$  matrix  $L_{\phi}(x)$  is required, with  $m$  denoting the length of hash codes. The elements of  $L_{\phi}(x)$  are often designed as the outputs of neural networks parameterized by  $\phi$ . Therefore, if  $m$  is large, the number of neural network outputs will be too large. To overcome this issue, a more parameter-efficient strategy called Low-Rank Perturbation is employed, which restricts covariance matrix to the form
$$
\Sigma = D + U U ^ {\top}, \tag {25}
$$
where  $D$  is a diagonal matrix with positive entries and  $U = [u_{1}, u_{2}, \dots, u_{v}]$  is a low-rank perturbation matrix with  $u_{i} \in \mathbb{R}^{m}$  and  $v \ll m$ . Under this low-rank perturbed  $\Sigma$ , the Gaussian samples can be reparameterized as
$$
r _ {\phi} = \mu_ {\phi} (x) + D _ {\phi} ^ {1 / 2} (x) \cdot \epsilon_ {1} + U _ {\phi} (x) \cdot \epsilon_ {2}, \tag {26}
$$