SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\begin{array}{l} \widetilde {\mathcal {L}} _ {k + 1} - \widetilde {\mathcal {L}} _ {k} \\ = \mathbb {E} _ {r ^ {1.. k}} \left[ \mathrm {K L} \left(h _ {k} (s) | | q (s | x)\right) \right] \\ - \mathbb {E} _ {r ^ {1.. k + 1}} [ \mathrm {K L} (h _ {k + 1} (s) | | q (s | x)) ] \\ = \mathbb {E} _ {r 1.. k + 1} [ \mathrm {K L} (h _ {k} (s) | | q (s | x)) \\ - \operatorname {K L} \left(h _ {k + 1} (s) | | q (s | x)\right) ] \\ = \mathbb {E} _ {r ^ {1.. k + 1}} \left[ \mathbb {E} _ {h _ {k} (s)} \log h _ {k} (s) - \mathbb {E} _ {h _ {k + 1} (s)} \log h _ {k + 1} (s) \right] \\ = \mathbb {E} _ {r ^ {1.. k + 1}} \left[ \mathbb {E} _ {h _ {k} (s)} \log h _ {k} (s) - \mathbb {E} _ {h _ {k} (s)} \log h _ {k + 1} (s) \right] \\ = \mathbb {E} _ {r ^ {1 \dots k + 1}} \left[ \mathrm {K L} \left(h _ {k} (s) | | h _ {k + 1} (s)\right) \right] \geq 0. \\ \end{array}
$$
We now show that  $\lim_{k\to \infty}\widetilde{\mathcal{L}}_k = \mathcal{L}$ . According to the strong law of large numbers,  $h_k(s) = \frac{1}{k}\sum_j^k q(s|r^{(j)})$  converges to  $\mathbb{E}_{q(r|x)}[q(s|r)] = q(s|x)$  almost surely. We then have
$$
\lim  _ {k \rightarrow \infty} \mathbb {E} _ {r ^ {1.. k}} \left[ \mathrm {K L} \left(h _ {k} (s) | | q (s | x)\right)\right] = 0.
$$
Therefore,  $\widetilde{\mathcal{L}}_k$  approaches  $\mathcal{L}$  as  $k$  approaches infinity.
A.3 Derivation of reparameterization for  $h_k(s)$
Recall that  $h_k(s) = \frac{1}{k} \sum_{j=1}^{k} q(s | r_\phi^{(j)})$ . We show that it can be easily reparameterized. Specifically, we could sample from such a mixture distribution through a two-stage procedure: (i) choosing a component  $c \in \{1, 2, \dots, k\}$  from a uniform discrete distribution, which is then transformed as a  $k$ -dimensional one-hot vector  $\tilde{c}$ ; (ii) drawing a sample from the selected component, i.e.  $q(s | r_\phi^{(c)})$ . Moreover, we define a matrix  $R_\phi(x) \in \mathbb{R}^{m \times k}$  with its columns consisting of  $r_\phi^{(1)}, r_\phi^{(2)}, \dots, r_\phi^{(k)}$ , each of which can be also reparameterized. In this way, a sample  $\tilde{s}_\phi$  from the distribution  $h_k(s)$  can be simply expressed as
$$
\tilde {s} _ {\phi} = \frac {\operatorname {s i g n} \left(\sigma \left(R _ {\phi} \tilde {c}\right) - u\right) + 1}{2}
$$
which can be seen as selecting a sample  $r_{\phi}^{(c)}$  and then passing it through a perturbed sigmoid function. Therefore, during training, the gradients of  $\phi$  are simply back-propagated through the chosen sample  $r_{\phi}^{(c)}$ .
A.4 Comparisons between RBSH and our method
As discussed before, the main reason that we cited this paper but didn't compare with it is that the
Number of Bits	20Newsgroup	TMC
RBSH	Ours	RBSH	Ours
8	0.5190	0.5393	0.7620	0.7667
16	0.6087	0.6275	0.7959	0.7975
32	0.6385	0.6647	0.8138	0.8203
64	0.6655	0.6941	0.8224	0.8289
128	0.6668	0.7005	0.8193	0.8324
Table 6: Precision of the top 100 received documents on 20Newsgroup and TMC datasets.
datasets in (Hansen et al., 2019) are preprocessed differently as ours. Therefore, it is inappropriate to include the performance of the model from (Hansen et al., 2019) into the comparisons of our paper directly. Our work is a direct extension along the research line of VDSH and NASH. In our experiments, we followed their setups and used the preprocessed datasets that are publicized by them. However, in (Hansen et al., 2019), the datasets are preprocessed by themselves. The preprocessing procedure influences the final performance greatly, as observed in the reported results.
To see how our model performs compared to (Hansen et al., 2019), we evaluate our model on the 20Newsgroup and TMC datasets that are preprocessed by the method in (Hansen et al., 2019). The results are reported in Table 6, where RBSH is the model from (Hansen et al., 2019). We can see that using the same preprocessed datasets, our model overall performs better than RBSH, especially in the case of long codes. It should be emphasized that the correlation-introducing method proposed in this paper can be used with all existing VAE-based hashing models. In this paper, the base model is NASH, and when they are used together, we see a significant performance improvement. Since the RBSH is also a VAE-based hashing model, the proposed method can also be used with it to introduce correlations into the code bits, and significant improvements can also be expected.