SlowGuess/ACL_2020_OCR huggingface dataset:
where  $\epsilon_1\sim \mathcal{N}(0,I_m)$  and  $\epsilon_{2}\sim \mathcal{N}(0,I_{v})$  . We can simply replace (17) with the above expression in any place that uses  $r$  . In this way, the number of neural network outputs can be dramatically reduced from  $m^2$  to  $mv$
Semantic Hashing (Salakhutdinov and Hinton, 2009) is a promising technique for fast approximate similarity search. Locality-Sensitive Hashing, one of the most popular hashing methods (Datar et al., 2004), projects documents into low-dimensional hash codes in a randomized manner. However, the method does not leverage any information of data, and thus generally performs much worse than those data-dependent methods. Among the data-dependent methods, one of the mainstream methods is supervised hashing, which learns a function that could output similar hash codes for semantically similar documents by making effective use of
the label information (Shen et al., 2015; Liu et al., 2016).
Different from supervised methods, unsupervised hashing pays more attention to the intrinsic structure of data, without making use of the labels. Spectral hashing (Weiss et al., 2009), for instance, learns balanced and uncorrelated hash codes by seeking to preserve a global similarity structure of documents. Self-taught hashing (Zhang et al., 2010), on the other hand, focuses more on preserving local similarities among documents and presents a two-stage training procedure to obtain such hash codes. In contrast, to generate high-quality hash codes, iterative quantization (Gong et al., 2013) aims to minimize the quantization error, while maximizing the variance of each bit at the same time.
Among the unsupervised hashing methods, the idea of generative semantic hashing has gained much interest in recent years. Under the VAE framework, VDSH (Chaidaroon and Fang, 2017) was proposed to first learn continuous the documents' latent representations, which are then cast into binary codes. While semantic hashing is achieved with generative models nicely, the two-stage training procedure is problematic and is prone to result in local optima. To address this issue, NASH (Shen et al., 2018) went one step further and presented an integrated framework to enable the end-to-end training by using the discrete Bernoulli prior and the ST technique, which is able to estimate the gradient of functions with discrete variables. Since then, various directions have been explored to improve the performance of NASH. (Dong et al., 2019) proposed to employ the mixture priors to improve the model's capability to distinguish documents from different categories, and thereby improving the quality of hash codes. On the other hand, a more accurate gradient estimator called Gumbel-Softmax (Jang et al., 2017; Maddison et al., 2017) is explored in Doc2hash (Zhang and Zhu, 2019) to replace the ST estimator in NASH. More recently, to better model the similarities between different documents, (Hansen et al., 2019) investigated the combination of generative models and ranking schemes to generate hash codes. Different from the aforementioned generative semantic hashing methods, in this paper, we focus on how to incorporate correlations into the bits of hash codes.
Datasets Following previous works, we evaluate our model on three public benchmark datasets: i) Reuters21578, which consists of 10788 documents with 90 categories; ii) 20Newsgroups, which contains 18828 newsgroup posts from 20 different topics; iii) TMC, which is a collection of 21519 documents categorized into 22 classes.
Training Details For the conveniences of comparisons, we use the same network architecture as that in NASH and BMSH. Specifically, a 2-layer feed-forward neural network with 500 hidden units and a ReLU activation function is used as an inference network, which receives the TF-IDF of a document as input and outputs the mean and covariance matrix of the Gaussian random variables  $r$ . During training, the dropout (Srivastava et al., 2014) is used to alleviate the overfitting issue, with the keeping probability selected from  $\{0.8, 0.9\}$  based on the performance on the validation set. The Adam optimizer (Kingma and Ba, 2014) is used to train our model, with the learning rate set to 0.001 initially and then decayed for every 10000 iterations. For all experiments on different datasets and lengths of hash codes, the rank  $v$  of matrix  $U$  is set to 10 and the number of component  $k$  in the distribution  $h_k(s)$  is set to 10 consistently, although a systematic ablation study is conducted in Section 5.5 to investigate their impacts on the final performances.
Baselines The following unsupervised semantic hashing baselines are adopted for comparisons: Locality Sensitive Hashing (LSH) (Datar et al., 2004), Stack Restricted Boltzmann Machines (S-RBM) (Salakhutdinov and Hinton, 2009), Spectral Hashing (SpH) (Weiss et al., 2009), Self-Taught Hashing (STH) (Zhang et al., 2010), Variational Deep Semantic Hashing (VDSH) (Chaidaroon and Fang, 2017), Neural Architecture for Generative Semantic Hashing (NASH) (Shen et al., 2018), and Semantic Hashing model with a Bernoulli Mixture prior (BMSH)(Dong et al., 2019).
Evaluation Metrics The performance of our proposed approach is measured by retrieval precision i.e., the ratio of the number of relevant documents to that of retrieved documents. A retrieved document is said to be relevant if its label is the same as that of the query one. Specifically, during the eval
uating phase, we first pick out top 100 most similar documents for each query document according to the hamming distances of their hash codes, from which the precision is calculated. The precisions averaged over all query documents are reported as the final performance.
5.2 Results of Generative Semantic Hashing
The retrieval precisions on datasets TMC, Reuters and 20Newsgroups are reported in Tables 1, 2 and 3, respectively, under different lengths of hash codes. Compared to the generative hashing method NASH without considering correlations, we can see that the proposed method, which introduces correlations among bits by simply employing the distribution of Boltzmann machine as the posterior, performs significantly better on all the three datasets considered. This strongly corroborates the benefits of taking correlations into account when learning the hash codes. From the tables, we can also observe that the proposed model even outperforms the BMSH, an enhanced variant of NASH that employs more complicated mixture distributions as a prior. Since only the simplest prior is used in the proposed model, larger performance gains can be expected if mixture priors are used as in BMSH. Notably, a recent work named RBSH is proposed in (Hansen et al., 2019), which improves NASH by specifically ranking the documents according to their similarities. However, since it employs a different data preprocessing technique as the existing works, we cannot include its results for a direct comparison here. Nevertheless, we trained our model on their preprocessed datasets and find that our method still outperforms it. For details about the results, please refer to Appendix A.4.
Moreover, when examining the retrieval performance of hash codes under different lengths, it is observed that the performance of our proposed method never deteriorates as the code length increases, while other models start to perform poorly after the length of codes reaching a certain level. For the most comparable methods like VDSH, NASH and BMSH, it can be seen that the performance of 128 bits is generally much worse than that of 64 bits. This phenomenon is illustrated more clearly in Figure 1. This may attribute to the reason that for hash codes without correlations, the number of codes will increase exponentially as the code length increases. Because the code space is too large, the probability of assigning similar items
Method	8 bits	16 bits	32 bits	64 bits	128 bits
LSH	0.4388	0.4393	0.4514	0.4553	0.4773
S-RBM	0.4846	0.5108	0.5166	0.5190	0.5137
SpH	0.5807	0.6055	0.6281	0.6143	0.5891
STH	0.3723	0.3947	0.4105	0.4181	0.4123
VDSH	0.4330	0.6853	0.7108	0.4410	0.5847
NASH	0.5849	0.6573	0.6921	0.6548	0.5998
BMSH	n.a.	0.7062	0.7481	0.7519	0.7450
Ours	0.6959	0.7243	0.7534	0.7606	0.7632
Table 1: Precision of the top 100 retrieved documents on TMC dataset.
Method	8 bits	16 bits	32 bits	64 bits	128 bits
LSH	0.2802	0.3215	0.3862	0.4667	0.5194
S-RBM	0.5113	0.5740	0.6154	0.6177	0.6452
SpH	0.6080	0.6340	0.6513	0.6290	0.6045
STH	0.6616	0.7351	0.7554	0.7350	0.6986
VDSH	0.6859	0.7165	0.7753	0.7456	0.7318
NASH	0.7113	0.7624	0.7993	0.7812	0.7559
BMSH	n.a.	0.7954	0.8286	0.8226	0.7941
Ours	0.7589	0.8212	0.8420	0.8465	0.8482
Table 2: Precision of the top 100 retrieved documents on Reuters dataset.
to nearby binary codes may decrease significantly. But for the proposed model, since the bits of hash codes are correlated to each other, the effective number of codes can be determined by the strength of correlations among bits, effectively restricting the size of code space. Therefore, even though the code length increases continually, the performance of our proposed model does not deteriorate.
5.3 Empirical Study of Computational Efficiency
To show the computational efficiency of our proposed method, we also report the average running time per epoch in GPU on TMC dataset, which is of the largest among the considered ones, in Table 4. As a benchmark, the average training time of vanilla NASH is 2.553s per epoch. It can be seen that because of the use of low-rank parameterization of the covariance matrix, the proposed model can be trained almost as efficiently as vanilla NASH, but deliver a much better performance.
5.4 Hash Codes Visualization
To further investigate the capability of different models in generating semantic-preserving binary codes, we project the hash codes produced by VDSH, NASH and our proposed model on 20Newsgroups datasets onto a two-dimensional plane by using the widely adopted UMAP technique (McInnes
Figure 1: Retrieval precisions of unsupervised hashing methods on three datasets under different code lengths.
(a) VDSH
(b) NASH
(c) Ours
Figure 2: Visualization of the 128-bit hash codes learned by VDSH, NASH and our model on 20Newsgroups dataset respectively. Each data point in the figure above denotes a hash code of the corresponding document, and each color represents one category.
Method	8 bits	16 bits	32 bits	64 bits	128 bits
LSH	0.0578	0.0597	0.0666	0.0770	0.0949
S-RBM	0.0594	0.0604	0.0533	0.0623	0.0642
SpH	0.2545	0.3200	0.3709	0.3196	0.2716
STH	0.3664	0.5237	0.5860	0.5806	0.5443
VDSH	0.3643	0.3904	0.4327	0.1731	0.0522
NASH	0.3786	0.5108	0.5671	0.5071	0.4664
BMSH	n.a.	0.5812	0.6100	0.6008	0.5802
Ours	0.4389	0.5839	0.6183	0.6279	0.6359
et al., 2018) and then visualize them on the two-dimensional planes, as shown in Figure 2. It can be seen that the hash codes produced by VDSH are quite mixed for documents from different categories, while those produced by NASH are more distinguishable, consistent with the hypothesis that NASH is able to produce better codes than VDSH thanks to the end-to-end training. From the figure, we can further observe that the hash codes produced by our proposed method are the most distinguishable among all three methods considered, corroborating the benefits of introducing correlations among the bits of hash codes.
Table 3: Precision of the top 100 retrieved documents on 20Newsgroups dataset.
Value of v	Value of k	Avg. Time (seconds)
1	1	2.934
1	5	3.124
5	1	3.137
5	5	3.353
10	5	3.403
10	10	3.768
Table 4: Average running time per epoch on TMC dataset under different values of  $v$  and  $k$  .
5.5 Analyses on the Impacts of  $v$  and  $k$
$\mathbf{R}\mathbf{\nabla}v$  Low-rank perturbed covariance matrix enables the proposed model to trade-off between complexity and performance. That is, larger  $v$  allows the model to capture more dependencies among latent variables, but the required computational complexity also increases. To investigate its impacts, we evaluate the performance of the 64-bit hash codes obtained from the proposed model under different values of  $v$ , with the other key parameter  $k$  fixed to 10. The result is listed in the left half of Table 5. Notably, the proposed model with  $v = 0$  is equivalent to NASH since there is not any correlation between the binary random variables. It can be seen that as the number of ranks
Value of v	Precision	Value of k	Precision
0	0.7812	1	0.8300
1	0.8353	3	0.8391
5	0.8406	5	0.8395
10	0.8465	10	0.8465
Table 5: Left: Retrieval precisions under different values of  $v$  with  $k$  fixed to be 10 on Reuters dataset; Right: Retrieval precision under different values of  $k$  with  $v$  fixed to be 10 on Reuters dataset.
increases, the retrieval precisions also increase, justifying the hypothesis that employing the posteriors with correlations can increase the model's representational capacity and thereby improves the hash codes' quality in turn. It is worth noting that the most significant performance improvement is observed between the models with  $v = 0$  and  $v = 1$ , and then as the value of  $v$  continues to increase, the improvement becomes relatively small. This indicates that it is feasible to set the  $v$  to a relatively small value to save computational resources while retaining competitive performance.
The number of mixture components  $k$  As stated in Section 3.3, increasing the number of components  $k$  in the mixture distribution  $h_k(s)$  will reduce the gap between the lower bound  $\widetilde{\mathcal{L}}_k$  and the ELBO  $\mathcal{L}$ . To investigate the impacts of  $k$ , the retrieval precisions of the proposed model are evaluated under different values of  $k$ , while setting the other key parameter  $v = 10$ . It can be seen from the right half of Table 5 that as the number of components  $k$  increases, the retrieval precision also increases gradually, suggesting that a tighter lower bound  $\widetilde{\mathcal{L}}_k$  can always indicate better hash codes. Hence, if more mixture components are used, better hash codes can be expected. Due to the sake of complexity, only 10 components are used at most in the experiments.
In this paper, by employing the distribution of Boltzmann machine as the posterior, we show that correlations can be efficiently introduced into the bits. To facilitate training, we first show that the BM distribution can be augmented as a hierarchical concatenation of a Gaussian-like distribution and a Bernoulli distribution. Then, an asymptotically-exact lower bound of ELBO is further developed to tackle the tricky normalization term in Boltzmann machines. Significant performance gains are
observed in the experiments after introducing correlations into the bits of hash codes.
This work is supported by the National Natural Science Foundation of China (NSFC) (No. 61806223, U1711262, U1501252, U1611264, U1711261), National Key R&D Program of China (No. 2018YFB1004404), and Fundamental Research Funds for the Central Universities (No. 191gjc04). Also, CC appreciates the support from Yahoo! Research.
David H. Ackley, Geoffrey E. Hinton, and Terrence J. Sejnowski. 1985. A learning algorithm for boltzmann machines. Cognitive Science, 9(1):147-169.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv preprint arXiv:1308.3432.
Suthee Chaidaroon and Yi Fang. 2017. Variational deep semantic hashing for text documents. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '17, pages 75-84, New York, USA.
Mayur Datar, Nicole Immorlica, Piotr Indyk, and Vahab S. Mirrokni. 2004. Locality-sensitive hashing scheme based on p-stable distributions. In Proceedings of the Twentieth Annual Symposium on Computational Geometry, SCG '04, pages 253-262, New York, USA.
Guillaume Desjardins, Aaron Courville, Yoshua Bengio, Pascal Vincent, and Olivier Delalleau. 2010. Tempered markov chain monte carlo for training of restricted boltzmann machines. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 145-152, Chia Laguna Resort, Sardinia, Italy.
Wei Dong, Qinliang Su, Dinghan Shen, and Changyou Chen. 2019. Document hashing with mixture-prior generative models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5229-5238, Hong Kong, China.
Yunchao Gong, Svetlana Lazebnik, Albert Gordo, and Florent Perronnin. 2013. Iterative quantization: A procrustean approach to learning binary codes for large-scale image retrieval. IEEE Trans. Pattern Anal. Mach. Intell., 35(12):2916-2929.
Will Grathwohl, Dami Choi, Yuhuai Wu, Geoff Roeder, and David Duvenaud. 2018. Backpropagation through the void: Optimizing control variates for black-box gradient estimation. In International Conference on Learning Representations (ICLR 2018).
Casper Hansen, Christian Hansen, Jakob Grue Simonsen, Stephen Alstrup, and Christina Lioma. 2019. Unsupervised neural generative semantic hashing. In Proceedings of the 42Nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR'19, pages 735-744, New York, USA.
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparametrization with gumble-softmax. In International Conference on Learning Representations (ICLR 2017).
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv preprint arXiv:1412.6980.
Michael S. Lew, Nicu Sebe, Chabane Djeraba, and Ramesh Jain. 2006. Content-based multimedia information retrieval: State of the art and challenges. ACM Trans. Multimedia Comput. Commun. Appl., 2(1):1-19.
H. Liu, R. Wang, S. Shan, and X. Chen. 2016. Deep supervised hashing for fast image retrieval. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2064-2072.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables. In International Conference on Learning Representations (ICLR 2017).
Leland McInnes, John Healy, Nathaniel Saul, and Lukas Grossberger. 2018. Umap: Uniform manifold approximation and projection. The Journal of Open Source Software, 3(29):861.
Ruslan Salakhutdinov and Geoffrey Hinton. 2009. Semantic hashing. International Journal of Approximate Reasoning, 50(7):969 - 978.
Dinghan Shen, Qinliang Su, Paidamoyo Chapfuwa, Wenlin Wang, Guoyin Wang, Ricardo Henao, and Lawrence Carin. 2018. NASH: Toward end-to-end neural architecture for generative semantic hashing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2041-2050, Melbourne, Australia.
F. Shen, C. Shen, W. Liu, and H. T. Shen. 2015. Supervised discrete hashing. In 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 37-45.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: A simple way to prevent neural networks
from overfitting. Journal of Machine Learning Research, 15:1929-1958.
Benno Stein, Sven Meyer zu Eissen, and Martin Potthast. 2007. Strategies for retrieving plagiarized documents. In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '07, pages 825-826, New York, USA.
Qinliang Su, Lawrence Carin, et al. 2017a. A probabilistic framework for nonlinearities in stochastic neural networks. In Advances in Neural Information Processing Systems, pages 4486-4495.
Qinliang Su, Xuejun Liao, Chunyuan Li, Zhe Gan, and Lawrence Carin. 2017b. Unsupervised learning with truncated gaussian graphical models. In Thirty-First AAAI Conference on Artificial Intelligence.
Tijmen Tieleman. 2008. Training restricted boltzmann machines using approximations to the likelihood gradient. In Proceedings of the 25th International Conference on Machine Learning, ICML '08, pages 1064-1071, New York, USA.
George Tucker, Andriy Mnih, Chris J Maddison, John Lawson, and Jascha Sohl-Dickstein. 2017. Rebar: Low-variance, unbiased gradient estimates for discrete latent variable models. In Advances in Neural Information Processing Systems, pages 2627-2636.
Yair Weiss, Antonio Torralba, and Rob Fergus. 2009. Spectral hashing. In Advances in Neural Information Processing Systems, pages 1753-1760.
Mingzhang Yin and Mingyuan Zhou. 2019. ARM: Augment-REINFORCE-merge gradient for stochastic binary networks. In International Conference on Learning Representations.
Dell Zhang, Jun Wang, Deng Cai, and Jinsong Lu. 2010. Self-taught hashing for fast similarity search. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '10, pages 18-25, New York, USA.
Dongqing Zhang and Wu-Jun Li. 2014. Large-scale supervised multimodal hashing with semantic correlation maximization. In Twenty-Eighth AAAI Conference on Artificial Intelligence.
Yifei Zhang and Hao Zhu. 2019. Doc2hash: Learning discrete latent variables for documents retrieval. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2235-2240, Minneapolis, Minnesota.
Proof. Making use of completing the square technique, the joint distribution of  $r$  and  $s$  can be decomposed as:
$$
\begin{array}{l} q (s, r) = q (s | r) q (r) \\ = \frac {e ^ {- \frac {1}{2} (r - \mu) ^ {\top} \Sigma^ {- 1} (r - \mu) + r ^ {\top} s}}{| 2 \pi \Sigma | ^ {\frac {1}{2}} Z} \\ = \frac {e ^ {- \frac {1}{2} [ r - (\Sigma s + \mu) ] ^ {\top} \Sigma^ {- 1} [ r - (\Sigma s + \mu) ]}}{| 2 \pi \Sigma | ^ {\frac {1}{2}} Z} e ^ {\mu^ {\top} s + \frac {1}{2} s ^ {\top} \Sigma s} \\ = q (r | s) q (s), \\ \end{array}
$$
$$
\begin{array}{l} q (r | s) = \mathcal {N} (r; \Sigma s + \mu , \Sigma), \\ q (s) = \frac {1}{Z} e ^ {\mu^ {\top} s + \frac {1}{2} s ^ {\top} \Sigma s}. \\ \end{array}
$$
From above, we show that the marginal distribution  $q(s)$  is a Boltzmann machine distribution.
We show the following facts about the proposed lower bound of ELBO  $\tilde{\mathcal{L}}_k$ .
First, For any integer  $k$ , we have  $\widetilde{\mathcal{L}}_{k + 1} \geq \widetilde{\mathcal{L}}_k$ . For brevity we denote  $\mathbb{E}_{q_{\phi}(r^{(1,\dots ,k)}|x)}$  as  $\mathbb{E}_{r^{1..k}}$ . First, due to the symmetry of indices, the following equality holds:
$$
\mathbb {E} _ {r ^ {1.. k}} \mathbb {E} _ {q (s | r ^ {(1)})} \log h _ {k} (s) = \mathbb {E} _ {r ^ {1.. k}} \mathbb {E} _ {q (s | r ^ {(i)})} \log h _ {k} (s).
$$
$$
\begin{array}{l} \mathbb {E} _ {r ^ {1.. k}} \mathbb {E} _ {q (s | r ^ {(1)})} \log h _ {k} (s) \\ = \frac {1}{k} \sum_ {i = 1} ^ {k} \mathbb {E} _ {r ^ {1.. k}} \mathbb {E} _ {q (s | r ^ {(1)})} \log h _ {k} (s) \\ = \frac {1}{k} \sum_ {i = 1} ^ {k} \mathbb {E} _ {r ^ {1.. k}} \mathbb {E} _ {q (s | r ^ {(i)})} \log h _ {k} (s) \\ = \mathbb {E} _ {r ^ {1.. k}} \mathbb {E} _ {h _ {k} (s)} \log h _ {k} (s), \\ \end{array}
$$
$$
\begin{array}{l} \mathbb {E} _ {r ^ {1.. k + 1}} \mathbb {E} _ {h _ {k + 1} (s)} \log h _ {k + 1} (s) \\ = \frac {1}{k + 1} \sum_ {i = 1} ^ {k + 1} \mathbb {E} _ {r ^ {1.. k + 1}} \mathbb {E} _ {q (s | r ^ {(i)})} \log h _ {k + 1} (s) \\ = \mathbb {E} _ {r ^ {1.. k + 1}} \mathbb {E} _ {q (s | r ^ {(1)})} \log h _ {k + 1} (s) \tag {27} \\ = \frac {1}{k} \sum_ {i = 1} ^ {k} \mathbb {E} _ {r ^ {1.. k + 1}} \mathbb {E} _ {q (s | r ^ {(i)})} \log h _ {k + 1} (s) \\ = \mathbb {E} _ {r ^ {1.. k + 1}} \mathbb {E} _ {h _ {k} (s)} \log h _ {k + 1} (s). \\ \end{array}
$$
Applying the equality (27) gives us: