SlowGuess/ACL_2020_OCR huggingface dataset:
We see that using the encoder embeddings typically outperforms reembedding from scratch, and that global representations tend to outperform local ones, except in the full data regime. Furthermore, we see that the Categorical VAE and VQ-VAE are largely comparable on average, though we undertake a finer-grained comparison by dataset in Appendix A. Perhaps most interestingly, we note that when reembedding from scratch, Hard EM significantly outperforms the other approaches in the lowest data regimes (i.e., for 200 and 500 examples). In fact, Hard EM allows us to match the performance of the best previously reported results even when reembedding from scratch; see Table 3.
Table 2 shows the best combinations of model and hyperparameters when training with 200 labeled examples on AG News. These settings were used in obtaining the numbers in Figure 2, and are largely stable across datasets.
In Figure 3, we compare the average accuracy of our local and global model variants trained on 200 labeled examples, as we vary  $M$ . When reembedding, local representations tend to improve as we move from  $M = 1$  to  $M = 2$ , but not significantly after that. When reembedding global representations, performance increases as  $M$  does. Unsurprisingly, when not reembedding,  $M$  matters less.
Method	K	M
Local	CatVAE	4096	1
Local (re)	Hard EM	1024	1
Global	CatVAE	256	4
Global (re)	Hard EM	4096	4
Table 2: Best methods and settings of  $K$  and  $M$  when training on 200 labeled examples of the AG News corpus and evaluating on the development set. The "(re)" affix indicates that latent variables are reembedded from scratch.
Model	200	500	2500	Full
AG News
CBOW	63.4 (1.5)	72.9 (0.7)	82.1 (0.2)	90.0 (0.2)
VAMPIRE*	83.9 (0.6)	84.5 (0.4)	85.8 (0.2)	-
VAMPIRE	82.2 (0.8)	84.7 (0.2)	86.4 (0.4)	91.0 (0.1)
Local	82.7 (0.1)	84.3 (0.3)	85.0 (0.4)	86.6 (0.2)
Local (re)	82.7 (0.4)	84.0 (0.3)	85.4 (0.1)	87.1 (0.3)
Global	84.6 (0.1)	85.7 (0.1)	86.3 (0.2)	87.5 (0.6)
Global (re)	83.9 (0.5)	84.6 (0.2)	85.1 (0.3)	86.9 (0.1)
DBPedia
CBOW	72.7 (0.6)	84.7 (0.7)	92.8 (0.3)	97.7 (0.1)
VAMPIRE	89.1 (1.3)	93.7 (0.5)	95.7 (0.2)	98.2 (0.1)
Local	89.2 (0.2)	92.8 (0.4)	94.6 (0.2)	97.1 (0.3)
Local (re)	88.7 (0.2)	90.2 (0.3)	93.3 (0.1)	96.9 (0.2)
Global	91.8 (0.5)	94.3 (0.3)	95.0 (0.2)	95.6 (0.0)
Global (re)	88.5 (0.7)	92.3 (0.7)	94.6 (0.4)	95.8 (0.1)
Yelp Full
CBOW	31.0 (5.9)	41.1 (0.6)	48.4 (0.4)	58.9 (0.4)
VAMPIRE	41.4 (2.9)	47.2 (0.7)	52.5 (0.1)	60.3 (0.1)
Local	46.2 (0.8)	49.0 (0.5)	51.9 (0.5)	53.1 (0.3)
Local (re)	47.2 (0.7)	49.4 (0.7)	52.1 (0.2)	55.0 (0.6)
Global	48.5 (1.0)	50.1 (0.5)	53.0 (0.3)	54.9 (0.4)
Global (re)	46.0 (0.5)	47.4 (0.5)	48.8 (0.8)	53.8 (0.3)
Table 3: Test accuracy results by dataset and by the number of labeled examples used in training. The scores are averages over five random subsamples, with standard deviations in parentheses and column bests in bold. VAMPIRE* for AG News is reported by Gururangan et al. (2019) and VAMPIREs are from our experiments.
Finally, we show the final accuracies obtained by our best models on the test data of each dataset in Table 3. We see that on all datasets when there are only 200 or 500 labeled examples, our best model outperforms VAMPIRE and the CBOW baseline, and our models that reembed the latents from scratch match or outperform VAMPIRE. As noted in Table 2, it is Hard EM that is particularly performant in these settings.
8 Analysis and Discussion
8.1 Qualitative analysis
To gain a better understanding of what the learned clusters represent, we examine their patterns on the AG News dataset labeled with four classes. Since VQ-VAEs and Categorical VAEs exhibit similar patterns, we focus on the latter model.
Tables 4 and 5 show examples of sentence- and word-level clusters, respectively, induced by Categorical VAEs. The sentence-level model encodes each document into  $M = 4$  latents, each taking one of  $K = 256$  integers. The word-level model encodes each word into  $M = 1$  latent taking one of  $K = 1024$  integers. Since a word can be assigned multiple clusters, we take the majority cluster for illustration purposes.
We see that clusters correspond to topical aspects of the input (either a document or a word). In particular, in the sentence-level case, documents in the same cluster often have the same ground-truth label. We also find that each of  $M$  latents independently corresponds to topical aspects (e.g.,  $z_{1} = 65$  implies that the topic has to do with technology); thus, taking the combination of these latents seems to make the cluster "purer". The word-level clusters are also organized by topical aspects (e.g., many words in cluster 510 are about modern conflicts in the Middle East).
8.2 Effect of Alternating Optimization
While Hard EM achieves impressive performance when reembedding from scratch and when training on only 200 or 500 examples, we wonder whether this performance is due to the alternating optimization, to the multiple E-step updates per M-step update, or to the lack of sampling. We accordingly experiment with optimizing our VQ-VAE and CatVAE variants in an alternating way, allowing multiple inference network updates per update of the generative parameters  $\theta$ . We show the results on the AG News dataset in Table 6. We find that alternating does generally improve the performance of VQ-VAE and CatVAE as well, though Hard EM performs the best overall when reembedding from scratch. Furthermore, because Hard EM requires no sampling, it is a compelling alternative to CatVAE. For all three methods, we find that doing 3 inference network update steps during alternating optimization performs no better than doing a single one, which suggests that aggressively optimizing the inference network is not crucial in our setting.
Cluster	Class	Text
(23, 155, 24, 53)	World	a platoon in iraq is being investigated for allegedly refusing to carry out a convoy mission...
World	afp chechen warlord shamil basayev has claimed responsibility for the deadly school...
World	the federal government has sent a team of defence personnel to verify a claim that two...
World	an audio tape purportedly by osama bin laden praises gunmen who attacked a us consulate...
(41, 75, 175, 222)	Business	amazon com says it has reached an agreement to buy joyo com, the largest internet retailer...
Business	electronic data systems offered voluntary early retirement to about 9, 200 us employees...
Business	in the aftermath of its purchase of at amp t wireless, cingular wireless is selling several sets...
Sci/Tech	wired amp wireless continues its reign at the top spot among it priorities due to widespread...
(10, 208, 179, 180)	Sports	this is the week of the season when every giants defensive back needs to have shoulders as...
Sports	drew henson will have to wait before he's the star of the dallas cowboys offense right now...
Sports	st louis how do you beat the greatest show on turf with two rookie cornerbacks...
Sports	cincinnati bengals coach marvin lewis said yesterday that he expects quarterback carson...
(65, 224, 78, 114)	Sci/Tech	microsoft acknowledged on monday it continued to battle a technical glitch that prevented...
Sci/Tech	users of the music player should watch out for hacked themes a flaw allows would be...
World	microsoft's popular internet explorer has a serious rival in thefirefox browser
Sci/Tech	microsoft has doubled the period of time it will allow business users of windows xp to...
Table 4: Examples of sentence-level ( $M = 4$ ,  $K = 256$ ) clusters on AG News.
Cluster	Words
822	government of Indonesia guilty prison general prosecutors leader law german sex authorities charged marched issue
651	yankees veteran baltimore quarterback offense tampa steelers giants defensive cleveland minnesota pittsburgh
595	month currency low session dollar euro greenback yen monetary weakening lows versus maintained grip rebounded
305	if despite when although
304	core plans intel athlon operon processors chip hewlett packard strategy clearer forum designs desktop upped ante
802	bit cameras image pleasing integrates multimedia functions gprs automation self types btx supercomputers logic
298	president dick cheney john republicans kerry voters democrat javier sen kellogg
994	exploded bomb near killing injuring explosion eight residents firefighters leak central philippine 55 heavily lancun
484	apple atari san francisco sony toshiba anaheim finally assault famed mp3 freedom u2 accusations brook introduces
510	iraq killed car rebel iraq military suicide forces marines insurgents baghdad evacuation bomber strikes explosions
Table 5: Word-level ( $M = 1$ ,  $K = 1024$ ) clusters on AG News. We take the majority cluster for each word for illustration purposes.
Model	200	200 (re)	500	500 (re)
EM-Local	81.4	82.1	83.0	82.8
EM-Global	85.6	84.6	85.5	85.4
Cat-Local-Alt	83.3	82.9	84.8	84.1
Cat-Global-Alt	86.4	83.1	87.1	85.0
Cat-Local	83.2	82.5	85.3	84.8
Cat-Global	85.4	82.8	86.1	84.5
VQ-Local-Alt	82.9	81.1	84.8	81.4
VQ-Global-Alt	84.7	79.6	85.9	82.9
VQ-Local	82.6	78.7	83.6	81.3
VQ-Global	83.0	76.8	85.4	82.0
Table 6: Effect of alternating optimization on AG News classification with 200 and 500 labels. The “(re)” affix denotes reembedding. Accuracies are on development set with column highs in **bold**.
8.3 Compression
We briefly discuss in what sense discrete latent representations reduce storage requirements. Given a vocabulary of size 30,000, storing a  $T$ -length sentence requires  $T \log_2 30000 \approx 14.9T$  bits. Our
models require at most  $ML\log_2K$  bits to represent a sentence, which is generally smaller, and especially so when using a global representation. It is also worth noting that storing a  $d$ -dimensional floating point representation of a sentence (as continuous latent variable approaches might) costs  $32d$  bits, which is typically much larger.
While the above holds for storage, the space required to classify a sentence represented as  $ML$  integers using a parametric classifier may not be smaller than that required for classifying a sentence represented as a  $d$ -dimensional floating point vector. On the other hand, nearest neighbor-based methods, which are experiencing renewed interest (Guu et al., 2018; Chen et al., 2019; Wiseman and Stratos, 2019), should be significantly less expensive in terms of time and memory when sentences are encoded as  $ML$  integers rather than  $d$ -dimensional floating point vectors. In the next subsection we quantitatively evaluate our discrete representations in a nearest neighbor-based retrieval setting.
Discrete Embedding
M=4, K=256	M=8, K=128	M=16, K=256
Hard EM	76.1	79.6	78.8
CatVAE	77.5	73.7	78.5
VQ-VAE	69.1	73.5	71.2
Continuous Embedding (300d)
L2	COSINE
GloVe	76.4	76.6
fastText	72.8	74.1
Table 7: Unsupervised document retrieval on AG News dataset, measured by average label precision of top 100 nearest neighbors of the development set. Underlined score is the row best. Discrete representations use Hamming distance.
8.4 Nearest Neighbor-Based Retrieval
In the classification experiments of Section 5, we evaluated our discrete representations by training a small classifier on top of them. Here we evaluate our global discrete representations in a document retrieval task to directly assess their quality; we note that this evaluation does not rely on the learned code books, embeddings, or a classifier.
In these experiments we use each document in the development set of the AG News corpus as a query to retrieve 100 nearest neighbors in the training corpus, as measured by Hamming distance. We use average label precision, the fraction of retrieved documents that have the same label as the query document, to evaluate the retrieved neighbors. We compare with baselines that use averaged  $300d$  pretrained word vectors (corresponding to each token in the document) as a representation, where neighbors are retrieved based on cosine or  $\mathcal{L}_2$  distance. We use GloVe with a 2.2 million vocabulary (Pennington et al., 2014) and fastText with a 2 million vocabulary (Mikolov et al., 2018). The results are in Table 7. We see that CatVAE and Hard EM outperform these CBOW baselines (while being significantly more space efficient), while VQ-VAE does not. These results are in line with those of Figure 2, where VQ-VAE struggles when its code book vectors cannot be used (i.e., when reembedding from scratch).
In Figure 4 we additionally experiment with a slightly different setting: Rather than retrieving a fixed number of nearest neighbors for a query document, we retrieve all the documents within a neighborhood of Hamming distance  $\leq D$ , and calculate the average label precision. These results use global representations with  $M = 16$ , and we therefore examine thresholds of  $D \in \{0, \dots, 16\}$ . We
Figure 4: Retrieving document clusters with Hamming distance  $\leq D$ , for global models with  $M = 16$  and  $K = 256$ . Query and target documents are from AG News's development set and training set respectively. Dot size indicates the number of documents in a cluster.
see that for CatVAE and Hard EM, the document similarity (or label precision) has an approximately linear correlation with Hamming distance. On the other hand, VQ-VAE shows a more surprising pattern, where high precision is not achieved until  $D = 10$ , perhaps suggesting that a large portion of the latent dimensions are redundant.
We have presented experiments comparing the discrete representations learned by a Categorical VAE, a VQ-VAE, and Hard EM in terms of their ability to improve a low-resource text classification system, and to allow for nearest neighbor-based document retrieval. Our best classification models are able to outperform previous work, and this remains so even when we reembed discrete latents from scratch in the learned classifier. We find that amortized Hard EM is particularly effective in low-resource regimes when reembedding from scratch, and that VQ-VAE struggles in these settings.
This material is based upon work supported by the Air Force Office of Scientific Research under award number FA9550-18-1-0166.
Maruan Al-Shedivat and Ankur Parikh. 2019. Consistency by agreement in zero-shot neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),
pages 1184-1197, Minneapolis, Minnesota. Association for Computational Linguistics.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. 2013. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv:1308.3432.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning, pages 10-21, Berlin, Germany. Association for Computational Linguistics.
Peter F. Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The mathematics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2):263-311.
Mingda Chen and Kevin Gimpel. 2018. Smaller text classifiers with discriminative cluster embeddings. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 739-745, New Orleans, Louisiana. Association for Computational Linguistics.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019. A multi-task approach for disentangling syntax and semantics in sentence representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2453-2464.
Carl De Marcken. 1995. Lexical heads, phrase structure and the induction of grammar. In Third Workshop on Very Large Corpora.
Wei Dong, Qinliang Su, Dinghan Shen, and Changyou Chen. 2019. Document hashing with mixture-prior generative models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5229-5238.
Suchin Gururangan, Tam Dang, Dallas Card, and Noah A. Smith. 2019. Variational pretraining for semi-supervised text classification. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5880-5894, Florence, Italy. Association for Computational Linguistics.
Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, and Percy Liang. 2018. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437-450.
Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019. Lapping inference networks and posterior collapse in variational autoencoders. In Proceedings of International Conference on Learning Representations (ICLR).
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long Short-Term Memory. Neural Computation, 9:1735-1780.
Eric Jang, Shixiang Gu, and Ben Poole. 2017. Categorical reparameterization with Gumbel-Softmax. In Proceedings of International Conference on Learning Representations (ICLR).
Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. 1999. An introduction to variational methods for graphical models. Machine learning, 37(2):183-233.
Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. 2018. Fast decoding in sequence models using discrete latent variables. In Proceedings of the 35th International Conference on Machine Learning, pages 2390-2399, Stockholm, Sweden. PMLR.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751, Doha, Qatar. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of International Conference on Learning Representations (ICLR).
Diederik P. Kingma, Tim Salimans, Rafal Jozefowicz, Xi Chen, Ilya Sutskever, and Max Welling. 2016. Improving variational inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems, pages 4743-4751.
Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In Proceedings of International Conference on Learning Representations (ICLR).
Bohan Li, Junxian He, Graham Neubig, Taylor Berg-Kirkpatrick, and Yiming Yang. 2019. A surprisingly effective fix for deep latent variable modeling of text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3601-3612.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. 2017. The Concrete distribution: A continuous relaxation of discrete random variables. In Proceedings of International Conference on Learning Representations (ICLR).
Chris J. Maddison, Daniel Tarlow, and Tom Minka. 2014. A* sampling. In Advances in Neural Information Processing Systems, pages 3086-3094.
Tomas Mikolov, Edouard Grave, Piotr Bojanowski, Christian Puhrsch, and Armand Joulin. 2018. Advances in pre-training distributed word representations. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018).
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems, pages 3111-3119.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pages 6306-6315.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Ali Razavi, Aaron van den Oord, and Oriol Vinyals. 2019. Generating diverse high-fidelity images with VQ-VAE-2. In Advances in Neural Information Processing Systems, pages 14866-14876.
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014. Stochastic backpropagation and approximate inference in deep generative models. In Proceedings of the 31st International Conference on Machine Learning, pages 1278-1286, Beijing, China. PMLR.
Valentin I. Spitkovsky, Hiyan Alshawi, Daniel Jurafsky, and Christopher D. Manning. 2010. Viterbi training improves unsupervised dependency parsing. In Proceedings of the Fourteenth Conference on Computational Natural Language Learning, pages 9-17, Uppsala, Sweden. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Ronald J. Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8.
Sam Wiseman and Karl Stratos. 2019. Label-agnostic sequence labeling by copying nearest neighbors. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5363-5369.
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. In Proceedings of the 34th International Conference on Machine Learning, pages 3881-3890, Sydney, Australia. PMLR.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, pages 649-657.
Tiancheng Zhao, Kyusong Lee, and Maxine Eskenazi. 2018. Unsupervised discrete sentence representation learning for interpretable neural dialog generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1098-1107, Melbourne, Australia. Association for Computational Linguistics.
A Model Comparison by Datasets
We plot the development set classification performance of each method, this time distinguishing between datasets, in Figure 5.
Figure 5: The accuracies obtained by Hard EM, Categorical VAE, and VQ-VAE representations on the development datasets of AG News (top), DBPedia (middle), and Yelp Full (bottom), for different numbers of labeled training examples. Triangular and circular markers correspond to global and local models, respectively. Unshaded and shaded markers correspond to reembedding from scratch and using encoder embeddings, respectively.