SlowGuess/NAACL_2021_OCR huggingface dataset:
Datasets	Models	Topic Coherence / Topic Uniqueness
25	50	100	200
IMDB	LDA	0.395 / 20.3	0.387 / 30.1	0.383 / 33.9	0.391 / 34.4
sLDA	0.421 / 15.8	0.376 / 18.9	0.291 / 13.5	0.288 / 14.6
JST	0.472 / 22.7	0.526 / 26.8	0.527 / 29.3	0.530 / 31.1
NVDM	0.281 / 15.8	0.284 / 30.2	0.273 / 50.3	0.266 / 54.8
GSM	0.384 / 22.4	0.402 / 21.0	0.410 / 39.7	0.394 / 42.4
NTM	0.423 / 28.8	0.508 / 28.6	0.513 / 24.1	0.523 / 23.5
PROLDA	0.502 / 31.1	0.543 / 30.8	0.566 / 27.7	0.558 / 29.2
SCHOLAR	0.550 / 28.4	0.616 / 27.0	0.618 / 29.7	0.593 / 31.5
DIATOM	0.544 / 37.1	0.639 / 38.1	0.626 / 36.5	0.615 / 30.7
GoodReads	LDA	0.441 / 19.6	0.463 / 33.5	0.455 / 41.6	0.462 / 40.3
sLDA	0.432 / 34.4	0.387 / 47.3	0.313 / 25.6	0.315 / 23.8
JST	0.465 / 43.5	0.549 / 46.2	0.560 / 47.6	0.551 / 45.2
NVDM	0.294 / 40.8	0.323 / 30.2	0.287 / 48.3	0.264 / 46.9
GSM	0.411 / 24.8	0.481 / 40.1	0.482 / 38.1	0.473 / 41.4
NTM	0.421 / 23.5	0.523 / 47.6	0.493 / 33.4	0.465 / 38.7
PROLDA	0.551 / 30.3	0.562 / 41.8	0.564 / 39.8	0.556 / 37.7
SCHOLAR	0.545 / 38.3	0.603 / 42.0	0.681 / 41.2	0.664 / 38.4
DIATOM	0.582 / 54.0	0.634 / 52.9	0.628 / 54.9	0.609 / 48.7
Amazon	LDA	0.430 / 28.9	0.447 / 47.5	0.438 / 64.8	0.445 / 59.3
sLDA	0.421 / 67.7	0.393 / 62.1	0.323 / 87.5	0.331 / 74.8
JST	0.450 / 73.0	0.558 / 71.2	0.544 / 78.8	0.518 / 70.9
NVDM	0.278 / 42.4	0.310 / 32.5	0.281 / 38.4	0.261 / 49.1
GSM	0.441 / 53.2	0.451 / 60.0	0.433 / 61.7	0.427 / 64.4
NTM	0.493 / 52.8	0.501 / 53.1	0.547 / 55.3	0.508 / 59.3
PROLDA	0.492 / 63.4	0.543 / 51.4	0.528 / 58.7	0.551 / 62.1
SCHOLAR	0.548 / 60.5	0.587 / 65.1	0.641 / 63.2	0.629 / 68.2
DIATOM	0.563 / 82.0	0.598 / 82.3	0.626 / 80.8	0.636 / 78.5
Table 2: Topic Coherence and Topic Uniqueness results for 25/50/100/200 topics. The best result in each column and for each dataset is highlighted in bold.
We report the results in terms of topic coherence/uniqueness, sentiment classification and topic disentanglement rate. We also perform ablation studies to gain more insights into our model.
5.1 Topic Coherence and Uniqueness
We conduct thorough experimental evaluations to assess the quality and disentanglement rate of extracted topics. To assess the quality of topics, we compute their topic coherence (Röder et al., 2015) coupled with their topic uniqueness. We evaluate the topic coherence using the  $C_V$  metric, a slightly refined Normalized Pointwise Mutual Information (NPMI) score using a boolean sliding window to determine the words' context (Röder et al., 2015).
Additionally, we monitor the topic uniqueness (TU) to measure word redundancy across topics. Following Nan et al. (2019), we use  $cnt(l,k)$  to denote the total number of times the top word  $l$  in topic  $k$  appears among the top words across all topics, then  $TU(k) = \frac{1}{L}\sum_{l=1}^{L}\frac{1}{cnt(l,k)}$ . TU is inversely proportional to the number of times each word appears in topics; a higher TU score implies that the top words are rarely repeated and, therefore,
more diverse and unique topics.
In Table 2, we report the topic coherence and the topic uniqueness values. The supervised document label information was incorporated into sLDA, JST, SCHOLAR and DIATOM. Other models are purely unsupervised. We can observe that among conventional LDA-based models, JST performs significantly better compared to both LDA and sLDA for different topic settings and across all datasets. Neural topic models give mixed results. In terms of topic coherence, the trend is SCHOLAR  $>$  PROLDLA  $>$  NTM  $>$  GSM  $>$  NDVM. However, when we examine the topic uniqueness values, we can see that higher topic coherence values do not necessarily lead to higher topic uniqueness values. This shows that the topic coherence value could be misleading sometimes since a high topic coherence could be due to the redundancy of words across topics. We also notice that models with supervised document label information (except sLDA) generally outperform the unsupervised ones. This shows that the document label information can indeed help to extract more meaningful topics. When compared our proposed DI-ATOM with the baselines, we can observe that it achieves better coherence and topic uniqueness
Models	IMDB	GoodReads	Amazon
SVM
+ TFIDF	0.672 ± 0.02	0.711 ± 0.01	0.661 ± 0.02
+ TFIDF + Lexicon	0.683 ± 0.02	0.719 ± 0.02	0.667 ± 0.02
+ LDA	0.615 ± 0.02	0.659 ± 0.02	0.594 ± 0.01
sLDA	0.637 ± 0.01	0.652 ± 0.01	0.579 ± 0.01
JST	0.639 ± 0.01	0.518 ± 0.01	0.538 ± 0.01
SCHOLAR	0.645 ± 0.02	0.673 ± 0.03	0.613 ± 0.02
DIATOM	0.726 ± 0.03	0.704 ± 0.02	0.686 ± 0.02
- w/o Plot Network	0.734 ± 0.03	0.695 ± 0.03	0.603 ± 0.02
Table 3: Sentiment classification accuracy with 50 topics over the test set.
values most of the time, showing the benefit of separating opinion-bearing topics from plot topics by adversarial learning.
5.2 Sentiment Classification
In this section, we compare DIATOM with other supervised topic models for sentiment classification. The purpose of this evaluation is to highlight the discriminative power of the generated representations for the labels of interest while having attractive and unique properties as topic models, rather than confronting them with current state-of-the-art for text classification. We additionally report some baseline results using a Support Vector Machine (SVM) which has been widely employed on these task (Pang and Lee, 2004b) providing an understanding of the relative differences in performance of different approaches.
Table 3 shows the sentiment classification accuracy. In JST, the supervised document label information is only incorporated as prior to the model, while both sLDA and SCHOLAR treat the class label of each document as a response variable and jointly model both documents and their responses. We can observe that the latter is more effective in incorporating supervised information since both sLDA and SCHOLAR outperform JST in general. But DIATOM gives significantly better results all over the baselines with the improvement over the best baseline model, SCHOLAR, by  $3 - 8\%$ . In our models, features used for sentiment classification are opinion-bearing topics. This shows that separating opinion topics from plot/neutral topics is beneficial for sentiment classification. We also observe that the contribution of the plot network to sentiment classification is dataset-dependent. The use of plot network largely boosts the sentiment classification accuracy by over  $8\%$  on the Amazon dataset. But its effect is negligible on the other two datasets.
When compared with traditional sentiment classification models such as SVM, we found that DI-ATOM outperforms SVM trained with various features on both IMDB and Amazon. But it performs slightly worse than SVM trained with TFIDF features with or without an additional incorporation of sentiment lexicon features. Nevertheless, DI-ATOM gives superior performance compared to SVM trained on LDA topic features in the range of  $5 - 11\%$  , showing the effectiveness of using opinion topics for sentiment classification.
5.3 Topic Disentanglement
None of the aforementioned measures can, however, capture how opinion and plot topics are distributed. To this aim, we use topic labeling to assign a proxy label (Positive, Negative, Plot, None) to each topic and then measure the topic-disentanglement rate  $\rho$  as the proportion of opinion-bearing topics with respect to the overall set of topics, complementary to the proportion of plot/neutral topics:  $\rho = \frac{S}{S + K}$ , with  $S$  being the number of opinion topics and  $K$  the number of plot/neutral topics.
For each topic, we first calculate its embedding by taking the normalized weighted average of the vectors of its top  $N$  words:  $\vec{t}_z = \frac{1}{N}\sum_{i=1}^N\alpha_i \times \vec{w}_i$ , where  $\alpha_i$  is the normalized distribution of word  $w_i$  in topic  $z$ . We then retrieve the top 10 most similar sentences from the human-annotated sentence set measured by the cosine similarity between the topic embedding and each sentence embedding. The sentence embedding is computed using the Sentence-BERT encoder (Reimers and Gurevych, 2019). The most frequent label among the retrieved sentences is adopted as the topic's label.
To highlight the disentanglement capability of DIATOM, in Figure 3, we analyse how the proportion of opinion-bearing topics varies across standard and sentiment topic models. We notice that
Plot/Neutral Topics
Amazon - Topic 1 Dent, Gotham, City, Gordon, Bruce, Wayne, Harvey, Joker, Criminal, Nolan
1. Being imprisoned Batman has enough time to paint a gigantic flaming bat on a bridge while people are literally being executed on the hour.
2. Batman gets with Catwoman... after how hard she sold him out?
Amazon - Topic 2 Gandalf, Frodo, Jackson, Tolkien, Dwarf, Fellowship, Peter, Orc, Ring, Hobbit
1. [...] the myriad inhabitants of Middle-earth, the legendary Rings of Power, and the fellowship of hobbits, elves, dwarfs, and humans-led by the wizard Gandalf (Ian McKellen) and the brave hobbit Frodo.
2. This is the beginning of a trilogy; soon to be finalized.
Opinion-Bearing Topics
Amazon - Topic 1 Expectation, Quality, Definitely, Great, Good, Worth, Graphic, Predictable, Compare, Decent
1. Action is good.
2. Rachel Weisz was “mostly” good.
Amazon - Topic 2 Price, Shame, Service, Normally, Purchase, Connection, Greed, Stream, Watch, Frustrate
1. This experience leaves me skeptical of the Amazon Prime video service.
2. Look closely before purchasing.
Table 4: Example topics extracted by DIATOM from Amazon reviews and their associated most similar sentences.
Figure 3: Disentangling rate  $(\%)$  of topic models across different number of topics.
despite the signal from the document labels, sLDA and SCHOLAR tend to produce topics rather balanced in terms of neutral and opinion-bearing topics. JST has a more skewed distribution towards opinion topics. DIATOM instead generates an actual proportion of opinion topics approaching the expected proportion set up by the model, demonstrating the capability to control the generation of plot and opinion-bearing topics.
In Table 4, we show a set of topics grouped according to the disentanglement induced by DI-ATOM from Amazon<sup>8</sup>. For each topic, we report an excerpt of the most similar sentences retrieved. Aside from being overall coherent, we can guess more peculiar plots related for instance to 'The Hobbit' or 'Batman' as in the Plot/Neutral topics. The opinion-bearing topics report a collection of commonly appreciated or critic aspects; some of them are made up of mixed terms describing the
issues and the associated experience (e.g. Topic 2).
Another way to look at the disentangled topics is through the visualization of topic vectors.
As an example, we plot in Figure 4 the 2-dimensional representation of the topic distributions projected by t-SNE for the Amazon dataset. Different colors represent different types of topics generated by DIATOM, namely plot/neutral in blue and opinion in red. We notice how consistently plot/neutral topics tend to cluster together across different number of topics, with the boundary close to polarized topics likely to share common features, as shown in Figure 1 in which the plot topic and the negative topic share a common word 'Dwarf'.
We report in Table 5 the results of the ablation study on DIATOM. We observe that removing the orthogonal regularization has a limited effect on sentiment classification, but causes a fluctuation on topic coherence and a clear drop in topic uniqueness. A significant classification performance drop is observed by removing the sentiment classifier, which essentially reduces DIATOM to an unsupervised model. Removing both the orthogonal regularization and the sentiment classifier shows a major negative impact on both accuracy and the topic's quality. Finally, we assess the influence of the plot network (§3), and while we do not notice any consistent impact across the datasets in terms of sentiment classification, the topics have a notable drop in terms of coherence and diversity.
<sup>8</sup>Topic results from IMDB and GoodRead can be found in the Appendix.
Figure 4: Example of t-SNE projection for the Amazon dataset of the topic distribution for different number of topics. Color are assigned according to plot/neutral and opinion topics.
Datasets	Models	Accuracy	TC / TU
IMDB	DIATOM	0.726 ± 0.03	0.639 / 38.1
- w/o orth reg.	0.723 ± 0.01	0.582 / 27.5
- w/o sent. class.	0.491 ± 0.03	0.601 / 35.4
- w/o both	0.478 ± 0.03	0.544 / 25.4
- w/o Plot Net	0.734 ± 0.03	0.603 / 36.7
GoodReads	DIATOM	0.704 ± 0.02	0.634 / 52.9
- w/o orth. reg.	0.681 ± 0.02	0.612 / 41.1
- w/o sent. class.	0.446 ± 0.02	0.638 / 47.6
- w/o both	0.410 ± 0.02	0.552 / 39.6
- w/o Plot Net	0.695 ± 0.03	0.615 / 49.3
Amazon	DIATOM	0.686 ± 0.02	0.598 / 82.3
- w/o orth reg.	0.682 ± 0.01	0.605 / 55.3
- w/o sent. class.	0.601 ± 0.03	0.573 / 76.9
- w/o both	0.548 ± 0.03	0.567 / 52.1
- w/o Plot Net	0.603 ± 0.02	0.584 / 78.3
Table 5: Ablation study over DIATOM by removing the orthogonal regularization, the sentiment classifier or just the auxiliary Plot Network.
5.6 Further Discussion
Although the adversarial mechanism implemented in DIATOM is rather effective in disentangling opinion and neutral/plot topics, at times the opinion topics could exhibit terms of mixed polarities. An additional adversarial mechanism can be a viable solution at the cost of increasing the model's complexity.
In our current model, the latent plot topics  $z_{a}$  extracted from reviews are encouraged to have a similar discriminative power as the latent topic  $z_{d}$  learned from plots directly for predicting the plots. It is also possible to impose a Gaussian prior centred on  $z_{d}$  for the latent plot topics in reviews instead of using the Gaussian prior of zero mean and unit variance.
While we focus on separating opinion topics
from plot or neutral ones in movie and book reviews in this work, our proposed framework can be applicable in other scenarios. For example, in veracity classification of Twitter rumours, we want to disentangle latent factors which are indicative of veracity of tweets from those which are event-related. Our proposed framework provides a potential solution to it.
We have described DIATOM, a new neural topic model to generate disentangled topics through the combination of VAE and adversarial learning. The results on the novel MOBO dataset show that DIATOM generates better topics in terms of both topic coherence and topic uniqueness, and can disentangle opinion-bearing topics from plot/neutral ones. Finally, we have identified some existing limitations and provided viable solutions to be explored in the future.
Our gratitude goes to all the annotators and to Fabio Miceli and Gianluca Pergola for their relentless effort. This work is funded by the EPSRC (grant no. EP/T017112/1, EP/V048597/1). YH is supported by a Turing AI Fellowship funded by the UK Research and Innovation (UKRI) (grant no. EP/V020579/1).
Alessandro Achille and Stefano Soatto. 2018. Emergence of invariance and disentanglement in deep representations. Journal of Machine Learning Research (JMLR), 19(1):1947-1980.
Emmanuel Bengio, Valentin Thomas, Joelle Pineau, Doina Precup, and Yoshua Bengio. 2017. Independently controllable features. CoRR, abs/1703.07718.
Y. Bengio, A. Courville, and P. Vincent. 2013. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798-1828.
David M. Blei, John D. Lafferty, and Lafferty. 2006. Dynamic topic models. In Proceedings of the 23rd International Conference on Machine Learning, (ICML), pages 113-120.
David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research (JMLR), 3:993-1022.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning.
Dallas Card, Chenhao Tan, and Noah A. Smith. 2018. Neural Models for Documents with Metadata. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, (ACL), pages 2031-2040, Melbourne, Australia.
Tim Cooijmans, Nicolas Ballas, César Laurent, Caglar Gülçehre, and Aaron Courville. 2017. Recurrent batch normalization. In Proceedings of the 2017 International Conference for Learning Representations, (ICLR), Toulouse, France.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, (NAACL). Association for Computational Linguistics.
Adji B. Dieng, Chong Wang, Jianfeng Gao, and John W. Paisley. 2017. Topicnn: A recurrent neural network with long-range semantic dependency. In 5th International Conference on Learning Representations, (ICLR), Toulon, France, April 24-26, 2017, Conference Track Proceedings.
Ran Ding, Ramesh Nallapati, and Bing Xiang. 2018. Coherence-aware neural topic modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, (EMNLP), pages 830-836, Brussels, Belgium.
Babak Esmaeili, Hao Wu, Sarthak Jain, Alican Bozkurt, N Siddharth, Brooks Paige, Dana H. Brooks, Jennifer Dy, and Jan-Willem van de Meent. 2019. Structured disentangled representations. In In Proceedings of Machine Learning Research (PMLR), pages 2525-2534.
Shuyang Gao, Rob Brekelmans, Greg Ver Steeg, and Aram Galstyan. 2019. Auto-encoding total correlation explanation. volume 89 of Proceedings of Machine Learning Research, pages 1157-1166. (PMLR).
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In In Proceedings of the International Conference on Artificial Intelligence and Statistics, (AISTATS).
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative Adversarial Nets. In Advances in Neural Information Processing Systems 27, (NIPS), pages 2672-2680.
Lin Gui, Jia Leng, Gabriele Pergola, yu zhou, Ruifeng Xu, and Yulan He. 2019. Neural topic model with reinforcement learning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). (ACL).
I. Higgins, Loic Matthey, A. Pal, C. Burgess, Xavier Glorot, M. Botvinick, S. Mohamed, and Alexander Lerchner. 2017. Beta-VAE: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Representations, Workshop Track Proceedings, (ICLR).
Irina Higgins, David Amos, David Pfau, Sébastien Racanière, Loïc Matthey, Danilo J. Rezende, and Alexander Lerchner. 2018. Towards a definition of disentangled representations. CoRR, abs/1812.02230.
Tai Hoang, Huy Le, and Tho Quan. 2019. Towards autoencoding variational inference for aspect-based opinion summary. Applied Artificial Intelligence (API), 33:796-816.
Yohan Jo and Alice H Oh. 2011. Aspect and sentiment unification model for online review analysis. In Proceedings of the fourth ACM international conference on Web search and data mining, (WSDM), pages 815-824.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, (ACL), Florence, Italy.
Sudipta Kar, Suraj Maharjan, A. Pastor López-Monroy, and Thamar Solorio. 2018. MPST: A corpus of movie plot synopses with tags. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation, (LREC), Miyazaki, Japan.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proceedings of the 2015 International Conference for Learning Representations, (ICLR).
Diederik P. Kingma, Max Welling, and Welling. 2014. Auto-encoding variational bayes. In 2nd International Conference on Learning Representations, (ICLR).
Abhishek Kumar, Prasanna Sattigeri, and Avinash Balakrishnan. 2017. Variational inference of disentangled latent concepts from unlabeled observations. In 5th International Conference on Learning Representations, (ICLR).
C. Lin, Y. He, R. Everson, and S. Ruger. 2012. Weakly supervised joint sentiment-topic detection from text, 2012. IEEE Transactions on Knowledge and Data Engineering, (6):1134-1145.
Chenghua Lin and Yulan He. 2009. Joint sentiment/topic model for sentiment analysis. In Proceedings of the 18th ACM Conference on Information and Knowledge Management, (CIKM), pages 375-384.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, (ACL), pages 142-150.
James Martens. 2010. Deep learning via hessian-free optimization. In Proceedings of the 27th International Conference on International Conference on Machine Learning, (ICML), pages 735-742.
Tomonari Masada, Atsuhiro Takasu, and Takasu. 2018. Adversarial learning for topic models. In Advanced Data Mining and Applications, (ADMA).
Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton van den Hengel. 2015. Image-based recommendations on styles and substitutes. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval, (SIGIR).
Jon D. Mcauliffe, David M. Blei, and Blei. 2008. Supervised topic models. In Advances in Neural Information Processing Systems 20, (NIPS), pages 121-128.
Yishu Miao, Edward Grefenstette, and Phil Blunsom. 2017. Discovering discrete latent topics with neural variational inference. In Proceedings of The 34th International Conference on Machine Learning, (ICML).
Yishu Miao, Lei Yu, and Phil Blunsom. 2016. Neural variational inference for text processing. In Proceedings of the 33nd International Conference on Machine Learning, (ICML), pages 1727-1736.