SlowGuess/NAACL_2021_OCR huggingface dataset:
Andriy Mnih, Karol Gregor, and Gregor. 2014. Neural variational inference and learning in belief networks. In Proceedings of the 31st International Conference on International Conference on Machine Learning, (ICML).
Feng Nan, Ran Ding, Ramesh Nallapati, and Bing Xiang. 2019. Topic modeling with Wasserstein autoencoders. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, (ACL), pages 6345-6381, Florence, Italy.
Bo Pang and Lillian Lee. 2004a. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 271-278.
Bo Pang and Lillian Lee. 2004b. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL), Barcelona, Spain.
Xingchao Peng, Zijun Huang, Xineng Sun, and Kate Saenko. 2019. Domain agnostic learning with disentangled representations. In Proceedings of the 36th International Conference on Machine Learning, (ICML), pages 5102-5112.
Gabriele Pergola, Lin Gui, and Yulan He. 2019. TDAM: A Topic-Dependent Attention Model for Sentiment Analysis. Information Processing & Management, 56(6):102084.
Gabriele Pergola, Yulan He, and David Lowe. 2018. Topical phrase extraction from clinical reports by incorporating both local and global context. In The 2nd AAAI Workshop on Health Intelligence, (AAAI), pages 499-506.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
Michael Röder, Andreas Both, and Alexander Hinneburg. 2015. Exploring the space of topic coherence measures. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, (WSDM), pages 399-408.
Michal Rosen-Zvi, Thomas Griffiths, Mark Steyvers, and Padhraic Smyth. 2004. The author-topic model for authors and documents. In Proceedings of the 20th Conference on Uncertainty in Artificial Intelligence, (UAI), pages 487-494.
Akash Srivastava, Charles A. Sutton, and A. Sutton. 2017. Autoencoding variational inference for topic models. In 5th International Conference on Learning Representations, (ICLR).
Valentin Thomas, Emmanuel Bengio, William Fedus, Jules Pondard, Philippe Beaudoin, Hugo Larochelle, Joelle Pineau, Doina Precup, and Yoshua Bengio. 2017. Disentangling the independently controllable factors of variation by interacting with the world. pages 2590-2599.
Mengting Wan, Rishabh Misra, Ndapa Nakashole, and Julian J. McAuley. 2019. Fine-grained spoiler detection from large-scale review corpora. In Proceedings of the 57th Conference of the Association for Computational Linguistics, (ACL), Florence, Italy, pages 2605-2610.
Rui Wang, Deyu Zhou, and Yulan He. 2019. Atm: Adversarial-neural topic model. Information Processing & Management, 56(6):102098.
Xuerui Wang, Andrew McCallum, and McCallum. 2006. Topics over time: A non-markov continuous-time model of topical trends. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, (SIGKDD), pages 424-433.
Steven Wilson, Rada Mihalcea, Ryan Boyd, and James Pennebaker. 2016. Disentangling topic models: A cross-cultural analysis of personal values through words. In Proceedings of the First Workshop on NLP and Computational Social Science.
Weidi Xu, Haoze Sun, Chao Deng, and Ying Tan. 2017. Variational autoencoder for semi-supervised text classification. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, (AAAI).
Zichao Yang, Zhiting Hu, Ruslan Salakhutdinov, and Taylor Berg-Kirkpatrick. 2017. Improved variational autoencoders for text modeling using dilated convolutions. In Proceedings of the 34th International Conference on Machine Learning, (ICML), pages 3881-3890.
Hyperparameters We tune the models' hyperparameters on the development set via a random search over combinations of learning rate  $\lambda \in [0.001, 0.5]$ , dropout  $\delta \in [0.0, 0.6]$  and topic vector size  $\gamma_t \in [25, 50, 100, 200]$ . Encoder and decoder are configured following (Srivastava et al., 2017). The hidden representation of documents is set to 100 and sentiment classifier's hidden size to 50. Matrices are randomly initialized with the Xavier and sparse methods (Glorot and Bengio, 2010; Martens, 2010). We employ the Adam optimizer (Kingma and Ba, 2015), set the batch size to 64 and apply batch normalization as additional regularizer (Coolijmans et al., 2017).
Sequential Unfreezing Instead of simultaneously training all the model components, we unfreeze them sequentially. We first freeze the sentiment classifier and update only the autoencoder. At the  $e_{th}$  epoch, we unfreeze the sentiment classifier uniquely on the polarized features to let the classifier training. Finally, at the  $(e + n)_{th}$  epoch,
we unfreeze the adversarial mechanism to drive the generation of neutral features fooling the classifier. We follow an analogous approach with regards to the plot classifier. The values of  $e$  and  $a$  are treated as hyperparameters and chosen through the random search. We found that the sequential unfreezing scheme leads to better topic disentanglement.
A.2 Example Topic Results
In Table A1, we show a set of topics grouped according to the disentanglement induced by DI-ATOM from IMDB and GoodReads. For each topic, we report an excerpt of the most similar sentences retrieved. Aside from being overall coherent, we can guess rather paradigmatic themes as the Topic 1 about peace and war between countries in IMDB-Topic 1. It is worth having a closer look at the IMDB-Topic 2, which despite the negative theme of depression and suicide, the model is able to correctly gather those words under the same plot topic. The opinion-bearing topics report a collection of commonly appreciated or critic aspects; some of them are mainly collections of related adjectives with the same polarity (e.g. IMDB-Topic 1).
Plot/Neutral Topics
IMDB - Topic 1 Government, Country, Peace, Information, Free, Plane, Theory, Anti, Soldier, Hitler
1. Groundbreaking in the realm of socially relevant drama, it dealt with issues such as abortion, domestic violence, student protest, child neglect, illiteracy, slumlords, the anti-war movement, [...].
2. This effort by Charlie ultimately evolves into a major portion of the U.S. foreign policy known as the Reagan Doctrine, under which the U.S. expanded assistance beyond just the [...]
IMDB - Topic 2 Window, Hospital, Apartment, Suicide, Commit, Pitt, Serial, Strange, Killer, Mental
1. Even re-think why two boys/young men would do what they did - commit mutual suicide via slaughtering their classmates.
2. It's the patented scene where the killer creeps up behind the victim.
GoodReads - Topic 1 Cure, Plague, Trial, Betray, Thomas, Secret, Dashner, Ball, Betrayal, Wicked
1. Blaming Cinder for her daughter's illness, Cinder's stepmother volunteers her body for plague research, an "honor" that no one has survived.
2. By age thirteen, she has undergone countless surgeries, transfusions, and shots so that her older sister, Kate, can somehow fight the leukemia that has plagued her since childhood.
GoodReads - Topic 2 Teenager, Fault, Illness, Mental, Depression, Maddy, Grief, Bully, Topic, Greg
1. She's got a lot of mental strength, having been ostracized for most of her life.
2. She went through a divorce, a crushing depression, another failed love, and the eradication of everything she ever thought she was supposed to be.
Opinion-Bearing Topics
IMDB - Topic 1 Badly, Stock, Remove, Poorly, Hype, Ridiculous, Insult, Disaster, Excuse, Lame
1. I can't imagine how anyone could have read this badly written script and given it the greenlight.
2. Although there has obviously been a lot of money spent on them the numbers are badly staged and poorly photographed.
IMDB - Topic 2 Exceptional, Recommend, Excellent, Craft, Believable, Overlook, Vhs, Solid, Festival, Amaze
1. Overall, this is a good film and an excellent adaption.
2. It's great acting, superb cinematography and excellent writing.
GoodReads - Topic 1 Negative, Judge, Note, Pretend, Embarrass, Quality, Extreme, Guilty, Fake, Borrow
1. Can you give something negative stars?
2. And while it must be hard reading negative reviews you need to be able to deal with this in a graceful way (no one likes a sore loser).
GoodReads - Topic 2 Teen, Nice, Normally, Little, Genre, Amuse, Theme, Enjoyment, Blow, Reread
1. What would have made the book a lot more fun to read was more meatier characters in the other girls.
2. But I feel like that was part of the fun of it.
Table A1: Example topics extracted by DIATOM from IMDB and GoodReads and their associated most similar sentences.