SlowGuess/NAACL_2021_OCR huggingface dataset:
Gabriele Pergola, Lin Gui, Yulan He
Department of Computer Science, University of Warwick, UK
{gabriele.pergola, lin GUI, yulan.he}@warwick.ac.uk
The flexibility of the inference process in Variational Autoencoders (VAEs) has recently led to revising traditional probabilistic topic models giving rise to Neural Topic Models (NTM). Although these approaches have achieved significant results, surprisingly very little work has been done on how to disentangle the latent topics. Existing topic models when applied to reviews may extract topics associated with writers' subjective opinions mixed with those related to factual descriptions such as plot summaries in movie and book reviews. It is thus desirable to automatically separate opinion topics from plot/neutral ones enabling a better interpretability. In this paper, we propose a neural topic model combined with adversarial training to disentangle opinion topics from plot and neutral ones. We conduct an extensive experimental assessment introducing a new collection of movie and book reviews paired with their plots, namely MOBO dataset, showing an improved coherence and variety of topics, a consistent disentanglement rate, and sentiment classification performance superior to other supervised topic models.
Variational Autoencoders (VAEs) (Kingma et al., 2014) allow to design complex generative models of data. In the wake of the renewed interest for VAEs, traditional probabilistic topic models (Blei et al., 2003) have been revised giving rise to several Neural Topic Model (NTM) variants, such as NVDM (Miao et al., 2016), ProdLDA (Srivastava et al., 2017), and NTM-R (Ding et al., 2018). Although these approaches have achieved significant results via the neural inference process, existing topic models when applied to user reviews may extract topics with writers' subjective opinions mixed with those related to factual descriptions such as plot summaries of movies and books (Lin et al., 2012). Yet surprisingly very little work has been
Disentangled Topics
Figure 1: Disentangled topics extracted by DIATOM from the Amazon reviews for "The Hobbit".
done on how to disentangle the inferred topic representations.
Disentangled representations can be defined as representations where individual latent units are sensitive to variations of a single generative factor, while being relatively invariant to changes of other factors (Bengio et al., 2013; Higgins et al., 2017). Inducing such representations has been shown significantly beneficial for their generalization and interpretability (Achille and Soatto, 2018; Peng et al., 2019). In the context of topic modeling, we propose to consider latent topics as generative factors to be disentangled to improve their interpretability and discriminative power. Disentangled topics are topics invariant to the factors of variation of text, which for instance, in the context of book and movie reviews could be the author's opinion (e.g. positive/negative), the salient parts of a plot or other auxiliary information reported. An illustration of this is shown in Figure 1 in which opinion topics are separated from plot topics.
However, models relying solely on sentiment information are easily misled and not suitable to disentangle opinion from plots, since even plot descriptions frequently make large use of sentiment expressions (Pang and Lee, 2004a). Consider for example the following sentence: "The ring holds a
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2870-2883 June 6-11, 2021. ©2021 Association for Computational Linguistics
dark power, and it soon begins to exert its evil influence on Bilbo", an excerpt from a strong positive Amazon's review.
Therefore, we propose to distinguish opinion-bearing topics from plot/neutral ones combining a neural topic model architecture with adversarial training. In this study, we present the DIsentangled Adversarial TOpic Model (DIATOM)<sup>1</sup>, aiming at disentangling information related to the target labels (i.e. the review score), from other distinct aspects yet possibly still polarised (e.g. plot descriptions). We also introduce a new dataset, namely the MOBO dataset<sup>1</sup>, made up of movie and book reviews, paired with their related plots. The reviews come from different publicly available datasets: IMDB (Maas et al., 2011), GoodReads (Wan et al., 2019) and Amazon reviews (McAuley et al., 2015), and encompass a wide spectrum of domains and styles. We conduct an extensive experimental assessment of our model. First, we assess the topic quality in terms of topic coherence and diversity and compare DIATOM with other supervised topic models on the sentiment classification task; then, we analyse the disentangling rate of topics to quantitatively assess the degree of separation between actual opinion and plot/neutral topics.
- We propose a new model, DIATOM, which is able to generate disentangled topics through the combination of VAE and adversarial learning;
- We introduce the MOBO dataset, a new collection of movie and book reviews paired with their plots;
- We conduct an experimental assessment of our model, highlighting more interpretable topics with better topic coherence and diversity scores compared to others state-of-the-art supervised topic models, and improved discriminative power on sentiment classification, and a consistent topic-disentanglement rate.
Our work is closely related to three lines of research: sentiment-topic models, neural topic models and learning disentangled representations.
Sentiment-Topic Models. Probabilistic graphical models for topic extraction have been extensively studied. Beyond LDA (Blei et al., 2003),
a wide spectrum of models has specialised it to more particular tasks using contextual information (Rosen-Zvi et al., 2004; Wang et al., 2006; Blei et al., 2006; Pergola et al., 2018, 2019). Supervised-LDA (sLDA) (Mcauliffe et al., 2008) is a general-purpose supervised extension which builds on top of LDA by adding a response variable associated with each document (e.g. a review's rating). A category of extensions particularly relevant for this work is the sentiment-topic models. Examples include the Joint Sentiment-Topic (JST) model (Lin and He, 2009; Lin et al., 2012) and Aspect and Sentiment Unification Model (ASUM) (Jo and Oh, 2011). These models are able to extract informative topics grouped under different sentiment classes. Although they do not rely on document labels, they require word prior polarity information to be incorporated into the learning process in order to generate consistent results. Nevertheless, The possibility to supervise the learning process with document labels makes JST suitable for a fair comparison. Compared to DIATOM, the discussed sentiment topic models can only distinguish between polarity-bearing topics and neutral ones, remaining strictly aligned to the provided labels. Instead, DIATOM is able to generate opinion-bearing topics and plot topics which may still be polarized but not carrying any user's opinion.
Neural Topic Models. Neural models provide a more generic and extendable alternative to topic modeling, and therefore, have recently gained increasing interest. Some of them use belief networks (Mnih et al., 2014), or enforce the Dirichlet prior on the document-topic distribution via Wasserstein Autoencoders (Nan et al., 2019). Others adopt continuous representations to capture long-term dependencies or preserve word order via sequence-to-sequence VAE (Dieng et al., 2017; Xu et al., 2017; Bowman et al., 2016; Yang et al., 2017) whose time complexity and difficulty of training, however, have limited their applications. Neural Variational Document Model (NVDM) (Miao et al., 2016) is a direct extension of VAE used for topic detection in text. In NVDM, the prior of latent topics is assumed to be a Gaussian distribution. This is not ideal since it cannot mimic the simplex in the latent topic space. To address this problem, LDA-VAE (Srivastava et al., 2017) instead used the logistic normal distribution to approximate the Dirichlet distribution. ProdLDA (Srivastava et al., 2017) extended LDA-VAE by replacing the mixture model of LDA with a product of experts. SCHOLAR is a
<sup>1</sup>Source code and dataset available at: https://github.com/gabrer/diatom.
Figure 2: The DIATOM Architecture.
neural framework for topic models with metadata incorporation (Card et al., 2018). When metadata are document labels, the model infers topics that are relevant to those labels. Although some studies have applied the adversarial approach (Goodfellow et al., 2014) and reinforcement learning (Gui et al., 2019) to topic models setting a Dirichlet prior on the generative network (Wang et al., 2019; Masada et al., 2018), it is still unexplored how to use this mechanism to disentangle opinion-bearing topics from plot or neutral topics.
Representation Disentanglement. Among the slightly different versions of representation disentanglement proposed (Bengio et al., 2013; Higgins et al., 2018; Gao et al., 2019), the one achieved in DIATOM is analogous to Thomas et al. (2017) and Bengio et al. (2017), where they impose additional constraints to the representations controlled using a reinforcement learning mechanism determining the disentangled factors. Alternatively, in DIATOM we make use of an adversarial approach over the available target labels. Application in text processing has shown promising results (John et al., 2019; Kumar et al., 2017; Hoang et al., 2019; Esmaeili et al., 2019), yet applications to topic modeling are still limited (Wilson et al., 2016) and to the best of our knowledge, there is no work in separating opinion-bearing topics from plot/neutral topics.
3 DIATOM architecture
Our proposed DIATOM model is shown in Figure 2. Assuming a document  $\pmb{x}$  is associated with a sentiment label  $y_{s}$ , and each document can be represented by latent topics associated with semi-
ments  $(z_{s})$  and plots $^{2}$ $(z_{a})$ , we aim to learn a model maximizing the joint data-label log-likelihood,  $\log p(\boldsymbol{x}, y_{s})$ :
$$
\begin{array}{l} \log p (\boldsymbol {x}, y _ {s}) = \log \int \int p (\boldsymbol {x}, y _ {s}, \boldsymbol {z} _ {a}, \boldsymbol {z} _ {s}) d \boldsymbol {z} _ {a} d \boldsymbol {z} _ {s} \\ \geq \mathbb {E} _ {q _ {\phi} (\boldsymbol {z} _ {a} | \boldsymbol {x}), q _ {\psi} (\boldsymbol {z} _ {s} | \boldsymbol {x}, y _ {s})} [ \log p _ {\theta} (\boldsymbol {x} | \boldsymbol {z} _ {a}, \boldsymbol {z} _ {s}) ] \\ + \mathbb {E} _ {q _ {\phi} \left(\boldsymbol {z} _ {a} | \boldsymbol {x}\right), q _ {\psi} \left(\boldsymbol {z} _ {s} | \boldsymbol {x}, y _ {s}\right)} [ \log p _ {\pi} \left(y _ {s} | \boldsymbol {x}\right) ] \\ - \operatorname {K L} \left(q _ {\phi} \left(\boldsymbol {z} _ {a} | \boldsymbol {x}\right) | | p \left(\boldsymbol {z} _ {a}\right)\right) \\ - \operatorname {K L} \left(q _ {\psi} \left(\boldsymbol {z} _ {s} \mid \boldsymbol {x}, y _ {s}\right) \| p \left(\boldsymbol {z} _ {s}\right)\right) \tag {1} \\ \end{array}
$$
Inspired by Miao et al. (2016) and Card et al. (2018), we assume the document-level topic distribution for plots can be approximated by a multilayer perceptron (MLP) taking as input a multivariate Gaussian distribution, and similarly for the topic distribution for sentiments. The multinomial distribution over words under a plot topic and an opinion topic can be parametrised by a weight matrix  $W$ . The generative process is shown below.
- For each document  $d \in \{1,..,D\}$ ,
- Draw the latent plot-topics,
$$
\hat {\phi} \sim \mathcal {N} (\mu_ {\phi}, \Sigma_ {\phi}), \quad \boldsymbol {z} _ {a} = f _ {\hat {\phi}} (\hat {\phi})
$$
- Draw the latent opinion-topics,
$$
\hat {\psi} \sim \mathcal {N} (\mu_ {\psi}, \Sigma_ {\psi}), \quad z _ {s} = f _ {\hat {\psi}} (\hat {\psi})
$$
- For each word  $n \in \{1, \dots, N_d\}$  in  $d$ ,
$$
* \operatorname {D r a w} x _ {d, n} \sim p (x _ {d, n} | \boldsymbol {W}, \boldsymbol {z} _ {a}, \boldsymbol {z} _ {s})
$$
- Generate the document-level sentiment label,  $y_{s} \sim p(y_{s} | f_{y}(\boldsymbol{z}_{s}))$
where  $f_{\hat{\phi}}$ ,  $f_{\hat{\psi}}$  and  $f_{y}$  are MLPs,  $z_{a}$  is a  $K$ -dimensional latent topic representation of plots for document  $d$ ,  $z_{s}$  is a  $S$ -dimensional latent topic representation of sentiments for document  $d$ . The probability of word  $x_{d,n}$  can be parametrised by another network:
$$
p \left(x _ {d, n} \mid \boldsymbol {W}, \boldsymbol {z} _ {a}, \boldsymbol {z} _ {s}\right) \propto \exp \left(\boldsymbol {m} _ {d} + \boldsymbol {W} \cdot \left(\boldsymbol {z} _ {a} \| \boldsymbol {z} _ {s}\right)\right) \tag {2}
$$
where  $\pmb{m}_d$  is the  $V$ -dimensional background log-frequency word distribution, and  $\pmb{W} \in \mathbb{R}^{V \times (K + S)}$ , while  $z_a \parallel z_s$  is the concatenation of the two latent topic vectors.
Plot Inference Network. Following the idea of VAE which computes a variational approximation to an intractable posterior using MLPs, we define two inference networks  $f_{\mu_{\phi}}$  and  $f_{\Sigma_{\phi}}$  which takes as input the word counts in documents:
$$
\mu_ {\phi} = f _ {\mu_ {\phi}} (\boldsymbol {x}) \quad \Sigma_ {\phi} = \operatorname {d i a g} \left(f _ {\Sigma_ {\phi}} (\boldsymbol {x})\right) \tag {3}
$$
2These are the topics not associated with the target sentiments, which can be either plot topics or neutral topics (not about plots). For notational convenience, we call both plot topics.
The outputs of both networks are vectors in  $\mathbb{R}^K$ . Here, 'diag' converts a column vector to a diagonal matrix. For a document  $x$ ,  $q(\phi) \simeq \mathcal{L}\mathcal{N}(\mu_{\phi}, \Sigma_{\phi})$ . With such a formulation, we can generate samples from  $q(\phi)$  by first sampling  $\epsilon \sim \mathcal{N}(0, I)$  and then computing  $\hat{\phi} = \sigma (\mu_{\phi} + \Sigma_{\phi}^{1 / 2}\epsilon)$ .
Sentiment Inference Network. Similarly, to compute a variational approximation to  $q(\psi)$ , we define two inference networks  $f_{\mu_{\psi}}$  and  $f_{\Sigma_{\psi}}$  which takes as input the word counts in documents:
$$
\mu_ {\psi} = f _ {\mu_ {\psi}} (\boldsymbol {x}) \quad \Sigma_ {\psi} = \operatorname {d i a g} \left(f _ {\Sigma_ {\psi}} (\boldsymbol {x})\right) \tag {4}
$$
The outputs of both networks are vectors in  $\mathbb{R}^S$ . For a document  $x$ ,  $q(\psi) \simeq \mathcal{LN}(\mu_{\psi}, \Sigma_{\psi})$ . We then generate samples from  $q(\psi)$  by first sampling  $\epsilon \sim \mathcal{N}(0, I)$  and then computing  $\hat{\psi} = \sigma (\mu_{\psi} + \Sigma_{\psi}^{1/2}\epsilon)$ .
Overall Objective. With the sampled  $\hat{\phi}$  and  $\hat{\psi}$ , for each document  $x$ , we can minimise the reconstruction loss with a Monte Carlo approximation using  $L$  independent samples:
$$
\begin{array}{l} \mathcal {L} _ {\boldsymbol {x}} \approx \frac {1}{L} \sum_ {l = 1} ^ {L} \sum_ {n = 1} ^ {N _ {d}} \log p _ {\theta} (x _ {d, n} | \hat {\phi} ^ {(l)}, \hat {\psi} ^ {(l)}) \\ - \operatorname {K L} \left(q \left(\boldsymbol {z} _ {a} \mid \boldsymbol {x}\right) | | p \left(\boldsymbol {z} _ {a}\right)\right) \\ - \operatorname {K L} \left(q \left(\boldsymbol {z} _ {s} \mid \boldsymbol {x}, y _ {s}\right) \mid p \left(\boldsymbol {z} _ {s}\right)\right) \tag {5} \\ \end{array}
$$
where the first term in the RHS is given by Eq. (2). It has been previously shown in Kingma et al. (2014), if a standard multivariate normal prior is placed on the latent variables  $\mathbf{z}_a$  and  $\mathbf{z}_s$ , then there is a closed form solution to the KL divergence terms above.
We assume that the latent topics associated with plots,  $z_{a}$ , are independent of sentiment classes, and hence, when fed into a sentiment classifier, should generate a uniform sentiment class distribution (similar to adversarial learning). On the contrary, the latent topics associated with sentiments,  $z_{s}$ , should bear essential information to discriminate between sentiment classes. Therefore, we define the following two objectives for sentiment classification; the former being the expected KL divergence with the uniform distribution  $\mathcal{U}$ , and the latter a cross-entropy loss:
$$
\begin{array}{l} \mathcal {L} _ {a d v} = - \mathbb {E} _ {q _ {\phi} (\boldsymbol {z} _ {a})} \left[ \operatorname {K L} \left(\mathcal {U} (0, M) \mid | p (\hat {y} | \boldsymbol {z} _ {a})\right) \right] \\ \mathcal {L} _ {s e n t} = - \mathbb {E} _ {q _ {\psi} \left(\boldsymbol {z} _ {s}\right)} \sum_ {c = 1} ^ {M} y _ {c} \log \left(p \left(\hat {y} _ {c} \mid \boldsymbol {z} _ {s}\right)\right) \tag {6} \\ \end{array}
$$
where  $M$  is the total number of sentiment classes, and  $\mathcal{U}(0,M)$  represents the uniform sentiment class distribution.
To further disentangle the latent topics associated with plots and sentiments, while concurrently minimise the redundancy in the final topic matrix, we apply an orthogonal regularizer over the decoder matrix  $\mathbf{W}$ .  $\mathcal{L}_{\text{orth}}$  reaches its minimum value when the dot product between different topic distributions goes close to zero:
$$
\mathcal {L} _ {\text {o r t h}} = \left| \left| \boldsymbol {W} \cdot \boldsymbol {W} ^ {T} - \mathbb {I} \right| \right| \tag {7}
$$
Our final objective function is:
$$
\mathcal {L} = - \alpha \mathcal {L} _ {\boldsymbol {x}} + \beta \mathcal {L} _ {\text {a d v}} + \gamma \mathcal {L} _ {\text {s e n t}} + \eta \mathcal {L} _ {\text {o r t h}} \tag {8}
$$
where  $\alpha, \beta, \gamma$  and  $\eta$  control the relative contribution of various loss functions.
Plot Network. An additional VAE is plugged to the model providing a supplementary signal for the latent plot topic extraction. This mechanism preserves the plot information that might contain some sentiment words and thus, be wrongly regard as a user's opinion. The inference network is defined analogously to Eq. 3, which instead of taking a review document, takes a plot summary as input. An additional cross-entropy objective is minimized to drive the latent plot topics  $(z_{a})$  which would have a similar discriminative power as the features  $(z_{d})$  directly derived from the plots when used for plot classification:
$$
\mathcal {L} _ {\boldsymbol {d}} = \mathbb {E} _ {q _ {\omega} (\boldsymbol {z} _ {d} | d)} [ p _ {\zeta} (d | \boldsymbol {z} _ {d}) ] - \operatorname {K L} \left(q (\boldsymbol {z} _ {d} | \boldsymbol {d}) \mid \mid p (\boldsymbol {z} _ {d})\right)
$$
$$
\mathcal {L} _ {p l o t _ {z _ {a}}} = - \mathbb {E} _ {q _ {\phi} (\boldsymbol {z} _ {a})} \sum_ {p = 1} ^ {P} y _ {p} \log (p \big (\hat {y} _ {p} | \boldsymbol {z} _ {a} \big))
$$
$$
\mathcal {L} _ {\text {p l o t} z _ {d}} = - \mathbb {E} _ {q _ {\omega} (\boldsymbol {z} _ {d})} \sum_ {p = 1} ^ {P} y _ {p} \log \left(p \left(\hat {y} _ {p} \mid \boldsymbol {z} _ {d}\right)\right) \tag {9}
$$
where  $P$  denotes the total number of plots in each dataset. Finally,  $-\mathcal{L}_d$  and  $\mathcal{L}_{plot}$  are added to the overall loss defined in Eq. 8.
We conduct thorough experimental evaluations to assess the quality and disentanglement rate of extracted topics. To assess the quality of topics, we compute their topic coherence (Röder et al., 2015) coupled with their topic uniqueness. Then, we additionally look at the discriminative power of the disentangled features on the sentiment classification task. To fully assess the disentanglement rate of different methods, we perform topic labeling
to compute the sentiment polarity of each topic (if any) and then measure the overall disentanglement rate ( $\S 5.3$ ). As a result, we obtain an estimate of the extent to which different models can accurately control the topic disentanglement rate. We introduce and use a new dataset, named the MOBO dataset, pairing movie/book plots with their users' reviews, and including human-annotated sentences. MOBO Dataset. The MOBO dataset is a collection of reviews and plots about MOvie and BOok, associated to human-annotated sentences: while the pairs of reviews and plots are used to enhance the generation of plot topics, the human-annotated sentences provide the necessary ground-truth to automatically evaluate the topics' polarity.
Movie and book reviews were collected and paired from 3 public datasets: the Stanford's IMDB movie reviews (Maas et al., 2011), the GoodReads (Wan et al., 2019) and the Amazon reviews dataset (McAuley et al., 2015). Among all the available reviews in the IMDB dataset, we keep the ones with a corresponding plot in the MPST dataset (Kar et al., 2018), a corpus of movie synopses. The Goodreads dataset comes already with books' reviews paired with the related plots; while from the Amazon dataset, among all the product reviews, we keep only the ones related to movies available on the store and whose descriptions consist of the movie plots<sup>3</sup>. With the help of 15 annotators we further labeled more than 18,000 reviews' sentences ( $\sim$ 6000 per corpus), marking the sentence polarity (Positive, Negative), or whether a sentence describes its corresponding movie/book Plot, or none of the above (None)<sup>4</sup>. We ensured that each sentence was labelled by at least 2 annotators by assigning overlapping subsets of  $\sim$ 2400 sentences. In case of disagreement, when no consensus was reached, a final choice was made through a majority vote involving a third annotator. The final inter-annotator agreement (Cohen's kappa) was computed between each pair of annotators sharing a common subsets, with a minimum value of 0.572 and maximum of 0.831, for a resulting average of  $0.758^{5}$ . We report the dataset statistics in Table 1. We report the dataset statistics in Table 1.
Baselines. We compare the experimental results with the following baselines:
Statistics	IMDB	GoodReads	Amazon
# plots	1,131	150	100
# reviews	25,836	83,852	32,375
% Pos. reviews	0.46	0.33	0.32
% Neg. reviews	0.54	0.50	0.46
% Neu. reviews	0	0.17	0.22
Training set	20,317	65,816	25,883
Dev. set	2,965	9,007	3,275
Test set	2,554	9,029	3,217
# annotated sent.	6,000	6,000	6,000
Table 1: The MOBO dataset statistics.
sLDA (Mcauliffe et al., 2008): a supervised extension of LDA adding a response variable associated with each document.
JST (Lin and He, 2009): Joint Sentiment-Topic model built on LDA which is able to extract polarity-bearing topics.
NVDM (Miao et al., 2016): a VAE with an encoder network mapping the bag-of-words representations into continuous latent distributions, and a generative network for the document reconstruction.
GSM (Miao et al., 2017): based upon NVDM, the Gaussian Softmax topic model generating the topic distribution by applying a softmax function on the hidden representations of documents.
NTM (Ding et al., 2018): Neural Topic Model is a variation of NVDM by plugging the topic coherence metric directly into the model's objective.
PRODLDA (Srivastava et al., 2017): ProdLDA introduces a Dirichlet prior in place of Gaussian prior over the latent topic variable.
SCHOLAR (Card et al., 2018): a neural framework based on variational inference for the generation of topic while incorporating metadata information.
Parameter Setting. We perform tokenization and sentence splitting with  $\mathrm{SpaCy}^6$ . When available, we keep the default preprocessing, as it is the case for sLDA and SCHOLAR. Along with stopwords, we also remove tokens shorter than three characters and those with just digits or punctuation. We set the vocabulary to the 2,000 most common words as the best trade-off for each dataset. The 300-dimensional word vectors are initialized with a pretrained BERT embedding (Devlin et al., 2019). Sentence embeddings are generated from the SentenceBERT using a pretrained BERT-large with meantokens pooling (Reimers and Gurevych, 2019). We use the predefined split of the MOBO dataset into training, development and test set in the proportion of 80/10/10 and average all the results over 5 runs.
$^{7}$ Hyperparameter setting and training details are in Appendix.