Venue: NAACL 2022

# Disentangled Learning of Stance and Aspect Topics for Vaccine Attitude Detection in Social Media

Lixing Zhu1, Zheng Fang1, Gabriele Pergola1, Rob Procter1,2, Yulan He1,2

###### Abstract

Building models to detect vaccine attitudes on social media is challenging because of the composite, often intricate aspects involved, and the limited availability of annotated data. Existing approaches have relied heavily on supervised training that requires abundant annotations and pre-defined aspect categories. Instead, with the aim of leveraging the large amount of unannotated data now available on vaccination, we propose a novel semi-supervised approach for vaccine attitude detection, called VADet. A variational autoencoding architecture based on language models is employed to learn from unlabelled data the topical information of the domain. Then, the model is fine-tuned with a few manually annotated examples of user attitudes. We validate the effectiveness of VADet on our annotated data and also on an existing vaccination corpus annotated with opinions on vaccines. Our results show that VADet is able to learn disentangled stance and aspect topics, and outperforms existing aspect-based sentiment analysis models on both stance detection and tweet clustering. Our source code and dataset are available at [http://github.com/somethingx1202/VADet](http://github.com/somethingx1202/VADet).

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote †: footnotemark: [ENDFOOTNOTE]

+
Footnote†: footnotemark: [ENDFOOTNOTE]


Welling, 2014) and Independent Component Analysis (ICA) (Khemakhem et al., 2020) sheds light on this problem. VAE is an unsupervised method that can be used to glean information that must be retained from the vaccine-related corpus. Meanwhile, a handful of annotations would induce the separation of independent factors following the ICA requirement for prior knowledge and inductive biases (Hyvarinen et al., 2019; Locatello et al., 2020a, b). To this end, we could disentangle the latent factors that are either specific to the aspect or to the stance, and improve the quality of the latent semantics learned from unannotated data.

We frame the problem of vaccine attitude detection as a joint aspect span detection and stance classification task, assuming that a tweet, which is limited to 280 characters, would usually only discuss one aspect. In particular, we extend a pre-trained language model (LM) by adding a topic layer, which aims to model the topical theme discussed in a tweet. In the absence of annotated data, the topic layer is trained to reconstruct the input message built on VAE. Given the annotated data, where each tweet is annotated with an aspect span and a stance label, the learned topic can be disentangled into a stance topic and an aspect topic. The stance topic is used to predict the stance label of the given tweet, while the aspect topic is used to predict the start and the ending positions of the aspect span. By doing so, we can effectively leverage both unannotated and annotated data for model training.

To evaluate the effectiveness of our proposed model for vaccine attitude detection on Twitter, we have collected over 1.9 million tweets relating to COVID vaccines between February and April 2021. We have further annotated 2,800 tweets with both aspect spans and stance labels. In addition, we have also used an existing Vaccination Corpus1 in which 294 documents related to the online vaccination debate have been annotated with opinions towards vaccination. Our experimental results on both datasets show that the proposed model outperforms existing opinion triple extraction model and BERT QA model on both aspect span extraction and stance classification. Moreover, the learned latent aspect topics allow the clustering of user attitudes towards vaccines, facilitating easier discovery of positive and negative attitudes in social media. The contribution of this work can be summarised as follows:

Footnote 1: [https://github.com/cltl/VaccinationCorpus](https://github.com/cltl/VaccinationCorpus)

* We have proposed a novel semi-supervised approach for joint latent stance/aspect representation learning and aspect span detection;
* The developed disentangled representation learning facilitates better attitude detection and clustering;
* We have constructed an annotated dataset for vaccine attitude detection.

## 2 Related Work

Our work is related to three lines of research: aspect-based sentiment analysis, disentangled representation learning, and vaccine attitude detection.

**Aspect-Based Sentiment Analysis** (ABSA) aims to identify the aspect terms and their polarities from text. Much work has been focusing on this task. The techniques used include Conditional Random Fields (CRFs) (Marcheggiani et al., 2014), Bidirectional Long Short-Term Memory networks (BiLSTMs) (Baziotis et al., 2017), Convolutional Neural Networks (CNNs) (Zhang et al., 2015b), Attention Networks (Yang et al., 2016; Pergola et al., 2021b), DenseLSTMs (Wu et al., 2018), NestedLTSMs (Moniz and Krueger, 2017), Graph Neural Networks (Zhang et al., 2019) and their combinations (Wang et al., 2018; Zhu et al., 2021; Wan et al., 2020), to name a few.

Zhang et al. (2015a) framed this task as text span detection, where they used text spans to denote aspects. The same annotation scheme was employed in (Li et al., 2018b), where intra-word attentions were designed to enrich the representations of aspects and predict their polarities. Li et al. (2018c) formalized the task as a sequence labeling problem under a unified tagging scheme. Their follow-up work (Li et al., 2019) explored BERT for end-to-end ABSA. Peng et al. (2020) modified this task by introducing opinion terms to shape the polarity. A similar modification was made in (Zhao et al., 2020) to extract aspect-opinion pairs. Position-aware tagging was introduced to entrench the offset between the aspect span and opinion term (Xu et al., 2020). More recently, instead of using pipeline approaches or sequence tagging, Barnes et al. (2021) adapted syntactic dependency parsing to perform aspect and opinion expression extraction, and polarity classification, thus formalizing the task as structured sentiment analysis.

 

Disentangled representation learningDeep generative models learn the hidden semantics of text, of which many attempt to capture the independent latent factor to steer the generation of text in the context of NLP (Hu et al., 2017; Li et al., 2018a; Pergola et al., 2019; John et al., 2019; Li et al., 2020). The majority of the aforementioned work employs VAE (Kingma et al., 2014) to learn controllable factors, leading to the abundance of VAE-based models in disentangled representation learning (Higgins et al., 2017; Burgess et al., 2018; Chen et al., 2018). However, previous studies show that unsupervised learning of disentanglement by optimising the marginal likelihood in a generative model is impossible (Locatello et al., 2019). While it is also the case that non-linear ICA is unable to uncover the true independent factors, Khemakhem et al. (2020) established a connection between those two strands of work, which is of particular interest to us since the proposed framework learns to approximate the true factorial prior given few examples, recovering a disentangled latent variable distribution on top of additionally observed variables. In this paper, stance labels and aspect spans are additionally observed on a handful of data, which could be used as inductive biases that make disentanglement possible.

Vaccine attitude detectionVery little literature exists on attitude detection for vaccination. In contrast, there is growing interest in Covid-19 corpus construction (Shuja et al., 2021). Of particular interest to us, Banda et al. (2021) built an on-going tweet dataset that traces the development of Covid-19 by 3 keywords: "coronavirus", "2019nCoV" and "corona virus". Hussain et al. (2021) utilized hydrated tweets from the aforementioned corpus to analyze the sentiment towards vaccination. They used lexicon-based methods (i.e., VADER and TextBlob) and pre-trained BERT to classify the sentiment in order to gain insights into the temporal sentiment trends. A similar approach has been proposed in (Hu et al., 2021). Lyu et al. (2021) employed a topic model to discover vaccine-related themes in twitter discussions and performed sentiment classification using lexicon-based methods. However, none of the work above constructed datasets about vaccine attitudes, nor did they train models to detect attitudes. Morante et al. (2020) built the Vaccination Corpus (VC) with events, attributions and opinions annotated in the form of text spans, which is the only dataset available to us to perform attitude detection.

## 3 Methodology

The goal of our work is to detect the stance expressed in a tweet (i.e., '_pro-vaccination_', '_anti-vaccination_', or '_neutral_'), identify a text span that indicates the concerning aspect of vaccination, and cluster tweets into groups that share similar aspects. To this end, we propose a novel latent representation learning model that jointly learns a stance classifier and disentangles the latent variables capturing stance and aspect respectively. Our proposed Vaccine Attitude Detection (VADet) model is firstly trained on a large amount of unannotated Twitter data to learn latent topics via masked Language Model (LM) learning. It is then fine-tuned on a small amount of Twitter data annotated with stance labels and aspect text spans for simultaneously stance classification and aspect span start/end position detection. The rationale is that the inductive bias imposed by the annotations would encourage the disentanglement of latent stance topics and aspect topics. In what follows, we will present our proposed VADet model, first under the masked LM learning and later extended to the supervised setting for learning disentangled stance and aspect topics.

VAdet in the masked LM learningWe insert a topic layer into a pre-trained language model such as ALBERT, as shown in Figure 2, allowing the network to leverage pre-trained information while fine-tuned on an in-domain corpus. We assume that there is a continuous latent variable \(z\) involved in the language model to reconstruct the original text from the masked tokens. We retain the weights of a language model and lear

Figure 2: VADet in masked language model learning. The latent variables are encoded via the topic layers incorporated into the masked language model.

 

tion during the fine-tuning. More concretely, the topic layer partitions a language model into lower layers and higher layers denoted as \(\psi\) and \(\theta\), respectively. The lower layers constitute the Encoder that parameterizes the variational posterior distribution denoted as \(q_{\phi}(z|\psi(\mathbf{w}))\), while the higher layers reconstruct the input tokens, which is referred to as the Decoder.

The objective of VAE is to minimize the KL-divergence between the variational posterior distribution and the approximated posterior. This is equivalent to maximizing the Evidence Lower BOund (ELBO) expressed as:

\[\mathbb{E}_{q_{\phi}(z|\psi(\mathbf{w}))}[\log p_{\theta}(\textbf{w}^{H}|z, \psi(\mathbf{w}))]-\mathrm{KL}[q_{\phi}(z|\psi(\mathbf{w}))||p(z)], \tag{1}\]

where \(q_{\phi}(z|\psi(\mathbf{w}))\) is the encoder and \(p_{\theta}(\textbf{w}^{H}|z,\psi(\mathbf{w}))\) is the decoder. Here, \(\textbf{w}=[w_{\texttt{CLS}},w_{\texttt{I:n}}]\), since the special classification embedding \(w_{\texttt{CLS}}\) is automatically prepended to the input sequence Devlin et al. (2019), \(\textbf{w}^{H}\) denotes the reconstructed input.

Following Kingma and Welling (2014), we choose a standard Gaussian distribution as the prior, denoted as \(p(z)\), and the diagonal Gaussian distribution \(z\sim\mathcal{N}(\mu_{\phi}(\psi(\textbf{w})),\sigma_{\phi}^{2}(\psi(\textbf{ w})))\) as the variational distribution. The decoder computes the probability of the original token given the latent variable sampled from the Encoder. We use the Memory Scheme Li et al. (2020) to concatenate \(z\) and \(\psi(\textbf{w})\), making the latent representation compatible for higher layers of the language model. Then the latent presentation \(z\) is passed to \(\theta\) to reconstruct the original text.

VADet with disentanglement of aspect and stanceOne of the training objectives of vaccine attitude detection is to detect the text span that indicates the aspect and to predict the associated stance label. Existing approaches rely on structured annotations to indicate the boundary and dependency between aspect span and opinion words Xu et al. (2020); Barnes et al. (2021), or use a two-stage pipeline to detect the aspect span and the associated opinion separately Peng et al. (2020). The problem is that the opinion expressed in a tweet and the aspect span often overlap. To mitigate this issue, we instead separate the stance and aspect from their representations in the latent semantic space, that is, disentangling latent topics learned by VADet into latent stance topics and latent aspect topics. A recent study in disentangled representation learning Locatello et al. (2019) shows that unsupervised learning of disentangled representations is theoretically impossible from i.i.d. observations without inductive biases, such as grouping information Bouchacourt et al. (2018) or access to labels Locatello et al. (2020); Trauble et al. (2021). As such, we extend our model to a supervised setting in which disentanglement of the latent vectors can be trained on annotated data.

Figure 3 outlines the overall structure of VADet

Figure 3: VADet in supervised learning. The text segment highlighted in blue is the annotated aspect span. The right part learns latent aspect topic \(z_{a}\) from aspect text span \([w_{a}:w_{b}]\) only under masked LM learning. The left part learns jointly latent stance topic \(z_{s}\) and latent aspect topic \(z_{w}\) from the whole input text, and trained simultaneously for stance classification and aspect start/end position detection.

 

in the supervised setting. On the right hand side, we show VADet learned from the annotated aspect text span \([w_{a}:w_{b}]\) under masked LM learning. The latent variable \(z_{a}\) encodes the hidden semantics of the aspect expression. We posit that the aspect span is generated from a latent representation with a standard Gaussian distribution being its prior. The ELBO for reconstructing the aspect text span is:

\[\begin{split}\mathcal{L}_{A}=\mathbb{E}_{q_{\phi}(z_{a}|\psi(w_{a :b}))}[\log p_{\theta}(w^{H}_{a:b}|z_{a},\psi(w_{a:b}))]\\ -\mathrm{KL}[q_{\theta}(z_{a}|\psi(w_{a:b}))||p(z_{a})],\end{split} \tag{2}\]

where \(w^{H}_{a:b}\) denotes the reconstructed aspect span. Ideally, the latent variable \(z_{a}\) does not encode any stance information and only captures the aspect mentioned in the sentence. Therefore, the \(z_{s}\) for the language model on the right hand side is detached and the reconstruction loss for [CLS] is set free.

On the left hand side of Figure 3, we train VADet on the whole sentence. The input to VADet is formalized as: '[CLS]text'. Instead of mapping an input to a single latent variable \(z\), as in masked LM learning of VADet, the input is now mapped to a latent variable decomposing into two components, \([z_{s},z_{w}]\), one for the stance and another for the aspect. We use a conditionally factorized Gaussian prior over the latent variable \(z_{w}\sim p_{\theta}(z_{w}|w_{a:b})\), which enables the separation of \(z_{s}\) and \(z_{w}\) since the diagonal Gaussian is factorized and the conditioning variable \(w_{a:b}\) is observed.

We establish an association between \(z_{w}\) and \(z_{a}\) by specifying \(p_{\theta}(z_{w}|w_{a:b})\) to be the encoder network of \(q_{\phi}(z_{a}|w_{a:b})\), since we want the latent semantics of aspect span to encourage the disentanglement of attitude in the latent space. In other words, the prior of \(z_{w}\) is configured as the approximate posterior of \(z_{a}\) to enforce the association between the disentangled aspect in sentence and the _de facto_ aspect. As a result, the ELBO for the original text is written as

\[\begin{split}&\mathbb{E}_{q_{\phi}(z_{w}|\psi(\mathbf{w}))}[\log p _{\theta}(\mathbf{w}^{H}|z_{w},\psi(\mathbf{w}))]\\ &\qquad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad\quad \quad 

(e.g., _Pfizer_, _AstraZeneca_, and _Moderna_). Our final dataset comprises \(1.9\) million English tweets collected between February 7th and April 3rd, 2021. We randomly sample a subset of tweets for annotation. Upon an initial inspection, we found that over \(97\%\) of tweets mentioned only one aspect. As such, we annotate each tweet with a stance label and a text span characterizing the aspect. In total, 2,800 tweets have been annotated in which 2,000 are used for training and the remaining 800 are used for testing. The statistics of the dataset is listed in Table 1. The stance labels are imbalanced. On the other hand, the average opinion length is longer than the average aspect length, and is close to the average tweet length. For the purpose of evaluation on tweet clustering and latent topic disentanglement, we further annotate tweets with a categorical label indicating the aspect category. Inspired by Morante et al. (2020), we identify \(24\) aspect categories3 and each tweet is annotated with one of these categories. It is worth mentioning that aspect category labels are not used for training.

Footnote 3: The full list of aspect categories is shown in Table A1.

VcMorante et al. (2020) is a vaccination corpus consisting of 294 Internet documents about online vaccine debate annotated with events, 210 of which are annotated with opinions (in the form of text spans) towards vaccines. The stance label is considered to be the stance for the whole sentence. Those sentences with conflicting stance labels are regarded as neutral. We split the dataset into a ratio of 2:1 for training and testing. This eventually left us with 1,162 sentences for training and 531 sentences for testing.

BaselinesWe compare the experimental results with the following baselines: BertQA Li et al. (2018): a pre-trained language model well-suited for span detection. With BertQA, attitude detection is performed by first classifying stance labels then predicting the answer queried by the stance label. The text span is configured as the ground-truth answer. We rely on its HuggingFace4Wolf et al. (2020) implementation. We employ ALBERT Lan et al. (2020) as the backbone language model for both BertQA and VADet.

Footnote 4: [https://huggingface.com/transformers/model_doc/albert.html#albertforquestionanswering](https://huggingface.com/transformers/model_doc/albert.html#albertforquestionanswering)

ASTEPeng et al. (2020): a pipeline approach consisting of aspect extraction Li et al. (2018) and sentiment labelling Li et al. (2018).

Evaluation MetricsFor stance classification, we use accuracy and Macro-averaged F1 score. For aspect span detection, we follow Rajpurkar et al. (2016) in adopting exact match (EM) accuracy of the starting-ending position and Macro-averaged F1 score of the overlap between the prediction and ground truth aspect span. For tweet clustering, we follow Xie et al. (2016) and Zhang et al. (2021) and use the Normalized Mutual Information (NMI) metric to measure how the clustered group aligns with ground-truth categories. In addition, we also report the clustering accuracy.

### Experimental Results

In all our experiments, VADet is firstly pre-trained in an unsupervised way on our collected 1.9 million tweets before fine-tuned on the annotated training set from the VAD or VC corpora.

Stance Classification and Aspect Span DetectionIn Table 2, we report the performance on attitude detection. In stance classification, our model outperforms both baselines with more significant improvements on ASTE. On aspect span extraction, VADet yields even more noticeable improvements, with a \(2.3\%\) increase in F1 over BertQA on VAD, and \(2.7\%\) on VC. These results indicate that the successful prediction relies on the hidden representation learned in the unsupervised training. The disentanglement of stance and aspect may have also contributed to the improvement.

ClusteringTo assess whether the learned latent aspect topics would allow meaningful categorization of documents into attitude clusters, we perform

\begin{table}
\begin{tabular}{l c c c c} \hline \hline  & \multicolumn{2}{c}{VAD} & \multicolumn{2}{c}{VC} \\ \cline{2-5} Specification & Train & Test & Train & Test \\ \hline \# tweets & 2000 & 800 & 1162 & 531 \\ \# anti-vac. & 638 & 240 & 822 & 394 \\ \# neutral & 142 & 76 & 41 & 27 \\ \# pro-vac. & 1220 & 484 & 299 & 110 \\ \hline Avg. length & 33.5 & 34.13 & 29.6 & 30.24 \\ len(aspect) & 17.5 & 18.75 & 1.03 & 1.08 \\ len(opinion) & 27.97 & 29.01 & 3.25 & 3.15 \\ \hline \# tokens & 67k & 27.3k & 34.4k & 16.8k \\ \hline \hline \end{tabular}
\end{table}
Table 1: Dataset Statistics. ‘# tweets’ denotes the number of tweets in VAD, and for VC it is the number of sentences. ’anti-vac.’ means _anti-vaccination_ while ’pro-vac.’ means _pro-vaccination_. ‘Avg. length’ and ‘# token’ measure the number of word tokens.

 

clustering using the disentangled representations that encode aspects, i.e., \(z_{w}\). Deep Embedding Clustering (DEC) [23] is employed as the backend. For comparison, we also run DEC on the aspect representations of documents returned by BertQA. For each document, its aspect representation is obtained by average over the fine-tuned ALBERT representations of the constituent words in its aspect span. To assess the quality of clusters, we need the annotated aspect categories for documents in the test set. In VAD, we use the annotated aspect labels as the ground-truth categories whereas in VC we use the annotated event types. Results are presented in the lower part of Table 2. We found a prominent increase in NMI score over the baselines. Using the learned latent aspect topics as features, DEC (VADet) outperforms DEC (BertQA) by \(4.6\%\) and \(1.9\%\) in accuracy on VAD and VC, respectively. We also notice that using K-means as the clustering approach directly on the BERT-encoded tweet representations gives worse results compared to DEC. A similar trend is observed on the NMI metric. The improvements are shown visually in Figure 4 where the clustered groups produced by VADet are more identifiable. In the absence of categorical labels, the perspective expressed by each group can be inferred from the constituent tweets. For example, the tweet _'@user Georgian nurse dies of allergic reaction after receiving AstraZeneca Covid19 vaccine'_ lies in the centroid of the red group, which relates to safety concerns.

Cluster Semantic Coherence EvaluationThe semantic coherence is the extent to which tweets within a cluster belong to each other, which is employed as an evaluation metric for cluster quality evaluation in an unsupervised way. Recent work of Bilal et al. (2021) found that Text Generation Metrics (TGMs) align well with human judgement in evaluating clusters in the context of microblog posts. TGM by definition measures the similarity between the ground-truth and the generated text. The rationale is that a high TGM score means sentence pairs are semantically similar. Here, two metrics are used: _BERTScore_, which calculates the similarity of two sentences as a sum of cosine similarities between their tokens' embeddings [75], and _BLEURT_, a pre-trained adjudicator that fine-tunes BERT on an external dataset of human ratings [8]. As in [10], we adopt the Exhaustive Approach that for a cluster \(C\), its coherence score is the average TGM score of every possible tweet pair in the cluster:

\[f(C)=\frac{1}{N^{2}}\sum_{i,j\in[1,N],i<j}\mathrm{TGM}(\mathrm{tweet}_{i}, \mathrm{tweet}_{j}).\]

Figure 5 shows the BERTScore and the BLEURT score of VADet and baselines on two datasets. The VADet shows consistent improvements across the datasets. This indicates that tweets clustered using the latent aspect topics generated by VADet are semantically more similar, thus validating the assumption that disentangled representations are more effective in bringing together tweets of a similar gist.

Conditional PerplexityFew metrics have been proposed to evaluate the quality of disentangled representations [10]. Therefore, we

\begin{table}
\begin{tabular}{l c c c c} \hline \hline Model & \multicolumn{2}{c}{VAD} & \multicolumn{2}{c}{VC} \\ \cline{2-5} _Stance_ & Acc. & F1 & Acc. & F1 \\ \hline BertQA & 0.754 & 0.742 & 0.719 & 0.708 \\ ASTE & 0.723 & 0.710 & 0.704 & 0.686 \\ VADet & **0.763** & **0.756** & **0.727** & **0.713** \\ \hline _Aspect Span_ & Acc. & F1 & Acc. & F1 \\ \hline BertQA & 0.546 & 0.722 & 0.525 & 0.670 \\ ASTE & 0.508 & 0.684 & 0.467 & 0.652 \\ VADet & **0.556** & **0.745** & **0.541** & **0.697** \\ \hline _Cluster_ & Acc. & NMI & Acc. & NMI \\ \hline DEC (BertQA) & 0.633 & 58.1 & 0.586 & 52.8 \\ K-means (BERT) & 0.618 & 56.4 & 0.571 & 50.1 \\ DEC (VADet) & **0.679** & **60.7** & **0.605** & **54.7** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Results for stance classification, aspect span extraction and aspect clustering on both VAD and VC corpora.

Figure 4: Clustered groups of VADet and BertQA on the VAD dataset. Each color indicates a ground truth aspect category. The clusters are dominated by: _(1) Red: the (adverse) side effects of vaccines; (2) Green: explaining personal experiences with any aspect of vaccines; and (3) Cyan: the immunity level provided by vaccines_.

 

adopt the language model perplexity conditioned on \(z_{a}\) to evaluate the extent to which the disentangled representation improves language generation on held-out data. Perplexivity is widely used in the literature of text style transfer John et al. (2019); Yi et al. (2020), where the probability of the generated language is calculated conditioned on the controlled latent code. A lower perplexity score indicates better language generation performance. Following John et al. (2019), we compute an estimated aspect vector \(\hat{z}^{(k)}_{a}\) of a cluster \(k\) in the training set as

\[\hat{z}^{(k)}_{a}=\frac{\sum_{i\in\mathrm{cluster~{}}k}z^{(k)}_{a,i}}{\text{ \# tweets in cluster~{}}k},\]

where \(z^{(k)}_{a,i}\) is the learned aspect vector of the \(i\)-th tweet in the \(k\)-th cluster. For the stance vector \(z_{s}\), we sample one value per tweet. The stance vector is concatenated with the aspect vector \(\hat{z}^{(k)}_{a}\) to calculate the probability of generating the held-out data, i.e., the testing set. For the baseline models, we choose \(\beta\)-VAE Higgins et al. (2017) and SCHOLAR Card et al. (2018). We train \(\beta\)-VAE on the same data with \(\beta\) set to different values. SCHOLAR is trained on tweet content and stance labels. For both the baselines we use ELBO on the held-out data as an upper bound on perplexity.

Figure 6 plots the perplexity score achieved by all the methods. Our model achieves the lowest perplexity score on both datasets. It managed to decrease the perplexity value by roughly 200 compared to the baseline models. SCHOLAR outperforms \(\beta\)-VAE under three settings of \(\beta\) value. We speculate that this might be due to the incorporation of the class labels in the training of SCHOLAR. Nevertheless, VADet produces congenial sentences in aspect groups, with latent codes tweaked to proxy centroids, showing that the disentangled representation does capture the desired factor.

AblationsWe conduct ablation studies to investigate the effect of semi-supervised learning that uses the variational latent representation learning approach and aspect-stance disentanglement on the latent semantics. We study their effects on stance classification and aspect span detection. The results are reported in Table 3.

We can observe that on VAD without disentangled learning or unsupervised pre-training results in the degradation of the stance classification performance. However, on VC, we see a slight increase in classification accuracy without disentangled learning. We attribute this to the vagueness of the stance which might cause the model to disentangle more than it should be. On the aspect span detection task, we observe consistent performance drop across all metrics and on both datasets. In particular, without the pre-training module, the performance drops more significantly. These results indicate that semi-supervised learning is highly effective with VAE, and the disentanglement of stance and aspect serves as a useful component, whic

Figure 5: Semantic coherence evaluated in two metrics.

Figure 6: Conditional perplexity on two corpora.

 

improvements.

## 5 Conclusions

In this work, we presented a semi-supervised model to detect user attitudes and distinguish aspects of interest about vaccines on social media. We employed a Variational Auto-Encoder to encode the main topical information into the language model by unsupervised training on a massive, unannotated dataset. The model is then further trained under a semi-supervised setting that leverages annotated stance labels and aspect spans to induce the disentanglement between stances and aspects in a latent semantic space. We empirically showed the benefits of such an approach for attitude detection and aspect clustering over two vaccine corpora. Ablation studies show that disentangled learning and unsupervised pre-training are important to effective vaccine attitude detection. Further investigations on the quality of the disentangled representations verify the effectiveness of the disentangled factors. While our current work mainly focuses on short text of social media data where a sentence is assumed to discuss a single aspect, it would be interesting to extend our model to deal with longer text such as online debates in which multiple arguments or aspects may appear in a single sentence.

## Acknowledgements

This work was funded by the the UK Engineering and Physical Sciences Research Council (grant no. EP/T017112/1, EP/V048597/1). LZ is supported by a Chancellor's International Scholarship at the University of Warwick. YH is supported by a Turing AI Fellowship funded by the UK Research and Innovation (grant no. EP/V020579/1).

## References

* M Banda et al. (2021) Juan M Banda, Ramya Tekumalla, Guanyu Wang, Jingyuan Yu, Tuo Liu, Yuning Ding, Ekaterina Artemova, Elena Tutubulana, and Gerardo Chowell. 2021. A large-scale covid-19 twitter chatter dataset for open scientific research--an international collaboration. _Epidemologia_, 2(3):315-324.
* Barnes et al. (2021) Jeremy Barnes, Robin Kurtz, Stephan Oepen, Lilja Ovrelid, and Erik Welldal. 2021. Structured sentiment analysis as dependency graph parsing. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_, pages 3387-3402. Association for Computational Linguistics.
* Baziotis et al. (2017) Christos Baziotis, Nikos Peleks, and Christos Doulkeridis. 2017. DataStories at SemEval-2017 task 4: Deep LSTM with attention for message-level and topic-based sentiment analysis. In _Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017)_, pages 747-754, Vancouver, Canada. Association for Computational Linguistics.
* Bilal et al. (2021) Iman Munire Bilal, Bo Wang, Maria Liakata, Rob Procter, and Adam Tsakalidis. 2021. Evaluation of thematic coherence in microblogs. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing_, pages 6800-6814. Association for Computational Linguistics.
* Bouchacourt et al. (2018) Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. 2018. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. _Proceedings of the AAAI Conference on Artificial Intelligence_, 32(1).
* Burgess et al. (2018) Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthey, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. 2018. Understanding disentangling in \(\beta\)-vae. _arXiv preprint arXiv:1804.03599_.
* Card et al. (2018) Dallas Card, Chenhao Tan, and Noah A. Smith. 2018. Neural models for documents with metadata. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_, pages 2031-2040, Melbourne, Australia. Association for Computational Linguistics.
* Chen et al. (2018) Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. 2018. Isolating sources of disentanglement in variational autoencoders. In _Advances in Neural Information Processing Systems_, volume 31. Curran Associates, Inc.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* He et al. (2019) Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2019. An interactive multi-task learning network for end-to-end aspect-based sentiment analysis. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 504-515, Florence, Italy. Association for Computational Linguistics.
* Higgins et al. (2017) Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-vae: Learning basic visual concepts with a constrained variational framework. In _5th International Conference on Learning Representations, ICLR 2017, 

Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
* Hu et al. (2021) Tao Hu, Siqin Wang, Wei Luo, Mengxi Zhang, Xiao Huang, Yingwei Yan, Regina Liu, Kelly Ly, Viraj Kacker, Bing She, and Zhenlong Li. 2021. Revealing public opinion towards covid-19 vaccines with twitter data in the united states: Spatiotemporal perspective. _J Med Internet Res_, 23(9):e30854.
* Hu et al. (2017) Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In _Proceedings of the 34th International Conference on Machine Learning_, volume 70 of _Proceedings of Machine Learning Research_, pages 1587-1596. PMLR.
* Hussain et al. (2021) Amir Hussain, Ahsen Tahir, Zain Hussain, Zakariya Sheikh, Mandar Gogate, Kia Dashtipour, Azhar Ali, and Aziz Sheikh. 2021. Artificial intelligence-enabled analysis of public attitudes on facebook and twitter toward covid-19 vaccines in the united kingdom and the united states: Observational study. _J Med Internet Res_, 23(4):e26627.
* Hyvarinen et al. (2019) Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. 2019. Nonlinear ica using auxiliary variables and generalized contrastive learning. In _Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics_, volume 89 of _Proceedings of Machine Learning Research_, pages 859-868. PMLR.
* John et al. (2019) Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 424-434, Florence, Italy. Association for Computational Linguistics.
* Khemakhem et al. (2020) Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. 2020. Variational autoencoders and nonlinear ica: A unifying framework. In _Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics_, volume 108 of _Proceedings of Machine Learning Research_, pages 2207-2217. PMLR.
* Kingma et al. (2014) Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling. 2014. Semi-supervised learning with deep generative models. In _Proceedings of the 27th International Conference on Neural Information Processing Systems_, NIPS'14, page 3581-3589, Cambridge, MA, USA. MIT Press.
* Kingma and Welling (2014) Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In _2nd International Conference on Learning Representations, ICLR 2014, Banffi, AB, Canada, April 14-16, 2014, Conference Track Proceedings_.
* Kanneman et al. (2020) Florian Kanneman, Mattijs Lambooij, Albert Wong, Antal Van Den Bosch, and Liesbeth Mollema. 2020. Monitoring stance towards vaccination in twitter messages. _BMC medical informatics and decision making_, 20(1):1-14.
* Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Li et al. (2020) Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiuijun Li, Yizhe Zhang, and Jianfeng Gao. 2020. Optimus: Organizing sentences via pre-trained modeling of a latent space. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_, pages 4678-4699. Association for Computational Linguistics.
* Li et al. (2018a) Juncen Li, Robin Jia, He He, and Percy Liang. 2018a. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1865-1874, New Orleans, Louisiana. Association for Computational Linguistics.
* Li et al. (2018b) Xin Li, Lidong Bing, Wai Lam, and Bei Shi. 2018b. Transformation networks for target-oriented sentiment classification. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics_, pages 946-956, Melbourne, Australia. Association for Computational Linguistics.
* Li et al. (2018c) Xin Li, Lidong Bing, Piji Li, Wai Lam, and Zhimou Yang. 2018c. Aspect term extraction with history attention and selective transformation. In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_, pages 4194-4200. International Joint Conferences on Artificial Intelligence Organization.
* Li et al. (2019) Xin Li, Lidong Bing, Wenxuan Zhang, and Wai Lam. 2019. Exploiting BERT for end-to-end aspect-based sentiment analysis. In _Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019)_, pages 34-41, Hong Kong, China. Association for Computational Linguistics.
* Locatello et al. (2019) Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnara Ratesch, Sylvain Gelly, Bernhard Scholkopf, and Olivier Bachem. 2019. Challenging common assumptions in the unsupervised learning of disentangled representations. In _Proceedings of the 36th International Conference on Machine Learning_, volume 97 of _Proceedings of Machine Learning Research_, pages 4114-4124. PMLR.
* Locatello et al. (2020a) Francesco Locatello, Ben Poole, Gunnar Raetsch, Bernhard Scholkopf, Olivier Bachem, and Michael Tschannen. 2020a. Weakly-supervised disentanglement without compromises. In _Proceedings of the 37th International Conference on Machine Learning_ 

volume 119 of _Proceedings of Machine Learning Research_, pages 6348-6359. PMLR.
* Locatello et al. (2020) Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Ratsch, Bernhard Scholkopf, and Olivier Bachem. 2020b. Disentangling factors of variations using few labels. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Lyu et al. (2021) Joanne Chen Lyu, Eileen Le Han, and Garving K Luli. 2021. Covid-19 vaccine-related discussion on twitter: topic modeling and sentiment analysis. _Journal of medical Internet research_, 23(6):e24435.
* Ma et al. (2018) Yukun Ma, Haiyun Peng, and Erik Cambria. 2018. Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive lstm. _Proceedings of the AAAI Conference on Artificial Intelligence_, 32(1).
* Marcheggiani et al. (2014) Diego Marcheggiani, Oscar Tackstrom, Andrea Esuli, and Fabrizio Sebastiani. 2014. Hierarchical multi-label conditional random fields for aspect-oriented opinion mining. In _Advances in Information Retrieval_, pages 273-285, Cham. Springer International Publishing.
* Moniz and Krueger (2017) Joel Ruben Antony Moniz and David Krueger. 2017. Nested lstms. In _Proceedings of the Ninth Asian Conference on Machine Learning_, volume 77 of _Proceedings of Machine Learning Research_, pages 530-544, Yonsei University, Seoul, Republic of Korea. PMLR.
* Morante et al. (2020) Roser Morante, Chantal van Son, Isa Maks, and Piek Vossen. 2020. Annotating perspectives on vaccination. In _Proceedings of the 12th Language Resources and Evaluation Conference_, pages 4964-4973, Marseille, France. European Language Resources Association.
* Narisnick et al. (2016) Eric Narisnick, Lars Hertel, and Padhraic Smyth. 2016. Approximate inference for deep latent gaussian mixtures. In _NIPS Workshop on Bayesian Deep Learning_, volume 2, page 131.
* Europe_, 1:100012.
* Peng et al. (2020) Haijun Peng, Lu Xu, Lidong Bing, Fei Huang, Wei Lu, and Luo Si. 2020. Knowing what, how and why: A near complete solution for aspect-based sentiment analysis. _Proceedings of the AAAI Conference on Artificial Intelligence_, 34(05):8600-8607.
* Pergola et al. (2019) Gabriele Pergola, Lin Gui, and Yulan He. 2019. TDAM: A topic-dependent attention model for sentiment analysis. _Information Processing & Management_, 56(6):102084.
* Pergola et al. (2021a) Gabriele Pergola, Lin Gui, and Yulan He. 2021a. A disentangled adversarial neural topic model for separating opinions from plots in user reviews. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2870-2883. Association for Computational Linguistics.
* Pergola et al. (2021b) Gabriele Pergola, Elena Kochkina, Lin Gui, Maria Liakata, and Yulan He. 2021b. Boosting low-resource biomedical QA via entity-aware masking strategies. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 1977-1985. Association for Computational Linguistics.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,00+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
* Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7881-7892. Association for Computational Linguistics.
* Shuja et al. (2021) Junaid Shuja, Eisa Alanazi, Waleed Alasmary, and Abdulaziz Alashaikh. 2021. Covid-19 open source data sets: a comprehensive survey. _Applied Intelligence_, 51(3):1296-1325.
* Trauble et al. (2021) Frederik Trauble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Scholkopf, and Stefan Bauer. 2021. On disentangled representations learned from correlated data. In _Proceedings of the 38th International Conference on Machine Learning_, volume 139 of _Proceedings of Machine Learning Research_, pages 10401-10412. PMLR.
* Wan et al. (2020) Hai Wan, Yufei Yang, Jianfeng Du, Yanan Liu, Kunxun Qi, and Jeff Z. Pan. 2020. Target-aspect-sentiment joint detection for aspect-based sentiment analysis. _Proceedings of the AAAI Conference on Artificial Intelligence_, 34(05):9122-9129.
* Wang et al. (2018) Jingjing Wang, Jie Li, Shoushan Li, Yangyang Kang, Min Zhang, Luo Si, and Guodong Zhou. 2018. Aspect sentiment classification with both word-level and clause-level attention networks. In _Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18_, pages 4439-4445. International Joint Conferences on Artificial Intelligence Organization.
* Wolf et al. (2018) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Fantowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jenrite, Julien Flu, Canwen Xu, 

Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45. Association for Computational Linguistics.
* Wu et al. (2018) Chuhan Wu, Fangzhao Wu, Sixing Wu, Junxin Liu, Zhigang Yuan, and Yongfeng Huang. 2018. THU_NGN at SemEval-2018 task 3: Tweet irony detection with densely connected LSTM and multi-task learning. In _Proceedings of The 12th International Workshop on Semantic Evaluation_, pages 51-56, New Orleans, Louisiana. Association for Computational Linguistics.
* Xie et al. (2016) Janyuan Xie, Ross Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analysis. In _Proceedings of The 33rd International Conference on Machine Learning_, volume 48 of _Proceedings of Machine Learning Research_, pages 478-487, New York, New York, USA. PMLR.
* Xu et al. (2020) Lu Xu, Hao Li, Wei Lu, and Lidong Bing. 2020. Position-aware tagging for aspect sentiment triplet extraction. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing_, pages 2339-2349. Association for Computational Linguistics.
* Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1480-1489, San Diego, California. Association for Computational Linguistics.
* Yi et al. (2020) Xiaoyuan Yi, Zhenghao Liu, Wenhao Li, and Maosong Sun. 2020. Text style transfer via learning style instance supported latent space. In _Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20_, pages 3801-3807. International Joint Conferences on Artificial Intelligence Organization. Main track.
* Zhang et al. (2019) Chen Zhang, Qiuchi Li, and Dawei Song. 2019. Aspect-based sentiment classification with aspect-specific graph convolutional networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 4568-4578, Hong Kong, China. Association for Computational Linguistics.
* Zhang et al. (2021) Dejiao Zhang, Feng Nan, Xiaokai Wei, Shang-Wen Li, Henghui Zhu, Kathleen McKeown, Ramesh Nallapati, Andrew O. Arnold, and Bing Xiang. 2021. Supporting clustering with contrastive learning. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5419-5430. Association for Computational Linguistics.
* Zhang et al. (2015a) Meishan Zhang, Yue Zhang, and Duy-Tim Vo. 2015a. Neural networks for open domain targeted sentiment. In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_, pages 612-621, Lisbon, Portugal. Association for Computational Linguistics.
* Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with BERT. In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
* Zhang et al. (2015b) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015b. Character-level convolutional networks for text classification. In _Advances in Neural Information Processing Systems_, volume 28. Curran Associates, Inc.
* Zhao et al. (2020) He Zhao, Longtao Huang, Rong Zhang, Quan Lu, and Hui Xue. 2020. SpanMlt: A span-based multi-task learning framework for pair-wise aspect and opinion terms extraction. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3239-3248. Association for Computational Linguistics.
* Zhu et al. (2021)Lixing Zhu, Gabriele Pergola, Lin Gui, Deyu Zhou, and Yulan He. 2021. Topic-driven and knowledge-aware transformer for dialogue emotion detection. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1571-1582, Online. Association for Computational Linguistics.

 

Derivation of the Decomposed ELBO

Unsupervised training is based on maximizing the Evidence Lower Bound (ELBO):

\[{\mathbb{E}_{q_{\phi}(z_{s},z_{w}|\psi(\textbf{w}))}}[\log p_{\theta}( \textbf{w}|z_{s},z_{w},\psi(\textbf{w}))]\] \[-\mathrm{KL}[q_{\phi}(z_{s},z_{w}|\psi(\textbf{w}))||p(z_{s},z_{ w})],\]

where \(z\) is partitioned into \(z_{s}\) and \(z_{w}\). Like standard VAE Kingma and Welling (2014), the variational distribution is a multivariate Gaussian with a diagonal covariance:

\[q_{\phi}(z_{s},z_{w}|\psi(\textbf{w}))=\mathcal{N}(z_{s},z_{w}|\mu,\sigma^{2}I),\]

where \(\mu=[\mu^{s},\mu^{w}]\) and \(\sigma=[\sigma^{s},\sigma^{w}]\). Since the coverage matrix is diagonal, \(z_{s}\) and \(z_{w}\) are uncorrelated. Therefore, the joint probability is decomposed into:

\[q_{\phi}(z_{s},z_{w}|\psi(\textbf{w}))=q_{\phi}(z_{s}|\psi(\textbf{w}))q_{\phi }(z_{w}|\psi(\textbf{w})),\]

where \(q_{\phi}(z_{s}|\psi(\textbf{w}))=\mathcal{N}(z_{s}|\mu^{s},\sigma^{s})\), \(\phi\) are the variational parameters. The prior of \([z_{s},z_{w}]\sim\mathcal{N}(z_{s},z_{w}|\textbf{0},I)\) can also be decomposed into the product of \(p(z_{s})\) and \(p(z_{w})\), then the KL term becomes:

\[\mathrm{KL}[q_{\phi}(z_{s}|\psi(\textbf{w}))||p(z_{s})]+\mathrm{KL}[q_{\phi} (z_{w}|\psi(\textbf{w}))||p(z_{w})].\]

As for the decoder \(p_{\theta}(\textbf{w}|z_{s},z_{w},\psi(\textbf{w}))\), the reconstruction of each masked token and \(w_{\texttt{[CLS]}}\) are independent from each other, i.e., they are not predicted in an autoregressive way. Therefore, the joint probability is decomposed into:

\[p_{\theta}(\textbf{w}|z_{s},z_{w},\psi(\textbf{w}))\] \[=p_{\theta}(w_{\texttt{[CLS]}}|z_{s},z_{w},\psi(\textbf{w}))\,p_{ \theta}(w_{1:n}|z_{s},z_{w},\psi(\textbf{w}))\]

We customize the decoder network to make \(w_{\texttt{[CLS]}}\) solely dependent on \(z_{s}\), and obtain

\[{\mathbb{E}_{q_{\phi}(z_{s})}}{\mathbb{E}_{q_{\phi}(z_{w})}}[\log p _{\theta}(w_{\texttt{[CLS]}}|z_{s},\psi(\textbf{w}))+\] \[\log p_{\theta}(w_{1:n}|z_{w},\psi(\textbf{w}))]\]

Here, we omit \(\psi(\textbf{w})\) for notational simplicity. Given the supervision of annotated aspect spans, the prior of \(z_{w}\) is constrained by \(q_{\phi}(z_{w}|\psi(w_{a:b}))\) (a.k.a., the encoder of \(w_{a:b}\)), this will change the KL term into:

\[\begin{split}&\mathrm{KL}[q_{\phi}(z_{s}|\psi(\textbf{w}))||p( z_{s})]\\ &+\mathrm{KL}[q_{\phi}(z_{w}|\psi(w_{1:n}))||q_{\phi}(z_{w}|\psi( w_{a:b}))],\end{split}\]

and finally the ELBO is expressed as

\[{\mathbb{E}_{q_{\phi}(z_{s})}}[\log p_{\theta}(w_{\texttt{[CLS]}} |z_{s},\psi(\textbf{w}))]\] \[+{\mathbb{E}_{q_{\phi}(z_{w})}}[\log p_{\theta}(w_{1:n}|z_{w},\psi (\textbf{w}))]\] \[-\mathrm{KL}[q_{\phi}(z_{s}|\psi(\textbf{w}))||p(z_{s})]\] \[-\mathrm{KL}[q_{\phi}(z_{w}|\psi(w_{1:n}))||q_{\phi}(z_{w}|\psi( w_{a:b}))].\]

## Appendix B Data Collection and Preprocessing

We are qualified Twitter Academic Research API 5 users. We obtained the ethical approval for our proposed research from the university's ethics committee before the start of our work. We collected tweets between February 7th and April 3rd, 2022 using 60 vaccine-related keywords. The exhaustive list is: _'covid-19 vax'_, _'covid-19 vaccine'_, _'covid-19 vaccines'_, _'covid-19 vaccination'_, _'covid-19 vaccinations'_, _'covid-19 jabs'_, _'covid19 vaccine'_, _'covid19vaccines'_, _'covid19 vaccination'_, _'covid19 jac'_, _'covid19 jab'_, _'covid19 jabs'_, _'covid19 yacine'_, _'covid19 vaccinations'_, _'covid19 jab'_, _'covid19 jabs'_, _'covid wax'_, _'covid Vaccine'_, _'covid vaccines'_, _'covid jack'_, _'covid jabs'_, _'coronavirus vaccines'_, _'coronavirus vaccinated'_, _'coronavirus vaccinations'_, _'coronavirus jabs'_, _'Pfizer Vaccine'_, _'BioNTech vaccine'_, _'Oxford vaccine'_, _'AstraZeneca Vaccine'_, _'Moderna vaccine'_, _'Sputnik vaccine'_, _'Sinovac vaccine'_, _'Singubarm vaccine'_, _'Pfizer jab'_, _'BioTTech job'_, _'Oxford jab'_, _'Maderna Zeneca jab'_, _'Shotnik yield'_, _'Silnovac jab'_, _'Sinopharm jab'_, _'Pflicer voxx'_, _'Oxdord vaccinate'_, _'AstraZeneca vaccinate'_, _'Satnuik vaccinate'_, _'Sinovac vaccinate'_, _'Sinovac vaccinate'_.

Footnote 5: [https://developer.twitter.com/en/products/twitter-api/academic-research/application-info](https://developer.twitter.com/en/products/twitter-api/academic-research/application-info)

Only tweets in English were collected. Retwets were discarded. For pre-processing, hyperlinks, usernames and irregular symbols were removed. Emojis and emoticons were converted to their literal meanings using an emoticon dictionary6.

Footnote 6: [https://wwwrock.fr/en/t/kaomoji/](https://wwwrock.fr/en/t/kaomoji/)

## Appendix C Hyper-parameters and Training Details

The dimensions of \(z_{a}\), \(z_{w}\) and \(z_{s}\) are \(768\), \(768\) and \(32\), respectively. For each tweet, the number of samples from \(\epsilon\sim\mathcal{N}(\mathbf{0},\mathbf{I})\) is \(1\). We modified the LM-fine-tuning script7 from the HuggingFace library to implement VADet in the masked LM learning. We use default settings for the training 

script (i.e., Trainer in the HuggingFace library8), except for the batch size which is set to \(128\). The data pre-processor for the masked language model is the data collator for language modeling9, which provides the function of randomly masking the tokens. The tokenizer for the data collator is the ready-to-use ALBERT tokenizer10. For the pretrained language model (i.e., ALBERT) employed in this model, we inherit the default setting from the AlbertConfig class. We train VADet for \(5\) epochs on the un-annotated corpus.

Footnote 7: [https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transforms.Trainer](https://huggingface.co/docs/transformers/master/en/main_classes/trainer#transforms.Trainer)

Footnote 8: [https://huggingface.co/docs/transformers/main_classes/data_collator](https://huggingface.co/docs/transformers/main_classes/data_collator)

In the supervised training of VADet, we use a batch size of \(64\). The learning rate is initialized to \(2e-5\) with a linear warm-up schedule. We employ \(5\)-fold training in which the training set is split into \(5\) subsets, of which 4 are used for training and the rest is for validation at the end of each epoch, and the final prediction is an ensemble of 5 independently-saved models. We train each model for 5 epochs, which takes roughly \(2\) hours on a node of single Nvidia RTX 2080 GPU.

## Appendix D Annotation Guidelines

We invited two annotators who are PhD students and proficient in English to label each tweet with a stance label and an aspect span. Each annotator was instructed to answer four questions in a row. The four questions are:

* What is the stance towards vaccination?
* What is the Aspect Span? (i.e., Events or targets, it can be nouns, noun phrase, clause or sentence with verbal predicates).
* What is the opinion term/span? It should be opinion expressions, comprising both explicit and implicit expressions of stance.
* What is the Aspect category? It should be one of the pre-defined aspect categories (shown in Table A1).

The annotators have the choice to skip some of the questions if they find it difficult to answer. Taking the tweet '_Very grateful to those at Oxford. I've got my first #Covid19 vaccine._' as an example, the annotators are expected to answer with: '_Pro-vaccine_', '_I've got my first #Covid19 vaccine_', '_Very grateful to those at Oxford. I've got my first #Covid19 vaccine_', '2'. If an annotator chooses to skip a tweet at any step of the process, this tweet will be recorded as skipped and the annotator will not be assigned with similar tweets.

We first had a trial run where each annotator was asked to annotate the same set of tweets. Any disagreement was recorded and discussed to refine our annotation guideline in order to achieve consistency between the annotators.

## Appendix E Predefined Aspect Categories

Table A1 shows our pre-defined aspect categories, partly inspired by Morante et al. (2020). These categories are only used in the evaluation of tweet clustering results, not for training.

 



\begin{table}
\begin{tabular}{c|l} Label & Definition \\ \hline \hline
1 & AstraZeneca: How health organisations/institution, communities, groups, individuals and other entities position themselves towards vaccines \\ \hline
2 & AstraZeneca: Explaining personal experiences with any aspect of vaccines \\ \hline
3 & AstraZeneca: The achievement that vaccines have brought (vaccines save lives, protect the community, protect future generations) \\ \hline
4 & AstraZeneca: The (adverse) side effects of vaccines: illnesses, symptoms, deaths \\ \hline
5 & AstraZeneca: The immunity level provided by vaccines \\ \hline
6 & AstraZeneca: The economic effect of vaccination (less illnesses, less expenses for family and society) \\ \hline
7 & AstraZeneca: Discussing the personal freedom to choose in relation to vaccines \\ \hline
8 & AstraZeneca: Discussing the relation between vaccines and religion, conspiracy or moral attitudes \\ \hline
9 & Pfizer or Moderna: How health organisations/institution, communities, groups, individuals and other entities position themselves towards vaccines \\ \hline
10 & Pfizer or Moderna: Explaining personal experiences with any aspect of vaccines \\ \hline
11 & Pfizer or Moderna: The achievement that vaccines have brought (vaccines save lives, protect the community, protect future generations) \\ \hline
12 & Pfizer or Moderna: The (adverse) side effects of vaccines: illnesses, symptoms, deaths \\ \hline
13 & Pfizer or Moderna: The immunity level provided by vaccines \\ \hline
14 & Pfizer or Moderna: The economic effect of vaccination (less illnesses, less expenses for family and society) \\ \hline
15 & Pfizer or Moderna: Discussing the personal freedom to choose in relation to vaccines \\ \hline
16 & Pfizer or Moderna: Discussing the relation between vaccines and religion, conspiracy or moral attitudes \\ \hline
17 & Other Brands or not mentioned: How health organisations/institution, communities, groups, individuals and other entities position themselves towards vaccines \\ \hline
18 & Other Brands or not mentioned: Explaining personal experiences with any aspect of vaccines \\ \hline
19 & Other Brands or not mentioned: The achievement that vaccines have brought (vaccines save lives, protect the community, protect future generations) \\ \hline
20 & Other Brands or not mentioned: The (adverse) side effects of vaccines: illnesses, symptoms, deaths \\ \hline
21 & Other Brands or not mentioned: The immunity level provided by vaccines \\ \hline
22 & Other Brands or not mentioned: The economic effect of vaccination (less illnesses, less expenses for family and society) \\ \hline
23 & Other Brands or not mentioned: Discussing the personal freedom to choose in relation to vaccines \\ \hline
24 & Other Brands or not mentioned: Discussing the relation between vaccines and religion, conspiracy or moral attitudes \\ \hline \end{tabular}
\end{table}
Table A1: The predefined aspect categories and their definitions.