SlowGuess/NAACL_2022_OCR huggingface dataset:
(e.g., Pfizer, AstraZeneca, and Moderna). Our final dataset comprises 1.9 million English tweets collected between February 7th and April 3rd, 2021. We randomly sample a subset of tweets for annotation. Upon an initial inspection, we found that over  $97\%$  of tweets mentioned only one aspect. As such, we annotate each tweet with a stance label and a text span characterizing the aspect. In total, 2,800 tweets have been annotated in which 2,000 are used for training and the remaining 800 are used for testing. The statistics of the dataset is listed in Table 1. The stance labels are imbalanced. On the other hand, the average opinion length is longer than the average aspect length, and is close to the average tweet length. For the purpose of evaluation on tweet clustering and latent topic disentanglement, we further annotate tweets with a categorical label indicating the aspect category. Inspired by (Morante et al., 2020), we identify 24 aspect categories<sup>3</sup> and each tweet is annotated with one of these categories. It is worth mentioning that aspect category labels are not used for training.
VC (Morante et al., 2020) is a vaccination corpus consisting of 294 Internet documents about online vaccine debate annotated with events, 210 of which are annotated with opinions (in the form of text spans) towards vaccines. The stance label is considered to be the stance for the whole sentence. Those sentences with conflicting stance labels are regarded as neutral. We split the dataset into a ratio of 2:1 for training and testing. This eventually left us with 1,162 sentences for training and 531 sentences for testing.
Baselines We compare the experimental results with the following baselines:
BertQA (Li et al., 2018c): a pre-trained language model well-suited for span detection. With BertQA, attitude detection is performed by first classifying stance labels then predicting the answer queried by the stance label. The text span is configured as the ground-truth answer. We rely on its Hugging-Face $^4$  (Wolf et al., 2020) implementation. We employ ALBERT (Lan et al., 2020) as the backbone language model for both BertQA and VADET.
ASTE (Peng et al., 2020): a pipeline approach consisting of aspect extraction (Li et al., 2018c) and sentiment labelling (Li et al., 2018b).
Evaluation Metrics For stance classification, we use accuracy and Macro-averaged F1 score. For aspect span detection, we follow Rajpurkar et al. (2016) in adopting exact match (EM) accuracy of the starting-ending position and Macro-averaged F1 score of the overlap between the prediction and ground truth aspect span. For tweet clustering, we follow Xie et al. (2016) and Zhang et al. (2021) and use the Normalized Mutual Information (NMI) metric to measure how the clustered group aligns with ground-truth categories. In addition, we also report the clustering accuracy.
In all our experiments, VADET is firstly pre-trained in an unsupervised way on our collected 1.9 million tweets before fine-tuned on the annotated training set from the VAD or VC corpora.
Stance Classification and Aspect Span Detection In Table 2, we report the performance on attitude detection. In stance classification, our model outperforms both baselines with more significant improvements on ASTE. On aspect span extraction, VADET yields even more noticeable improvements, with a  $2.3\%$  increase in F1 over BertQA on VAD, and  $2.7\%$  on VC. These results indicate that the successful prediction relies on the hidden representation learned in the unsupervised training. The disentanglement of stance and aspect may have also contributed to the improvement.
Clustering To assess whether the learned latent aspect topics would allow meaningful categorization of documents into attitude clusters, we perform
4https://huggingface.co/ transformers/model_doc/albert.html# albertforquestionanswering
3The full list of aspect categories is shown in Table A1.
Model	VAD	VC
Stance	Acc.	F1	Acc.	F1
BertQA	0.754	0.742	0.719	0.708
ASTE	0.723	0.710	0.704	0.686
VADET	0.763	0.756	0.727	0.713
Aspect Span	Acc.	F1	Acc.	F1
BertQA	0.546	0.722	0.525	0.670
ASTE	0.508	0.684	0.467	0.652
VADET	0.556	0.745	0.541	0.697
Cluster	Acc.	NMI	Acc.	NMI
DEC (BertQA)	0.633	58.1	0.586	52.8
K-means (BERT)	0.618	56.4	0.571	50.1
DEC (VADET)	0.679	60.7	0.605	54.7
Table 2: Results for stance classification, aspect span extraction and aspect clustering on both VAD and VC corpora.
clustering using the disentangled representations that encode aspects, i.e.,  $z_{w}$ . Deep Embedding Clustering (DEC) (Xie et al., 2016) is employed as the backend. For comparison, we also run DEC on the aspect representations of documents returned by BertQA. For each document, its aspect representation is obtained by averaging over the fine-tuned ALBERT representations of the constituent words in its aspect span. To assess the quality of clusters, we need the annotated aspect categories for documents in the test set. In VAD, we use the annotated aspect labels as the ground-truth categories whereas in VC we use the annotated event types. Results are presented in the lower part of Table 2. We found a prominent increase in NMI score over the baselines. Using the learned latent aspect topics as features, DEC (VADET) outperforms DEC (BertQA) by  $4.6\%$  and  $1.9\%$  in accuracy on VAD and VC, respectively. We also notice that using K-means as the clustering approach directly on the BERT-encoded tweet representations gives worse results compared to DEC. A similar trend is observed on the NMI metric. The improvements are shown visually in Figure 4 where the clustered groups produced by VADET are more identifiable. In the absence of categorical labels, the perspective expressed by each group can be inferred from the constituent tweets. For example, the tweet '@user Georgian nurse dies of allergic reaction after receiving AstraZeneca Covid19 vaccine' lies in the centroid of the red group, which relates to safety concerns.
Cluster Semantic Coherence Evaluation The semantic coherence is the extent to which tweets within a cluster belong to each other, which is em
(a) VADET
(b) BertQA
Figure 4: Clustered groups of VADET and BertQA on the VAD dataset. Each color indicates a ground truth aspect category. The clusters are dominated by: (1) Red: the (adverse) side effects of vaccines; (2) Green: explaining personal experiences with any aspect of vaccines; and (3) Cyan: the immunity level provided by vaccines.
ployed as an evaluation metric for cluster quality evaluation in an unsupervised way. Recent work of Bilal et al. (2021) found that Text Generation Metrics (TGMs) align well with human judgement in evaluating clusters in the context of microblog posts. TGM by definition measures the similarity between the ground-truth and the generated text. The rationale is that a high TGM score means sentence pairs are semantically similar. Here, two metrics are used: BERTScore, which calculates the similarity of two sentences as a sum of cosine similarities between their tokens' embeddings (Zhang et al., 2020), and BLEURT, a pre-trained adjudicator that fine-tunes BERT on an external dataset of human ratings (Sellam et al., 2020). As in (Bilal et al., 2021), we adopt the Exhaustive Approach that for a cluster  $C$ , its coherence score is the average TGM score of every possible tweet pair in the cluster:
$$
f (C) = \frac {1}{N ^ {2}} \sum_ {i, j \in [ 1, N ], i <   j} \mathrm {T G M} (\mathrm {t w e e t} _ {i}, \mathrm {t w e e t} _ {j}).
$$
Figure 5 shows the BERTScore and the BLEURT score of VADET and baselines on two datasets. The VADET shows consistent improvements across the datasets. This indicates that tweets clustered using the latent aspect topics generated by VADET are semantically more similar, thus validating the assumption that disentangled representations are more effective in bringing together tweets of a similar gist.
Conditional Perplexity Few metrics have been proposed to evaluate the quality of disentangled representations (Pergola et al., 2021a). Therefore, we
Figure 5: Semantic coherence evaluated in two metrics.
adopt the language model perplexity conditioned on  $z_{a}$  to evaluate the extent to which the disentangled representation improves language generation on held-out data. Perplexity is widely used in the literature of text style transfer (John et al., 2019; Yi et al., 2020), where the probability of the generated language is calculated conditioned on the controlled latent code. A lower perplexity score indicates better language generation performance. Following John et al. (2019), we compute an estimated aspect vector  $\hat{z}_{a}^{(k)}$  of a cluster  $k$  in the training set as
$$
\hat {z} _ {a} ^ {(k)} = \frac {\sum_ {i \in \operatorname {c l u s t e r} k} z _ {a , i} ^ {(k)}}{\# \text {t w e e t s i n c l u s t e r} k},
$$
where  $z_{a,i}^{(k)}$  is the learned aspect vector of the  $i$ -th tweet in the  $k$ -th cluster. For the stance vector  $z_{s}$  we sample one value per tweet. The stance vector is concatenated with the aspect vector  $\hat{z}_a^{(k)}$  to calculate the probability of generating the held-out data, i.e., the testing set. For the baseline models, we choose  $\beta$ -VAE (Higgins et al., 2017) and SCHOLAR (Card et al., 2018). We train  $\beta$ -VAE on the same data with  $\beta$  set to different values. SCHOLAR is trained on tweet content and stance labels. For both the baselines we use ELBO on the held-out data as an upper bound on perplexity.
Figure 6 plots the perplexity score achieved by all the methods. Our model achieves the lowest perplexity score on both datasets. It managed to decrease the perplexity value by roughly 200 compared to the baseline models. SCHOLAR outperforms  $\beta$ -VAE under three settings of  $\beta$  value. We speculate that this might be due to the incorporation of the class labels in the training of SCHOLAR. Nevertheless, VADET produces congenial sentences in aspect groups, with latent codes tweaked to proxy centroids, showing that the disentangled representation does capture the desired
Figure 6: Conditional perplexity on two corpora.
Ablations We conduct ablation studies to investigate the effect of semi-supervised learning that uses the variational latent representation learning approach and aspect-stance disentanglement on the latent semantics. We study their effects on stance classification and aspect span detection. The results are reported in Table 3.
Model	VAD	VC
Stance	Acc.	F1	Acc.	F1
VADET	0.763	0.756	0.727	0.713
VADET-D	0.751	0.746	0.736	0.716
VADET-U	0.741	0.734	0.712	0.698
Aspect Span	Acc.	F1	Acc.	F1
VADET	0.556	0.745	0.541	0.697
VADET-D	0.540	0.728	0.537	0.684
VADET-U	0.528	0.712	0.525	0.653
Table 3: Results of stance classification and aspect span detection of VADET without disentanglement (-D) or unsupervised pre-training (-U).
We can observe that on VAD without disentangled learning or unsupervised pre-training results in the degradation of the stance classification performance. However, on VC, we see a slight increase in classification accuracy without disentangled learning. We attribute this to the vagueness of the stance which might cause the model to disentangle more than it should be. On the aspect span detection task, we observe consistent performance drop across all metrics and on both datasets. In particular, without the pre-training module, the performance drops more significantly. These results indicate that semi-supervised learning is highly effective with VAE, and the disentanglement of stance and aspect serves as a useful component, which leads to noticeable
In this work, we presented a semi-supervised model to detect user attitudes and distinguish aspects of interest about vaccines on social media. We employed a Variational Auto-Encoder to encode the main topical information into the language model by unsupervised training on a massive, unannotated dataset. The model is then further trained under a semi-supervised setting that leverages annotated stance labels and aspect spans to induce the disentanglement between stances and aspects in a latent semantic space. We empirically showed the benefits of such an approach for attitude detection and aspect clustering over two vaccine corpora. Ablation studies show that disentangled learning and unsupervised pre-training are important to effective vaccine attitude detection. Further investigations on the quality of the disentangled representations verify the effectiveness of the disentangled factors. While our current work mainly focuses on short text of social media data where a sentence is assumed to discuss a single aspect, it would be interesting to extend our model to deal with longer text such as online debates in which multiple arguments or aspects may appear in a single sentence.
This work was funded by the UK Engineering and Physical Sciences Research Council (grant no. EP/T017112/1, EP/V048597/1). LZ is supported by a Chancellor's International Scholarship at the University of Warwick. YH is supported by a Turing AI Fellowship funded by the UK Research and Innovation (grant no. EP/V020579/1).
Juan M Banda, Ramya Tekumalla, Guanyu Wang, Jingyuan Yu, Tuo Liu, Yuning Ding, Ekaterina Artemova, Elena Tutubalina, and Gerardo Chowell. 2021. A large-scale pandemic-19 twitter chatter dataset for open scientific research—an international collaboration. Epidemiologia, 2(3):315-324.
Jeremy Barnes, Robin Kurtz, Stephan Oepen, Lilja Øvrelid, and Erik Velldal. 2021. Structured sentiment analysis as dependency graph parsing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 3387-3402. Association for Computational Linguistics.
Christos Baziotis, Nikos Pelekis, and Christos Doukkeridis. 2017. DataStories at SemEval-2017 task 4: Deep LSTM with attention for message-level and topic-based sentiment analysis. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 747-754, Vancouver, Canada. Association for Computational Linguistics.
Iman Munire Bilal, Bo Wang, Maria Liakata, Rob Procter, and Adam Tsakalidis. 2021. Evaluation of the thematic coherence in microblogs. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, pages 6800-6814. Association for Computational Linguistics.
Diane Bouchacourt, Ryota Tomioka, and Sebastian Nowozin. 2018. Multi-level variational autoencoder: Learning disentangled representations from grouped observations. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).
Christopher P Burgess, Irina Higgins, Arka Pal, Loic Matthew, Nick Watters, Guillaume Desjardins, and Alexander Lerchner. 2018. Understanding disentangling in  $\beta$ -vae. arXiv preprint arXiv:1804.03599.
Dallas Card, Chenhao Tan, and Noah A. Smith. 2018. Neural models for documents with metadata. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 2031-2040, Melbourne, Australia. Association for Computational Linguistics.
Ricky T. Q. Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. 2018. Isolating sources of disentanglement in variational autoencoders. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Ruidan He, Wee Sun Lee, Hwee Tou Ng, and Daniel Dahlmeier. 2019. An interactive multi-task learning network for end-to-end aspect-based sentiment analysis. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 504-515, Florence, Italy. Association for Computational Linguistics.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. 2017. beta-vae: Learning basic visual concepts with a constrained variational framework. In 5th International Conference on Learning Representations, ICLR 2017,
Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net.
Tao Hu, Siqin Wang, Wei Luo, Mengxi Zhang, Xiao Huang, Yingwei Yan, Regina Liu, Kelly Ly, Viraj Kacker, Bing She, and Zhenlong Li. 2021. Revealing public opinion towards COVID-19 vaccines with twitter data in the united states: Spatiotemporal perspective. J Med Internet Res, 23(9):e30854.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1587-1596. PMLR.
Amir Hussain, Ahsen Tahir, Zain Hussain, Zakariya Sheikh, Mandar Gogate, Kia Dashtipour, Azhar Ali, and Aziz Sheikh. 2021. Artificial intelligence-enabled analysis of public attitudes on facebook and twitter toward Covid-19 vaccines in the united kingdom and the united states: Observational study. J Med Internet Res, 23(4):e26627.
Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. 2019. Nonlinear ica using auxiliary variables and generalized contrastive learning. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 859-868. PMLR.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 424-434, Florence, Italy. Association for Computational Linguistics.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. 2020. Variational autoencoders and nonlinear ica: A unifying framework. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Research, pages 2207-2217. PMLR.
Diederik P. Kingma, Danilo J. Rezende, Shakir Mohamed, and Max Welling. 2014. Semi-supervised learning with deep generative models. In Proceedings of the 27th International Conference on Neural Information Processing Systems, NIPS'14, page 3581-3589, Cambridge, MA, USA. MIT Press.
Diederik P. Kingma and Max Welling. 2014. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings.
Florian Kunneman, Mattijs Lambooij, Albert Wong, Antal Van Den Bosch, and Liesbeth Mollema. 2020.
Monitoring stance towards vaccination in twitter messages. BMC medical informatics and decision making, 20(1):1-14.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020. ALBERT: A lite BERT for self-supervised learning of language representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Chunyuan Li, Xiang Gao, Yuan Li, Baolin Peng, Xiujun Li, Yizhe Zhang, and Jianfeng Gao. 2020. Optimus: Organizing sentences via pre-trained modeling of a latent space. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 4678-4699. Association for Computational Linguistics.
Juncen Li, Robin Jia, He He, and Percy Liang. 2018a. Delete, retrieve, generate: a simple approach to sentiment and style transfer. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1865-1874, New Orleans, Louisiana. Association for Computational Linguistics.
Xin Li, Lidong Bing, Wai Lam, and Bei Shi. 2018b. Transformation networks for target-oriented sentiment classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, pages 946-956, Melbourne, Australia. Association for Computational Linguistics.
Xin Li, Lidong Bing, Piji Li, Wai Lam, and Zhimou Yang. 2018c. Aspect term extraction with history attention and selective transformation. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 4194-4200. International Joint Conferences on Artificial Intelligence Organization.
Xin Li, Lidong Bing, Wenxuan Zhang, and Wai Lam. 2019. Exploiting BERT for end-to-end aspect-based sentiment analysis. In Proceedings of the 5th Workshop on Noisy User-generated Text (W-NUT 2019), pages 34-41, Hong Kong, China. Association for Computational Linguistics.
Francesco Locatello, Stefan Bauer, Mario Lucic, Gunnar Raetsch, Sylvain Gelly, Bernhard Scholkopf, and Olivier Bachem. 2019. Challenging common assumptions in the unsupervised learning of disentangled representations. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 4114-4124. PMLR.
Francesco Locatello, Ben Poole, Gunnar Raetsch, Bernhard Scholkopf, Olivier Bachem, and Michael Tschannen. 2020a. Weakly-supervised disentangle-ment without compromises. In Proceedings of the 37th International Conference on Machine Learning,
volume 119 of Proceedings of Machine Learning Research, pages 6348-6359. PMLR.
Francesco Locatello, Michael Tschannen, Stefan Bauer, Gunnar Ratsch, Bernhard Scholkopf, and Olivier Bachem. 2020b. Disentangling factors of variations using few labels. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Joanne Chen Lyu, Eileen Le Han, and Garving K Luli. 2021. Covid-19 vaccine-related discussion on twitter: topic modeling and sentiment analysis. Journal of medical Internet research, 23(6):e24435.
Yukun Ma, Haiyun Peng, and Erik Cambria. 2018. Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive LSTM. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1).
Diego Marcheggiani, Oscar Tackstrom, Andrea Esuli, and Fabrizio Sebastiani. 2014. Hierarchical multilabel conditional random fields for aspect-oriented opinion mining. In Advances in Information Retrieval, pages 273-285, Cham. Springer International Publishing.
Joel Ruben Antony Moniz and David Krueger. 2017. Nested lstms. In Proceedings of the Ninth Asian Conference on Machine Learning, volume 77 of Proceedings of Machine Learning Research, pages 530-544, Yonsei University, Seoul, Republic of Korea. PMLR.
Roser Morante, Chantal van Son, Isa Maks, and Piek Vossen. 2020. Annotating perspectives on vaccination. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 4964-4973, Marseille, France. European Language Resources Association.
Eric Nalisnick, Lars Hertel, and Padhraic Smyth. 2016. Approximate inference for deep latent gaussian mixtures. In NIPS Workshop on Bayesian Deep Learning, volume 2, page 131.
Elise Paul, Andrew Steptoe, and Daisy Fancourt. 2021. Attitudes towards vaccines and intention to vaccinate against Covid-19: Implications for public health communications. The Lancet Regional Health - Europe, 1:100012.
Haiyun Peng, Lu Xu, Lidong Bing, Fei Huang, Wei Lu, and Luo Si. 2020. Knowing what, how and why: A near complete solution for aspect-based sentiment analysis. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):8600-8607.
Gabriele Pergola, Lin Gui, and Yulan He. 2019. TDAM: A topic-dependent attention model for sentiment analysis. Information Processing & Management, 56(6):102084.