SlowGuess/NAACL_2022_OCR huggingface dataset:
Gabriele Pergola, Lin Gui, and Yulan He. 2021a. A disentangled adversarial neural topic model for separating opinions from plots in user reviews. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2870-2883. Association for Computational Linguistics.
Gabriele Pergola, Elena Kochkina, Lin Gui, Maria Liakata, and Yulan He. 2021b. Boosting low-resource biomedical QA via entity-aware masking strategies. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1977-1985. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881-7892. Association for Computational Linguistics.
Junaid Shuja, Eisa Alanazi, Waleed Alasmary, and Abdulaziz Alashaikh. 2021. Covid-19 open source data sets: a comprehensive survey. Applied Intelligence, 51(3):1296-1325.
Frederik Trauble, Elliot Creager, Niki Kilbertus, Francesco Locatello, Andrea Dittadi, Anirudh Goyal, Bernhard Scholkopf, and Stefan Bauer. 2021. On disentangled representations learned from correlated data. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 10401-10412. PMLR.
Hai Wan, Yufei Yang, Jianfeng Du, Yanan Liu, Kunxun Qi, and Jeff Z. Pan. 2020. Target-aspect-sentiment joint detection for aspect-based sentiment analysis. Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):9122-9129.
Jingjing Wang, Jie Li, Shoushan Li, Yangyang Kang, Min Zhang, Luo Si, and Guodong Zhou. 2018. Aspect sentiment classification with both word-level and clause-level attention networks. In Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence, IJCAI-18, pages 4439-4445. International Joint Conferences on Artificial Intelligence Organization.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu,
Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45. Association for Computational Linguistics.
Chuhan Wu, Fangzhao Wu, Sixing Wu, Junxin Liu, Zhi-gang Yuan, and Yongfeng Huang. 2018. THU_NGN at SemEval-2018 task 3: Tweet irony detection with densely connected LSTM and multi-task learning. In Proceedings of The 12th International Workshop on Semantic Evaluation, pages 51-56, New Orleans, Louisiana. Association for Computational Linguistics.
Junyuan Xie, Ross Girshick, and Ali Farhadi. 2016. Unsupervised deep embedding for clustering analysis. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 478-487, New York, New York, USA. PMLR.
Lu Xu, Hao Li, Wei Lu, and Lidong Bing. 2020. Position-aware tagging for aspect sentiment triplet extraction. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, pages 2339-2349. Association for Computational Linguistics.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480-1489, San Diego, California. Association for Computational Linguistics.
Xiaoyuan Yi, Zhenghao Liu, Wenhao Li, and Maosong Sun. 2020. Text style transfer via learning style instance supported latent space. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pages 3801-3807. International Joint Conferences on Artificial Intelligence Organization. Main track.
Chen Zhang, Qiuchi Li, and Dawei Song. 2019. Aspect-based sentiment classification with aspect-specific graph convolutional networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4568-4578, Hong Kong, China. Association for Computational Linguistics.
Dejiao Zhang, Feng Nan, Xiaokai Wei, Shang-Wen Li, Henghui Zhu, Kathleen McKeown, Ramesh Nallapati, Andrew O. Arnold, and Bing Xiang. 2021. Supporting clustering with contrastive learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5419-5430. Association for Computational Linguistics.
Meishan Zhang, Yue Zhang, and Duy-Tin Vo. 2015a. Neural networks for open domain targeted sentiment. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 612-621, Lisbon, Portugal. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. *Bertscore: Evaluating text generation with BERT*. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015b. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
He Zhao, Longtao Huang, Rong Zhang, Quan Lu, and Hui Xue. 2020. SpanMlt: A span-based multi-task learning framework for pair-wise aspect and opinion terms extraction. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3239-3248. Association for Computational Linguistics.
Lixing Zhu, Gabriele Pergola, Lin Gui, Deyu Zhou, and Yulan He. 2021. Topic-driven and knowledge-aware transformer for dialogue emotion detection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1571-1582, Online. Association for Computational Linguistics.
A Derivation of the Decomposed ELBO
Unsupervised training is based on maximizing the Evidence Lower Bound (ELBO):
$$
\begin{array}{l} \mathbb {E} _ {q _ {\phi} (z _ {s}, z _ {w} | \psi (\mathbf {w}))} [ \log p _ {\theta} (\mathbf {w} | z _ {s}, z _ {w}, \psi (\mathbf {w})) ] \\ - \mathrm {K L} [ q _ {\phi} (z _ {s}, z _ {w} | \psi (\mathbf {w})) | | p (z _ {s}, z _ {w}) ], \\ \end{array}
$$
where  $z$  is partitioned into  $z_{s}$  and  $z_{w}$ . Like standard VAE (Kingma and Welling, 2014), the variational distribution is a multivariate Gaussian with a diagonal covariance:
$$
q _ {\phi} \left(z _ {s}, z _ {w} \mid \psi (\mathbf {w})\right) = \mathcal {N} \left(z _ {s}, z _ {w} \mid \mu , \sigma^ {2} I\right),
$$
where  $\mu = [\mu^s,\mu^w ]$  and  $\sigma = [\sigma^s,\sigma^w ]$ . Since the covariance matrix is diagonal,  $z_{s}$  and  $z_{w}$  are uncorrelated. Therefore, the joint probability is decomposed into:
$$
q _ {\phi} \left(z _ {s}, z _ {w} \mid \psi (\mathbf {w})\right) = q _ {\phi} \left(z _ {s} \mid \psi (\mathbf {w})\right) q _ {\phi} \left(z _ {w} \mid \psi (\mathbf {w})\right),
$$
where  $q_{\phi}(z_s|\psi (\mathbf{w})) = \mathcal{N}(z_s|\mu^s,\sigma^s)$ $\phi$  are the variational parameters. The prior of  $[z_s,z_w]\sim$ $\mathcal{N}(z_s,z_w|\mathbf{0},I)$  can also be decomposed into the product of  $p(z_{s})$  and  $p(z_{w})$  , then the KL term becomes:
$$
\operatorname {K L} \left[ q _ {\phi} \left(z _ {s} \mid \psi (\mathbf {w})\right) \| p \left(z _ {s}\right) \right] + \operatorname {K L} \left[ q _ {\phi} \left(z _ {w} \mid \psi (\mathbf {w})\right) \| p \left(z _ {w}\right) \right].
$$
As for the decoder  $p_{\theta}(\mathbf{w}|z_s,z_w,\psi (\mathbf{w}))$  , the reconstruction of each masked token and  $w_{[CLS]}$  are independent from each other, i.e., they are not predicted in an autoregressive way. Therefore, the joint probability is decomposed into:
$$
\begin{array}{l} p _ {\theta} (\mathbf {w} \mid z _ {s}, z _ {w}, \psi (\mathbf {w})) \\ = p _ {\theta} \left(w _ {[ \mathrm {C L S} ]} \left| z _ {s}, z _ {w}, \psi (\mathbf {w})\right)\right) p _ {\theta} \left(w _ {1: n} \left| z _ {s}, z _ {w}, \psi (\mathbf {w})\right)\right) \\ \end{array}
$$
We customize the decoder network to make  $w_{[CLS]}$  solely dependent on  $z_{s}$ , and obtain
$$
\begin{array}{l} \mathbb {E} _ {q _ {\phi} (z _ {s})} \mathbb {E} _ {q _ {\phi} (z _ {w})} [ \log p _ {\theta} (w _ {[ \mathrm {C L S} ]} | z _ {s}, \psi (\mathbf {w})) + \\ \left. \log p _ {\theta} \left(w _ {1: n} \mid z _ {w}, \psi (\mathbf {w})\right) \right] \\ \end{array}
$$
Here, we omit  $\psi (\mathbf{w})$  for notational simplicity. Given the supervision of annotated aspect spans, the prior of  $z_{w}$  is constrained by  $q_{\phi}(z_w|\psi (w_{a:b}))$  (a.k.a., the encoder of  $w_{a:b}$ ), this will change the KL term into:
$$
\begin{array}{l} \mathrm {K L} \left[ q _ {\phi} \left(z _ {s} \mid \psi (\mathbf {w})\right) \mid \mid p (z _ {s}) \right] \\ + \mathrm {K L} [ q _ {\phi} (z _ {w} | \psi (w _ {1: n})) | | q _ {\phi} (z _ {w} | \psi (w _ {a: b})) ], \\ \end{array}
$$
and finally the ELBO is expressed as
$$
\begin{array}{l} \mathbb {E} _ {q _ {\phi} (z _ {s})} \left[ \log p _ {\theta} \left(w _ {[ \mathrm {C L S} ]} \mid z _ {s}, \psi (\mathbf {w})\right) \right] \\ + \mathbb {E} _ {q _ {\phi} (z _ {w})} [ \log p _ {\theta} (w _ {1: n} | z _ {w}, \psi (\mathbf {w})) ] \\ - \operatorname {K L} \left[ q _ {\phi} \left(z _ {s} \mid \psi (\mathbf {w})\right) \right] | | p (z _ {s}) ] \\ - \operatorname {K L} \left[ q _ {\phi} \left(z _ {w} | \psi \left(w _ {1: n}\right)\right) | | q _ {\phi} \left(z _ {w} | \psi \left(w _ {a: b}\right)\right) \right]. \\ \end{array}
$$
B Data Collection and Preprocessing
We are qualified Twitter Academic Research API  $^{5}$  users. We obtained the ethical approval for our proposed research from the university's ethics committee before the start of our work. We collected tweets between February 7th and April 3rd, 2022 using 60 vaccine-related keywords. The exhaustive list is: 'covid-19 vax', 'covid-19 vaccine', 'covid-19 vaccines', 'covid-19 vaccination', 'covid-19 vaccinations', 'covid-19 jab', 'covid-19 jabs', 'covid19 vax', 'covid19 vaccine', 'covid19 vaccines', 'covid19 vaccination', 'covid19 vaccines', 'covid19 jab', 'covid19 jabs', 'covid vax', 'covid vaccine', 'covid vaccines', 'covid vaccination', 'covid vaccines', 'covid jab', 'covid jabs', 'coronavirus vax', 'coronavirus vaccine', 'coronavirus vaccines', 'coronavirus vaccination', 'coronavirus vaccinations', 'coronavirus jab', 'coronavirus jabs', 'Pfizer vaccine', 'BioNTech vaccine', 'Oxford vaccine', 'AstraZeneca vaccine', 'Moderna vaccine', 'Sputnik vaccine', 'Sinovac vaccine', 'Sinopharm vaccine', 'Pfizer jab', 'BioNTech jab', 'Oxford jab', 'AstraZeneca jab', 'Moderna jab', 'Sputnik jab', 'Sinovac jab', 'Sinopharm jab', 'Pfizer vax', 'BioNTech vax', 'Oxford vax', 'AstraZeneca vax', 'Moderna vax', 'Sputnik vax', 'Sinovac vax', 'Sinopharm vax', 'Pfizer vaccinate', 'BioNTech vaccinate', 'Oxford vaccinate', 'AstraZeneca vaccinate', 'Moderna vaccinate', 'Sputnik vaccinate', 'Sinovac vaccinate', 'Sinopharm vaccinate'.
Only tweets in English were collected. Retweets were discarded. For pre-processing, hyperlinks, usernames and irregular symbols were removed. Emojis and emoticons were converted to their literal meanings using an emoticon dictionary<sup>6</sup>.
C Hyper-parameters and Training Details
The dimensions of  $z_{a}, z_{w}$  and  $z_{s}$  are 768, 768 and 32, respectively. For each tweet, the number of samples from  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$  is 1. We modified the LM-fine-tuning script from the HuggingFace library to implement VADET in the masked LM learning. We use default settings for the training
5https://developer.twitter.com/en/ products/twitter-api/academic-research/ application-info
<https://wprock.fr/en/t/kaomoji/>
$^{7}$ https://github.com/huggingface/transformers/blob/master/examples/pytorch/language-modeling/run_m1m.py
script (i.e., Trainer in the HuggingFace library $^{8}$ ), except for the batch size which is set to 128. The data pre-processor for the masked language model is the data collator for language modeling $^{9}$ , which provides the function of randomly masking the tokens. The tokenizer for the data collator is the ready-to-use ALBERT tokenizer $^{10}$ . For the pretrained language model (i.e., ALBERT) employed in this model, we inherit the default setting from the AlbertConfig class. We train VADET for 5 epochs on the un-annotated corpus.
In the supervised training of VADET, we use a batch size of 64. The learning rate is initialized to  $2e - 5$  with a linear warm-up schedule. We employ 5-fold training in which the training set is split into 5 subsets, of which 4 are used for training and the rest is for validation at the end of each epoch, and the final prediction is an ensemble of 5 independently-saved models. We train each model for 5 epochs, which takes roughly 2 hours on a node of single Nvidia RTX 2080 GPU.
We invited two annotators who are PhD students and proficient in English to label each tweet with a stance label and an aspect span. Each annotator was instructed to answer four questions in a row. The four questions are:
- What is the stance towards vaccination?
- What is the Aspect Span? (i.e., Events or targets, it can be nouns, noun phrase, clause or sentence with verbal predicates).
- What is the opinion term/ span? It should be opinion expressions, comprising both explicit and implicit expressions of stance.
- What is the Aspect category? It should be one of the pre-defined aspect categories (shown in Table A1).
The annotators have the choice to skip some of the questions if they find it difficult to answer. Taking the tweet 'Very grateful to those at Oxford.
I've got my first #Covid19 vaccine.' as an example, the annotators are expected to answer with: 'Pro-vaccine', 'I've got my first #Covid19 vaccine', 'Very grateful to those at Oxford. I've got my first #Covid19 vaccine', '2'. If an annotator chooses to skip a tweet at any step of the process, this tweet will be recorded as skipped and the annotator will not be assigned with similar tweets.
We first had a trial run where each annotator was asked to annotate the same set of tweets. Any disagreement was recorded and discussed to refine our annotation guideline in order to achieve consistency between the annotators.
E Predefined Aspect Categories
Table A1 shows our pre-defined aspect categories, partly inspired by (Morante et al., 2020). These categories are only used in the evaluation of tweet clustering results, not for training.
$^{8}$ https://huggingface.co/docs/  
transformers/master/en/main_classesers/  
trainer#transformers.Trainer  
 $^{9}$ https://huggingface.co/docs/  
transformers/main_classeses/data.collator  
 $^{10}$ https://huggingface.co/docs/  
transformers/master/en/model_doc/albert#  
transformers.AlbertTokenizer
Label
Definition
1	AstraZeneca: How health organisations/institution, communities, groups, individuals and other entities position themselves towards vaccines
2	AstraZeneca: Explaining personal experiences with any aspect of vaccines
3	AstraZeneca: The achievement that vaccines have brought (vaccines save lives, protect the community, protect future generations)
4	AstraZeneca: The (adverse) side effects of vaccines: illnesses, symptoms, deaths
5	AstraZeneca: The immunity level provided by vaccines
6	AstraZeneca: The economic effect of vaccination (less illnesses, less expenses for family and society)
7	AstraZeneca: Discussing the personal freedom to choose in relation to vaccines
8	AstraZeneca: Discussing the relation between vaccines and religion, conspiracy or moral attitudes
9	Pfizer or Moderna: How health organisations/institution, communities, groups, individuals and other entities position themselves towards vaccines
10	Pfizer or Moderna: Explaining personal experiences with any aspect of vaccines
11	Pfizer or Moderna: The achievement that vaccines have brought (vaccines save lives, protect the community, protect future generations)
12	Pfizer or Moderna: The (adverse) side effects of vaccines: illnesses, symptoms, deaths
13	Pfizer or Moderna: The immunity level provided by vaccines
14	Pfizer or Moderna: The economic effect of vaccination (less illnesses, less expenses for family and society)
15	Pfizer or Moderna: Discussing the personal freedom to choose in relation to vaccines
16	Pfizer or Moderna: Discussing the relation between vaccines and religion, conspiracy or moral attitudes
17	Other Brands or not mentioned: How health organisations/institution, communities, groups, individuals and other entities position themselves towards vaccines
18	Other Brands or not mentioned: Explaining personal experiences with any aspect of vaccines
19	Other Brands or not mentioned: The achievement that vaccines have brought (vaccines save lives, protect the community, protect future generations)
20	Other Brands or not mentioned: The (adverse) side effects of vaccines: illnesses, symptoms, deaths
21	Other Brands or not mentioned: The immunity level provided by vaccines
22	Other Brands or not mentioned: The economic effect of vaccination (less illnesses, less expenses for family and society)
23	Other Brands or not mentioned: Discussing the personal freedom to choose in relation to vaccines
24	Other Brands or not mentioned: Discussing the relation between vaccines and religion, conspiracy or moral attitudes
Table A1: The predefined aspect categories and their definitions.