SlowGuess/NAACL_2021_OCR huggingface dataset:
Emily Allaway and Kathleen McKeown. 2020. Zero-shot stance detection: A dataset and model using generalized topic representations. In EMNLP.
Isabelle Augenstein, Tim Rocktäschel, Andreas Vlachos, and Kalina Bontcheva. 2016. Stance detection with bidirectional conditional encoding. In EMNLP.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv, abs/1810.04805.
Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In ICML.
S. Hochreiter and J. Schmidhuber. 1997. Long short-term memory. Neural Computation, 9:1735-1780.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Dilek Küçuk. 2017. Stance detection in turkish tweets. ArXiv, abs/1706.06894.
Mirko Lai, Alessandra Teresa Cignarella, D. I. H. Farias, Cristina Bosco, V. Patti, and P. Rosso. 2020. Multilingual stance detection in social media political debates. Comput. Speech Lang., 63:101075.
J. Lin. 1991. Divergence measures based on the shannon entropy. IEEE Trans. Inf. Theory, 37:145-151.
Saif M. Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiao-Dan Zhu, and Colin Cherry. 2016. Semeval-2016 task 6: Detecting stance in tweets. In SemEval@NAACL-HLT.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
2https://developer.twitter.com/en/ developer-terms/agreement-and-policy
Barbara Plank and Gertjan van Noord. 2011. Effective measures of domain similarity for parsing. In ACL.
Sebastian Ruder and Barbara Plank. 2017. Learning to select data for transfer learning with bayesian optimization. In EMNLP.
Mariona Taule, Maria Antonia Martí, Francisco M. Rangel Pardo, Paolo Rosso, Cristina Bosco, and Viviana Patti. 2017. Overview of the task on stance and gender detection in tweets on catalan independence. In IberEval@SEPLN.
Adam Tsakalidis, Nikolaos Aletras, Alexandra I. Cristea, and Maria Liakata. 2018. Nowcasting the stance of social media users in a sudden vote: The case of the greek referendum. Proceedings of the 27th ACM International Conference on Information and Knowledge Management.
Jannis Vamvas and Rico Sennrich. 2020. X -stance: A multilingual multi-target dataset for stance detection. In KONVENS.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of the Thirty-First Annual Conference on Neural Information Systems.
Penghui Wei and W. Mao. 2019. Modeling transferable topics for cross-target stance detection. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval.
Wan Wei, Xiao Zhang, Xuqin Liu, Wei Chen, and Tengjiao Wang. 2016. pkudblab at semeval-2016 task 6: A specific convolutional neural network system for effective stance detection. In SemEval@NAACL-HLT.
Chang Xu, Cécile Paris, Surya Nepal, and Ross Sparks. 2018. Cross-target stance classification with self-attention networks. In ACL.
Guido Zarrella and Amy Marsh. 2016. Mitre at semeval-2016 task 6: Transfer learning for stance detection. In SemEval@NAACL-HLT.
B. Zhang, M. Yang, Xutao Li, Yunming Ye, Xiaofei Xu, and Kuai Dai. 2020. Enhancing cross-target stance detection with transferable semantic-emotion knowledge. In ACL.
Yuan Zhang, Regina Barzilay, and Tommi S. Jaakkola. 2017. Aspect-augmented adversarial networks for domain adaptation. Transactions of the Association for Computational Linguistics, 5:515-528.
Our models are implemented using Pytorch<sup>3</sup>. We implement our K-means clustering with Scikit-learn<sup>4</sup>. Our models are trained using one Titan Xp P8 GPU, but, as noted in the paper, they can also be trained on the CPU with a minimal increase in computation time.
We train TOAD, TOAD without adversary, and BiCond for a maximum of 100 epochs with early stopping on the development set, computed using  $F_{avg}$ . We use Adam (Kingma and Ba, 2015) to optimize and for the adversarial model we decay the learning rate in relation to the discriminator strength hyperparameter  $\rho$  (Ganin and Lempitsky, 2015). Specifically, until epoch 50, the learning rate is fixed at  $l$  and the value of  $\rho$  remains 0. If total number of epochs is  $t$ , for an epoch  $e > 50$  we compute,  $p = (e - 50) / t$ . The learning rate at epoch  $e$  is computed as  $l / (1 + \alpha \cdot p)^{\beta}$  and the value of  $\rho$  at epoch  $e$  is computed as  $2 / (1 + e^{-\gamma \cdot p}) - 1$  where  $\alpha, \beta$  and  $\gamma$  are hyperparameters.
For the BERT baseline we fine-tune for 10 epochs using the implementation of BERT from the Hugging Face Transformers library. We use a batch size of 16 and a learning rate of  $2e - 5$  with linear decay after the first  $10\%$  of training steps. We optimize using AdamW. To prevent exploding gradients, we apply gradient clipping to 1.0.
We report validation performance of our models on stance classification (see Table 5) as well as the score of the topic-discriminator on the training set, since it is not computed on the development set (see Table 6). We also show the average number of parameters and runtime for all models averaged over all topics (see Table 7).
We tune the hyperparameters for our adversarial model using uniform sampling on the development set with 20 search trials. We select the best hyperparameter setting using the average rank of the stance classifier F1 (higher is better) and topic discriminator F1 (lower is better). We remove settings where the discriminator F1 is  $< 0.01$ , under the assumption that such low performance is the result of overly corrupt representations that will not generalize. In all models, we use pre-trained
100-dimensional GloVe vectors (Pennington et al., 2014) in our models. We show hyperparameter configurations and search space for TOAD (Table 8), TOAD without the adversary (Table 9) and BiCond (Table 10). Note that there are no hyperparemters to tune for the BERT baseline.
A.3 Data
We preprocess tweets by removing URLs and mentions. We remove the # symbol from hashtags in tweets and tokenize the hashtags. We remove emojis and punctuation from tweets. We convert tweets to lowercase and remove stopwords from the tweets. We show the class distribution in Table 11.
A.4 Topic Divergence
Jensen-Shannon divergence (Lin, 1991) is a smoothed, symmetric variant of KL divergence. Let  $t^{(1)}$  and  $t^{(2)}$  be two topics and  $P$  and  $Q$  be word-distributions for the topics respectively. Then the KL divergence is defined as  $D_{KL}(P||Q) = \sum_{i} p_{i} \log \frac{p_{i}}{q_{i}}$ . However,  $D_{KL}(P||Q)$  is undefined if  $q_{i} = 0$  for any  $q_{i} \in Q$ . Therefore, Jensen-Shannon divergence uses the average distribution  $M = \frac{1}{2}(P + Q)$  and is defined as
$$
D _ {J S} (P | | Q) = \frac {1}{2} \left(D _ {K L} (P | | M) + D _ {K L} (Q | | M)\right).
$$
We follow Plank and van Noord (2011) in computing word distributions for each topic pair. Let  $V = \cup_{t} V_{t}$  be the union of the vocabularies for all topics  $t$ . Then for the topic pair  $t^{(1)}$  and  $t^{(2)}$ , the distribution for one topic is either  $t \in \mathbb{R}^{|V_{t^{(1)}} \cup V_{t^{(2)}}|}$  or  $t \in \mathbb{R}^{|V_{t^{(1)}}|}$ , where  $t_{i}$  is the probability of the  $i$ -th word in the vocabulary. Note, we use  $V_{t^{(1)}} \cup V_{t^{(2)}}$  or  $V_{t^{(1)}}$  rather than  $V$  to ensure that  $m_{i} \neq 0$  for all  $m_{i} \in M$ , regardless of choice of topics. Also note that when using only the vocabulary from  $t^{(1)}$ ,  $D_{JS}(P||Q)$  is no longer symmetric, since the size of  $t$  depends of which topic is  $t^{(1)}$ .
A.5 Ablation Results
We report full ablation results on all components of the adversarial model, on all six topics on the development sets (see Table 12).
We also report the results of applying K-means clustering on the representations extracted from the models trained in each setup. For clustering, we extract representations for the entire dataset (train, dev, and test). Then we randomly split the dataset into train and test with no zero-shot topic. We fit
<sup>4</sup>https://scikit-learn.org/stable/
DT	HC	FM	LA	A	CC
BERT	45.0	42.8	41.8	42.9	42.9	41.5
BiCond	66.7	68.7	65.6	66.9	64.8	61.3
(64.6 ± 0.01)	(65.8 ± 0.01)	(64.2 ± 0.006)	(64.4 ± 0.02)	(62.0 ± 0.01)	(59.7 ± 0.006)
TOAD	66.1	65.9	62.2	64.9	64.6	64.8
(64.4 ± 0.09)	(64.1 ± 0.18)	(59.8 ± 0.09)	(63.0 ± 0.07)	(62.0 ± 0.06)	(58.8 ± 0.13)
- adv	69.3	72.6	68.2	69.2	66.5	65.3
(68.1 ± 0.008)	(70.7 ± 0.008)	(66.7 ± 0.007)	(66.8 ± 0.01)	(65.3 ± 0.006)	(63.9 ± 0.02)
Table 5:  $F_{avg}$  results on the development sets for each topic, with mean and variance shown for models with hyperparameter tuning.
DT	HC	FM	LA	A	CC
TOAD	1.9	2.2	28.7	1.3	26.5	4.2
Table 6: Topic-discriminator F1 on the training set for TOAD across topics.
K-means clustering on the training portion and evaluate on the test portion. We use the same train/test split for all clusterings. We evaluate using homogeneity (evaluates whether each cluster contains only examples of one topic) and completeness (all examples from one topic are in one cluster) (see Table 13).
BERT	BiCond	TOAD	TOAD -adv
# parameters	110 million	358926	554152	632915
avg. runtime	15min	5min	20min	20min
Table 7: Search trials, time, and parameters for models. We average across all six topics for each model.
Hyperparameter	Search space	Best assignment
DT	HC	FM	LA	A	CC
BiLSTM hidden size	unifrom-integer[40-150]	80	105	113	115	111	105
Stance classifier hidden size	uniform-integer[80-300]	147	278	201	222	213	254
Topic discriminator hidden size	uniform-integer[40-150]	85	95	140	120	143	90
λrec	choice[1]	1	1	1	1	1	1
λtr	choice[0.1, 1, 10]	0.1	0.1	10.0	10.0	1.0	0.1
γ	uniform-integer[10-15]	14	12	14	11	11	10
α	choice[10]	10	10	10	10	10	10
β	choice[0.25]	0.25	0.25	0.25	0.25	0.25	0.25
l	choice[0.001]	0.001	0.001	0.001	0.001	0.001	0.001
Table 8: Hyperparameter search space and best settings for TOAD.
Hyperparameter	Search space	Best assignment
DT	HC	FM	LA	A	CC
BiLSTM hidden size	unifrom-integer[40-150]	96	96	140	134	140	115
Stance classifier hidden size	uniform-integer[80-300]	137	137	228	166	228	222
Table 9: Hyperparameter search space and best settings for TOAD without the adversary.
Hyperparameter	Search space	Best assignment
DT	HC	FM	LA	A	CC
BiLSTM hidden size	unifrom-integer[40-150]	74	94	128	78	141	104
Dropout	uniform-flat[0.1-0.4]	0.2380	0.3220	0.4015	0.3086	0.3912	0.2501
Table 10: Hyperparameter search space and best setting for BiCond.
Topic	%Pro	%Con	%Neither	# Total
DT	20.9	42.3	36.8	707
HC	16.6	57.4	26.0	984
FM	28.2	53.8	18.0	949
LA	17.9	58.3	23.8	933
A	16.9	63.3	19.8	733
CC	59.4	4.6	36.0	564
Table 11: Class distributions for each of the six topics.
DT	HC	FM	LA	A	CC
F1	Δ	F1	Δ	F1	Δ	F1	Δ	F1	Δ	F1	Δ
TOAD	49.5	51.2	54.1	46.2	46.1	30.9
- transformation	38.8	-10.7	43.2	-8.0	51.9	-2.2	43.6	-2.6	44.5	-1.6	44.4	+13.5
- Ltr	34.8	-14.7	48.0	-3.1	47.9	-6.2	39.7	-6.4	41.9	-4.3	4.5	-26.4
- Ltrec	44.6	-4.9	52.5	+1.3	35.0	-19.1	46.9	+0.7	38.7	-7.4	4.4	-26.5
- Ldrec	36.9	-12.6	46.3	-4.9	49.7	-4.4	48.3	+2.1	43.1	-3	18.1	-12.8
- Ltrec & - Ldrec	43.0	-6.5	43.5	-7.7	40.1	-14.0	43.3	-2.9	39.5	-6.6	37.3	+6.4
- residual topic	39.3	-10.2	43.4	-7.8	45.4	-8.7	43.3	-2.9	44.6	-1.5	37.3	+6.4
-Dul	40.0	-9.5	51.1	-0.1	44.0	-10.1	46.2	-0.0	40.3	-5.8	26.1	-4.8
Table 12: Full component ablation on test sets for all six topics.
DT	HC	FM	LA	A	CC
Hom.	Com.	Hom.	Com.	Hom.	Com.	Hom.	Com.	Hom.	Com.	Hom.	Com.
TOAD	0.034	0.034	0.118	0.120	0.293	0.302	0.091	0.093	0.091	0.092	0.144	0.149
- adv	0.102	0.104	0.135	0.142	0.078	0.081	0.075	0.078	0.033	0.034	0.097	0.1
Table 13: Homogeneity (Hom.) and completeness (Com.) for clusters computed with the representations extracted from models with each of the six topics as the test set.