SlowGuess/ACL_2020_OCR huggingface dataset:
- BiLSTM: This method uses BiLSTM to encode the sentence and target separately. The hidden states from both directions are combined to infer the stance label.
- BiCond (Augenstein et al., 2016): This method is similar to BiLSTM but uses a conditional encoding method that learns a target-dependent sentence representation for stance detection.
- CrossNet (Xu et al., 2018): This model is a variant of BiCond, which leverages a self
Source-Target:	FM→LA	LA→FM	HC→DT	DT→HC	HC→TP	TP→HC	DT→TP	TP→DT
BiLSTM	0.448	0.412	0.298	0.358	0.291	0.395	0.311	0.341
BiCond	0.450	0.416	0.297	0.358	0.292	0.402	0.317	0.347
CrossNet	0.454	0.433	0.431	0.362	0.298	0.417	0.314	0.374
VTN	0.473	0.478	0.479	0.364	-	-	-	-
BERT	0.479	0.339	0.436	0.365	0.261	0.231	0.241	0.456
CrossNet-C	0.449	0.439	0.442	0.369	0.297	0.413	0.324	0.355
CrossNet-CF	0.467	0.457	0.457	0.396	0.307	0.411	0.377	0.398
CrossNet-CA	0.473	0.475	0.455	0.407	0.301	0.442	0.409	0.396
TextCNN-E	0.469	0.458	0.380	0.404	0.309	0.450	0.356	0.396
SEKT (Ours)	0.536	0.513	0.477	0.420	0.335	0.460	0.444	0.395
Table 2: Performance comparison of cross-target stance detection in terms of  $F1_{avg}$  on 8 tasks.
Source-Target:	FM→LA	LA→FM	HC→DT	DT→HC	HC→TP	TP→HC	DT→TP	TP→DT
BiLSTM	0.401	0.379	0.433	0.401	0.236	0.418	0.207	0.389
BiCond	0.403	0.392	0.442	0.408	0.239	0.424	0.207	0.396
CrossNet	0.442	0.431	0.461	0.418	0.244	0.425	0.211	0.407
BERT	0.499	0.395	0.412	0.399	0.353	0.295	0.391	0.478
CrossNet-C	0.473	0.399	0.439	0.403	0.251	0.428	0.221	0.414
CrossNet-CF	0.497	0.438	0.434	0.404	0.280	0.437	0.302	0.428
CrossNet-CA	0.507	0.434	0.452	0.401	0.283	0.453	0.375	0.440
TextCNN-E	0.513	0.466	0.360	0.385	0.283	0.472	0.191	0.433
SEKT (Ours)	0.523	0.510	0.463	0.432	0.300	0.489	0.391	0.435
Table 3: Performance comparison of different models for cross-target stance detection.
attention layer to capture important words in the input text.
- VTN (Wei and Mao, 2019): The model utilizes the latent topics shared between the two targets as transferable knowledge for cross-target adaptation.
- BERT (Devlin et al., 2019): The method fine-tunes a pre-trained BERT model to perform cross-target detection. Specifically, we convert the given context and target to "[CLS] + target + [SEP] + context" structure for source and target domain, respectively.
We also extend CrossNet and TextCNN to incorporate external knowledge (SE-graph), resulting in stronger competitors.
- CrossNet-C: Similar to (Margatina et al., 2019), we extend the original CrossNet model by incorporating external knowledge. Here, three variants are considered, where CrossNet-C adopts the attentional concatenation, CrossNet-CF uses the feature-based gating mechanism, and CrossNet-CA adopts an attentional affine transformation.
- TextCNN-E: TextCNN (Kim, 2014) is an important baseline for text classification. Here, we extend TextCNN to the cross
target setting, denoted as TextCNN-E. Specifically, each word is represented as a 3D tensor by concatenating the embeddings of  $k$  semantically/emotionally-related words.
4.5 Overall Performance
We report the experimental results in terms of  $\mathrm{F1}_{avg}$  and  $\mathrm{F1}_m$  in Table 2 and Table 3, respectively. From the results, we can observe that BiLSTM has the worst performance because BiLSTM neither exploits the target information nor considers knowledge transfer for the cross-target stance detection. BiCond performs slightly better than BiLSTM, since it explicitly encodes the target information. As an extension to BiCond by introducing the attention mechanism, CrossNet shows a marginal improvement (e.g.,  $13.4\%$  on  $\mathrm{HC} \rightarrow \mathrm{DT}$  for  $\mathrm{F1}_{avg}$ ,  $3.9\%$  on  $\mathrm{LA} \rightarrow \mathrm{FM}$  for  $\mathrm{F1}_m$ ). This may be because that the attention mechanism can learn the informative stance-aware sentence representation. However, this knowledge transfer scheme is based on word-level information, which often suffers from the data scarcity problem. VTN, which is a concept-level knowledge transfer model, achieves the best performance among all the baseline methods. It is noteworthy that the performance of BERT is not stable. Promising results are achieved on  $\mathrm{FM} \rightarrow \mathrm{LA}$  and  $\mathrm{HC} \rightarrow \mathrm{DT}$ , but it performs unsatisfactorily on other tasks. The reason may be that BERT does not explicitly employ any knowledge transfer
SEKT	w/o SE	w/o KAMU
FM→LA	0.536 (0.523)	0.461 (0.492)	0.471 (0.499)
LA→FM	0.513 (0.510)	0.443 (0.455)	0.475 (0.469)
HC→DT	0.477 (0.463)	0.449 (0.439)	0.449 (0.450)
DT→HC	0.420 (0.432)	0.400 (0.404)	0.411 (0.407)
HC→TP	0.335 (0.279)	0.314 (0.278)	0.321 (0.280)
TP→HC	0.460 (0.489)	0.448 (0.466)	0.453 (0.471)
DT→TP	0.444 (0.391)	0.407 (0.371)	0.411 (0.376)
TP→DT	0.395 (0.435)	0.394 (0.420)	0.395 (0.431)
strategy. The proposed SEKT method yields better performance than all the baselines in most of the tasks. For example, our method improves  $5.7\%$  on  $\mathrm{FM} \rightarrow \mathrm{LA}$ ,  $3.5\%$  on  $\mathrm{LA} \rightarrow \mathrm{FM}$ ,  $5.5\%$  on  $\mathrm{DT} \rightarrow \mathrm{HC}$  over the best competitors in terms of  $\mathrm{F1}_{avg}$ . The advantage of SEKT comes from its two characteristics: (i) we develop a GCN based model to fully exploit the external knowledge from both semantic and emotion lexicons; (ii) a knowledge-aware memory unit is proposed to better fuse the external knowledge.
We also compare our SEKT model with the competitors that also integrate the semantic-emotion knowledge graph with GCN, e.g., CrossNet-C, CrossNet-CF, CrossNet-CA and TextCNN-E. The results are demonstrated in Table 2 and Table 3. CrossNet-C produces the worst performance in general. The reason is that concatenating the external knowledge and context representation could make the external knowledge lost in the sentence encoding process. CrossNet-CF and CrossNet-CA perform better than CrossNet-C since they incorporate the external knowledge into the hidden layers of BiLSTM. As expected, SEKT achieves the best performance, which verifies the effectiveness of the KAMU model.
To investigate the impact of each part on our SEKT model, we perform the ablation test by discarding SE graph knowledge (denoted as w/o SE) and knowledge-aware memory unit (denoted as w/o KAMU), respectively. Specifically, for the w/o SE model, the external knowledge is expressed by a weighted sum of the embeddings of four semantically/emotionally-related words. For the w/o KAMU model, we replace the KE-BiLSTM structure by the standard BiLSTM layer, and the external knowledge is combined in the hidden layer.
Table 4: Ablation test results in terms of  $F1_{avg}$  and  $F1_m$  (in the parentheses) by discarding SE graph (w/o SE) and knowledge-aware memory unit (w/o KAMU).
hop No.	DT→HC	LA→FM	DT→TP
1	0.401	0.489	0.431
2	0.417	0.513	0.444
3	0.420	0.479	0.424
4	0.374	0.369	0.408
Table 5: The experimental results with respect to varying number of hops in GCN.
The ablation results are summarized in Table 4. From the results, we observe that both the SE graph and KAMU make great improvements to our SEKT method. The external semantic and emotional knowledge can help SEKT to capture multi-hop semantic correlations between words or emotion tags. On the one hand, KAMU helps to fully incorporate the external knowledge into the BiLSTM network, which makes the representation learning model more general to new targets.
Number of Hops Based on our empirical observation, capturing the multi-hop semantic correlation is one of the most important parts for the overall performance of SEKT. Thus, we also investigate the impact of the number of hops used in GCN. In particular, we evaluate the performance of SEKT by varying the number of hops from 1 to 4 with a step size of 1. From Table 5, we can observe that the best results are achieved when the number of hops is 2 or 3. This is because GCN with a mediate hop number can capture semantic correlations between words while preventing from introducing unnecessary noises.
5 Error Analysis
To better understand the limitations of SEKT, we additionally carry out an analysis of the errors made by SEKT. Specifically, we randomly select 100 instances that are incorrectly predicted by SEKT from the expanded SemEval-2016 dataset. We revealed several reasons for the classification errors, which can be divided into the following categories. First, SEKT fails to classify some sentences that contain latent opinions or require deep comprehension. For example, for the sentence "I guess NBC does not like to hear the truth.[favor]" with a target "Donald Trump", SEKT tends to predict an incorrect against stance. This is because the SEKT model cannot learn the implicit relationship between  $\mathrm{NBC^{*}}$  and TRUMP, which is not acquirable from the semantic-emotion lexicons. The
* National Broadcasting Company
second error category is caused by special hashtags with implicit meanings. For example, SEKT cannot correctly predict the stance for the sentence "The gift that keeps on giving. #makeitstop #SemST" [against]. This may be because the information in the sentence is not sufficient enough such that SEKT cannot capture the sequential patterns of the stance-related words. It suggests that certain data augmentation strategy needs to be devised in the future so as to capture the sequential patterns between stance-related words from short texts.
In this paper, we proposed a semantic-emotion knowledge transferring (SEKT) model for cross-target stance classification, which used the external knowledge from semantic and emotion lexicons as commonsense knowledge to bridge the gap across different targets. Specifically, we first built a SE-graph from semantic and emotion lexicons, which leveraged external knowledge from both word-level and concept-level. Second, the GCN was employed to learn the graph representation that captured multi-hop semantic connections between words or emotion tags. Third, we extend the standard BiLSTM classifier to fully integrate the external knowledge by adding a novel knowledge-aware memory unit to the BiLSTM cell. The experimental results demonstrated that the SEKT model significantly outperformed the state-of-the-art methods for cross-target stance detection.
This research was supported in part by the National Key R&D Program of China, 2018YFB2101100, 2018YFB2101101 and NSFC under Grant Nos. U1836107, 61972111, 61572158 and 61602132. Min Yang was partially supported by National Natural Science Foundation of China (No. 61906185), Guangdong Basic and Applied Basic Research Foundation (No. 2019A1515011705 and No. 2018A030313943), and the project AWS13C008.
I Augenstein, T Rocktaeschel, A Vlachos, and K Bontcheva. 2016. Stance detection with bidirectional conditional encoding. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Sheffield.
Erik Cambria, Soujanya Poria, Devamanyu Hazarika, and Kenneth Kwok. 2018. Senticnet 5: Discov-
ering conceptual primitives for sentiment analysis by means of context embeddings. In Thirty-Second AAAI Conference on Artificial Intelligence.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4186. Association for Computational Linguistics.
Kuntal Dey, Ritvik Shrivastava, and Saroj Kaushik. 2018. Topical stance detection for twitter: A two-phase LSTM model using attention. In European Conference on Information Retrieval, pages 529-536. Springer.
Mauro Dragoni and Giulio Petrucci. 2018. A fuzzy-based strategy for multi-domain sentiment analysis. International Journal of Approximate Reasoning, 93:59-73.
Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017. Stance classification with target-specific neural attention networks. International Joint Conferences on Artificial Intelligence.
Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. 2016. Harnessing deep neural networks with logic rules. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2410-2420.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751.
Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.
Zeyang Lei, Yujiu Yang, and Min Yang. 2018. SAAN: A sentiment-aware attention network for sentiment analysis. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, SIGIR 2018, Ann Arbor, MI, USA, July 08-12, 2018, pages 1197-1200. ACM.
Yukun Ma, Haiyun Peng, and Erik Cambria. 2018. Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive LSTM. In Thirty-Second AAAI Conference on Artificial Intelligence.
Katerina Margatina, Christos Baziotis, and Alexandros Potamianos. 2019. Attention-based conditioning methods for external knowledge integration. arXiv preprint arXiv:1906.03674.
Saif M Mohammad and Peter D Turney. 2013. Crowdsourcing a word-emotion association lexicon. Computational Intelligence, 29(3):436-465.
Swapna Somasundaran and Janyce Wiebe. 2009. Recognizing stances in online debates. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP: Volume 1-Volume 1, pages 226-234. Association for Computational Linguistics.
Qingying Sun, Zhongqing Wang, Qiaoming Zhu, and Guodong Zhou. 2018. Stance detection with hierarchical attention network. In Proceedings of the 27th International Conference on Computational Linguistics, pages 2399-2409.
Matt Thomas, Bo Pang, and Lillian Lee. 2006. Get out the vote: Determining support or opposition from congressional floor-debate transcripts. In Proceedings of the 2006 conference on empirical methods in natural language processing, pages 327-335. Association for Computational Linguistics.
Marilyn A Walker, Pranav Anand, Robert Abbott, and Ricky Grant. 2012. Stance classification using dialogic properties of persuasion. In Proceedings of the 2012 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 592-596. Association for Computational Linguistics.
Penghui Wei, Junjie Lin, and Wenji Mao. 2018. Multi-target stance detection via a dynamic memory-augmented network. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1229-1232. ACM.
Penghui Wei and Wenji Mao. 2019. Modeling transferable topics for cross-target stance detection. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1173-1176. ACM.
Chang Xu, Cecile Paris, Surya Nepal, and Ross Sparks. 2018. Cross-target stance classification with self-attention networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 778-783.
Bowen Zhang, Xiaofei Xu, Min Yang, Xiaojun Chen, and Yunming Ye. 2018. Cross-domain sentiment classification by capsule network with semantic rules. IEEE Access, 6:58284-58294.
Chen Zhang, Qiuchi Li, and Dawei Song. 2019a. Aspect-based sentiment classification with aspect-specific graph convolutional networks. arXiv preprint arXiv:1909.03477.
Jingqing Zhang, Piyawat Lertvittayakumjorn, and Yike Guo. 2019b. Integrating semantic knowledge to tackle zero-shot text classification. In Proceedings
of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1031-1040.