SlowGuess/ACL_2020_OCR huggingface dataset:
Bowen Zhang $^{1}$ , Min Yang $^{2}$ , Xutao Li $^{3*}$ , Yunming Ye $^{3*}$ , Xiaofei Xu $^{1}$ , Kuai Dai $^{3}$
<sup>1</sup>Harbin Institute of Technology, Harbin, China
$^{2}$ Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, China
$^{3}$ Harbin Institute of Technology, Shenzhen, China
Stance detection is an important task, which aims to classify the attitude of an opinionated text towards a given target. Remarkable success has been achieved when sufficient labeled training data is available. However, annotating sufficient data is labor-intensive, which establishes significant barriers for generalizing the stance classifier to the data with new targets. In this paper, we proposed a Semantic-Emotion Knowledge Transferring (SEKT) model for cross-target stance detection, which uses the external knowledge (semantic and emotion lexicons) as a bridge to enable knowledge transfer across different targets. Specifically, a semantic-emotion heterogeneous graph is constructed from external semantic and emotion lexicons, which is then fed into a graph convolutional network to learn multi-hop semantic connections between words and emotion tags. Then, the learned semantic-emotion graph representation, which serves as prior knowledge bridging the gap between the source and target domains, is fully integrated into the bidirectional long short-term memory (BiLSTM) stance classifier by adding a novel knowledge-aware memory unit to the BiLSTM cell. Extensive experiments on a large real-world dataset demonstrate the superiority of SEKT against the state-of-the-art baseline methods.
The goal of stance detection is to automatically predict the attitude (i.e., favor, against, or none) of an opinionated text towards a given target (Du et al., 2017). Recently, deep learning methods, such as convolutional neural network (CNN) and long short-term memory (LSTM) (Augenstein et al., 2016; Du et al., 2017), have dominated the study of stance detection. Impressive stance detection performances have been achieved when a large
number of labeled samples are available. However, obtaining rich annotated data is a time-consuming and labor-intensive process. Conventional stance detection methods are struggling to cope well with the data across targets. This motivates the studies of cross-target stance detection (Wei and Mao, 2019), which infers the attitude of the destination target by leveraging a large amount of annotated data from the source target.
So far, several previous studies have been conducted for cross-target stance detection (Augenstein et al., 2016; Xu et al., 2018; Wei and Mao, 2019). These methods leverage either common words or concept-level knowledge shared by different targets to bridge the knowledge gap across the different targets. Such models suffer from two issues when they are applied to cross-target stance detection in practice. First, stance detection often involves analyzing the texts from social media that are short and informal, making it difficult to extract domain-independent common words shared by different targets from the training data. Second, users may express their stance towards a given target in an implicit way. Thus, the existing concept-level based methods may fail to distinguish implicit stance-carrying terms and context information.
To alleviate the aforementioned issues, we propose a semantic-emotion knowledge transferring (SEKT) model for cross-domain stance detection, which leverages external knowledge as a bridge between source and destination targets. The proposed model is motivated by the observation that the data with different targets usually shares certain common external knowledge that can be transferred from the source to destination targets. First, we build a semantic-emotion graph (SE-graph) from semantic-related and emotion-related lexicons, which incorporates external knowledge from both word-level and concept-level. In SE-graph, each node is either a word or an emotion tag, and
* corresponding authors: {lixutao, yym} @hit.edu.cn
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3188-3197 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
the edge between each node pair indicates the co-occurrences of the two nodes in the lexicons. Second, a graph convolutional network (GCN) (Kipf and Welling, 2016) is employed to learn the graph representation that captures the multi-hop semantic connections between words or emotion tags rather than one-hop connection. Third, we extend the standard bidirectional LSTM (BiLSTM) classifier to fully integrate the external knowledge (SEgraph) by adding an additional knowledge-aware memory unit (KAMU) to the LSTM cell. KAMU is capable of controlling the influence of the external knowledge in learning the hidden state of each word.
- We construct a semantic-emotion heterogeneous graph from external semantic and emotion lexicons, and employ GCN to learn the semantic graph representation. The external knowledge enriches the representation learning of the text and target and can be used as a bridge to enable knowledge transfer across different targets.
- We extend the standard LSTM cell with an additional memory unit, effectively integrating external knowledge into the classifier for stance detection.
- We conduct extensive experiments on a large dataset expanded from SemEval-2016 Task 6 to verify the effectiveness of our model for cross-domain stance detection. The experimental results show that our model consistently outperforms the compared methods.
2.1 In-domain Stance Detection
Stance detection aims to infer the attitude of a text towards specific target expression, which is related to argument mining, fact-checking, and aspect-level sentiment analysis. Early stance detection methods were concentrated on debates (Thomas et al., 2006; Somasundaran and Wiebe, 2009; Walker et al., 2012). In recent years, mining users' stance from social media has attracted increasing attention due to its broad applications (Du et al., 2017; Dey et al., 2018; Wei et al., 2018). For example, Du et al. (2017) incorporated target-specific information into stance classification with
an attention mechanism. Dey et al. (2018) proposed a two-phase RNN method, where the first phase is to filter the non-neutral text while the second phase is to classify the attitude. Wei et al. (2018) further extended the model to deal with multi-target stance detection and utilized a shared memory network to capture the stance related information towards multiple related targets. Sun et al. (2018) adopted a hierarchical attention method to construct text representation with various linguistic factors.
2.2 Cross-target Stance Detection
There are also several studies being developed for cross-target stance detection problems, which can be divided into two classes. The first one mainly focuses on word-level transfer, which utilizes the common words shared by two targets to bridge the knowledge gap. For example, Augenstein et al. (2016) proposed a bidirectional conditional encoding method by incorporating the target to learn the target-specific words. Xu et al. (2018) further utilized the self-attention mechanism to identify the word importance. The second type of approach attempts to address this transfer learning problem with concept-level knowledge shared by two targets. For example, Wei and Mao (2019) proposed a variational Transfer Network (VTN) method, which complements the commonly used knowledge by inferring the latent topics shared by the two targets.
2.3 Incorporating External Knowledge
There are also plenty of studies that incorporate external resources, such as prior knowledge, grammar rules, domain descriptions, into deep learning framework to address the data sparsity issue (Zhang et al., 2018; Dragoni and Petrucci, 2018; Zhang et al., 2019b; Hu et al., 2016). For example, Lei et al. (2018) integrated the external knowledge in the word embedding layer. Margatina et al. (2019) combined the external knowledge with the hidden layer acquired by RNN. However, these methods ignored the relations between external knowledge and input context. Ma et al. (2018) developed a Sentic LSTM method, which contained an additional affective gate mechanism in the LSTM cell to assist in learning knowledge-aware context representation.
Figure 1: The framework of the proposed SEKT model for cross-target stance detection. It consists of two main components, i.e., SE-graph and knowledge-enhanced BiLSTM.
3 Our Methodology
We use  $X^{s} = \{x_{i}^{s}, p_{i}^{s}\}_{i=1}^{N^{s}}$  to denote the collection of labeled data in the source domain, where each  $x$  denotes the input text and  $p$  denotes the corresponding target.  $N^{s}$  represents the number of instances in  $X^{s}$ . Each sentence-target pair  $(x^{s}, p^{s}) \in X^{s}$  has a stance label  $y^{s}$ . Given an input sentence  $x^{t}$  and a corresponding target  $p^{t}$  in the target domain, this study aims to predict a stance label for the input sentence  $x^{t}$  towards the given target  $p^{t}$  by using the model learned with the labeled data  $X^{s}$  in source domain.
As illustrated in Figure 1, our model consists of two primary components: a semantic-emotion graph (SE-graph) network and a knowledge-enhanced BiLSTM network. First, we build SE-graph from semantic-related and emotion-related lexicons, where GCN is employed to learn the graph representation that captures the semantic connections between words or emotion tags with the multi-hop connection. Then, we extend the BiLSTM classifier to fully integrate the SE-graph by adding a novel knowledge-aware memory unit (KAMU) to the LSTM cell. Next, we will introduce the main components of our model in detail.
3.2 Semantic-Emotion Knowledge Graph Construction
The data in different domains usually shares certain background knowledge that can possibly be transferred from the source domain to the target domain. Thus, we leverage external knowledge as a bridge between the source and target domains.
To this end, we build a semantic-emotion knowledge graph (SE-graph) to represent the external knowledge that may contribute to cross-target stance detection. The SE-graph utilizes the words or emotion tags in the semantic and emotion lexicons as nodes, and constructs weighted edges between words or emotion tags based on their co-occurrence frequency. First, we utilize the whole words from the semantic lexicon SenticNet (Cambria et al., 2018) as the word-nodes and add edges between the semantic words that capture the word-word semantic connections. Second, we attempt to assign emotion tags to the words in SenticNet by looking for the emotion lexicon EmoLex (Mohammad and Turney, 2013), and add edges between the words and emotion tags that capture the word-tag connection. For example, for a word "mad" in SenticNet, its semantic-related words from SenticNet are "resent, malice, rage, temper", and the corresponding emotion tags from EmoLex are "#anger", #disgust". In this way, we can construct a weighted SE graph  $G$ . However, each emotion tag (node) represents a concept-level knowledge,
which tends to have many connected nodes. As a result, emotional knowledge may dominate the input text. To alleviate this issue, we re-scale the weights of the word-tag edges by a constant.
The SE-graph can capture the semantic connections between words and emotion tags with multihop connections. It can help the stance detector to differentiate the important and appropriate words for knowledge transfer. Intuitively, the nodes with high degrees can be considered as the words that contain common background knowledge, which often act as a bridge between different targets.
3.3 SE-graph Embedding
We learn the embedding of each node in the SE-graph with graph convolutional network (GCN), aiming to fully exploit the multi-hop semantic and emotional connections between the nodes. Due to the semantic locality between the words, we extract a  $k$ -hop subgraph from SE-graph for each word. The subgraph is then fed into a GCN to learn the graph representation. Here, we adopt GCN because it has been proved to be effective and efficient to learn graph embedding (Zhang et al., 2019a).
Formally, let  $E \in \mathbb{R}^{v \times d}$  be a matrix containing all  $v$  nodes in SE-graph with their features, where  $d$  is the size of the node embedding. For each node, we extract a  $k$ -hop subgraph  $G_{s}$  from the whole graph, which has a degree matrix  $D$  and adjacency matrix  $A$ . The normalized symmetric adjacency matrix of subgraph  $G_{s}$  can be calculated as:  $\tilde{A} = D^{-\frac{1}{2}}AD^{-\frac{1}{2}}$ . By feeding the subgraph  $G_{s}$  into a two-layer GCN, the corresponding subgraph representation  $L \in \mathbb{R}^{n \times c}$  with  $n$  nodes can be calculated by:
$$
L = \sigma (\tilde {A} \sigma (\tilde {A} E W _ {0}) W _ {1}) \tag {1}
$$
where  $\sigma$  represents a non-linear function,  $W_0\in \mathbb{R}^{d*v}$  and  $W_{1}\in \mathbb{R}^{d*c}$  are trainable parameters. To obtain a more compact graph representation, we further feed  $L$  into a fully-connected layer, producing a final graph representation  $M\in \mathbb{R}^d$
3.4 Knowledge-enhanced BiLSTM
Preliminary (Vanilla BiLSTM) Generally, two independent BiLSTM networks (denoted as  $\mathrm{BiLSTM}^x$  and  $\mathrm{BiLSTM}^p$ ) are employed to encode the input sentence  $x$  and the target  $p$ , respectively. BiLSTM can capture the left and right context of each word in the input. In particular, for the  $t$ -th word  $w_t$  in the input sequence of the target,
$\overrightarrow{h}_t^p$  computes its forward hidden state  $\overleftarrow{h}_t^p$  and backward hidden state  $\overleftarrow{h}_t^p$ . We concatenate both the forward and backward hidden states to form the final hidden state  $h_t^p = [\overrightarrow{h}_t^p \oplus \overleftarrow{h}_t^p]$  for word  $w_t$  at the  $t$ -th position of the input target. After learning the contextual representation of the target, we learn a target-aware sentence representation  $H^s$  by initializing  $\mathrm{BiLSTM}^x$  with the final hidden state of  $\mathrm{BiLSTM}^p$ .
The background knowledge contained in external lexicons is the collection of facts that individuals are expected to know, and plays a crucial role in reading comprehension. We propose a knowledge-enhanced BiLSTM (KE-BiLSTM) model, which incorporates the external background knowledge contained in the semantic-emotion knowledge graph into the BiLSTMs via a novel knowledge-aware memory unit (KAMU). KE-BiLSTM helps to identify discriminative semantic and emotion knowledge from the input text. It is motivated by two considerations:
- The external commonsense knowledge provides rich information of entities and relations between them, and highlights the features that are essential for stance detection. For example, with the external semantic lexicon, we can correctly understand the unusual word "zugzwang" through the semantically related words "chess", "strategy", "forced" contained in the semantic lexicon. Hence, we devise KE-BiLSTM to effectively leverage the graph embedding of SE-graph and fully explore the external knowledge from both word-level and concept-level.
- There exist dynamic interaction patterns and complementarity between the context and the external knowledge within the input sequence for stance detection. Instead of leveraging only the input context in each BiLSTM unit, we take external commonsense knowledge into consideration by adding a novel knowledge-aware memory unit to the BiLSTM, which dynamically controls the amount of external knowledge at each encoding step and thus balances the contextual and knowledge information for stance detection.
As illustrated in Figure 2, KE-BiLSTM consists of two primary parts: a BiLSTM network (depicted in blue) and a knowledge-aware memory unit (depicted in green). Similar to the standard BiLSTM
Figure 2: The structure of the knowledge-enhanced BiLSTM unit.
network, KE-BiLSTM also computes forward and backward hidden sequences, which are then combined to form the output representation. Due to limited space, we solely introduce the implementation details of the forward layer. The forward and backward knowledge-enhance LSTMs can be computed in a similar way.
In KE-BiLSTM, the BiLSTM network learns the sequential features of the input text. Formally, in the forward layer of BiLSTM, the input gate  $i_{t}$ , forget gate  $f_{t}$ , output gate  $g_{t}$ , and the memory cell  $\overrightarrow{C}_{t}$  are updated as:
$$
i _ {t} = \sigma \left(W _ {i} w _ {t} + U _ {i} \overrightarrow {h} _ {t - 1} + V _ {i} \overrightarrow {C} _ {t - 1}\right) \tag {2}
$$
$$
f _ {t} = \sigma \left(W _ {f} w _ {t} + U _ {f} \overrightarrow {h} _ {t - 1} + V _ {f} \overrightarrow {C} _ {t - 1}\right) \tag {3}
$$
$$
\begin{array}{l} g _ {t} = \tanh  \left(W _ {g} w _ {t} + U _ {g} \overrightarrow {h} _ {t - 1} + V _ {g} \overrightarrow {C} _ {t - 1}\right) (4) \\ \vec {C} _ {t} = f _ {t} \odot \vec {C} _ {t - 1} + i _ {t} \odot g _ {t} (5) \\ \end{array}
$$
where  $\sigma$  represents the sigmoid function.  $W$ ,  $U$ , and  $V$  are trainable parameters.  $w_{t}$  is the  $t$ -th word of the input text.  $\overrightarrow{h}_{t-1}$  is the hidden state for the  $t-1$ -th word.
We propose a knowledge-aware memory component to incorporate the external knowledge into BiLSTM. For each word  $w_{t}$ , we extract the corresponding entity from SE-graph by performing n-gram matching and acquire a subgraph representation  $M_t^0$ . A new knowledge memory  $\vec{M}_t$  at time  $t$  is computed with a linear interpolation between the previous  $M_t^0$  and its candidate activation  $\delta_t$ :
$$
\overrightarrow {M} _ {t} = z _ {t} \odot M _ {t} ^ {0} + (1 - z _ {t}) \odot \delta_ {t} \tag {6}
$$
where  $z_{t}\in [0,1]$  is utilized to balance the importance of  $M_t^0$  and  $\delta_t$ , which can be computed by:
$$
z _ {t} = \sigma \left(W _ {z} w _ {t} + U _ {z} M _ {t} ^ {0}\right) \tag {7}
$$
where  $W_{z}$  and  $U_{z}$  are parameters to be learned. The candidate activation  $\delta_t$  is updated as:
$$
\delta_ {t} = \tanh  \left(W _ {\delta} w _ {t} + U _ {\delta} \left(r _ {t} \odot M _ {t} ^ {0}\right)\right) \tag {8}
$$
where  $W_{\delta}$  and  $U_{\delta}$  are parameters to be learned.  $r_t$  is the reset gate which aims to combine the knowledge in  $M_t^0$  and  $w_t$ , which is defined as:
$$
r _ {t} = \sigma \left(W _ {r} w _ {t} + U _ {i} M _ {t} ^ {0}\right) \tag {9}
$$
where  $W_{r}$  and  $U_{r}$  are projection parameters.
Finally, the linear transformation of  $w_{t}$ ,  $h_{t-1}$ ,  $\overrightarrow{M}_{t}$  and  $C_{t}$  are combined to calculate the output  $\overrightarrow{\sigma}_{t}$  of the forward KE-BiLSTM layer:
$$
\begin{array}{l} \vec {\sigma} _ {t} = \sigma \left(W _ {o} w _ {t} + U _ {o} \vec {h} _ {t - 1} + V _ {o} \vec {M} _ {t} + Q _ {o} \vec {C} _ {t}\right) (10) \\ \vec {h} _ {t} = o _ {t} \odot \tanh  (\vec {C} _ {t} + \vec {M} _ {t}) (11) \\ \end{array}
$$
where  $\vec{o}_t$  and  $\vec{h}_t$  denote the output gate and the hidden state of the forward network of KE-BiLSTM unit at time step  $t$ . The hidden state  $\vec{h}_t$  of the backward network at time step  $t$  can be computed in a same way. We can get the overall hidden state  $h_t = [\vec{h}_t \oplus \vec{h}_t]$  for word  $w_t$ .
Finally, we can use KE-BiLSTM to learn knowledge-enhanced sentence representation  $H^{s} = \{h_{1}^{s},\dots ,h_{n}^{s}\}$  and knowledge-enhanced target representation  $H^{p} = \{h_{1}^{p},\dots ,h_{m}^{p}\}$ , where  $n$  and  $m$  denote the lengths of sentence  $x$  and given target  $p$ , respectively.
3.5 Stance Detection
We employ an attention mechanism to characterize the effect of the target on enforcing our SEKT model to pay more attention to the important words of the context. In particular, we use the target representation  $H^{p}$  as the attention source to calculate the attention weight  $\alpha_{t}$  for the  $t$ -th word:
$$
\alpha_ {t} = \operatorname {s o f t m a x} \left(\bar {h ^ {p}} ^ {\mathrm {T}} h _ {t} ^ {x}\right) \tag {12}
$$
where  $h^p$  denote the average vector of target representation  $H^p$ . We can learn the attentive sentence representation  $emb$  by congregating the embeddings of hidden states  $H^s$  with attention vector  $\alpha$ :
$$
e m b = \sum_ {t = 1} ^ {n} \alpha_ {t} h _ {t} ^ {x} \tag {13}
$$
Target	Favor/Against/None	Avg-length
DT	148/299/260	17.1
HC	163/565/256	17.0
FM	268/ 511/170	18.4
LA	167/544/222	19.0
TP	333/452/460	33.3
Table 1: The statistics of our experimental data extended from SemEval-2016 Task 6.
Finally, the sentence representation  $emb$  is fed into a fully-connected layer followed by a softmax layer to compute a stance probability distribution:
$$
\hat {y} = \operatorname {s o f t m a x} \left(W _ {y} \operatorname {e m b} + b _ {y}\right) \tag {14}
$$
where  $W_{y}$  is a projection parameter and  $b_{y}$  is a bias term.  $\hat{y}$  denotes the predicted stance probability for the input sentence  $x$  and target  $p$ . Given an annotated training set  $X^{s}$ , we utilize the cross-entropy between the predicted stance  $\hat{y}$  and the ground-truth stance  $y$  as our loss function for stance detection:
$$
\mathcal {L} = - \sum_ {i = 1} ^ {N} \sum_ {j = 1} ^ {C} y _ {i j} \log \hat {y} _ {i j} \tag {15}
$$
where  $N$  represents the number of instances in the training set.  $C$  denotes the number of possible stance categories.  $y_{i}$  represents the one-hot represented ground-truth label for the  $i$ -th instance.  $\hat{y}_i$  is the predicted stance probability vector. This model can be optimized with the standard gradient descent algorithm.
4.1 Experimental Data
We extend the SemEval-2016 Task 6 dataset (denoted as SemEval-2016) to evaluate the performance of our SEKT model for cross-target stance detection. SemEval-2016 is the first stance detection dataset collected from Twitter, which contains 4870 stance-bearing tweets towards different targets. Each tweet is classified as "favor", "against" or "none". Following the previous work (Wei and Mao, 2019), we use the tweets from four targets, including Donald Trump (DT), Hillary Clinton (HC), Legalization of Abortion (LA), and Feminist Movement (FM). These targets are commonly utilized to evaluate the cross-target stance classification.
In addition to the four targets in SemEval-2016, we introduce an additional Trade Policy (TP) target as the fifth target, which is an incredibly hot topic nowadays. Specifically, 1245 tweets related
to TP are collected and manually labeled as "favor", "against" and "none". The statistics of this expanded dataset are reported in Table 1.
Concerning the targets, the expanded dataset can be divided into two groups: Women's Right (FM, LA) and American Politics (HC, DT, TP). Thus, we constructed 8 cross-target stance detection tasks (  $DT\rightarrow HC$ $HC\rightarrow DT$ $FM\rightarrow LA$ $LA\rightarrow FM$ $TP\rightarrow HC$ $HC\rightarrow TP$ $TP\rightarrow DT$ $DT\rightarrow TP$  ). Here, the left side of the arrow corresponds to the source target and the right side of the arrow denotes the destination target.
Two evaluation metrics are adopted to verify our SEKT model. First, following (Wei and Mao, 2019), we leverage the average F1-score as one evaluation metric (denoted as  $F_{avg}$ ). Second, since the targets in the dataset are imbalanced, we also compute both the micro-averaged F1 (dominating large class) and macro-averaged F1 (dominating small class), and treat their average as another evaluation metric:  $F1_{m} = (F1_{micro} + F1_{macro}) / 2$ .
In the experiments, we use the 300-dimensional word2vec pre-trained on English Google News corpus to initialize the word embeddings. Follow (Augenstein et al., 2016), the node features is pretrained on unlabelled corpora. The hidden size of LSTM is set to 100. Dropout (dropout rate = 0.2) is used to avoid overfitting. The Adam optimizer is applied to train the model, with the mini-batch size of 8 and the learning rate of 0.001.
4.4 Baseline Methods
We evaluate and compare our model with several strong baselines, which are described as follows: