SlowGuess/NAACL_2022_OCR huggingface dataset:
Wenqian Zhang\* Shangbin Feng\* Zilong Chen\* Zhenyu LeiJundong Li Minnan Luo\*
School of Computer Science and Technology, Xi'an Jiaotong University School of Science, Xi'an Jiaotong University, University of Virginia
{2194510944, wind_binteng, luoyangczl, Fischer}@stu.xjtu.edu.cn
jundong@virginia.edu minnluo@xjtu.edu.cn
Political perspective detection has become an increasingly important task that can help combat echo chambers and political polarization. Previous approaches generally focus on leveraging textual content to identify stances, while they fail to reason with background knowledge or leverage the rich semantic and syntactic textual labels in news articles. In light of these limitations, we propose KCD, a political perspective detection approach to enable multi-hop knowledge reasoning and incorporate textual cues as paragraph-level labels. Specifically, we firstly generate random walks on external knowledge graphs and infuse them with news text representations. We then construct a heterogeneous information network to jointly model news content as well as semantic, syntactic and entity cues in news articles. Finally, we adopt relational graph neural networks for graph-level representation learning and conduct political perspective detection. Extensive experiments demonstrate that our approach outperforms state-of-the-art methods on two benchmark datasets. We further examine the effect of knowledge walks and textual cues and how they contribute to our approach's data efficiency.
Political perspective detection aims to identify ideological stances of textual data such as social media posts and news articles. Previous approaches generally leverage the textual content of news articles with various text modeling techniques to identify political stances. Those works (Jiang et al., 2019; Li and Goldwasser, 2019, 2021; Feng et al., 2021a) leveraged diversified text models, such as recurrent neural networks (Yang et al., 2016), word embedding techniques (Pennington et al., 2014; Peters et al., 2018), convolutional neural networks (Jiang et al., 2019), and pre-trained language models (Devlin et al., 2019; Liu et al., 2019), to encode
Figure 1: Multi-hop knowledge reasoning and implicit textual indicators that facilitate perspective detection.
news paragraphs and classify them into different perspective labels. Later approaches incorporate information sources beyond text to facilitate argument mining and boost task performance. News discussion on social networks (Li and Goldwasser, 2019), social and linguistic information about news articles (Li and Goldwasser, 2021), media sources and information (Baly et al., 2020) as well as external knowledge from knowledge graphs (Feng et al., 2021a) are introduced in the task of political perspective detection and achieve better performance.
Although these methods attempted to leverage more than news content, they fail to present a framework capable of reasoning with background knowledge and leveraging implicit semantic and syntactic indicators such as sentiment and tense of news articles. For example, Figure 1 presents a typical news
Human Language Technologies, pages 4129 - 4140
article from Daily Kos<sup>1</sup>. This article discusses remarks from the Trump campaign team about Wik-ileaks and its effect on Hillary Clinton's bid for president. Individuals often rely on the multi-hop reasoning that Clinton and Trump are from opposite political parties and run against each other to inform their perspective analysis process. Besides, the negative sentiment expressed in satiric tones and the quotation of Trump campaign staff also give away the author's denial and left-leaning perspective. That being said, knowledge reasoning and implicit textual indicators are essential in the news bias detection process.
In light of these limitations, we propose a political perspective detection framework KCD (Knowledge Walks and Textual Cues Enhanced Political Perspective Detection). Specifically, KCD generates multi-hop knowledge walks, aggregates them based on semantic relevance and incorporates them in textual representations with multi-head attention. KCD then constructs a heterogeneous information network to jointly model knowledge-enriched news content and diversified textual cues as paragraph-level labels. Finally, KCD learns graph representations with relational graph neural networks and conduct perspective detection with different aggregation strategies. Our main contributions are summarized as follows:
- We propose knowledge walks, a strategy to incorporate multi-hop knowledge reasoning in textual representations for knowledge-aware political perspective detection.
- We propose to construct a heterogeneous information network to represent news articles, which jointly models knowledge-enriched news content and implicit textual cues in news articles.
- Extensive experiments demonstrate that our approach consistently outperforms state-of-the-art methods on two widely adopted benchmarks. Further analysis bears out the necessity of knowledge walks and textual cues in our approach.
2.1 Political Perspective Detection
Political perspective detection aims to identify the ideological stances of news articles, which is widely studied to help strengthen the online information landscape (Li and Goldwasser, 2019) and
mitigate ideological echo chambers (Li and Goldwasser, 2021; Feng et al., 2021a). Early approaches leverage text analysis techniques for bias detection, such as sentiment analysis (Jiang et al., 2011; Wang et al., 2017), bias feature extraction (Horne et al., 2018), word embeddings (Jiang et al., 2019; Li and Goldwasser, 2019), and different neural network architectures (Augenstein et al., 2016; Du et al., 2017; Xu et al., 2018; Yang et al., 2016; Jiang et al., 2019; Feng et al., 2021b; Li and Goldwasser, 2021; Feng et al., 2021a). In addition to textual content of news articles, social media users also become the focus of perspective detection research (Bel-Enguix et al., 2021). User interactions (Magdy et al., 2016), user clustering (Darwish et al., 2020), and label propagation (Stefanov et al., 2020) are leveraged to identify the ideological preferences on social media. Fusing both news text and social network analysis directions, Li and Goldwasser (2019) propose to enrich news text with the content and structure of social media discussions about these news articles. Recent state-of-the-art approaches chart a new path by incorporating social and political external knowledge into stance detection. Baly et al. (2020) propose adversarial media adaptation and leverage source background knowledge for political perspective detection. Li and Goldwasser (2021) combine language encoders with pre-training tasks of social and linguistic information. Feng et al. (2021a) propose to construct and leverage political knowledge graphs as domain-specific external knowledge. In this paper, we build on these works to examine and explore the effect of multi-hop knowledge reasoning and diversified textual cues in the task of political perspective detection.
2.2 Knowledge Graph in NLP
Knowledge graphs (KGs) are effective representations of real-world entities, relations, and knowledge. Generic (Fellbaum, 2010; Pellissier Tanon et al., 2020; Bollacker et al., 2008; Speer et al., 2017) and domain-specific KGs (Feng et al., 2021a; Chang et al., 2020) are widely adopted in NLP tasks as external knowledge sources. These approaches could mainly be categorized into feature extraction, language model and graph-based methods. For feature extraction approaches, KG embedding technique TransE (Bordes et al., 2013) is leveraged to learn features for knowledge injection (Ostendorff et al., 2019; Hu et al., 2021). For language model approaches, the adapter architecture is leveraged
<sup>1</sup>https://www.dailykos.com/
Figure 2: Overview of our proposed framework KCD.
to fine-tune on KG-related tasks (Majewska et al., 2021; Meng et al., 2021; Wei et al., 2021). In addition, Wang et al. (2021) propose a unified model to combine knowledge embedding with language representation pre-training. For graph-based approaches, KG entities and relations are injected into graphs and heterogeneous information networks (Hu et al., 2021; Feng et al., 2021a; Lu et al., 2021). Graph neural networks are then adopted to learn knowledge-aware text representations. In this paper, we propose knowledge walk, a novel strategy to infuse multi-hop knowledge reasoning into language representations and apply them in political perspective detection.
Figure 2 presents an overview of our proposed political perspective detection framework KCD (Knowledge Walks and Textual Cues Enhanced Political Perspective Detection). We firstly generate knowledge walks on the external knowledge graph. These knowledge walks are then selected based on semantic relevance and injected into textual representations with multi-head attention. We then construct a heterogeneous information network to jointly model knowledge-enriched news content and diversified textual cues as paragraph-level labels and supernodes. Finally, we adopt relational graph neural networks and different aggregation strategies to learn graph-level representation and conduct political perspective detection.
3.1 Knowledge Walks and Infusion
We firstly propose the novel strategy of knowledge walks and combine them with textual rep
resentations to enable multi-hop knowledge reasoning. We partition an  $n$ -paragraph news document into different paragraphs and denote them as  $\mathcal{S} = \{s_1, \dots, s_n\}$ . We encode each paragraph by averaging the embeddings of words from pre-trained RoBERTa (Liu et al., 2019):
$$
v _ {i} ^ {s} = \operatorname {R o B E R T a} \left(s _ {i}\right), \quad 1 \leq i \leq n \tag {1}
$$
We use a political knowledge graph $^2$  as external knowledge for perspective detection. Let the  $i$ -th triple in the knowledge graph be  $(e_{ih}, r_i, e_{it})$ , where  $e_{ih}$  and  $e_{it}$  denote the head and tail entity and  $r_i$  represents the relation of the  $i$ -th triple.
3.1.1 Knowledge Walk Generation
We firstly use TagMe (Ferragina and Scaiella, 2012) to identify mentioned KG entities in each paragraph  $s_i$ . For each mentioned entity, we use it as the starting point  $e_{(0)}$  in a  $K$ -hop knowledge walk:
$$
k w _ {i} = \left\{e _ {(0)}, r _ {0, 1}, e _ {(1)}, \dots , r _ {K - 1, K}, e _ {(K)} \right\} \tag {2}
$$
where  $e_{(i - 1)}$  and  $r_{i - 1,i}$  denote the  $i$ -th triple's head entity and relation. Specifically, a knowledge walk is generated by adopting biased random walk of length  $K$  starting from  $e_{(0)}$ . The conditional probability of arriving at  $e_{(i)}$  from  $e_{(i - 1)}$  through  $r_{i - 1,i}$  is formulated as
$$
P \left(e _ {(i)} \mid e _ {(i - 1)}, r _ {i - 1, i}\right) = \frac {\exp \left(p \left(r _ {i - 1 , i}\right)\right)}{\sum_ {j = 1} ^ {\left| N _ {r} (i - 1) \right|} \exp \left(p \left(r _ {j}\right)\right)} \tag {3}
$$
$^{2}$ https://github.com/BunsenFeng/news_stance_detector
where  $N_{r}(i - 1)$  denotes the neighboring relations of  $e_{(i - 1)}, p(r)$  is the importance score of KG relation  $r$ , which could be tuned by domain experts for human-in-the-loop knowledge walk generation. In this way, we generate multiple knowledge walks for each paragraph based on its mentioned entities, which models the multi-hop reasoning process with external knowledge.
3.1.2 Semantic-Guided Selection
After obtaining multiple knowledge walks for a single news paragraph, we propose a selection and aggregation process guided by textual content to differentiate essential knowledge walks from the irrelevant ones. We firstly transform each knowledge walk  $kw_{i}$  into a sentence  $t_i$  by concatenating the textual description of entities and relations. We then encode the knowledge walk sentence  $t_i$  with pre-trained RoBERTa (Liu et al., 2019):
$$
v _ {i} ^ {k} = R o B E R T a \left(t _ {i}\right) \tag {4}
$$
Suppose a total of  $m$  knowledge walks  $\{kw_{i,j}\}_{j = 1}^{m}$  are generated for paragraph  $s_i$ , we then aggregate their knowledge walk sentence embeddings  $\{v_{i,j}^k\}_{j = 1}^m$  as follows:
$$
v _ {i} ^ {p} = \sum_ {j = 1} ^ {m} \frac {\exp \left(\alpha \cdot v _ {i , j} ^ {k}\right)}{\sum_ {q = 1} ^ {m} \exp \left(\alpha \cdot v _ {i , q} ^ {k}\right)} v _ {i, j} ^ {k} \tag {5}
$$
where  $\alpha$  denotes the learnable attention vector guided by paragraph semantics:
$$
\alpha = \phi \left(W _ {a} v _ {i} ^ {s} + b _ {a}\right) \tag {6}
$$
where  $W_{a}$  and  $b_{a}$  are learnable parameters of the attention module and we use Leaky-ReLU for  $\phi$ .  $v_{i}^{s}$  is the sentence embedding from equation 1. In this way, we aggregate  $m$  knowledge walks based on semantic relevance to the paragraph to filter and retain important knowledge reasoning paths.
3.1.3 Knowledge Infusion
After representing multi-hop knowledge reasoning for paragraph  $s_i$  with  $v_i^p$ , we conduct document-wise multi-head self-attention to infuse knowledge walks into textual representations  $v_i^s$ . We concatenate knowledge walk and text representations:
$$
T = \operatorname {c o n c a t} \left(\left[ v _ {1} ^ {s}, v _ {1} ^ {p}, \dots , v _ {n} ^ {s}, v _ {n} ^ {p} \right]\right) \tag {7}
$$
where  $T$  is the input for multi-head self-attention:
$$
\tilde {T} = \text {M u l t i H e a d} (Q, K, V) \tag {8}
$$
where  $Q = K = V = T$  and the output  $\tilde{T} =$  concat([v1,s,v1p,...,v,,v]). In this way, we obtain language representations of news paragraphs  $\{\tilde{v_i^s}\}_{i = 1}^n$  , which jointly models textual content and related multi-hop knowledge reasoning paths.
3.2 Textual Cues and Graph Construction
We construct a heterogeneous information network (HIN) as in Figure 2 "Graph Construction" to jointly represent knowledge-enriched news content and diversified textual cues in news articles. Specifically, we use paragraph nodes to represent the news content and connect them with different paragraph-level labels with heterogeneous edges. Firstly, for paragraph nodes:
$\mathcal{V}1$  and  $\mathcal{R}1$ : Paragraph Nodes We use one node in  $\mathcal{V}1$  to represent each paragraph in the news article to partition the entire document and allow fine-grained analysis. We adopt the knowledge-enriched representations  $\{\tilde{v_i^s}\}_{i = 1}^n$  in Section 3.1 as initial node features for  $\mathcal{V}1$ . We then use relation  $\mathcal{R}1$  to connect adjacent paragraphs to preserve the original flow of the news article.
3.2.1 Semantic Cues
We further analyze the topic and sentiment of news paragraphs, extract paragraph-level labels and inject them into our news HIN structure.
$\nu 2$  and  $\mathcal{R}2$ : Topic Cues The topics and frequent topic switching in news articles often give away the stance and argument of authors. We train LDA to extract the topics in each political perspective detection corpus and use one node to represent each topic. We then encode the topic text with pretrained RoBERTa as node attributes. We then use  $\mathcal{R}2$  to connect each paragraph node in  $\nu 1$  with its affiliated topic node in  $\nu 2$  with the help of Bert-Topic (Grootendorst, 2020).
$\mathcal{V}3$  and  $\mathcal{R}3$ : Sentiment Cues The sentiment of news articles signal the authors' approval or denial, which helps identify their stances towards individuals and issues. We use two nodes to represent positive and negative sentiment and we make their node attributes learnable. We then conduct sentiment analysis (Wolf et al., 2020) to identify paragraph sentiment and use  $\mathcal{R}3$  to connect  $\mathcal{V}1$  with their corresponding sentiment nodes in  $\mathcal{V}3$ .
3.2.2 Syntactic Cues
Apart from semantic cues, syntactic information in news articles also contribute to the perspective analysis process (Dutta et al., 2022). In light of
this, we analyze the tense of news paragraphs and whether it contains direct quotation and use them as paragraph-level labels in our constructed HIN.
$\mathcal{V}4$  and  $\mathcal{R}4$ : Tense Cues The tense of news paragraphs helps separate facts from opinions. For example, simple past tense often indicates factual statements while simple future tense suggests opinions and projections that might not be factual. We use 17 nodes in  $\mathcal{V}4$  to represent 17 possible tenses in our constructed news HIN. We use NLTK (Bird et al., 2009) to extract paragraph tenses and use  $\mathcal{R}4$  to connect paragraph nodes in  $\mathcal{V}1$  with  $\mathcal{V}4$ .
$\nu 5$  and  $\mathcal{R}5$ : Quotation Cues It is common for authors to directly quote others' words in news articles, which helps to identify the basis of the author's argument. We use two nodes to differentiate between whether a news paragraph quotes someone or not. Specifically, we identify quotation marks in news paragraphs and use  $\mathcal{R}6$  to connect  $\nu 1$  with  $\nu 6$  based on whether direct quotation is detected.
3.2.3 Entity Cues
$\mathcal{V}6$  and  $\mathcal{R}6$ : Entity Cues We follow previous works (Feng et al., 2021a; Hu et al., 2021) to use one node to represent each entity in the external knowledge graph. We adopt TransE (Bordes et al., 2013) to learn knowledge graph embeddings and use them as initial node features for  $\mathcal{V}6$ . We then adopt entity linking tool TagMe (Ferragina and Scaiella, 2012) to align news paragraphs with their mentioned entities and use  $\mathcal{R}6$  to connect  $\mathcal{V}1$  with  $\mathcal{V}6$  correspondingly.
In this way, we obtain a heterogeneous information network for news articles that jointly models knowledge-enriched news content and diversified textual cues in news articles. Our approach could be similarly extended to other textual cues and paragraph-level labels that would be helpful in political perspective detection and related tasks.
3.3 Learning and Optimization
Upon obtaining the news HINs, we adopt relational graph neural networks for representation learning and conduct political perspective detection as graph-level classification. Specifically, we follow Feng et al. (2021a) and use gated R-GCN to ensure a fair comparison and highlight the effectiveness of knowledge walks and textual cues. After  $L$  layers of gated R-GCN, we denote the learned node representations as  $\overline{v}$  and obtain graph-level representation  $v_{g}$  with three different aggregation strategies: Paragraph Average (PA), Cue Average
Hyperparameter	Value
GNN input size	768
GNN hidden size	512
GNN layer L	2
# epoch	150
batch size	16
dropout	0.6
# knowledge walk	30,114
p(r) in Equ. (3)	constant c
# head in Equ. (8)	SE: 8, AS: 32
λ in Equ. (11)	1e-4
learning rate	1e-3
lr_schedule_patience	20
lr_schedule_step	0.1
# early stop epoch	40
Optimizer	Adam
Table 1: Hyperparameter settings of KCD. SE and AS denote the datasets SemEval and Allsides.
(CA) and Global Average (GA):
$$
v _ {g} = \left\{ \begin{array}{l l} \frac {1}{| \mathcal {V} 1 |} \sum_ {v \in \mathcal {V} 1} \bar {v} & \text {i f P a r a g r a p h A v e r a g e ;} \\ \frac {1}{| \mathcal {V} - \mathcal {V} 1 |} \sum_ {v \notin \mathcal {V} 1} \bar {v} & \text {i f C u e A v e r a g e ;} \\ \frac {1}{| \mathcal {V} |} \sum_ {v \in \mathcal {V}} \bar {v} & \text {i f G l o b a l A v e r a g e .} \end{array} \right. \tag {9}
$$
where  $\mathcal{V} = \bigcup_{i=1}^{6} \mathcal{V}_i$  represents the set of all nodes in our HIN. We then transform the graph-level representation  $v_g$  with a softmax layer and classify news articles into perspective labels:
$$
\hat {y} = \operatorname {s o f t m a x} \left(W _ {o} \cdot v _ {g} + b _ {o}\right) \tag {10}
$$
where  $W_{o}$  and  $b_{o}$  are learnable parameters and  $\hat{y}$  is our model's prediction. We optimize the end-to-end process with cross entropy loss and  $L_{2}$  regularization.
We make use of two real-world political perspective detection datasets SemEval (Kiesel et al., 2019) and Allsides (Li and Goldwasser, 2019), which are widely adopted in various previous works (Li and Goldwasser, 2019, 2021; Feng et al., 2021a). We follow the same evaluation settings as in previous works so that our results are directly comparable. Section B in the appendix provides more dataset details to facilitate reproduction.
We compare KCD with the following competitive baselines and state-of-the-art methods:
- CNN (Jiang et al., 2019) is the first-place solution in the SemEval 2019 Task 4 contest (Kiesel et al., 2019). It combines convolutional neural networks with Glove (Jiang et al., 2019) and ELMo (Peters et al., 2018) for political perspective detection on the SemEval dataset.
- HLSTM (Yang et al., 2016) is short for hierarchical long short-term memory networks. Li and Goldwasser (2019) uses HLSTMs and different word embeddings for news bias detection.
- HLSTM_Embed and HLSTM_Output (Li and Goldwasser, 2021) leverage entity information with masked entity models in addition to news content for political perspective detection.
- Word2Vec (Mikolov et al., 2013), GloVe (Pennington et al., 2014), ELMo (Peters et al., 2018), pre-trained BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) are leveraged by Feng et al. (2021a) as textual features and political perspective detection is further conducted with two fully connected layers.
- MAN (Li and Goldwasser, 2021) incorporates social and linguistic information with pre-training tasks and conducts fine-tuning on the task of political perspective detection.
- KGAP (Feng et al., 2021a), short for Knowledge Graph Augmented Political perspective detection, leverages knowledge graphs and graph neural networks for a knowledge-aware approach. We compare our gated R-GCN based approach with KGAP's gated R-GCN setting.
We implement our KCD framework with pytorch (Paszke et al., 2019), pytorch lightning (Falcon and The PyTorch Lightning team, 2019), pytorch geometric (Fey and Lenssen, 2019), and the transformers library (Wolf et al., 2020). We present our hyperparameter settings in Table 1 to facilitate reproduction. We adhere to these settings throughout all experiments in the paper unless stated otherwise. Our implementation is trained on a Titan X GPU with 12GB memory. We make our code and data publicly available<sup>3</sup>.
Method	Setting	SemEval	AllSides
Acc	MaF	Acc	MaF
CNN	GloVe	79.63	N/A	N/A	N/A
ELMo	84.04	N/A	N/A	N/A
HLSTM	GloVe	81.58	N/A	N/A	N/A
ELMo	83.28	N/A	N/A	N/A
Embed	81.71	N/A	76.45	74.95
Output	81.25	N/A	76.66	75.39
Text Model	Word2Vec	70.27	39.37	48.58	34.33
GloVe	80.71	63.64	71.01	69.81
ELMo	86.78	80.46	81.97	81.15
BERT	86.92	80.71	82.46	81.77
RoBERTa	87.08	81.34	85.35	84.85
MAN	GloVe	81.58	79.29	78.29	76.96
ELMo	84.66	83.09	81.41	80.44
Ensemble	86.21	84.33	85.00	84.25
KGAP	GRGCN	89.56	84.94	86.02	85.52
KCD	GA	88.52	84.13	86.02	85.53
CA	89.77	85.26	81.28	80.39
PA	90.87	87.87	87.38	87.14
KCD (PA)	- w/o TC	88.22	83.53	86.08	85.58
- w/o KW	87.29	81.77	85.51	85.00
Table 2: Political perspective detection performance on two benchmark datasets. Acc and MaF denote accuracy and macro-averaged F1-score. N/A indicates that the result is not reported in previous works. TC and KW indicate textual cues and knowledge walks respectively.
We present model performance on two benchmark datasets in Table 2, which demonstrates that
- KCD, especially with the PA aggregation strategy, consistently outperforms state-of-the-art methods on both benchmark datasets.
- KGAP and KCD, which incorporate knowledge graphs, outperform other baselines. This indicates that external knowledge is essential in providing background information and political context to analyze ideological perspectives. KCD without textual cues performs better than baseline methods except KGAP and performs close to KGAP. These suggests that KGAP's method of infusing knowledge as HIN nodes and our method of infusing knowledge as knowledge walks are both effective.
- PA outperforms CA and GA on both datasets, which suggests that the aggregation strategy is important since subsidiary nodes like textual cues may result in noise. As a result, we should focus on paragraph nodes in our heterogeneous information networks.
<sup>3</sup>https://github.com/Wenqian-Zhang/KCD
Figure 3: Our approach's performance when the maximum length of knowledge walk generation is specified from 1 to 10 knowledge graph triples.