SlowGuess/NAACL_2022_OCR huggingface dataset:
- Removing textual cues and knowledge walks in KCD result in substantial performance drop, which demonstrates the effectiveness of textual cues and knowledge walks.
In the following, we examine the effect of knowledge walks and textual cues in our approach. We also explore how our approach performs with limited data compared to baseline methods.
4.5 Knowledge Walks Study
We propose knowledge walks, an approach to conduct multi-hop reasoning on knowledge graphs and inject them into textual representations. We study the effect of knowledge walk length and knowledge infusion strategies on our model's performance.
4.5.1 Knowledge Walks Length
Our proposed knowledge walks could be of any length, where shorter walks provide more condensed knowledge and longer walks provide more diverse knowledge. To examine the effect of knowledge walk length, we generate  $5,088^{4}$  knowledge walks of 1 to 10 triples and present model performance in Figure 3. It is illustrated that longer knowledge walks (8 or 9 for SemEval, 7 or 8 for Allsides) perform better than shorter ones, indicating the necessity of multi-hop knowledge reasoning in the task of political perspective detection.
4.5.2 Knowledge Infusion Strategy
We propose a two-step approach to infuse multi-hop knowledge reasoning into textual representations of news articles:
Figure 4: Model performance with different knowledge infusion strategies at two aggregation steps. The horizontal and vertical axis represent the first and second aggregation.  $h_k$  denotes multi-head attention with  $k$  heads, mp and ap stand for max and average pooling.
- First Aggregation: We firstly aggregate different generated knowledge walks based on semantic relevance in Equ. (5) and Equ. (6).
- Second Aggregation: We then use multi-head attention to aggregate all paragraphs and knowledge representations with Equ. (7) and Equ. (8).
To examine the effect of our knowledge infusion strategy, we substitute these two aggregation steps with different multi-head attention settings as well as max and average pooling. Results in Figure 4 demonstrate significant performance difference on the horizontal axis. This suggests that our semantic relevance-based knowledge walks aggregation strategy in Equ. (5) and Equ. (6) successfully filters out irrelevant knowledge reasoning and contributes to model performance. Besides, according to the vertical axis, our adopted multi-head attention in Equ. (7) and Equ. (8) is generally effective and does not rely on specific attention head settings.
4.6 Textual Cues Study
We propose to leverage semantic, syntactic and entity textual cues as paragraph-level labels to leverage implicit indicators in news articles for political perspective detection. To examine the effectiveness of these textual cues, we randomly remove them with probability  $p$  and present model performance in Figure 5. It is illustrated that:
so that there is a knowledge walk beginning with every possible (entity, relation) in the knowledge graph.
Figure 5: Model performance when five different types of textual cues are gradually removed.
- A performance boost is observed between  $0\%$  and  $100\%$  for all five textual cues, suggesting the necessity of modeling implicit textual indicators. Besides, adding only part of textual cues sometimes leads to a decrease in performance, which implies that incomplete cues may be counterproductive and introduce noise.
- Among five different cues, entity and quotation cues contribute more to model performance than others. This suggests some implicit textual cues are more important than others in analyzing the ideological perspectives of news articles.
- The effect of textual cues is larger on the dataset SemEval, which is significantly smaller than All-sides. This suggests that we alleviate the data-hungry problem by introducing diversified textual cues as paragraph-level labels and contribute to model performance.
4.7 Data Efficiency Study
As Li and Goldwasser (2021) point out, supervised data annotations could be difficult and expensive to obtain for the task of political perspective detection in news media. Our proposed knowledge walks and textual cues serve as additional information and might help mitigate this issue. To examine whether we have achieved this end, we train KCD, KGAP (Feng et al., 2021a) as well as various text models with reduced training sets of SemEval and Allsides. Results in Figure 6 demonstrate that
- KCD has better data efficiency and achieves steady performance with smaller training sets. This observation is especially salient on Allsides where the news articles are longer (Li and Goldwasser, 2021), thus more knowledge walks and
Figure 6: Model performance when KCD and various competitive baselines are trained with  $10\%$  to  $100\%$  of the training set on SemEval and Allsides.
textual cues could be extracted and incorporated to alleviate data dependence.
- Both KCD and KGAP leverage external knowledge and are more robust to reduced datasets. Our approach further leverages textual cues and has better data deficiency. This suggests that a solution to limited data could be incorporating information in addition to news content.
- With only  $10\%$  training set, KCD outperforms all baselines by at least  $5.68\%$  and  $9.71\%$  in accuracy on two datasets. This suggests that our approach is simple, effective, and not data-hungry under limited data settings.
In this paper, we propose KCD, a political perspective detection approach that reasons with multi-hop external knowledge and leverages diversified implicit textual indicators. We firstly generate multi-hop knowledge walks, dynamically aggregate them
based on semantic relevance and infuse into news text representations. We then construct a heterogeneous information network to jointly model knowledge-enriched news content and diversified textual cues as paragraph-level labels. Finally, we learn graph representations with relational graph neural networks under different aggregation settings and conduct political perspective detection as graph-level classification. Extensive experiments demonstrate that our approach consistently outperforms state-of-the-art baselines on two benchmark datasets. Further experiments also bear out the necessity of knowledge walks and textual cues in modeling political perspectives in news media.
We would like to thank the reviewers and chairs for their feedback and suggestions. We would also like to thank all LUD Lab members for our collaborative research environment. This study was supported by the National Key Research and Development Program of China (No. 2020AAA0108800), National Nature Science Foundation of China (No. 61872287, No. 62192781, No. 61937001, No. 62050194, No. 62137002), Innovative Research Group of the National Natural Science Foundation of China (61721002), Innovation Research Team of Ministry of Education (IRT_17R86), Project of China Knowledge Center for Engineering Science and Technology, and Project of Chinese academy of engineering "The Online and Offline Mixed Educational Service System for 'The Belt and Road' Training in MOOC China".
Isabelle Augenstein, Tim Roktaschel, Andreas Vlachos, and Kalina Bontcheva. 2016. Stance detection with bidirectional conditional encoding. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 876-885, Austin, Texas. Association for Computational Linguistics.
Ramy Baly, Giovanni Da San Martino, James Glass, and Preslav Nakov. 2020. We can detect your bias: Predicting the political ideology of news articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4982-4991, Online. Association for Computational Linguistics.
Gemma Bel-Enguix, Helena Gómez-Adorno, Alejandro Pimentel, Sergio-Luis Ojeda-Trueba, and Brian
Aguilar-Vizuet. 2021. Negation detection on mexican spanish tweets: The t-mexneg corpus. Applied Sciences, 11(9).
Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. "O'Reilly Media, Inc."
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: A collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference on Management of Data, SIGMOD '08, page 1247-1250, New York, NY, USA. Association for Computing Machinery.
Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi-relational data. In Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc.
David Chang, Ivana Balažević, Carl Allen, Daniel Chawla, Cynthia Brandt, and Andrew Taylor. 2020. Benchmark and best practices for biomedical knowledge graph embeddings. In Proceedings of the 19th SIGBioMed Workshop on Biomedical Language Processing, pages 167-176, Online. Association for Computational Linguistics.
Kareem Darwish, Peter Stefanov, Michael Aupetit, and Preslav Nakov. 2020. Unsupervised user stance detection on twitter. In Proceedings of the International AAAI Conference on Web and Social Media, volume 14, pages 141-152.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jiachen Du, Ruifeng Xu, Yulan He, and Lin Gui. 2017. Stance classification with target-specific neural attention. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 3988-3994.
Subhabrata Dutta, Samiya Caur, Soumen Chakrabarti, and Tanmoy Chakraborty. 2022. Semi-supervised stance detection of tweets via distant network supervision. arXiv preprint arXiv:2201.00614.
William Falcon and The PyTorch Lightning team. 2019. PyTorch Lightning.
Christiane Fellbaum. 2010. Wordnet. In Theory and applications of ontology: computer applications, pages 231-243. Springer.
Shangbin Feng, Zilong Chen, Wenqian Zhang, Qingyao Li, Qinghua Zheng, Xiaojun Chang, and Minnan Luo. 2021a. Knowledge graph augmented political perspective detection in news media. arXiv preprint arXiv:2108.03861.
Shangbin Feng, Zhaoxuan Tan, Zilong Chen, Peisheng Yu, Qinghua Zheng, Xiaojun Chang, and Minnan Luo. 2021b. Legislator representation learning with social context and expert knowledge. arXiv preprint arXiv:2108.03881.
Paolo Ferragina and Ugo Scaiella. 2012. Fast and accurate annotation of short texts with wikipedia pages. IEEE Software, 29(1):70-75.
Matthias Fey and Jan Eric Lenssen. 2019. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv:1903.02428.
Maarten Grootendorst. 2020. Bertopic: Leveraging bert and c-tfidf to create easily interpretable topics.
Xu Han, Shulin Cao, Xin Lv, Yankai Lin, Zhiyuan Liu, Maosong Sun, and Juanzi Li. 2018. OpenKE: An open toolkit for knowledge embedding. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 139-144, Brussels, Belgium. Association for Computational Linguistics.
Benjamin D Horne, Sara Khedr, and Sibel Adali. 2018. Sampling the news producers: A large news and feature data set for the study of the complex media landscape. In Twelfth International AAAI Conference on Web and Social Media.
Linmei Hu, Tianchi Yang, Luhao Zhang, Wanjun Zhong, Duyu Tang, Chuan Shi, Nan Duan, and Ming Zhou. 2021. Compare to the knowledge: Graph neural fake news detection with external knowledge. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 754-763, Online. Association for Computational Linguistics.
Long Jiang, Mo Yu, Ming Zhou, Xiaohua Liu, and Tiejun Zhao. 2011. Target-dependent Twitter sentiment classification. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 151-160, Portland, Oregon, USA. Association for Computational Linguistics.
Ye Jiang, Johann Petrak, Xingyi Song, Kalina Bontcheva, and Diana Maynard. 2019. Team bertha von suttner at SemEval-2019 task 4: Hyperpartisan news detection using ELMo sentence representation convolutional network. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 840-844, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Johannes Kiesel, Maria Mestre, Rishabh Shukla, Emmanuel Vincent, Payam Adineh, David Corney, Benno Stein, and Martin Potthast. 2019. SemEval 2019 task 4: Hyperpartisan news detection. In Proceedings of the 13th International Workshop on Semantic Evaluation, pages 829-839, Minneapolis, Minnesota, USA. Association for Computational Linguistics.
Chang Li and Dan Goldwasser. 2019. Encoding social information with graph convolutional networks for Political perspective detection in news media. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2594-2604, Florence, Italy. Association for Computational Linguistics.
Chang Li and Dan Goldwasser. 2021. Using social and linguistic information to adapt pretrained representations for political perspective identification. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 4569-4579, Online. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Yinquan Lu, Haonan Lu, Guirong Fu, and Qun Liu. 2021. Kelm: Knowledge enhanced pretrained language representations with message passing on hierarchical relational graphs. arXiv preprint arXiv:2109.04223.
Walid Magdy, Kareem Darwish, Norah Abokhodair, Afshin Rahimi, and Timothy Baldwin. 2016. #isisisnotislam or #deportall muslims? predicting unspoken views. In Proceedings of the 8th ACM Conference on Web Science, WebSci '16, page 95-106, New York, NY, USA. Association for Computing Machinery.
Olga Majewska, Ivan Vulic, Goran Glavaš, Edoardo Maria Ponti, and Anna Korhonen. 2021. Verb knowledge injection for multilingual event processing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 6952-6969, Online. Association for Computational Linguistics.
Zaiqiao Meng, Fangyu Liu, Thomas Clark, Ehsan Shareghi, and Nigel Collier. 2021. Mixture-of-partitions: Infusing large biomedical knowledge graphs into BERT. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4672-4681, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Tomás Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In 1st International Conference
on Learning Representations, ICLR 2013, Scottsdale, Arizona, USA, May 2-4, 2013, Workshop Track Proceedings.
Malte Ostendorff, Peter Bourgonje, Maria Berger, Julian Moreno-Schneider, Georg Rehm, and Bela Gipp. 2019. Enriching bert with knowledge graph embeddings for document classification. arXiv preprint arXiv:1909.08402.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Thomas Pellissier Tanon, Gerhard Weikum, and Fabian Suchanek. 2020. Yago 4: A reason-able knowledge base. In *The Semantic Web*, pages 583–596, Cham. Springer International Publishing.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, page 4444-4451. AAAI Press.
Peter Stefanov, Kareem Darwish, Atanas Atanasov, and Preslav Nakov. 2020. Predicting the topical stance and political leaning of media using tweets. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 527-537, Online. Association for Computational Linguistics.
Bo Wang, Maria Liakata, Arkaitz Zubiaga, and Rob Procter. 2017. TDParse: Multi-target-specific sentiment recognition on Twitter. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 483-493, Valencia, Spain. Association for Computational Linguistics.
Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021. Kepler: A unified model for knowledge embedding and pre-trained language representation. Trans. Assoc. Comput. Linguistics, 9:176-194.
Xiaokai Wei, Shen Wang, Dejiao Zhang, Parminder Bhatia, and Andrew Arnold. 2021. Knowledge enhanced pretrained language models: A comprehensive survey. arXiv preprint arXiv:2110.08455.
Chang Xu, Cecile Paris, Surya Nepal, and Ross Sparks. 2018. Cross-target stance classification with self-attention networks. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 778-783, Melbourne, Australia. Association for Computational Linguistics.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1480-1489, San Diego, California. Association for Computational Linguistics.
Our proposed model has two minor limitations:
- We propose to model news articles with heterogeneous information networks. This graph-based approach might not fit well with shorter news articles with only a few paragraphs. This issue might be addressed by using sentence nodes instead of paragraph nodes for shorter articles.
- For very large knowledge graphs with many different types of relations, it might be hard for domain experts to help set  $p(r)$  for every knowledge graph relation. This issue might be addressed by only setting a larger  $p(r)$  for several important  $rs$  according to domain expert.
Dataset	# Articles	# Class	Class Distribution
SemEval	645	2	407 / 238
Allsides	10,385	3	4,164 / 3,931 / 2,290
Table 3: Details of two datasets SemEval and Allsides.
Datasets	Setting	Acc	MaF
SemEval	GA	88.62 ± 0.32	83.86 ± 0.27
CA	89.30 ± 0.24	84.85 ± 0.30
PA	89.90 ± 0.68	86.11 ± 1.18
AllSides	GA	84.88 ± 2.90	84.31 ± 2.96
CA	79.85 ± 2.58	79.41 ± 2.56
PA	87.17 ± 0.24	86.72 ± 0.35
Table 4: Average performance and standard deviation of three different aggregation strategies for five runs.
We used the same datasets as in previous works (Li and Goldwasser, 2019, 2021; Feng et al., 2021a), namely SemEval (Kiesel et al., 2019) and Allsides (Li and Goldwasser, 2019). We follow the same 10-fold setting for SemEval and 3-fold setting for Allsides (Li and Goldwasser, 2021). We use the exact same folds so that the results are directly comparable. A minor difference would be that we have to discard a few news articles on Allsides since theirurls have expired and we could not retrieve their original news article. We report the statistical information of SemEval and Allsides in Table 3.
C Computation Details
Our proposed approach has a total of 7.8M learnable parameters. It takes approximately 0.7 and 1.6 GPU hours to train our approach on two datasets respectively. We train our model on one Titan X GPU with 12GB memory.
C.2 Experiment Runs
We run our approach with three different aggregation strategies five times and report the average accuracy and macro F1-score with standard deviation in Table 4. For experiments in Section 4.5, 4.6 and 4.7, we do not have enough computational resources to run five times, thus we report the performance of a single run.
D Scientific Artifact Usage
We provide additional details about used scientific artifacts and specifically how we used them.
- NLTK (Bird et al., 2009): We use NLTK to extract the tense of news articles. Specifically, we first use NLTK POS-tagsger to process new paragraphs and attach speech tag to each word. Then we align verb tags with NLTK tagset to identify the tense of paragraphs.
- BertTopic (Grootendorst, 2020): We use BertTopic to mine the topics of news corpus. Specifically, we use BertTopic topic model to learn dataset-specific topic models. For SemEval we obtained 197 topics and for Allsides we obtained 1225 topics. Next, we predict topics for each news paragraph. Each topic consists of ten topic words with scores and we select the top five to serve as the news paragraph's topic.
- Huggingface Transformers (Wolf et al., 2020): We use the pipeline module for sentiment analysis. Specifically, we use the sentiment analysis API in the text classification pipeline to generate a sentiment label and score for news paragraphs. We then use the sentiment label as the sentiment cues for news paragraphs.
- TagMe (Ferragina and Scaiella, 2012): We use TagMe to align news articles with entities in the knowledge graph. Specifically, we use TagMe to annotate named entities in news paragraphs and save the entities with a score higher than 0.1 for further alignment. We then calculate the similarity score between TagMe annotated entities and political knowledge graph entities. We recognize the entities with a score higher than 0.9 as entity cues in our constructed HIN.
- Political knowledge graph (Feng et al., 2021a): We use the political knowledge graph collected in Feng et al. (2021a) for external knowledge in political perspective detection.
- OpenKE (Han et al., 2018): We use OpenKE to train TransE (Bordes et al., 2013) knowledge graph embeddings for the political knowledge graph. Specifically, we set the TransE hidden size to 768 and train the model with other default hyperparameters in OpenKE.