Cross-context News Corpus for Protest Events related Knowledge Base Construction

1 . Introduction . Socio-political event knowledge bases enable comparative social and political studies . The spatio-temporal distribution of these events sheds light on the causes and effects of governmental service management and political discourse resonate in the society . Consequently , construction of these knowledge bases a relatively long history . We focus on the protest events as a type of socio-political events . These events are in the scope of contentious politics ( CP ) and characterized by riots , strikes , and social movements , i.e . the “ repertoire of contention ” [ Giugni , 1998 , Tarrow , 1994 ] . Continuity of news articles and the significant impact of socio-political events direct social and political scientists to exploit news data to create knowledge bases of these events [ Chenoweth and Lewis , 2013 , Weidmann and Rød , 2019 , Raleigh et al. , 2010 ] . The need for collecting socio-political event data , which is protest and conflict data in the context of our work , has been satisfied manually [ Yoruk , 2012 ] , semi-automatically [ Nardulli et al. , 2015 ] , and automatically [ Leetaru and Schrodt , 2013 , Boschee et al. , 2013 , Schrodt et al. , 2014 , Sönmez et al. , 2016 ] . Protest knowledge base creation is either too expensive for manual approaches or has serious limitations when it is performed automatically [ Wang et al. , 2016 , Ettinger et al. , 2017 , Ward et al. , 2013 ] . Moreover , there has not been any common ground across projects that would enable to compare results of these studies [ Lorenzini et al. , 2016 ] . Therefore , as Emerging Welfare ( EMW ) project , we took on the challenge to create the common basis in terms of required high quality data and state-of-the-art tools for fully automating the creation of reliable and valid protest knowledge bases in a way that would serve as a benchmark and enable protest event collection studies benefit from it . This effort yielded a gold standard corpus ( GSC ) that will serve the machine learning and computational linguistics communities to study text processing tool development for constructing knowledge bases of protest events as well . The GSC consists of English news articles from various local and international sources 1 from India , China , and South Africa . Variety of the sources has allowed studying crosscontext robustness and generalizability , therefore addressing style and content change across sources , which are critical requirements of the ML models . The annotations were applied at document , sentence , and token level subsequently . Each instance is annotated by two people and adjudicated by the annotation supervisor for each level . Moreover , a detailed manual and semi-automatic quality check and error analysis were applied on the annotations . The corpus contains more than 10,000 news articles labelled as protest and non-protest ; protest containing articles , which are more than 800 , were further labelled as containing event information or not at the sentence level , and annotated at the token level for detailed event information such as the trigger ( s ) , semantic category , place , time and actor ( s ) of events , and co-reference information . The corpus has enabled the development of a pipeline of machine learning ( ML ) models that extract protest events from an archive of news articles . Moreover , some parts of the corpus has enabled shared-tasks that validated the corpus for cross-context document and sentence classification and token extraction [ Hürriyetoğlu et al. , 2019a , b ] 2 and event sentence coreference identification [ Hürriyetoğlu et al. , 2020 ] . The contributions of this paper are 1 ) a robust methodology for creating a corpus that enables the creation of robust ML-based text processing tools , 2 ) insights from applying this methodology to news archives across contexts , 3 ) a corpus that contains data from multiple contexts and annotations at various levels of granularity , 4 ) results of a pipeline consisting of automated tools that are created using the corpus , and 5 ) first state-of-the-art recall quantification on real random data in contrast to recall measurement in limited settings . We will describe the context of our work in reference to recent related work in Section 2 . Next we will introduce our methodology , the manuals we prepared , and the corpus we have created in Sections 3 , 4 , and 5 , respectively . Section 6 reports the results of the ML tools that were created using the corpus . Finally , Section 7 concludes this report by drawing overall results and point to future steps we are planning to perform in the near future . 2 . Relevant work . Automated socio-political event collection is shaped around language resources ; automated tools that exploit these resources ; the assumptions made to complete the design of an event 1. International sources were filtered based on meta-information to focus on the case countries . 2. https : //emw.ku.edu.tr/clef-protestnews-2019/ , https : //emw.ku.edu.tr/ ? event= challenges-and-opportunities-in-automated-coding-of-contentious-political-events collection system ; and datasets that are output of these systems . The shared resources are rare and the accessible tools are a few . Moreover , the assumptions made in delivering a resulted dataset are not examined in diverse settings . Automated tools for event information collection are designed in terms of pipelines that receive news articles from one or more news sources and yield records of event information . Each tool is inherently limited to the language resources it facilitates and the setting it is validated . Therefore , the quality of the result when an automated tool is used to analyze different sources , i.e . cross-context performance of these pipelines is rarely evaluated . One of the first steps of these pipelines , which is discriminating between relevant and irrelevant documents , has been extensively reported in Croicu and Weidmann [ 2015 ] and Hanna [ 2017 ] . Key term lists and labelled documents aid in determining which news reports contain relevant events . Each study reports their own key term lists and the way they use it . Moreover , labelled documents are released in terms of their URLs or document IDs in some collection without their content [ Makarov et al. , 2015 ] . Accessing the dataset using this limited information is the responsibility of the people who want to use these resources . Our work should be considered as a new version that does not restrict itself with keywords and apply state-of-the-art ML models to tackle the problem of selecting documents that contain event information , which is known as report selection problem in this field . The second main step of this task is extracting relevant tokens . ACE [ Doddington et al. , 2004 ] and TAC-KBP [ Mitamura et al. , 2015 ] are language resources that can be exploitet for token level event extraction . The size and scope of the data for protest events provided by these resources is quite limited . Moreover , the event definition of ACE and TAC-KBP does not capture the CP events . For instance , the ACE definition of the event type DEMONSTRATE , in itself , is too restrictive to be applicable in terms of a broad understanding of CP for two reasons . First , as it seems to limit the scope of this event type to spontaneous ( that is unorganized ) gatherings of people , it excludes certain actions of political and/or grassroots organizations such as political parties and NGOs . Protest actions of such organizations sometimes do not involve mass participation despite aiming at challenging authorities , raising their political agendas or issuing certain demands . Putting up posters , distributing brochures , holding press declarations in public spaces are examples of such protest events . Secondly , the requirement of mass participation in a public area leaves many protest actions such as on-line mass petitions and boycotts , which are not necessarily tied to specific locations where people actually gather , and actions of individuals or small groups such as hunger strikes and self-immolation . Consequently , protest event specific annotation schemas and datasets were proposed [ Huang et al. , 2016 , Sönmez et al. , 2016 , Makarov et al. , 2016 ] . These kind of resources are mainly created using key terms for a single context and it is a challenge to obtain the datasets on the basis of the shared limited information . We follow the detailed protest event information tradition as proposed by Lorenzini et al . [ 2016 ] and Gerner et al . [ 2002 ] , work on data not restricted with key terms , and make our data available to researchers in terms of shared-tasks and sufficient information to enable any researcher to work on it . Recent studies often assume one or more of the following i ) analyzing thousands or millions of sources will compensate for low tool performance , i.e . recall , ii ) a news report contains information about a single event , iii ) analyzing a sentence individually is sufficient to extract relevant information about an event , and iv ) tool performance on a new source will be comparable to the performance on the validation setting [ Weischedel and Boschee , 2018 , Tanev et al. , 2008 ] . Quantifying effect of these assumptions is not a simple task , therefore they are rarely tested . We either provide our observations that shed light on the effect of these assumptions or do not make them . 3 . Methodology . A gold standard corpus of protest events that can enable large scale , multi-source sociopolitical studies should be representative of the content it aims to capture . Moreover , it should enable quantifying the performance of the automation across contexts . Therefore , using available corpora that are already being allowed to be facilitated in research such as English Gigaword [ Parker et al. , 2011 ] is not an option for this setting . In order to satisfy these requirements , our methodology is designed to contain and incorporate multiple sources and countries and apply a detailed annotation methodology without sacrificing quality . We collect online local news articles , and international sources when local sources are not accessible , from India , China , and South Africa . We first download URLs of the freely accessible parts of online news archives such as Indian Express ( IEX ) , New Indian Express ( NIEX ) , The Hindu ( TH ) , Times of India ( ToI ) and South China Morning Post ( SCMP ) , People ’ s Daily ( PD ) .3 Then , for each source , we take a random sample of these URLs and download their content for labelling and annotation.4 The random sampling approach makes the task challenging and closer to reality . Key term list causes some events such as the ones reported without using common terms of protest concept ( the phrase “ classrooms empty ” can be used to report on a strike organized by teachers ) to be missed [ Hürriyetoğlu et al. , 2019a ] . Moreover , lexical variance across contexts can not always be captured using key terms . For instance , the terms “ bandh ” and “ idol immersion ” are event types that are specific to India and not covered by any generalpurpose protest key terms list . Our evaluation of four key term lists , which are reported by Huang et al . [ 2016 ] , Wang et al . [ 2016 ] , Weidmann and Rød [ 2019 ] , and Makarov et al . [ 2016 ] , yielded .68 and .80 precision and recall on our randomly sampled batches at best . Our annotation is based on an annotation manual created by an expert , randomly sampling documents from various sources and periods , and continuously monitoring the annotations to achieve a high inter-annotator agreement ( IAA ) . The same manuals are applied on data collected from different sources and countries , which enables obtaining comparable measures of automatic tool performance across contexts . Finally , in order to eliminate the risk of wrong labelling due to lack of knowledge about a country , a domain expert in politics of the target country instructs the annotators before they start the annotation . The annotation team consists of a supervisor , who is a social scientist and responsible for maintaining the annotation manuals and resolving the disagreements between the annotators , and master students or PhD candidates in social or political sciences , working in pairs . Throughout the annotation , the overlap ratio of annotated articles between pairs is 3 . The period covered by these archives is between around 2000 and 2017 . 4 . Only publicly accessible online information is processed and shared in terms of online URLs . We design our data collection , annotation , and tool development in a way that it would not yield any sensitive that could be used to target individuals by malicious state actors information about individuals . The precautions considered are : using and distributing data via URLs , and express personal characteristics in terms of broad categories such as student or worker . 100 % . The annotation starts by labelling whether a news article mentions a protest . Then , the sentence ( s ) that contain protest information are identified . Finally , protest information such as participants , place , and time is detected in the protest-related sentences at token level . The three levels of annotation , are separate but integrated in the sense that they form a pipeline in which a single document goes through each individual step , and each step is built upon the result of the previous step , which is about completing a batch of documents for a specific level before we start to annotate at next level . The aim here is to maximize time and resource efficiency and performance by utilizing the feedback of each level of annotation for the whole process . The lack of clear boundaries between these levels at the beginning of the annotation project had caused a relatively lower IAA and more time to be spent on the quality check and correction of the dataset . For these same reasons , we add a new step , namely sentence level , to the aforementioned main steps of protest event pipelines . This order of tasks enables error analysis and optimization possibility during annotation and tool development efforts . Each batch for document and sentence levels is corrected in terms of Spotchecks % 10 of the agreements were checked by the annotation supervisor . ML-internal 80 % of the batch is used to create an ML model . Next , the remaining 20 % is predicted using this model . This procedure was repeated until all instances are used at least once in training and once in test data . ML-external The annotated and corrected data from previous batches is used to create an automated classifier that is used to classify the newly annotated batch . The disagreements between the classifiers and the annotations are checked manually for MLinternal and -external . Annotations that were checked based on spotcheck , ML-internal , and -external were found to be annotated wrongly at around 2 % , 50 % , and 10 % respectively . In total , around 10 % of the annotations were corrected using these measures . In the cases we need to improve the performance of a tool on an already covered source , adapt it to a recent period , or start the analysis of a new source , we apply a recall optimized active learning ( AL ) based sampling from a news archive . We first train multiple ML-based classifiers ( three or more ) on the available corpus , then predict a random batch from the new context . To achieve elevated recall scores , we take the logical-or of all classifiers as final prediction , and select positive samples to be annotated . Although the recall decrease from 100 % to 97 % in such a sample , the precision of it increase from around 5 % to around 70 % in comparison to a random sample.5 AL significantly decreases the effort the annotators should spend on annotation [ Settles , 2009 ] . Since annotators observe many more positive samples in this setting , we expect the decrease of recall to be a minor issue at the moment .

Summary: The paper describes a corpus of news articles annotated for protest events. Overall, this is an interesting corpus with a lot of potential for re-use, however, the paper needs some clarifications. A key contribution of the paper is that the initial candidate document retrieval is not based purely on keyword matching, but rather uses a random sampling and active learning based approach to find relevant documents. This is motivated by the incompleteness of dictionaries for protest events. While this might be true, it would have been good to see an evaluation of this assumption with the current data. It is a bit unclear in the paper, but were the K and AL methods run over the same dataset? What are the datasets for which the document relevance precision & recall are reported on page 8?