Question: Are there any difficulties or limitations in the process of obtaining historical content from aggregators?
Evidence: One of the defining traits of previous event-data projects is the method through which they were generated. The original datasets such as WEIS and COPDAB were created by human coders who read news stories and coded events. Future datasets such as KEDS and Phil Schrodt's Levant dataset were created using automated coding software, such as KEDS or TABARI, and news stories download from content aggregators such as Lexis Nexis or Factiva. Both pieces of coding software made use of a technique referred to as shallow parsing BIBREF13 . Shallow parsing is best understood in contrast to a deep parsing method. In deep parsing, the entire syntactic structure of a sentence is used and understood. This syntactic structure includes things such as prepositional phrases, direct and indirect objects, and other grammatical structures. A shallow parse, however, focuses solely on, as the name implies, shallow aspects such as the part of speech of the words within the sentence.
The second major dimension that differentiates event datasets is how news content was acquired. For WEIS and COPDAB this was as simple as subscribing to the New York Times and coding from there. Later datasets, such as those created in conjunction with the Kansas Event Data Project, obtained historical content from aggregators, as mentioned above. This difficulty of this process changed at various points in time, with something like full automation possible at some points while human downloading of stories was required at others. There are often gaps in this historical content since the content aggregators catalog of different news services changes at various points and is often fairly limited. Updating datasets based on this type of content was also fairly labor intensive since new content had to be downloaded, cleaned, and run for every update. While orders of magnitude faster than human coding, this remained an involved process.
Taken together, these two aspects of event data generation, shallow parsing and content acquisition, form the basis for where the next generation of political event data can improve upon previous efforts. In short, a shift to deep parsing based on relatively recent advances in open-source natural language processing software, combined with realtime acquisition of news content and aggressive strategies for acquiring historical material, provide the motivation for the next generation of political event data. The following section provides greater detail regarding the implementation of these new features.
Later datasets, such as those created in conjunction with the Kansas Event Data Project, obtained historical content from aggregators, as mentioned above. This difficulty of this process changed at various points in time, with something like full automation possible at some points while human downloading of stories was required at others. There are often gaps in this historical content since the content aggregators catalog of different news services changes at various points and is often fairly limited.
Answer: This difficulty of this process changed at various points in time, with something like full automation possible at some points while human downloading of stories was required at others. There are often gaps in this historical content since the content aggregators catalog of different news services changes at various points and is often fairly limited.