SlowGuess/ACL_2020_OCR huggingface dataset:
Prafulla Kumar Choubey $^{1}$  Aaron Lee $^{1}$  Ruihong Huang $^{1}$  Lu Wang $^{2}$
$^{1}$  Department of Computer Science and Engineering, Texas A&M University
2 Khoury College of Computer Sciences, Northeastern University
(prafulla.choubey, aaronlee, huangrh)@tamu.edu
(luwang)@ccs.neu.edu
Understanding discourse structures of news articles is vital to effectively contextualize the occurrence of a news event. To enable computational modeling of news structures, we apply an existing theory of functional discourse structure for news articles that revolves around the main event and create a human-annotated corpus of 802 documents spanning over four domains and three media sources. Next, we propose several document-level neural-network models to automatically construct news content structures. Finally, we demonstrate that incorporating system predicted news structures yields new state-of-the-art performance for event coreference resolution. The news documents we annotated are openly available and the annotations are publicly released for future research<sup>1</sup>.
Detecting and incorporating discourse structures is important for achieving text-level language understanding. Several well-studied discourse analysis tasks, such as RST (Mann and Thompson, 1988) and PDTB style (Prasad et al., 2008) discourse parsing and text segmentation (Hearst, 1994), generate rhetorical and content structures that have been shown useful for many NLP applications. But these widely applicable discourse structures overlook genre specialties. In this paper, we focus on studying content structures specific to news articles, a broadly studied text genre for many NLP tasks and applications. We believe that genre-specific discourse structures can effectively complement genre independent discourse structures and are essential for achieving deep story-level text understanding.
What is in a news article? Normally, we expect a news article to describe well verified facts of newly
happened events, aka the main events. However, almost no news article limits itself to reporting only the main events. Most news articles also report context-informing contents, including recent precursor events and current general circumstances, that are meant to directly explain the cause or the context of main events. In addition, they often contain sentences providing further supportive information that is arguably less relevant to main events, comprising of unverifiable or hypothetical anecdotal facts, opinionated statements, future projections and historical backgrounds. Apparently, the relevance order of sentences is not always aligned with their textual order, considering that sentences in a news article are ordered based on their vague importance that is generally determined by multiple factors, including content relevance as well as other factors such as the focus of an article, the author's preferences and writing strategies.
While a number of theoretical studies for news discourse exist, little prior effort has been put on computational modeling and automatic construction of news content structures. We introduce a new task and a new annotated text corpus for profiling news discourse structure that categorizes contents of news articles around the main event. The NewsDiscourse corpus consists of 802 news articles (containing 18,155 sentences), sampled from three news sources (NYT, Xinhua and Reuters), and covering four domains (business, crime, disaster and politics). In this corpus, we label each sentence with one of eight content types reflecting common discourse roles of a sentence in telling a news story, following the news content schemata proposed by Van Dijk (Teun A, 1986; Van Dijk, 1988a,b) with several minor modifications.
Next, we present several baselines for automatically identifying the content type of sentences. The experimental results show that a decent performance can be obtained using a basic neural
$^{1}$ Dataset can be found at https://github.com/prafulla77/Discourse_Profiling
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5374-5386 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
network-based multi-way classification approach. The sentence classification performance can be further improved by modeling interactions between sentences in a document and identifying sentence types in reference to the main event of a document.
We envision that the news discourse profiling dataset as well as the learnt computational systems are useful to many discourse level NLP tasks and applications. As an example, we analyze correlations between content structures and event coreference structures in news articles, and conduct experiments to incorporate system predicted sentence content types into an event coreference resolution system. Specifically, we analyze the lifespan and spread of event coreference chains over different content types, and design constraints to capture several prominent observations for event coreference resolution. Experimental results show that news discourse profiling enables consistent performance gains across all the evaluation metrics on two benchmark datasets, improving the previous best performance for the challenging task of event coreference resolution.
Several well-studied discourse analysis tasks have been shown useful for many NLP applications. The RST (Mann and Thompson, 1988; Soricut and Marcu, 2003; Feng and Hirst, 2012; Ji and Eisenstein, 2014; Li et al., 2014a; Liu et al., 2019) and PDTB style (Prasad et al., 2008; Pitler and Nenkova, 2009; Lin et al., 2014; Rutherford and Xue, 2016; Qin et al., 2016; Xu et al., 2018) discourse parsing tasks identify discourse units that are logically connected with a predefined set of rhetorical relations, and have been shown useful for a range of NLP applications such as text quality assessment (Lin et al., 2011), sentiment analysis (Bhatia et al., 2015), text summarization (Louis et al., 2010), machine translation (Li et al., 2014b) and text categorization (Ji and Smith, 2017). Text segmentation (Hearst, 1994; Choi, 2000; Eisenstein and Barzilay, 2008; Koshorek et al., 2018) is another well studied discourse analysis task that aims to divide a text into a sequence of topically coherent segments and has been shown useful for text summarization (Barzilay and Lee, 2004), sentiment analysis (Sauper et al., 2010) and dialogue systems (Shi et al., 2019).
The news discourse profiling task is complementary to the well-established discourse analysis
tasks and is likely to further benefit many NLP applications. First, it studies genre-specific discourse structures, while the aforementioned discourse analysis tasks study genre independent general discourse structures and thus fail to incorporate domain knowledge. Second, it focuses on understanding global content organization structures with the main event at the center, while the existing tasks focus on either understanding rhetorical aspects of discourse structures (RST and PDTB discourse parsing) or detecting shallow topic transition structures (text segmentation).
Genre-specific functional structures have been studied based on different attributes, but mostly for genres other than news articles. Liddy (1991), Kircz (1991) and Teufel et al. (1999) used rhetorical status and argumentation type to both define functional theories and create corpora for scientific articles. Mizuta et al. (2006), Wilbur et al. (2006), Waard et al. (2009) and Liakata et al. (2012) extensively studied functional structures in biological domain with multiple new annotation schemata.
Past studies on functional structures of news articles have been mainly theoretical. Apart from Van Dijk's theory of news discourse (Teun A, 1986; Van Dijk, 1988b), Pan and Kosicki (1993) proposed framing-based approach along four structural dimensions: syntactic, script, thematic and rhetorical, of which syntactic structure is similar to the Dijk's theory. Owing to the high specificity of the Dijk's theory, Yarlott et al. (2018) performed a pilot study for its computational feasibility and annotated a small dataset of 50 documents taken from the ACE Phase 2 corpus (Doddington et al., 2004). However, as mentioned in the paper, their annotators were given minimal training prior to annotations, consequently, the kappa inter-agreement  $(55\%)$  between two annotators was not satisfactory. In addition, coverage of their annotated dataset on broad event domains and media sources was unclear. The only studies on functional structure of news article with sizable dataset include Baiamonte et al. (2016) that coarsely separates narration from descriptive contents and Friedrich and Palmer (2014) that classify clauses based on their aspectual property.
3 Elements of Discourse Profiling
We consider sentences to be units of discourse and define eight schematic categories to study their roles within the context of the underlying topic. The original Van Dijk's theory was designed for
5375
Main Content	Fine-grained type
(1) U.S. President Donald Trump tried on Tuesday to calm a storm over his failure to hold Russian
President Vladimir Putin accountable for meddling in the 2016 U.S. election, saying he misspoke in a
joint news conference in Helsinki.
(2) The rouble fell 1.2 percent on Tuesday following Trump's statement.	Main Event
Consequence
Context-informing Content	Fine-grained type
(3) Trump praised the Russian leader for his “strong and powerful” denial of the conclusions of U.S.
intelligence agencies that the Russian state meddling in the election.
(4) Special Counsel Robert Mueller is investigating that allegation and any possible collusion by
Trump's campaign.	Previous Event
Current Context
Additional Supportive Content	Fine-grained type
(5) Congress passed a sanctions law last year targeting Moscow for election meddling.
(6) “The threat of wider sanctions has grown,” a businessman told Reuters, declining to be named
because of the subject’s sensitivity.
(7) Republicans and Democrats accused him of siding with an adversary rather than his own country.
(8) McConnell and House Speaker Paul Ryan, who called Russia’s government “menacing,” said their
chambers could consider additional sanctions on Russia.	Historical Event
Anecdotal Event
Evaluation
Expectation
Table 1: Examples for eight Fine-grained content types.
analyzing discourse functions of individual paragraphs w.r.t the main event, and the pilot study done by Yarlott et al. (2018) also considered paragraphs as units of annotations. Observing that some paragraphs contain more than one type of contents, we decided to conduct sentence-level annotations instead to minimize disagreements between annotators. and allow consistent annotations $^{2}$ .
Table 1 contains an example for each content type. Consistent with the theory presented by Van Dijk, the categories are theoretical and some of them may not occur in every news article.
3.1 Main Contents
Main content describes what the text is about, the most relevant information of the news article. It describes the most prominent event and its consequences that render the highest level topic of the news report. Main Event (M1) introduces the most important event and relates to the major subjects in a news report. It follows strict constraints of being the most recent and relevant event, and directly monitors the processing of remaining document. Categories of all other sentences in the document are interpreted with respect to the main event. Consequence (M2) informs about the events that are triggered by the main news event. They are either temporally overlapped with the main event or happens immediately after the main event.
3.2 Context-informing Contents
Context-informing sentences provide information related to the actual situation in which main event occurred. It includes the previous events and other contextual facts that directly explain the circumstances that led to the main event. Previous Event (C1) describes the real events that preceded the main event and now act as possible causes or preconditions for the main event. They are restricted to events that have occurred very recently, within last few weeks. Current Context (C2) covers all the information that provides context for the main event. They are mainly used to activate the situation model of current events and states that help to understand the main event in the current social or political construct. They have temporal co-occurrence with the main event or describe the ongoing situation.
3.3 Additional Supportive Contents
Finally, sentences containing the least relevant information, comprising of unverifiable or hypothetical facts, opinionated statements, future projections and historical backgrounds, are classified as distantly-related content. Historical Event (D1) temporally precedes the main event in months or years. It constitutes the past events that may have led to the current situation, or indirectly relates to the main event or subjects of the news article. Anecdotal Event (D2) includes events with specific participants that are difficult to verify. It may include fictional situations or personal account of incidents of an unknown person especially aimed to exaggerate the situation. Evaluation (D3) introduces reactions from immediate participants, ex
2Our two annotators agreed that the majority of sentences describe one type of content. For a small number of sentences that contain a mixture of contents, we ask our annotators to assign the label that reflects the main discourse role of a sentence in the bigger context.
perts or known personalities that are opinionated and may also include explicit opinions of the author or those of the news source. They are often meant to describe the social or political implications of the main event or evaluation of the current situation. Typically, it uses statements from influential people to selectively emphasize on their viewpoints. Expectation (D4) speculates on the possible consequences of the main or contextual events. They are essentially opinions, but with far stronger implications where the author tries to evaluate the current situation by projecting possible future events.
3.4 Speech vs. Not Speech
In parallel with discourse profiling annotations, we also identify sentences that contain direct quotes or paraphrased comments stated directly by a human and label them as Speech. We assign a binary label, Speech vs. Not Speech, to each sentence independently from the annotations of the above eight schematic discourse roles. Note that Speech sentences may perfectly be annotated with any of the eight news discourse roles based on their contents, although we expect Speech sentences to serve certain discourse roles more often, such as evaluation and expectation.
3.5 Modifications to the Van Dijk Theory
The Van Dijk's theory was originally based on case studies of specific news reports. To accommodate wider settings covering different news domains and sources, we made several minor modifications to the original theory. First, we label both comments made by external sources (labeled as "verbal reactions" in the original theory) and comments made by journalistic entities as speech, and label speech with content types as well. Second, we added a new category, anecdotal event (D2), to distinguish unverifiable anecdotal facts from other contents. Anecdotal facts are quite prevalent in the print media. Third, we do not distinguish news lead sentences that summarize the main story from other Main Event (M1) sentences, considering that lead sentences pertain to the main event and major subjects of a news.
4 Dataset Creation and Statistics
The NewsDiscourse corpus consists of 802 openly accessible news articles containing 18,155 sentences<sup>3</sup> annotated with one of the eight content
types or  $N / A$  (sentences that do not contribute to the discourse structure such as photo captions, text links for images, etc.) as well as Speech labels. The documents span across the domains of business, crime, disaster and politics from three major news sources that report global news and are widely used: NYT (USA), Reuters (Europe) and Xinhua (China). We include 300 articles each (75 per domain) from Reuters and Xinhua that are collected by crawling the web and cover news events between 2018- '19. NYT documents are taken from existing corpora, including 102 documents from KBP  $2015^{4}$  (Elliset al., 2015) and 100 documents (25 per domain) from the annotated NYT corpus (Evan, 2008).
We trained two annotators for multiple iterations before we started the official annotations. In the beginning, each annotator completed 100 common documents (Eight from each of the domains and sources and four from the KBP) within the corpus to measure annotator's agreement. The two annotators achieved Cohen's  $\kappa$  score (Cohen, 1968) of 0.69144,0.72389 and 0.87525 for the eight fine-grained, three coarse-grained and Speech label annotations respectively. Then, the remaining documents from each domain and news source were split evenly between the two annotators.
Detailed distributions of the created corpus, including distributions of different content types across domains and media sources are reported in Tables 2 and 3 respectively. We find that distributions of content types vary depending on either domains or media sources. For instance, disaster documents report more consequences (M2) and anecdotal events (D2), crime documents contain more previous events (C1) and historical events (D1), while politics documents have the most opinionated contents (sentences in categories D3 and D4) immediately followed by business documents. Furthermore, among different sources, NYT articles are the most opinionated and describe historical events most often, followed by Reuters. In contrast, Xinhua articles has relatively more sentences describing the main event.
Speech labels and content type labels are separately annotated and each sentence has both a content type label and a speech label (binary, speech
as independent content. We used NLTK (Bird et al., 2009) to identify sentence boundaries in the body text. Occasionally, one sentence is wrongly split into multiple sentences, the annotators were instructed to assign them with the same label.
$^{4}$ KBP documents are not filtered for different domains due to the small size of corpus.
3Note that only sentences within the body of the news article are considered for annotation and headlines are considered
M1	M2	C1	C2	D1	D2	D3	D4	N/A
Business	336(8.5)	40(1.0)	225(5.8)	1,041(26.6)	238(6.1)	70(1.8)	1,049(26.8)	545(13.9)	368(9.4)
Crime	374(10.4)	78(2.2)	271(7.5)	941(26.1)	510(14.2)	77(2.1)	816(22.7)	204(5.7)	328(9.1)
Disaster	407(10.6)	206(5.3)	223(5.8)	1,032(26.8)	139(3.6)	330(8.6)	741(19.2)	405(10.5)	368(9.5)
Politics	475 (10.4)	21(0.4)	218(4.8)	954(20.9)	228(5.0)	85(1.9)	1,492(32.7)	679(14.9)	414(9.1)
Table 2: Distribution of Content type labels across domains, with percentages shown within parentheses.
M1	M2	C1	C2	D1	D2	D3	D4	N/A
NYT	492(8.4)	97(1.7)	342(5.8)	1401(24.0)	714(12.2)	197(3.4)	1876(32.1)	532(9.1)	197(3.3)
Xinhua	667(13.6)	95(1.9)	361(7.4)	1249(25.5)	214(4.4)	96(2.0)	953(19.5)	525(10.7)	736(15.0)
Reuters	624(8.4)	195(2.6)	391(5.1)	1837(24.8)	571(7.7)	316(4.3)	1867(25.2)	924(12.5)	686(9.3)
NYT_KBP	191(8.6)	42(1.9)	157(7.0)	519(23.3)	384(17.3)	47(2.1)	598(26.9)	148(6.7)	141(6.3)
Table 3: Distribution of Content type labels across media sources, with percentages shown within parentheses.
vs. not speech). In the created corpus, 5535 out of 18,155 sentences are labeled as speech.
5 Document-level Neural Network Model for Discourse Profiling
A wide range of computational models has been applied for extracting different forms of discourse structures. However, across several tasks, neural network methods (Ji and Eisenstein, 2015; Becker et al., 2017) are found the most effective, with relatively superior performance obtained by modeling discourse-level context (Dai and Huang, 2018a,b).
As an initial attempt, we use a hierarchical neural network to derive sentence representations and a document encoding, and model associations between each sentence and the main topic of the document when determining content types for sentences. Shown in Figure 1, it first uses a word-level bi-LSTM layer (Hochreiter and Schmidhuber, 1997) with soft-attention over word representations to generate intermediate sentence representations which are further enriched with the context information using another sentence-level bi-LSTM. Enriched sentence representations are then averaged with their soft-attention weights to generate document encoding. The final prediction layers model associations between the document encoding and each sentence encoding to predict sentence types.
Context-aware sentence encoding: Let a document be a sequence of sentences  $\{s_1,s_2\dots s_n\}$  which in turn are sequences of words  $\{(w_{11},w_{12}\dots)$  ..  $(w_{n1},w_{n2},\ldots)\}$ . We first transform a sequence of words in each sentence to contextualized word representations using ELMo (Peters et al., 2018) followed by a word-level biLSTM layer to obtain their hidden state representations  $H_{s}$ . Then, we take weighted sums of hidden representations using soft-attention scores to obtain intermediate sen
Figure 1: Neural-Network Architecture Incorporating Document Encoding for Content Type Classification
tence encodings  $(S_{i})$  that are uninformed of the contextual information. Therefore, we apply another sentence-level biLSTM over the sequence of sentence encodings to model interactions among sentences and smoothen context flow from the headline until the last sentence in a document. The hidden states  $(H_{t})$  of the sentence-level bi-LSTM are used as sentence encodings.
Document Encoding: We generate a reference document encoding, as a weighted sum over sentence encodings using their soft-attention weights.
Modeling associations with the main topic: Sentence types are interpreted with respect to the main event. However, while the sentence-level biLSTM augments sentence representations with the local context, they may be still unaware of the main topic. Therefore, we compute element-wise products and differences between the document encoding and a sentence encoding to measure their correlations, and further concatenate the products and differ
Models	M1	M2	C1	C2	D1	D2	D3	D4	Macro	Micro
F1	P	R	F1	F1
Feature-based (SVM)	34.0	8.0	18.0	44.0	45.0	14.0	52.0	44.0	39.1	37.9	38.3	45.7
Basic Classifier	42.5	24.7	18.2	55.4	59.6	28.5	66.1	52.5	52.6	47.9	48.8(±0.8)	57.5(±0.6)
Document LSTM	49.3	27.3	20.2	57.0	63.6	45.8	67.4	55.6	56.6	52.6	53.2(±0.7)	60.2(±1.0)
+Headline	49.8	30.0	21.8	56.7	63.2	42.7	66.8	58.7	57.3	52.9	53.8(±0.7)	60.4(±1.0)
+Document encoding	49.6	27.9	22.5	58.1	64.1	48.1	67.4	57.6	56.9	53.7	54.4(±0.8)	60.9(±0.7)
CRF Fine-grained	47.7	26.4	22.2	56.0	63.3	45.2	66.4	55.2	55.4	52.9	52.9(±1.4)	59.4(±1.1)
CRF Coarse-grained	48.4	29.3	21.6	55.9	62.9	47.2	66.7	54.2	55.6	53.4	53.5(±0.9)	59.6(±0.7)
Table 4: Performance of different systems on fine-grained discourse content type classification task. All results correspond to average of 10 training runs with random seeds. In addition, we report standard deviation for both macro and micro F1 scores.
ences with the sentence encoding to obtain the final sentence representation that is used for predicting its sentence type.
Predicting Sentence Types: First, we use a two layer feed forward neural network as a regular classifier to make local decisions for each sentence based on the final sentence representations. In addition, news articles are known to follow inverted pyramid (Bell, 1998) or other commonly used styles where the output labels are not independent. Therefore, we also use a linear chain CRF (Lafferty et al., 2001) layer on the output scores of the local classifier to model dependence among discourse labels.