SlowGuess/ACL_2020_OCR huggingface dataset:
We split 802 documents into training/dev/test sets of 502/100/200 documents. The training set includes 50 documents from each domain in Reuters and Xinhua, 9 documents from each domain in NYT and 66 documents from KBP; the dev set includes 8 documents from each domain and source and 4 documents from KBP; and the test set includes 17 documents from each domain in Reuters and Xinhua, 8 documents from each domain in NYT and 32 documents from KBP. The dataset is released with the standard split we used in our experiments. For evaluation, we calculate F1 score for each content type as well as micro and macro F1 scores.
Feature-based (SVM) uses linear SVM classifier (Pedregosa et al., 2011) over features used by Yarlott et al. (2018), including bag of words, tfidf and 100-dimensional paragraph vectors obtained through Doc2Vec (Le and Mikolov, 2014) implementation in Gensim (Rehürek and Sojka, 2010). Following Yarlott et al. (2018), we set minimum  $\alpha$  to 0.01, minimum word count to 5 for Doc2Vec
model and train it for 50 epochs. All three features are built on the entire training corpus and the value of  $C$  in SVM classifier is set to 10.
Basic Classifier uses only the word-level bi-LSTM with soft-attention to learn sentence representations followed by the local feed forward neural network classifier to make content type predictions.
6.2 Proposed Document-level Models
Document LSTM adds the sentence-level BiLSTM over sentence representations obtained from the word-level BiLSTM to enrich sentence representations with local contextual information.
+Document Encoding uses document encoding for modeling associations with the main topic and obtains the final sentence representations as described previously.
+Headline replaces document encoding with headline sentence encoding generated from the word-level biLSTM. Headline is known to be a strong predictor for the main event (Choubey et al., 2018).
CRF Fine-grained and CRF Coarse-grained adds a CRF layer to make content type predictions for sentences which models dependencies among fine-grained (eight content types) and coarse-grained (main vs. context-informing vs. supportive contents) content types respectively.
6.3 Implementation Details
We set hidden states dimension to 512 for both word-level and sentence-level biLSTMs in all our models. Similarly, we use two-layered feed forward networks with 1024-512-1 units to calculate attention weights for both the BiLSTMs. The final classifier uses two-layer feed forward networks with 3072-1024-9 units for predicting sentence types. All models are trained using Adam (Kingma and Ba, 2014) optimizer with the learning rate of 5e-5. For regularization, we use dropout (Srivastava et al., 2014) of 0.5 on the output activations
Systems	P	R	F1
Feature-based (SVM)	61.0	71.0	69.0
Basic Classifier	81.6	80.7	81.2(±0.4)
Document LSTM	80.7	83.6	82.2(±0.7)
of both BiLSTMs and all neural layers. Word embeddings are kept fixed during the training. All the neural model are trained for 15 epochs and we use the epoch yielding the best validation performance.
To alleviate the influence of randomness in neural model training and obtain stable experimental results, we run each neural model ten times with random seeds and report the average performance.
6.4 Results and Analysis
Tables 4 and 5 show the results from our experiments for content-type and speech label classification tasks. We see that a simple word-level biLSTM based basic classifier outperforms features-based SVM classifier (Yarlott et al., 2018) by  $10.5\%$  and  $11.8\%$  on macro and micro F1 scores respectively for content-type classification. Adding a sentence-level BiLSTM helps in modeling contextual continuum and improves performance by additional  $4.4\%$  on macro and  $2.7\%$  on micro F1 scores. Also, as content types are interpreted with respect to the main event, modeling associations between a sentence representation and the referred main topic representation using headline or document embeddings improves averaged macro F1 score by  $0.6\%$  and  $1.2\%$  respectively. Empirically, the model using document embedding performs better than the one with headline embedding by  $0.6\%$  implying skewed headlining based on recency which is quite prevalent in news reporting.
We further aim to improve the performance by using CRF models to capture interdependencies among different content types, however, CRF models using both fine-grained and coarse-grained label transitions could not exceed a simple classifier model. The inferior performance of CRF models can be explained by variations in news content organization structures (such as inverted pyramid, narrative, etc.), further implying the need to model those variations separately in future work.
Similarly, for speech label classification task, word-level biLSTM model achieves  $12.2\%$  higher F1 score compared to the feature-based SVM classifier which is further improved by  $1.0\%$  with
Table 5: Performance of different systems on Speech label classification task.
M1	M2	C1	C2	D1	D2	D3	D4	N/A
M1	88.0	2.6	9.0	38.2	14.6	0.4	123.2	28.	2.0
M2	6.4	32.4	0.0	28.4	2.0	0.0	3.4	5.4	0.0
C1	13.6	0.6	15.2	27.8	15.2	0.2	25.4	12.0	6.0
C2	39.6	19.2	22.8	483.6	53.2	5.6	134.6	37.6	14.8
D1	3.0	0.0	8.8	54.8	125.4	5.8	41.2	4.2	7.8
D2	1.6	1.6	1.8	9.4	4.0	37.8	41.2	2.8	1.8
D3	6.8	0.0	6.0	82.6	20.4	12.0	586.6	58.2	5.4
D4	4.2	1.2	0.8	29.0	0.4	1.0	63.2	111.4	1.8
NA	1.2	0.0	0.0	1.6	0.6	0.0	3.4	0.0	158.2
document-level biLSTM.
We generated confusion matrix (Table 6) for content-type classification based on prediction results of the best performing model Document LSTM + Document Encoding on the dev set. Prediction errors mainly occur between Main Event (M1) and Current Context / Evaluation (C2/D3), between Previous Event (C1) and Current Context (C2), between Evaluation (D3) and Expectation (D4), and between Current Context (C2) and Historical Event / Evaluation (D1/D3).
7 Utilizing Content Structure to Improve Event Coreference Resolution
Table 6: Confusion matrix for content-type classification based on prediction results of the model Document LSTM+Document Encoding on the dev set, averaged over 10 runs consistent with the results reported in Table 4.
M1	M2	C1	C2	D1	D2	D3	D4
51%	91%	79%	84%	86%	95%	84%	83%
Table 7: Percentages of Singleton events in sentences of each content type.
We envision that news discourse profiling can be useful to many discourse level NLP tasks and applications. As an example, we investigate uses of news structures for event coreference resolution by analyzing 102 documents from the KBP 2015 corpus included in our NewsDiscourse Corpus. We analyze the lifespan and spread of event coreference chains over different content types. First, table 7 shows the percentage of events that are singletons out of all the events that appear in sentences of each content type. We can see that in contrast to main event sentences (M1), other types of sentences are more likely to contain singleton events.
We further analyze characteristics of nonsingleton events, to identify positions of their coreferential mentions and the spread of coreference chains in a document. Motivated by van Dijk's theory, we hypothesize that the main events appear in each type of sentences, but the likelihoods of
M1	M2	C1	C2	D1	D2	D3	D4
58%	15%	23%	15%	10%	9%	14%	14%
Table 8: Percentages of Sentences of each content type that contain a headline main event.
M1	M2	C1	C2	D1	D2	D3	D4
13%	0%	33%	49%	69%	100%	49%	13%
Table 9: Percentages of Intra-type events out of nonsingleton events in sentences of each content type
seeing the main events in a sentence may vary depending on the sentence type. We consider events that appear in the news headline to approximate the main events of a news article. As shown in Table 8, around  $58\%^{5}$  of main event sentences (M1) contain at least one headline event, in addition, context-informing sentences (C1+C2), especially sentences focusing on discussing recent pre-cursor events (C1), are more likely to mention headline events as well.
Other than the main events, we observe that many events have all of their coreferential mentions appear within sentences of the same content type. We call such events intra-type events. In other words, an intra-type event chain starts from a sentence of any type will die out within sentences of the same content type. Table 9 shows the percentage of intra-type event chains out of all the event chains that begin in a certain type of sentence. We can see that non-main contents (e.g., content types C2-D3) are more likely to be self-contained from introducing to finishing describing an event. In particular, historical (D1) and anecdotal (D2) contents exhibit an even stronger tendency of having intra-type event repetitions compared to other non-main content types.
Incorporating Content Structure for Event Coreference Resolution: We incorporate news functional structures for event coreference resolution by following the above analysis and implementing content structure informed constraints in
an Integer Linear Programming (ILP) inference system to better identify singleton mentions, main event mentions and intra-type event mentions.
We use the Document LSTM+Document encoding classifier to predict sentence content types. In addition, we built a discourse-aware event singleton classifier, that resembles the sentence type classifier, to identify singleton event mentions in a document. Specifically, the singleton classifier combines document and sentence representations provided by the content type classifier with contextualized event word representations obtained from a separate word-level biLSTM layer with 512 hidden units. Then, the singleton classifier applies a two-layer feed forward neural network to identify event singletons, and the feed forward network has 3072-512-2 units.
We implement ILP constraints based on system predicted content types of sentences and singleton scores of event mentions. Detailed descriptions of ILP constraints we implemented and their equations are included in the appendix. The ILP formulation has been used in our previous work that yields the previous best system for event coreference resolution (Choubey and Huang, 2018), which aims to capture several specific document level distributional patterns of coreferential event mentions by simply using heuristics. For direct comparisons, we adopt the same experimental settings as in Choubey and Huang (2018), using KBP 2015 documents as the training data and using both KBP 2016 and KBP 2017 corpora for evaluation<sup>6</sup>. We retrained the sentence type classifier using 102 KBP 2015 documents annotated with content types, using 15 documents as the development set and the rest as the training data. We trained the event singleton classifier using the same train/dev split. In addition, we used the same event mentions and pairwise event coreference scores produced by a local pairwise classifier the same as in Choubey and Huang (2018)<sup>7</sup>.
Experimental Results: We compare the content
While all the main event sentences are expected to mention some main event, we use headline events to approximate main events and headline events do not cover all the main events of a news article. As shown in our previous work (Choubey et al., 2018), identifying main events is a challenging task in its own right and main events do not always occur in the headline of a news article. In addition, event annotations in the KBP corpora only consider a limited set of event types, seven types specifically, therefore, if main events do not belong to those seven types, they are not annotated as events, which also contributes to the imperfect percentage of main event sentences containing a headline event.
6 All the KBP corpora include documents from both discussion forum and news articles. But as the goal of this study is to leverage discourse structures specific to news articles for improving event coreference resolution performance, we only evaluate the ILP system using news articles in the KBP corpora. This evaluation setting is consistent with our previous work Choubey and Huang (2018). For direct comparisons, the results reported for all the systems and baselines are based on news articles in the test datasets as well
The classifier can be obtained from https://git.io/JeDw3
Model	KBP 2016	KBP 2017
\(B^{3}\)	\(CEAFe\)	MUC	BLANC	AVG	\(B^{3}\)	\(CEAFe\)	MUC	BLANC	AVG
Local classifier	51.47	47.96	26.29	30.82	39.13	50.24	48.47	30.81	29.94	39.87
+Content Structure	52.78	49.7	34.62	34.49	42.9	51.68	50.57	37.8	33.39	43.36
-Singletons	51.47	47.96	31.42	32.89	40.94	51.17	49.67	38.01	32.94	42.96
-Main Events	52.65	49.35	32.56	33.69	42.06	51.4	50.05	35.13	31.92	42.12
-Intra-type Events	52.62	49.63	32.97	34.07	42.32	51.62	50.45	37.54	33.42	43.26
Lu and Ng (2017)	50.16	48.59	32.41	32.72	40.97	-	-	-	-	-
Choubey and Huang (2018)	51.67	49.1	34.08	34.08	42.23	50.35	48.61	37.24	31.94	42.04
Table 10: Results for event coreference resolution systems on the benchmark datasets (KBP 2016 and 2017).
structure aware ILP system with a baseline system (the row Local classifier) that performs greedy merging of event mentions using local classifier predicted pairwise coreference scores as well as two most recent models for event coreference resolution, the heuristics-based ILP system (Choubey and Huang, 2018) and another recent system (Lu and Ng, 2017). We use the same evaluation method as in (Choubey and Huang, 2018) and evaluate event coreference resolution results directly without requiring event mention type match<sup>8</sup>.
Table 10 shows experimental results. Event coreference resolution is a challenging task as shown by the small margins of performance gains achieved by recent systems. The ILP model constrained by system predicted content structures (the row +Content Structure) outperforms the pairwise classifier baseline system as well as the two most recent systems consistently across all the evaluation metrics over the two benchmark datasets. In particular, our ILP system outperforms the previous state-of-the-art, the heuristics-based ILP system Choubey and Huang, with average F1 gains of  $0.67\%$  and  $1.32\%$  on KBP 2016 and KBP 2017 corpora respectively. The superior performance shows that systematically identified content structures are more effective than heuristics in guiding event linking, and establishes the usefulness of the new discourse profiling task.
To further evaluate the importance of ILP constraints on Singletons, Main events and Intra-type events, we perform ablation experiments by removing each constraint from the full ILP model. Based on the results in Table 10, all the three types of constraints have noticeable impacts to coreference performance, and singletons and main events constraints contribute the most.
Intuitively, news content structures can help in identifying other event relations as well, such as temporal and causal relations, and thus disentangling complete event structures. For instance, events occurring in C1 (Previous Event) sentences are probable cause for the main event which in turn causes events in M2 (Consequence) sentences (the same rationale can be applied for temporal order).
We have created the first broad-coverage corpus of news articles annotated with a theoretically grounded functional discourse structure. Our initial experiments using neural models ascertain the feasibility of this task. We conducted experiments and demonstrated the usefulness of news discourse profiling for event coreference resolution. In the future, we will further improve the performance of news discourse profiling by investigating subgenres of news articles, and extensively explore its usage for various other NLP tasks and applications.
We thank our anonymous reviewers for providing insightful review comments. We gratefully acknowledge support from National Science Foundation via the awards IIS-1942918 and IIS-1755943. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. Disclaimer: the views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of NSF or the U.S. Government.
Daniela Baiamonte, Tommaso Caselli, and Irina Prodanof. 2016. Annotating content zones in news articles. CLiC it, page 40.
The official KBP 2017 event coreference resolution scorer considers two event mentions coreferent if they strictly match on their event type and subtype, which requires building a high-performing event type identification system to enable an event coreference resolver to score well.
Regina Barzilay and Lillian Lee. 2004. Catching the Drift: Probabilistic Content Models, with Applications to Generation and Summarization.
Maria Becker, Michael Staniek, Vivi Nastase, Alexis Palmer, and Anette Frank. 2017. Classifying semantic clause types: Modeling context and genre characteristics with recurrent neural networks and attention. In Proceedings of the 6th Joint Conference on Lexical and Computational Semantics (* SEM 2017), pages 230-240.
Allan Bell. 1998. The discourse structure of news stories. In Approaches to media discourse, pages 64-104.
Parminder Bhatia, Yangfeng Ji, and Jacob Eisenstein. 2015. Better document-level sentiment analysis from RST discourse parsing. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 2212-2218, Lisbon, Portugal. Association for Computational Linguistics.
Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural language processing with Python: analyzing text with the natural language toolkit. "O'Reilly Media, Inc".
Freddy YY Choi. 2000. Advances in domain independent linear text segmentation. In Proceedings of the 1st North American chapter of the Association for Computational Linguistics conference, pages 26-33. Association for Computational Linguistics.
Prafulla Kumar Choubey and Ruihong Huang. 2018. Improving event coreference resolution by modeling correlations between event coreference chains and document topic structures. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 485-495.
Prafulla Kumar Choubey, Kaushik Raju, and Ruihong Huang. 2018. Identifying the most dominant event in a news article by mining event coreference relations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), volume 2, pages 340-345.
Jacob Cohen. 1968. Multiple regression as a general data-analytic system. *Psychological Bulletin*, 70:426-443.
Zeyu Dai and Ruihong Huang. 2018a. Building context-aware clause representations for situation entity type classification. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3305-3315.
Zeyu Dai and Ruihong Huang. 2018b. Improving implicit discourse relation classification by modeling inter-dependencies of discourse units in a paragraph. In Proceedings of the 2018 Conference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 141-151.
George R Doddington, Alexis Mitchell, Mark A Przybocki, Lance A Ramshaw, Stephanie M Strassel, and Ralph M Weischedel. 2004. The automatic content extraction (ace) program-tasks, data, and evaluation. In Lrec, volume 2, page 1.
Jacob Eisenstein and Regina Barzilay. 2008. Bayesian unsupervised topic segmentation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 334-343. Association for Computational Linguistics.
Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie Strassel. 2015. Overview of linguistic resources for the tac kbp 2015 evaluations: Methodologies and reults. In Proceedings of the TAC KBP 2015 Workshop, pages 16-17.
Sandhaus Evan. 2008. The new york times annotated corpus. LDC2008T19. DVD. Philadelphia: Linguistic Data Consortium.
Vanessa Wei Feng and Graeme Hirst. 2012. Text-level discourse parsing with rich linguistic features. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 60-68.
Annemarie Friedrich and Alexis Palmer. 2014. Situation entity annotation. In Proceedings of LAW VIII, The 8th Linguistic Annotation Workshop, pages 149-158.
Marti A Hearst. 1994. Multi-paragraph segmentation of expository text. In Proceedings of the 32nd annual meeting on Association for Computational Linguistics, pages 9-16. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Yangfeng Ji and Jacob Eisenstein. 2014. Representation learning for text-level discourse parsing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 13-24.
Yangfeng Ji and Jacob Eisenstein. 2015. One vector is not enough: Entity-augmented distributed semantics for discourse relations. Transactions of the Association for Computational Linguistics, 3:329-344.
Yangfeng Ji and Noah A. Smith. 2017. Neural discourse structure for text categorization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 996-1005, Vancouver, Canada. Association for Computational Linguistics.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Joost G Kircz. 1991. Rhetorical structure of scientific articles: the case for argumentational analysis in information retrieval. Journal of documentation, 47(4):354-372.
Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, and Jonathan Berant. 2018. Text segmentation as a supervised learning task. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 469-473, New Orleans, Louisiana. Association for Computational Linguistics.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data.
Quoc Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In International conference on machine learning, pages 1188-1196.
Jiwei Li, Rumeng Li, and Eduard Hovy. 2014a. Recursive deep models for discourse parsing. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2061-2069.
Junyi Jessy Li, Marine Carpuat, and Ani Nenkova. 2014b. Assessing the discourse factors that influence the quality of machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), volume 2, pages 283-288.
Maria Liakata, Shyamasree Saha, Simon Dobnik, Colin Batchelor, and Dietrich Rebholz-Schuhmann. 2012. Automatic recognition of conceptualization zones in scientific articles and two life science applications. Bioinformatics, 28(7):991-1000.
Elizabeth DuRoss Liddy. 1991. The discourse-level structure of empirical abstracts: An exploratory study. Information Processing & Management, 27(1):55-81.
Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2011. Automatically evaluating text coherence using discourse relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies-Volume 1, pages 997-1006. Association for Computational Linguistics.
Ziheng Lin, Hwee Tou Ng, and Min-Yen Kan. 2014. A pdtb-styled end-to-end discourse parser. Natural Language Engineering, 20(2):151-184.