Discourse Sense Flows: Modelling the Rhetorical Style of Documents across Various Domains
Authors: Rene Knaebel
Manfred Stede
Venue: EMNLP 2023 Findings (2023)
Keywords: rhetorical style
cross-domain
discourse parsing
discourse signals
connecting phrases
sense recognition
TL;DR: Our work extracts discourse relations and linearizes their senses for comparing the rhetorical style of documents across various corpora.

Abstract: Recent research on shallow discourse parsing has given renewed attention to the role of discourse relation signals, in particular explicit connectives and so-called alternative lexicalizations.
In our work, we first develop new models for extracting signals and classifying their senses, both for explicit connectives and alternative lexicalizations, based on the Penn Discourse Treebank v3 corpus. Thereafter, we apply these models to various raw corpora, and
we introduce 'discourse sense flows', a new way of modeling the rhetorical style of a document by the linear order of coherence relations, as captured by the PDTB senses. The corpora span several genres and domains, and we undertake comparative analyses of the sense flows, as well as experiments on automatic genre/domain discrimination using discourse sense flow patterns as features.
We find that n-gram patterns are indeed stronger predictors than simple sense (unigram) distributions.

Discourse Sense Flows: Modelling the Rhetorical Style of Documents across Various Domains
René Knaebel and Manfred Stede
Applied Computational Linguistics
Department of Linguistics
University of Potsdam Germany
Frene.knaebee manf red. meafrred.ttedeeuunl -potssdaapppttdmmmm.mm
Abstract
much work on genre/domain differentiation the basis of discourse features for large amounts of text documents.
Recent research on shallow discourse parsing has given renewed attention the role of dis- course relation signals, in particular explicit connectives and so-called alternative lexical- izations. In work, first develop new models for extracting signals and classifying their senses, both for explicit connectives and alternative lexicalizations, based the Penn Discourse Treebank v3 corpus. Thereafter, we apply these models to various raw corpora, and we introduce 'discourse sense flows', way of modeling the rhetorical style of a doc- ument by the linear order of coherence rela- tions, as captured by the PDTB senses. The corpora span several genres and domains, and we undertake comparative analyses of the sense flows, as well as experiments on automatic genre/domain discrimination using discourse sense flow patterss features. We find that n-gram patterns are indeed stronger predictors than simple sense (unigram) distributions.
We address this task from the perspective of dis- course relations as operationalized in the Penn Dis- course Treebank (PDTB) (Prasad et al., 2018). It represents discourse relations in the form of two text spans (arguments: Arg1 and Arg2) that sociated with a sense. These senses are organized in three-level hierarchy and most research focuses on the first two levels, coarse and fine (see Table 2). our work, we for now rely only on the dis- course signals and do not consider the arguments. Here is an example from the PDTB corpus with three signals:
(1)
"We had been soliciting opinions on it [long before]; Mr. Lawson's 's resignation, [and]e offer some of the collection for the benefit of his successor and one-time deputy, John Major. [To begin with,]3 should note that in contrast the U.S. deficit, Britain has been running largish budget surpluses."
1 Introduction
People write differently, and the style of writing highly depends on the author's intention. While texts from the news are largely informative, an argu- mentative essay, for example, is written to support particular opinion. Understanding these underlying purposes and structures in text-or why the text is arranged as is-is complex problem and re- quires giving more attention differences between genres (e.g.. news versus review) and domains (e.g.. economics versus science).
The PDTB distinguishes two groups of signaled re- lations: Explicit relations are signaled by a closed set of discourse connectives, e.g. because, and, if-then, and before. Alternative lexicalizations (AltLex) use phrases other than connectives, e.g. to begin with, for that reason, and it all adds up to. While the first two signals in ex. (1) are explicit sig- nals (and their senses are Temporal.Asynchronn and Expansion. Conjunction, respectively), the last one represents an alternative lexicalization with sense Expansion. Inriiiiinnnnttniiiiiion
Previous research has examined some genre dif- ferences within single newspaper corpus (Web- ber, 2009), and more recent work has shown the impact of joining diverse genres in training data for discourse parsing (Liu and Zeldes, 2023). Simi- larly, Stepanov and Riccardi (2014) demonstrated that it is difficult to generalize "discourse knowl- edge" from one domain to others, e.g., from news to biomedical text. However, there is so far not
In total, the PDTB (version 3) contains 25,878 signaled relations, 94% of which are of type 'ex- plicit', and the rest are AltLex phrases. Importantly, the PDTB also contains 'implicit' relations that are not lexically signaled all; for present pur- poses, do use them.

---

The condiotoo of cmal and counghyy An diaiiiiioo the
pogll
BSR
Aelew thy any
dhs lecaeiin of far allo th Shot of
Sengls
Fost
cner gir then small and enouging repiiiity fylor resl and works cossacieed by duecty benes,
small teneeghng
Sense Profile
and woble
peisa Duck hay sond was peaee save consdiaato ctpormment, onfewssoe Neaeue aed phame beee seake cletoole cenu sunders wurter to ras Epynialy santer the Jacdio
Dack nop sandy mudd peetle
hur
Honret
and abachee lokan and
thet Aowmatt
Sense Flow
Change Flow
Change
Original Discourse
Identified Discourse Signats
Discouran Flow
Discouran Profile
Figure I: Overview of the signal extraction process, from plain text via the modeling of sense flows to the abstraction of discourse prrfiles'. In this figure, we illustrate senses by only four classes.
In this paper, we develop the first models for identifying explicit and AltLex signals and their senses from the recent version 3 of the PDTB. Then, we use the models on corpora of raw texts from dif- ferent domains and genres, and we demonstrate that automatically produced signal/sense information can be used to differentiate those domains. Fig- ure I illustrates the steps from raw documents to "sense profiles" that we regard models of the doc- ument's s rhetorical style. They are operationalized by the distributions of n-gram patterns extracted from the document's 's sense flow.
model, early work by Pitler and Nenkova (2009) disambiguated explicit connectives and their associ- ated coarse sense based on syntactic features from gold constituency trees such as categories of the connective, its parent and siblings, as well as pair- wise interaction features. Later, the feature set was elaborated, e.g., with lexico-syntactic features and noo--to-connective category paths (Lin et al., 2014), and additional syntactic context information (Wang and Lan, 2015). Recent work integrated contextual- ized word embeddings for identification and sense classification (Knaebel and Stede, 2020).
Other work eliminated the dependency on pre- compiled lexicons by incorporating transformer- based sequence labeling approaches. Kurfalı (2020) used BERT successfully predict single and consecutive multi-word connectives. With re- spect the CoNLL shared task's 's exact evaluation scheme (Xue et al., 2016), he achieves a 96.77 Fl score for connective identification in the PDTB (v2). However, simplification has been introduced that replaces full connective by its correspond- ing head, e.g. after 10 minutes is mapped to after, which makes the task significantly casier.
Contributions In short, we (i) develop novel models for classifying senses of discourse sig- nals (connectives and AltLex phrases) based on the PDTB v3 corpus. We (ii) introduce a new concept called discourse sense flows, extract pat- terns of these flows, and provide statistical anal- ysis across nine corpora from various genres and domains. Based on this concept, we (iii) exam- ine statistical models' abilities to distinguish the rhetorical style of documents stemming from those genres/domains, in terms of sense n-grams.
We summarize the related work ($2) and the corpora we use ($3). Then, we describe our sig- nal extraction models ($4). Next, we introduce sense flows and study their predictions on texts from other domains ($5).
Recognizing AltLex signals in the PDTB cor- pus is challenging due to the higher complexity of the phrases when compared to explicit signals, and due to the small number of training examples. Prasad et al. (2010) analyzed regularities by defin- ing groups of phrases in the older version of the PDTB. In shallow discourse parsing, The identifi- cation of AltLex phrases was generally neglected, or the AltLex instances were simply merged with the implicit relations. Therefore, in the context of the
2 Related Work
Signal Identification Traditionally, explicit con- nectives were identified, first, by accessing pre- compiled connective lexicons and matching possi- ble candidates, before a statistical model learns to disambiguate these candidates for their 'discourse' or 'sentential' use. Using a maximum entropy
CoNLL shared task no dedi- cated previous results for AltLex sense classifica- tion were available. A first approach to predicting

---

the presence of AltLex phrases based on PDTB v3 was presented by Knaebel and Stede (2022). A shared task on multi-formalism discourse pars- ing (Zeldes et al., 2021> formed a new notion of signal identification by combining explicit and Alt- Lex relation signals, but it considered discontinu- ous signals (e.g.. if.. then) as separate relations. The task is thus different and results cannot be easily compared. Best performing systems in the connective identification track achieved scores of 92.02 F1 (Gessler et al., 2021) and 91.15 F1 (Bak- shi and Sharma, 2021) respectively, both using a combination of LLM and CRF for sentence label- ing.
Domain
Genre
#Docs
#EX
#AL
ABSTRACT
9,983
30,309
2,833
MED
ABSTRACT
10,000
32,498
3,607
ESSAY
1,562
31,456
1,077
ESSAY
402
6,395
440
NEWS
43,540
49,253
1,605
NEWS
2,225
25,116
1,211
WSJ
NEWS
2,161
24,218
1,536
SPEECH 2,459
172,526
5,825
UN
SPEECH
5,000
36,545
2,608
Table I: Corpus Statistics Overview: All relation counts
approximations based automatic predictions.
Rhetorical Style Pitler al. (2008) show that certain adjacent discourse relations are not inde- pendent of each other, thus motivating the identi- fication of certain local sense patterns. Lin al. (2011) use their previously developed PDTB parser to identify discourse relations and their senses (only coarse-grained), and they introduce the con- cept of discourse role matrix to model coherence in texts. The focus is on handling sentence per- mutation and applications such summariza- tion and argumentative zoning. Biran and McK- eown (2015) develop an n-gram model on PDTB senses that is used for planning a story. Their work shows performance differences depending on the n- gram model's training domain. Braud and Søgaard (2017) detect scientific fraud, with features includ- ing token n-grams and explicit connectives with their associated coarse senses. Ji and Smith (2017) 7) developed recursive neural model for document classification according to the rhetorical structure (RST) of the document.
corpus, in the following referred to domain, and also whether different domains have certain pat- terns in common. Therefore, prepare additional corpora, summarized in Table 1, to complement the data in the PDTB. We group domains into four spe- cific genre classes: ABSTRACT (scientific abstracts from two domains), ESSAY (argumentative learner essays). NEWS (two additional news wire agencies with various topics), and SPEECH (casual and polit- ical speech transcripts). We expect these genres to differ in their style of writing due their varying goals (e.g. informative, persuasive, entertaining) and target audiences, respectively. As some of the datasets were very large, we scaled them to a com- putationally feasible size as described below.
The Penn Discourse Treebank (PDTB) (Prasad et al., 2018) consists of 2,499 stories from a three- year Wall Street Joumaa (WSJ) collection, and thus belongs to the genre of News Articles. We use this corpus for training it is by far the largest available source for discourse relation annotations. While the previous version contains around 43K annotations with 18.4K explicit and 16K implicit relations, the recent version 3 introduces new annotation scheme with a revised sense hierarchy as well as 13K additional relations.
Wachsmuth et al. (2015) introduce sentiment flows and generalize from local sentiments to the global sentiment of review; they proposed using sentiment flow patterns for classification purposes. Building on this idea, Al-Khatib et al. (2017) adapt change flows argumentation mining for identify- ing patterns of evidence types and their correlations the topic of a document. In contrast to these two works that consider flows as a whole, we instead extract n-grams from discourse sense flows, for identifying patterns in rhetorical styles.
In the following, we briefly summarize our ad- ditional corpora grouped according to their genre.
News The New York Times Annotated Corpus (NYT)) contains about 1.8 million documents pub- lished by the New York Times between 1987 and 2007. We sample 500 documents per month from the years 2000-2007, but remove some of the ar- ticles due to peripheral contents, e.g., business
3 Data
This work examines the different rhetorical styles found in documents and investigates whether docu- ments can be associated with their corresponding
'nttps://doi.on http::///00...rg1..351111111 (tte//0..511/7/baaa10

---

market overviews, and thus focus on the more recent part of the dataset. The BBC News Arti- cles (BBC) (Greene and Cunningham, 2006) con-
Level-1
Level-2
#EX
#AL
SYNCHRONOUS
1,947
29
TEMPORAL
ASYNCHRONOUS
2,111
130
sists of 2225 documents from the BBC news web- site corresponding to stories in five topical areas (business, entertainment, polities, sport, tech) from 2004-2005.
CAUSE
2,395
915
CONTINGENCY
CONDITION
1,686
32
PURPOSE
381
34
CONCESSION
4,868
31
COMPARISON
CONTRAST
1,353
41
Essays The Automated Essay Scoring corpus (AES)² was created for shared task by the Hewlett Foundation in 2012. This corpus con- sists of manually-scored student essays for differ- ent prompts and grade levels. For our study, we selected only the 1,500 essays of the persuasive task written by 10th-grade students. The UKP Ar- gument Annotated Essays corpus (PEC) (Stab and Gurevych, 2017) consists of 402 persuasive essays written by language learners (crawled from an es- say website).
SIMILARITY
109
12
CONJUNCTION
8,799
138
DISJUNCTION
305
0
EQUIVALENCE
27
10
EXCEPTION
36
3
EXPANSION
INSTANTIATION
325
61
LEVELOFDETAIL
271
62
MANNER
543
3
SUBSTITUTION
219
29
Table 2: Simplified Sense Hierarchy: counts of explicit
relations (EX) and alternative lexicalizations (AL).
Speeches The TED Conferences Transcripts (TED) is a collection of English transcripts crawled from the TED site. It contains about 2500 talks with publication years ranging from 2006 2017. We add artificial paragraph breaks whenever audi- ence information such as upplause was noted. The UN Security Council Debates corpus (UN) (Schoen- feld et al., 2019) is based on 5,748 meeting pro- tocols dated between 1995 and 2020. The proto- cols are split into distinct speeches and result in 82,165 individual documents. Documents from 2018 2020 have been removed due some con- version errors in the dataset. We randomly select 5000 documents from the corpus.
Identification of Signaled Discourse Relations
For our study, we train models for both tasks, iden- tification of discourse signals and classifying their sense, separately. Similar to Kurfah and Östling (2021), for better context information we process full paragraphs instead of individual sentences. For signal labeling, we use pre-trained large language model (LLM) and fine-tune the base model com- bined with linear transformation and a conditional random field (CRF) (Lafferty et al., 2001) output layer. For properly encoding discourse signals, we use BIOES encoding. This has the advantage that can encode all types of discourse signals with- out loss: discontinuous signals such as not only [...] J but also [... and also signals that are nested within each other. We further add start and end tags for technical reasons related to the CRF
Abstracts The ACL Anthology Reference Corpus (ACL) (Bird et al., 2008) is continuously updated image of the publicly available ACL Anthology that collects conference and journal papers in the areas of natural language processing and compu- tational linguistics.. We sample 10,000 English abstracts from the years 2000-2019. Another pub- licly available source of scientific literature is the PubMed (MED)* search engine that provides access to the MEDLINE database. Here, we also sample 10,000 abstracts, but due to the missing publication meta information, we do not further restrict the data.
For relation sense identification, we follow most previous research and focus on the second level rather than the coarse senses. Due to the small amount of samples available for some classes, we ignore those that are distinguished by the features negation, speech act, and belief. Table 2 shows our proposed simplified sense hierarchy used through- out our work. For comparison purposes, how- ever, we will also show results on the original fine- grained (second-level) senses. We use the same LLM for signal labeling, but pre-compute token embeddings per paragraph and use them as fixed
hhttps://www www.we.gge...mm wttp:oooteeiiioonneooooo. tht:ps://www ted. com/
hhttpp::/pubmed..eeee http: nlm. alm.nih. nih.

---

Model
Partial-Match
Exact-Match
Coarse
Fine
Model
Eval
Acc
F1
Acc
Explicit (Linear)
95.74
95.69
Explicit (CRF)
95.96
95.91
altlex
altlex
91.22
81.11
86.91
60.73
AltLex (Linear)
58.47
56.94
Simplified
explicit
55.87
49.76
38.78
25.23
Altl.ex (CRF)
60.44
56.16
explicit
altiex
77.73
62.36
71.65
49.49
explicit
92.20
84.86
87.09
78.14
Table 3: Labeling performances: macro F1 for partial and exact matches, comparison of Explicit and AltLex models with linear and CRF prediction layers.
hoth
altlex
90. 14
78.60
86.70
62.62
explicit
92.02
86.38
86,90
78.17
altlex
altlex
91.73
81.23
86.
56.42
explicit
56.22
47.63
38.48
20.44
Original
explicit
altlex
77.58
66.87
69.25
44.08
explicit
92.20
86.62
86.72
68.38
both
altlex
90.32
80.91
87.71
61.85
explicit
92.00
86.42
86.57
66.81
inputs for our sense classifiers. Given a discourse signal, we aggregate all its token embeddings by mean and max pooling to get the same input size due to varying signal lengths. Furthermore, we con- catenate the respective surrounding embedding, left and right, to the model input, following (Knaebel and Stede, 2020). We jointly train coarse and fine senses, it has been shown beneficial for overall learning. (Long and Webber, 2022)
Table Sense classification performances (accuracy and macro fl score), comparison of models trained standard and simplified coarse and fine senses, either on one of the signals or on both.
(the linear classifier layer), with the extra CRF final layer. For both prediction tasks, using CRF is beneficial: while for the explicit connective iden- tification, the performance gains are rather small and the overall improvement is mainly rooted better recall values, we notice a higher increase in performance for AltLex signals of about 2% F1 score. We attribute this result to the different aver- age lengths of the signals, explicit signals tend to be shorter.
4.1 Quality of Extracted Discourse Signals
As suggested by Shi and Demberg (2017), we per- form validatiio of the results of our two mod- els by repeating the training 10 times with random initialization and present mean scores with stan- dard deviations. We split 10% of the PDTB for testing purposes and, further, split 10% of the re- maining data for validation during training. De- tailed parammterssssrss are shown in Appendix A. We train our labeling models in two stages: First, we optimize the linear classification layer that later provides emission scores to the CRF using cross- entropy loss. Second, we optimize the negative log-likelihood given the emission scores for se- quence of output tags using the forward algorithm. Final outputs for evaluation computed using the Viterbi algorithm.
Comparison with previous state-of-the- art (Gessler et al., 2021) is not trivial, due slightly different problem settings. The DISRPT shared task divided discontinuous signals into separate ones, thus simplifying the task; however, they use a stricter evaluation threshold. Combining our separate results for Exact-Match on explicit and AltLex signals by their relative occurrence within the PDTB, arrive at competitive result of 93.65 Fl score.
For the evaluation of discourse signal extraction, we use partial matching (Xue al., 2016) and de- fine the matching overlap based on the F1 score of two connecting phrases. Partial-Match and Exact- Match refer to 70% and 90% F1 matching thresh- olds, respectively. For example, when our model recognizes two of three words of the signal to begin with correctly, resulting in 0.66 recall, 1.0 preci- sion, and thus 0.83 F1, this signal would count as partially matched but not exactly. All experiments run on the same test set per iteration, with varying training and validation splits, 10 repetitions each. In Table 3, we compare the base architecture
For evaluating the raw sense classification (see Table 4), we compare both sense hierarchies, the official PDTB version and our simplified version. We train sense models on gold signals of either one of the signal types, AltLex or Explicit. Further, we train third model on the joint set of signals and thus investigate whether one model has advan- tages from learning from the other types of signals, Comparing both sense hierarchies, the coarse and fine sense predictions are rather mixed. For the original hierarchy, the coarse senses have better performance, while we have the opposite in the

---

Both Model
Partial-Match
Exact-Match
4.2
Cross-Domail Discourse Signals
AltLex (simple)
72.28
69.09
For cross-domain experiments, we train the same models as presented in Section 4, but omit the ad- ditional test split and thus have more training data available, which is especially helpful for alternative lexicalizations. Further, we train ensembles of size three for each of the tasks, separate explicit and AltLex signal labeling as well as joint sense clas- sification, and combine their outputs by majority voting and mean of output distributions, respec- tively. In this way, the combination of all models had access to all available training data, and the overall prediction results should improve." 5
AltLex (std)
71.76
68.66
Explicit (simple)
94.85
94.76
Explicit (std)
94.25
94.16
Table 5: Sense Classification performances with error propagation: macro F1 average for partial and exact matches, comparison of Explicit and AltLex model for standard (std) and simplified (simple) senses.
case of our simplified hierarchy. We hypothesize the model to focus on the easier task (classifying 4 types) rather than the more complex fine-sense classification, thus resulting in better coarse-sense scores. However, as we are mainly interested in fine senses, seeing coarse senses more as an auxil- iary task, we refrain from further investigations.
Explicit Connectives For comparison with the originally annotated connectives, we utilize the con- nective heads from the PDTB and we merge our extracted connectives into this list. Especially for temporal connective heads such as after, before, and later, often combined with numerical values and duration (e.g. 10 minutes, three days), we eas- ily find high number of variants recognized by our model. Also, adverbial modifications in com- bination with because produce more variants that are not recognized in training data, e.g. probably, mostly, and precisely.
For fine senses, performance increased in our simplified hierarchy compared to the original one. This can indicate that the ignored features (implicit beliefs, speech acts, negation), are too hard to learn for the model due to the missing argument spans. Training both signal types jointly seems to have mainly negative effects, as there is neither an ad- vantage over specialized models nor the simplified sense hicrarchy leading to better results.
Within the set of extracted connectives that do not match the connective heads list, we found some adjective/adverb ordinals, c.g. first, secondly, etc.. that would rather belong to the group of AltLex signals. Other adverbs such actually, essentially, fortunatel,, and obviously, also occur occasionally, which seem to not have any real discourse usage. Interestingly. we notice that misspellings of connec- tives, e.g., besause, beacause, and becasue, were identified as discourse signals.
Table 5 summarizes the results of our sense clas- sification model jointly trained on both signal types. The results are based on extracted signals and thus are similar to the actual parsing performance from previous shared tasks, for example. As in raw sense classification experiments, the standard hicrarchy results are outperformed by the simplified version. For our subsequent experiments, we continue with this pipeline and the simplified sense hierarchy.
The complete automatically compiled list, where connective heads are associated with their modified variants, is presented in Appendix D.
At the CoNLL Shared Task (Xue et al., 2016), the best system for partial matching of explicit re- lations (Wang and Lan, 2016) achieved 69.21 Fl, while we report simplified scores of 94.85 F1. Sim- plified senses seem much closer to the Shared Task senses, although they still have a smaller number, however, we also have to note that this result is highly influenced by the results on argument ex- traction. Comparing our 78. 78.14 F1 for explicit sense classification, (Kido and Aizawa, 2016) achieves 90.22 F1, but with gold arguments and fewer senses classify.
Alternative Lexicalizations We identified 41,308 instances (9,359 types) of alternative lexicalizations in all corpora. AltLex signals from the official PDTB have an average length of 2.35 (+ 1.68), while our predictions' averages range from 1.88-2.72, though with lower standard deviation. Interestingly, signals extracted from ESSAY and SPEECH are longer on average compared to the original PDTB.
Most of the unseen AltLex phrases belong to the
'Our models are available under https: thub. ctm/ rknaebel/emnlp- 2023-disseee wwwcoorss-seeeeeeee signal-flloooso

---

group of causal signals where we identified 1268 variants outside the PDTB. For signals associated with DISJUNCTION, EQUIVALENCE, EXCEPTION, MANNER, and SUBSTITUTION, all belonging to the Expansion class, we could not identify new instances. For more complete list of senses and their associated AltLex phrases see Appendix E.
coherence relations are left for future work. In order to impose linear order on the PDTB relations in text, we anchor every relation at the position of the first token of its signal, which yields a sequence of senses. In addition, following Wachsmuth et al. (2015), we map text's sense flow also to a change flow by collapsing contiguous instances of the same sense into single one (thus change flow thus does not contain any bigram of identical senses). Notice that for trigrams or larger n- n-grams, this elimination can result in new flow patterns that have not been observed before the climination. We add special tags to the beginning and the end of a sense flow so that these positions are explicitly represented in n-gram models. The resulting n-grams for the text constitute the fea- tures for our "style" classifier, introduced in the next section. For illustration, we provide a sample of bigrams extracted from our corpora in Table 7; the frequencies demonstrate some noticeable dif- ferences between the different domains. Longer lists of the 50 most frequent sense flows and sense change flows are given in Appendix B.
Jointly Classified Discourse Senses Table summarizes the relative frequencies of individual senses per corpus ordered by the overall usage. As expected, the most frequent relation by far is the CONJUNCTION relation, which occurs particularly often in SPEECH, 50.94% and 64.18%, and AB- STRACT genres, 58.99% and 57.54%, respectively. CONTINGENCY relations (CAUSE, CONDITION, PURPOSE) are used above average in the genres ESSAY and ABSTRACT, not surprisingly, as these genres are rather argumentative in nature. We no- tice that AES is slightly below average, possibly because of but errors due to incorrect spellings (learner language). Also, the higher frequency of PURPOSE and MANNER in the UNSC domain al- lows conclusions about the way politicians argue in their debates. For NEWS, we recognize very simi- lar proportions of CONCESSION and the temporal relations SYNCHRONOUS and ASYNCHRONOUS. The latter represents the main elements of a news article: chronologically describing the events that happened.
5.2 Discourse Senses for Text Classification
Given the differences in proportions of sense n- gram patterns, we hypothesize that these patterns are useful for distinguishing domains from each other and that patterns are indeed stronger features than models that simply contain distributions of the single senses. We test this by comparing the performance of linear models trained on varying n--ram sizes.
5 Modeling Rhetorical Style by the Flow of Discourse Signal Senses
5.1 Discourse Sense Flows
Model We process our data and sample at most 804 documents per domain per run to reduce ef- fects by different corpus sizes. This allows us to omit the linear model's intercept (bias) in order to focus the combination of features exclusively. We split 20% of the data for evaluation purposes before we extract n-gram patterns (n (1... 4}) from the sense flows of all texts. Then we use their Feight-weighted frequencies for training maxi- mum entropy (logistic regression) models because of their simplicity and interpretability. Besides ex- tracting features from raw discourse sense flows, we also compare models trained on change flows (as described above).
Approaches that have been suggested for model- ing the "thetorical style" of text documents include trees formed by coherence relations, as for instance in Rhetorical Structure Theory (Mann and Thomp- son, 1988), and "move" or "zone" analysis, where the text modeled as a sequence of contiguous functional units (Teufel and Moens, 2002). For large-scale automatic analysis, however, RST pars- ing does so far not robustly yield good results across domains; and the "zone" approach delib- crately uses genre-specific labels.
Alternatively, here propose model the global rhetorical style of a document as linear se- quence of discourse relations. We apply the signal identification models from the previous section and thus limit our model, for now, to signaled relations and thus speak of signal "sense flows"; implicit
Results Our goal is to exploit differences in rhetorical style for distinguishing documents from
"This is twice as many documents as are availahle for PEC, where only the available 402 documents could be used.

---

Discourse Sense
Mean
PEC
ACL
MED
BBC
NYT
TED
CONJUNCTION
48,914
34.018
40.173
58,993
57.545
45.623
45.633
43.103
50.945
64.189
CONCESSION
14.380
12.967
12.257
12.260
11.035
21.383
19.229
20.132
11.235
8.919
CAUSE
14.148
17.622
21.006
14.233
16.025
9.044
9.082
11.695
19.138
9,490
CONDITION
6.628
20,951
7.143
1.024
1.934
6.342
5.676
6.667
6.429
3.489
ASYNCHRONOUS
5.722
2.955
3.064
4.500
4.135
9.972
10.110
8.855
5.232
2.679
SYNCHRONOUS
2,945
2.506
2.346
1.345
0.965
3.459
5.626
3.467
3.474
3.315
PURPOSE
1,564
1.342
1.902
2.043
1.350
0.807
0.566
0.658
0.857
4.555
DISJUNCTION
1,479
4.297
1.775
1.081
0.965
0.882
1.333
1.248
1.049
0.683
INSTANTIATION
1,340
1.301
6.065
0.725
0.195
0.401
0.812
1.504
0.518
0.543
CONTRAST
1,273
0.790
2.240
1.112
4.894
0.486
0.227
0.202
SUBSTITUTION
0,848
1.395
0,866
0.221
0.943
0.889
0.598
0.748
MANNER
0.528
1.520
0.515
0.266
0.235
0.119
0.707
LEVELOFDETAIL
0,150
0.000
0.042
0.281
0.198
0.281
0.072
0.183
0.023
0,267
SIMILARITY
0.129
0.129
0.063
0.018
0.023
0.110
0.198
0.154
0,212
EXCEPTION
0.003
0.000
0.000
0.000
0.000
0.000
0.007
0.016
0.002
0.000
EQUIVALENCE
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.000
0.001
0.000
# Documents Relations
59,545
1,562
402
1,754
8,390
2,198
38,755
1,820
2,455
2,209
825,581
24,061
4.732
22,757
26,216
19,946
540,406
19.154
139,047
29,262
Relations per Doc
13.86
15.4
11.77
12.97
3.12
9.07
13.94
10.52
56.64
13.25
Table Distribution discourse senses on individual corpora, relative frequency in percent, entries sorted by the macro average per relation. Absolute counts are shown the bottom, well their sum.
Flow Pattern
Mean
BBC
Conjunction-<paul
2.37
1.26
2.38
5.18
3.82
2.50
1.03
2.25
0.83
2.12
40
Cause-Condittin
0.46
2.01
0.79
0.07
0.11
0.18
0.14
0.17
0.49
0.15
44
Caddition-Cause
0.44
2.01
0.79
0.07
0.11
0.09
0.13
0.17
0.42
0.15
46
0.43
0.25
0.00
0.15
0.22
0.98
0.66
1.21
0.28
0.08
47
Amncession-Synchronee
0.42
0.25
0.00
0.30
0.28
0.71
0.67
1.04
0.35
0.15
48
Asynchromous-Coneesee
0.41
0.13
0.00
0.15
0.28
1.16
0.67
1.04
0.21
0.08
49
Synchronous-Comeee Concession
0.41
0.25
0.00
0.22
0.22
0.71
0.65
1.04
0.35
0.23
Table Selected discourse sense flow patterns of n-gram size with relative frequency per corpus.
the various domains and their associated genres that we summarized in Section 3. A random stratified baseline trained on our sampled data achieves 0.11 macro F1 (=0.006) for the domain classification and 0.25 macro Fl (=0.007) for genre classifica- tion, respectively, which is clearly outperformed by our feature- approach. Figure 2 shows our results for different n-grams. Larger n-gram sizes outperform single senses (unigrams). Furthermore, it turns out that bi bi-grams show the best single per- formance, while combination of 1, 2, and 3-grams achieves the best overall performance. In our periments, noticed a decrease in performance whenever 4-grams were involved as features. In this regard, we hypothesize that a weak point of our modeling is the missing implicit relations, which account for half of the annotations. When lineariz- ing the discourse senses, we have certain (implicit relation) gaps in the flow. With longer n-grams, the chance increases that extracted patterns con- tain missing discourse information. These patterns seem rather misleading for the classification model.
cicoman
cigerre
st-ceman
ahgerra
1.
Figure 2: Pattern-based domain and genre classification. "Scope" indicates combination of flow type (sense flows (sf), change flows (cf)), and domain/genre distinction.

---

discourse sense flows, for example in essay scoring or in text-level sentiment analysis.
Irpesancc
-pasii on Hannee
Synch 1noso-rpes
Finally, assuming the availability robust RST- style parsing, adapting the extraction of discourse sense flows to tree structures might be useful, either by lincarizing the tree or extracting more complex tree patterns than sense n-grams.
spar Conceron
Herre
Specs
Ethics
We do not see any critical aspects our work with respect to ethical considerations.
Figure 3: Feature importance of domain ACI. based on the weights of linear model.
However, we want to highlight that our statistical models are trained only on English news-wire arti- cles from the Wall Street Journal published around 1990. We are aware, that this highly influences the predictions made on other domains' corpora. Un- fortunately, other resourees for shallow discourse annotations are very limited.
As for comparing sense flows change flows, there is no significant difference for domain classifica- tion, while for genre classification the sense flows often outperform change flows.
Limitations
As an example of feature importance derived from the weights learned by the statistical model, Figure 3 shows the 15 most important features for the domain of ACL abstracts. Besides the impor- tance of causal relations, the model also learns that common introductions for ACL. abstracts are CON- CESSION, MANNER, and INSTANTIATION. Ap- pendix C gives a complete view the remaining classes and their most important features.
Studying only the signaled relations provides first but limited view of document's rhetorical style. We think that adding the remaining relation classes of the theoretical framework will give much more valuable insights, also reflected by better prediction scores and better separations.
The model for signal extraction was trained and evaluated entirely on English data. However, due to the usage of neural language model for encod- ing paragraphs, adapting the model to different language straightforward.
6 Conclusions
We introduced the first models for identifying both explicit and AltLex signals and their senses in the PDTB v3 corpus. The resulting system was then used on unseen corpora from different domains, and we propose the notion of discourse sense flows to model the rhetorical style of documents. Reduc- ing complete flows to n-grams of different sizes en- ables us discriminate certain genres and domains with a classifier that uses merely these automati- cally generated n-grams as features.
Acknowledgements
This research has been supported by the German Research Foundation (DFG) with grant number 45591 1521, project "LARGA" in SPP "RATIO", We would like to thank the anonymous reviewers for their valuable feedback.
References
Our work is currently limited to discourse re- lations that contain lexical signals, e.g., explicit connectives and alternative lexicalizations. How- ever, given a classifier for implicit relations (a task that received much attention in the literature in re- cent years), those relations can be integrated, and assume this can lead more explanatory sense flows and thereby to improved genre/domain clas- sification performance.
Khalid Al-Khatib, Henning Wachsmuth, Matthias Ha- gen, and Benno Stein. 2017. Patterns of argumen- tation strategies across topics. Proceedings of the 2017 Conference on Empirical Methods in Natu- ral Language Processing, pages 1351-1357, Copen- hagen, Denmark. Association Computational Lin- guistics.
Sahil Bakshi and Dipti Sharma. 2021. A transformer based approach towards identification of discourse unit segments and connectives. In Proceedings of
In addition, we anticipate further applications of

---

the 2nd Shared lask on Discourse Relation Pars- ing and Treebanking (DISRPT 2021), pages 13-21, Punta Cana, Dominican Republic. Association for Computational Linguistics.
René Knaebel and Manfred Stede. 2022. Towards iden- tifying tnernntive-lexicaizzen relations. In Proceedings of the 29th International Conference on Computational Linguistics, pages 837- 850, Gyeongju. Republic of Korea. International Committee on Computational Linguistics.
signals of discourse
Or Biran and Kathleen McKeown. 2015. Discourse planning with an n-gram model of relations. In Pro- ceedings of the 2015 Conference on Empirical Meth- ods in Natural Language Processing, pages 1973- 1977, Lisbon, Portugal. Association for Computa- tional Linguisties.
Murathan Kurfah. 2020. Labeling explicit discourse relations using pre- trained language models. In Text, Speech, and Dialogue, pages 79-86, Cham. Springer International Publishing.
Steven Bird, Robert Dale, Bonnie Dorr, Bryan Gibson, Mark Joseph, Min-Yen Kan, Dongwon Lee, Brett Powley, Dragomir Radev, and Yee Fan Tan. 2008. The ACL Anthology reference corpus: A reference dataset for bibliographic research in computational linguistics. In Proceedings of the Sixth International Conference Language Resources and Evaluation (LREC708), Marrakech, Morocco. European Lan- guage Resources Association (EIRA)..
Murathan Kurfah and Robert Östling, 2021. Let's be explicit about that: Distant supervision for implicit discourse relation classification via connective pre- diction. In Proceedings of the 1st Workshop on Un- derstanding Implicit and Underspecifted Language. pages 1-10, Online. Association for Computational Linguisties.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling se- quence data. In Proceedings of the Eighteenth In- ternational Conference on Machine Learning, ICML "01, pages 282-289, San Francisco, CA, USA. Mor- gan Kaufmann Publishers Inc.
Chloé Braud and Anders Søgaard. 2017. Is writing style predictive of scientific fraud? In Proceedings of the Workshop on Stylistic Variation, pages 37 37-42, Copenhagen, Denmark. Association for Computa- tional Linguistics.
Luke Gessler, Shabnam Behzad, Yang Janet Liu, Siyao Peng. Yilun Zhu, and Amir Zeldes. 2021. Dis- CoDisCo at the DiSRPT2021 shared task: A system for discourse segmentation, classification, and con- nective detection. In Proceedings of the 2nd Shared Task on Discourse Relation Parsing and Treebank- ing (DISRPT 2021), pages 51-62, Punta Cana, Do- minican Republic. Association for Computational Linguistics.
Ziheng Lin, Hwee Tou Ng. and Min- Yen Kan. 2011. Au- tomatically evaluating text coherence using discourse relations. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Hu- man Language Technologies, pages 997-1006, Port- land, Oregon, USA. Association for Computational Linguistics.
Ziheng Lin, Hwee Tou Ng, and Min- Yen Kan, 2014. A pdtb-styled end-to-end discourse parser. Nat. Lang. Eng.. 20(2):151-184,
Derek Greene and Pádraig Cunningham. 2006. Practi- cal solutions to the problem of diagonal dominance kernel document clustering. In Proceedings of the 23rd International Conference on Machine Learn- ing, ICML '06, page 377-384, New York, NY, USA. Association for Computing Machinery.
Yang Janet Liu and Amir Zeldes. 2023. Why can't 't dis- course parsing generalize? a thorough investigation of the impact of data diversity. In Proceedings of the 17th Conference of the European Chapter of the As- sociation for Computational Linguistics, pages 3112- 3 3130, Dubrovnik, Croatia, Association for Computa- tional Linguistics.
Yangfeng Ji and Noah A. Smith. 2017. Neural dis- course structure for text categorization. In Proceed- ings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 996-1005, Vancouver, Canada. Association for Computational Linguistics.
Wanqiu Long and Bonnie Webber. 2022. Facilitating contrastive learning of discourse relational senses by exploiting the hierarchy of sense relations. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing, pages 10704- (0716, Abu Dhabi, United Arab Emirates. Associa- tion for Computational Linguistics.
Yusuke Kido and Akiko Aizawa. 2016. Discourse rela- tion sense classification with two-step classifiers. In Proceedings of the CoNLL-16 16 shared task, pages 129 135, Berlin, Germany. Association for Computational Linguistics.
William C. Mann and Sandra A. Thompson. 1988. Rhetorical structure theory: Toward functional the- ory of text organization. Text & Talk, 8:243 - 281.
René Knaebel and Manfred Stede. 2020. Contextual-
ized embeddings for connective disambiguation in shallow discourse parsing. In Proceedings of the First Workshop on Computational Approaches to Dis- course, pages 65-75, Online. Association for Com- putational Linguistics.
Emily Pitler and Ani Nenkova. 2009. Using syntax to disambiguate explicit discourse connectives in text. In Proceedings of the ACL-IJCNLP 2009 Conference Short Pupers, pages 13-16, Suntec, Singapore. Asso- ciation for Computational Linguistics.

---

Emily Pitler, Mridhula Raghupathy, Hena Mehta, Ani Nenkova, Alan Lee, and Aravind Joshi. 2008, Eas- ily identifiable discourse relations. In Coling 2008: Companion volume: Posters, pages 87-90, Manch- ester, UK. Coling 2008 Organizing Committee.
Bonnie Webber. 2009, Genre distinctions for discourse in the Penn TreeBank. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACI. and the 4th International Joint Conference on Natu- ral Language Processing of the AFNLP, pages 674 682, Suntec, Singapore. Association for Computa- tional Linguistics.
Rashmi Prasad, Aravind Joshi, and Bonnie Webber. 2010. Realization of discourse relations by other means: Alternative lexicalizations. Coling 20/0: Posters, pages 1023-1031, Beijing, China. Coling 2010 Organizing Committee.
Bonnie Webber, Rashmi Prasad, Alan Lee, and Aravind Joshi. 2019. The penn discourse treebank 3.0 annota- tion manual.
Rashmi Prasad, Bonnie Webber, and Alan Lee. 2018. Discourse annotation in the PDTB: The next genera- tion. In Proceedings 14th Joint ACL ISO Workshop on Interoperable Semantic Annotation, pages 87-97, Santa Fe. New Mexico, USA. Association for Com- putational Linguistics.
Nianwen Xue, Hwee Tou Ng. Sameer Pradhan, Attapol Rutherford, Bounie Webber, Chuan Wang, and Hong- min Wang. 2016. CoNLL 2016 shared task on multi- lingual shallow discourse parsing. In Proceedings of the CoNLL-16 shared task, pages 1-19, Berlin, Ger- many. Association for Computational Linguistics.
Mirco Schoenfeld, Steffen Eckhard. Ronny Patz, Hilde van Meegdenburg, and Antonio Pires. 2019. The UN Security Council Debates.
Amir Zeldes, Yang Janet Liu, Mikel Iruskieta, Philippe Muller, Chloé Braud, and Sonia Badene. 2021. The DISRPT 2021 shared task on elementary discourse unit segmentation, connective detection, and rela- tion classification. In Proceedings of the 2nd Shared Task on Discourse Relation Pursing and Treebanking (DISRPT 2021), pages I 1-12, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Wei Shi and Vera Demberg. 2017. On the need of cross validation for discourse relation classification. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Lin- guistics: Volume 2. Short Papers, pages 150-156, Valencia, Spain. Association for Computational Lin- guistics.
Christian Stab and Iryna Gurevych. 2017. Parsing argu- mentation structures in persuasive essays. Computa- tional Linguistics, 43(3):619-659.
Evgeny Stepanov and Giuseppe Riccardi. 2014. To- wards cross-domain PDTB-style discourse parsing. In Proceedings of the 5th International Workshop on Health Text Mining and Information Analysis (Louhi), pages 30-37, Gothenburg, Sweden. Association for Computational Linguistics.
Simone Teufel and Mare Moens. 2002. Articles sum- marizing scientific articles: Experiments with rele- vance and rhetorical status. Computational Linguis- tics, 28(4):409-445 445.
Henning Wachsmuth, Johannes Kiesel, and Benno Stein. 2015. Sentiment flow a general model of web re- view argumentation. In Proceedings of the 2015 Conference Empirical Methods in Natural Lan- guage Processing, pages 601-611, 1, Lisbon, Portugal. Association for Computational Linguistics.
Jianxiang Wang and Man Lan. 2015. A refined end- to-end discourse parser. In Proceedings of the Nine- teenth Conference on Computational Natural Lan- guage Learning- Shared Task, pages 17-24, Beijing, China. Association for Computational Linguistics.
Jianxiang Wang and Man Lan. 2016. Two end-to-end shallow discourse parsers for English and Chinese in CoNLI-2016 shared task. In Proceedings of the 1oNLL-16 shared task, pages 33-40, Berlin, Ger- many. Association for Computational Linguistics.

---

A Hyper Parameter Settings
zo00 + ... + 2z02 computed for each class and each zy input pattern multiplied by its corre- sponding weight 0j. We calculate the importance of each coefficient with respect to the learned class by e Note that we omit the intercept in order to entirely focus on the linear combination of features. Figure 4 shows the 15 most important features per class.
All models are implemented in PyTorch with addi- tional help from the Transformers library. Through- out our experiments, we use roberta-base architec- ture for encoding paragraphs. In the following, we summarize the settings for both model types that are finally used to extract signaling phrases and classify their associated sense.
Signal Labeling The signal labeling model uses a batch size of 16. We start by optimizing the lin- ear classifier and use Adam with a learning rate 5e 5 for the language model and le - for the linear classification layer. During iterations, the gradient norm is clipped to 5. The model is trained for 10 epochs, early stopping if there is no improvement for 3 epochs. As recommended by linear scheduler for the learning rate and 50 warmup steps. We par- tially tine-tune the language model, in particular, we freeze the first half of the layers and only fine- tune the second half. Also, we use a dropout of 0.3 right after the language model and before the linear classification layer. For measuring performance, we chose the macro F1 score. After the training has stopped, continue with CRF layer optimization. Therefore, we reduce the learning rate of the lan- guage model to 5c-5 and use learning rate of lc-4 for the CRF. The number of training epochs, early stopping, and validation performance are handled as before.
fine-tuning RoBERTa,
Sense Classification For sense classification, the batch size is 16. We have two hidden layers of size 256 and 64 before the actual classification layer. Each layer is followed by 0.3 dropout layer. The model is optimized by Adam and has learning rate of le-4. The loss combination coarse sense loss and fine sense loss in the ratio to We also use early stopping after 3 iterations without performance gain. Performance is measured by the mean of coarse and fine sense macro FI.
B Frequent Sense Flow Patterns
In Table 9 and Table 8, we present a list of the 50 most frequent patterns, sense flows, and change flows, respectively. The Tables are sorted by the macro average relative frequency of all domains.
Feature Importance
Linear regression models approximate linear rela- tions between inputs and outputs such that y =

---

Flow Patlern
Mean
ABS
ACL
BBC
PEC
NYT
WSI
MED
TED
UN
ComoswiionCConjuuae
4.26
2,88
3.75
5.45
6.03
3.71
6.15
3.63
3.27
3:48
Conjenction Saaiiinn-ooccssinn
4.03
Conjunctrro-qud>
3.55
1.56
7.43
3.57
3:45
1.47
3.16
6.04
1.18
4.11
Conjunction Cause
3.47
3.36
2.09
1.53
9.48
1.19
1.99
2.31
5.43
3.87
Cuno-Coojuuctton
3.44
3.36
2.16
9.48
1,22
2.16
2.09
5.08
3.79
<qadb- Conjinction
2.75
0.96
4.04
2.64
1.72
1.24
2.82
5.99
1.18
4.19
Conjunction Concession Conjunction
Condittoe-ooeuunttion
1.75
4.56
0.58
1.28
3:45
0.91
1,50
0.66
1.67
1.18
Synchrenous-Cmmuun Conjunitionn
10
Conjunction Condition
1.58
3.96
0.65
1,28
2.59
0.90
1.33
0.71
1.53
1.26
11
Conjencioo-AAsyaee Asynchrvoous
1,54
1.73
2.13
0.86
1,90
2.33
1.26
1.11
0.87
12
Asynchronous-C Coejunction
1.66
2.47
0.86
2.16
1.67
13
Conjunction Cause Conjunction
1.46
1.20
0.94
0,60
3.45
0.47
0.66
0.93
2.72
2.21
<pad> Concession
15
Amjjunctie-Synchhnnne
1.39
0.84
1.15
1.62
0.86
1,53
1,66
1.10
1.60
2.13
16
Concession-CC Cause
0.87
0.71
1,50
1.18
0.55
17
Cuss <pal>>
1.09
0.58
0,60
2.59
0.38
0.83
1.32
1.60
1,03
18
CunseConeession
1,05
1.32
0,50
0.94
2.59
0.65
1.33
0.44
1.18
0:47
19
ConsessoonCoonuuncciin Conjunetioo-Coneeeee
0.98
20
<pad> Cause
0.95
0.72
0.87
0.60
1.72
0.38
1.00
0.82
1660
0.87
Mamnss-Conjunction
22
Cause Conjunction Cause
0.81
0.72
0.29
0,17
2.59
0.18
0.33
0.38
1.81
0.79
23
Concession Conjunction
0.24
1,37
0.86
0.45
1.00
0.14
0.58
24
Concession Condition
0.78
0.22
0.85
0.86
0.59
1.16
0.38
0.56
0.32
25
Concssiion-quae
0.76
0.36
0.65
1,28
0.86
0.62
1.33
1.26
0.21
0.24
Conjunction Manner
27
<pud- Concession-Conie Conjunction
0.65
0.24
1.73
0.77
0.86
0.33
0.50
1.10
0.07
0.24
28
<padb Conjunction- Concersion
0.65
0.79
0.38
0.21
29
Concession Conjunction Cause
0.65
0.48
0.50
0.43
1.72
0.35
0.50
0.38
0.97
0.55
30
0.65
0.36
0.87
0.85
8.00
0.89
0.83
0.60
0.97
0.47
31
Comjunciion-Coolliiiot Conjunctiin
0.63
0.22
0.51
0.86
0.35
0.50
0.22
0.70
0.71
32
Parpose- Conjunction
0.63
0.60
0.72
0.17
0.86
0.15
0.17
0.33
0.42
2.29
Contrast Conjunction
34
Conjunction-Coeftee
0.60
0.24
0143
0.43
0.86
0.25
0.66
2.09
0.21
0.24
35
Conjunction Purpose
0.60
0.48
0.58
0.26
0.96
0.15
0.35
2.37
36
Case- C cojjunction<ppll
0.58
0.94
0.34
0.86
0.14
0.33
0.58
0.42
0.95
37
0.56
0.36
0,36
0.68
0.00
0.09
0.50
0144
0.84
1.18
38
Cause- Condition
0.54
0.07
0.86
0.17
0.70
39
Asynchromous Concession
0.53
0.24
0.14
1,36
0.00
0.82
1.33
0.33
0.35
0.16
Conditioe-Cosceee Colioos--occcsnnnnn
41
Conditico-Canee
0.51
2.28
0.07
0.17
0.86
0.15
0.17
0.11
0.56
0.24
42
Concession Synchronous
0.50
0.29
0.85
0.00
0.81
1.16
0.49
0.24
43
Muns-Conjunctions Concession
0.50
0.14
0.34
0.86
0.36
0.66
0.77
0.39
44
Concession Asynchronous
0.48
0.24
0.14
1.11
0.00
0.81
1.33
0.22
0.35
0.16
Synchronous-C Concestion
46
Concession Cause Conjunction
0.47
0.48
0.43
0.43
0.86
0.29
0.50
0.33
0.63
0.32
Conjunction-C coccesioo--padd
48
Instuntiaiion Conjunction
0.41
0.12
0.14
0.00
2.59
0.11
0.33
0.00
0.21
0.16
49
Conjuncioo Cause-cpal>
0.41
0.36
0.43
0.17
0.86
0.10
0.17
0.49
0.39
50
0.40
Table Relative frequency top 50 discourse sense flows with n-grams sizes are sorted by their macro average over all corpora.
The table entries

---

Flow Pattern
Mean
AES
AC1
BBC
PEC
NYT
WSJ
MED
TED
SenjusstioeCCvejuncttioo
9.09
5.65
8.95
8.21
12.70
5.21
8.29
9.11
8.83
14.90
Cunjunction- Kajjunciioe-Coqqjustiiee
Concession-Cnwn Crejunccion
3.30
2.39
2.81
4.37
4.76
2.96
5.01
2.75
2.43
2.19
Conjunction Concession
3.15
2.51
1.63
4.28
3.97
3.05
5.35
2.75
2.57
2:27
Cuos-Conjunction
2.64
2,89
1.85
1.25
7.14
0.98
1.63
3.76
2.34
6
Conjunction-Cane
2.63
2,89
1.78
1.25
7.14
0.96
1.58
1.80
3.96
2.34
Conjunctico- <ppl>>
2.12
quub-Conjunction
1.82
0.75
81
1.78
1.59
0,86
1,90
3.77
0.76
2.19
Conjunction-Core oonprstoeesoeliiccoooo-Coommaamo conursaeooccceiioooooooooooooooo
10
Condition Caaooo--Cnniggg Conjunction
1.38
3.89
0.52
1.07
2.38
0.74
1.21
0.56
1,25
0.76
11
Conjunction Condition
1.32
3.39
L.07
2.38
0.73
1.21
0.56
1.18
0.83
MechroonessCoojjuuna
1.26
1.45
1,36
1.32
1.29
Conjunction Asynchrooous 13
1.23
0.88
1.48
L.69
0.79
1,50
1.90
L.01
1.32
0.53
Conjunction-C Serrocdice-CCccsssiiiee
15
Asynchronous- Coejunntonnnn
1.19
0.63
1.33
1.96
0.79
1.51
1.73
D.96
1.25
0.53
Cause- Canse
1.18
1,76
0.45
0.98
17
Concession Concession
1.14
1,13
0.52
2.05
0.79
1.33
2.59
0.84
0.63
0.38
18
Concession Comoesion-Conjuustiioe Conjuuctiie-Conjjueee
1.09
0.63
1.11
1.61
1,01
1.55
1.01
0.97
1.13
Conjunction SepiuactionSSnchooontt
1.18
20
Conjunction Cosjunction <pud>
1.06
0.38
2.74
LO07
0.79
0.41
0,86
L.90
0.28
1.21
Conjunction CoeproooooojuncioooooCooo
22
quad Concession
1.02
0.50
2.59
1.16
1.59
0.53
1.04
1.35
0.14
0.30
23
Conjunction. Cause-Cone Conjunction
0.99
0.88
0.45
2.38
0.35
0,52
D.67
1.67
1.21
24
Comcesson Cause
0.92
1,38
0.74
0.89
1.59
0.58
1.21
0.62
0,90
0.38
25
CauseCCon Conjunction-Connnnciin
0.82
0.75
0.59
0.36
1.59
0.32
0,52
D.56
1.46
1.21
27
Conjunctice-Cveippn Crejiiio-CCeennctinnCCCueee
0.77
0.63
0.44
0.36
1.59
0.30
0.35
0.62
1.53
1.13
28
spad-Coon Conjunction-CC Conjanction
0.69
0.13
0.80
0.33
1.74
0.28
1,36
29
Manner Conjunction
0.69
0.25
2.37
0.36
0.79
0.32
0.35
0.73
0.21
0.83
30
Conoesion-Cxeddtioo
0.64
1,76
0.22
0.71
0.79
0.48
0,86
D.34
0.42
0.15
31
Concasion-quub
0.38
0.52
0.98
0.79
0.49
0.96
0.14
0.15
32
Conjunction Manner
0.58
0.25
1.70
0.36
0.79
0,28
0.38
0.56
0.21
0.68
Cuss <pal>
34
<uub-Cmuse
0.55
0.63
0.67
0.36
0.79
0.23
0.52
D.56
0.83
0.38
35
Concession Consenction.Coecession Concession
0.53
0.38
0.89
0.64
0.34
0.28
0.23
36
Contrast Conjunction
0.51
0.25
0.37
0.45
0.79
0.21
0.69
1.52
0.14
0.15
37
Conjsse-oojuunctioon
0.48
0.50
0.59
0.18
0.79
0.12
0.17
D.22
0.28
1.44
Conjunction-Coree Conttrio
0.13
0.27
0.79
0.20
1.57
0.15
39
Conjunction Conjunction Conjunction <pad>
0.46
0,13
1.33
0.45
0.00
0.18
0.35
0.84
0.14
0.76
Cuse- Cond-Conniion
41
Conditioo-C Coniiiso-CCocesssonn
0.45
1.38
0.15
0.54
0.00
0.42
0.86
D.17
0.35
0.15
42
Conjunction Condition Conjunction
0.45
1,13
0.15
0.36
0.79
0.26
0.35
0.17
0.49
0.38
43
Conjunctioe-Purooc
0.50
0.18
0.79
0.12
0.17
D.11
1.44
44
Condition Cause
0.44
2.01
0.07
0.09
0.79
0.13
0.17
D.11
0.42
0.15
Monjunetiio-AAyychmre Conjunctionn
46
Concession Asynchromous
0.43
0.25
0.15
0.98
0.00
0.66
1.21
0.22
0.28
0.08
47
Conession-Synchree
48
Asynchronous Concession
0.41
0.13
0.15
1.16
0.00
0.67
1.04
0.28
0.21
0.08
49
Synchromous Concession
0.41
0.25
0.71
0.65
1.04
0.22
0.35
0.23
Comdditoo-Coejjncciinnee Conterriion-Cnnnntttonn
1.01
0.36
0.38
Table Relative frequency of top 50 discourse sense sense,flows with n-grams of sizes 2 to The table entries are sorted by their macro average over all corpora.

---

Carre Con cenornttonnn
Or
Cano Seetie
(a) AES
(b) ACL
Dapa Ooleenters
Tedla fud
(c) BBC
(d) PEC
Oe
(e) NYT
(1) WSJ
Coseuciin wrtee Cremmare
Ond
MED
(h) TED
250
Figure 4: Feature importance plots for genre classification.

---

D Differences of Connectives and their Modifications
In the following, we automatically compile a table of connectives from the explicit relations we identified in the corpora. Similar to the annotation manual (Webber et al., 2019), we show the connective heads (left) and their corresponding modified occurrences (right) along with the counts. Here, we exclude all modified connectives identified in the original PDTB3 data. Due to the automatic mapping from modified occurrences to their possible heads, errors involve mismatches and non-explicit signals that have been identified.
about
2
ahout time (I); about the same that (1)
after
712
not long after (65), the day after (48), years after (46), hours after (44), days after (39), four years after (29). just days after (29), three days after (28), moments after (27), six months after (27), just hours after (25), four months after (22), five months after (18), three weeks after (16), one week after (16). few years after (15). five days after (15), weeks after (14), one year after (14), well after (13), one month after (13). eight years after (13), just weeks after (12), six days after (12). seconds after (11), a decade after (1 several months after (11), nine years after (1 (11), just months after (10), only days after (9), less than a year after (9), an hour after (9), less than week after (9), after (8). 20 years after (8). six weeks after (8), in the days after (8). less than 24 hours after (8), only hours after (7): just a week after (7), ten days after (7), two hours after (7), decades after (7), a night after (7), sometime after (6), less than two years after (6), just two years after (6), ahout a month after (6), just four days after (6), just a few months after (6), I...]
afterward
10
immediately afterward (12), only afterward (3), not long afterward (3), long afterward (2), even afterward (1), right afterward (1), week afterward (1), half hour afterward (1), before and afterward (1), very soon afterward (1)
afterwards
4
immediately afterwards (3), soon afterwards (1), not long afterwards (1), right afterwards (1)
also
4
not also (5), even also (1), only also (1), so also (1)
and
4
alternately and (7). simultaneously and (2), either and (1), at one and the same time (1)
as
84
(66), as such (25), as opposed to (24), only as (19), as fact (16). example (12), inasmuch (11), as example (8). fast (7). as to (5), as bonus (5), right as (5), exactly as (4), as soon (4), almost as (4), far as (3), least as (3), soon as (3),
consequence (69).
quickly as (3), not as (3), often as (3), precisely as (3), long (3), immediately as (2), as of fact
(2), as result (2), as early as (2), perhaps as (2), somewhat as (2), first as (2), bonus (2), just as (2), as an instance (I), where as (I), at long as (1), as long (I), while the same time as (l), as all as (I), just as fast as (I), both as (I), almost preciscly as (I), for (I), a (I), anxicty (I), because as (1), as long that (1), almost as quickly as (1), as a well as (1), hard as (1), [...]
as result
4
possibly as result (2), partly result (2). perhaps as a result (1), presumably as result (1)
as if
3
almost if (8), just as if (4), much as if (1)
as long as
5
just as long as (6), as long as then (1), at least as long us (1), as long as three months after (1),
for as long as (1)
as soon as as though as well as at the same time bocause
3
almost as soon as (23). nearly as soon as (I). perhaps as soon as (1)
1
almost though (1)
I
both as well as (2)
3
at the same time as (5), all at the same time (2), almost at the same time (2)
66
precisely hecause (57), mostly hecause (46), not least because (44), probably hecause (37),
maybe because (32). not just because (23), possibly because (19), either because (16), all because
(16), solely because (11), both because (10), specifically because (8), small part hecause (7),
not so much because (7), often because (7), whether because (6), in large measure because (6),
part because (6), at least in part because (5). not necessarily because (5), chiefly because (4), usually because (4), sometimes because (4), most likely because (3), more because (3), purely because (3), almost because (2), supposedly because (2), not simply because (2), ostensibly
because (2), reportedly because (2). less because (2), some because (2), for because (1). because than (1), no because (1), first off because (1), initially because (1), most notably because (1), not
merely because (1), obviously because (I), necessarily because (1), generally because (1), many
because (1), perhaps in part hecause (1), principally because (1), almost certainly because (1),
adventurous hecause (1), at in part because (1), in no small part hecause (1), [...]
because of
3
all because of (2), supposedly because of (I), in part because of (1)

---

before
303
not before (53), well before (48), hours before (33), a year before (25), right before (24), moments before (19), days before (18), months before (17), few days before (17), just hours before (16), minutes before (15), not long before (14), just weeks before (13), month before (12), a day before (12), four days before (12). two weeks before (12), weeks before (12), one day before (11), the day before (11), a few months before (11), an hour before (10), three days before (10), six months before (10). two hours before (9), three months before (9), a few hours before (8), only days before (7), the year hefore (7), three years before (6), in the months hefore (6), soon before (6), four months before (6), just minutes before (5), 10 days before (5), five years before (5), five months before (5), three weeks before (5), five days before (5), way before (5), 30 minutes before (4), less than two months before (4), six days before (4), several days before (4), a full year before (4), one year before (4), four years before (4), about a month before (4),
less than two weeks before (4), less than a week before (4), [...]
before and after
3
just before and after (2). both before and after (1), immediately before and after (1)
both and
2
both because and (1), both and also (1)
but
11
not but (29). only but (4), not merely but (4), just but (3), not so much but (2), not along but (1), not simply but (1), merely but (1), perhaps but (1), instead but (1), but example (1)
but ulso
6
not but also (10), only but also (3), not simply but also (3), not merely but also (3), just but also (I), hoth hut also (1)
by
.59
by the same token (31), either by (26), in part by (23), perhaps by (18), mainly by (144, primarily
by (13), not by (13), possibly by (10), especially by (9), first by (8), often by (8), usually by (7), whether by (5), specifically by (4), presumably by (3), both by (3), in large part by (3), by
extension (3), notably by (2), most by (2), part by (2), in by (2), generally by (2), most notably by
(2), precisely by (2), say by (2), by end (2), less by (2), apparently by (2). if by (2), principally by
(2), by the end (2), by analogy (2), including by (2), essentially by (1), even by (1 (I), sometimes
by (1). by example (I), basically by (1), by gum (1), by contract (1). by the time (1), now by
(1), typically by (1), by sharp contrast (I), by time (1), lately by (1), visibly by (1), by extreme contrast (1), definitively by (1), [...]
earlier
2
a moment earlier (1), two earlier (1)
either or
2
either else (3), either because or (1)
else
2
else (47), either else (1)
even after
I
not even after (3)
even hefore
2
perhaps even before (1), sometimes even before (1)
even
not even if (1)
even when
I
not even when (8)
for
9
for comparison (8), for starters (4), just for (4), for an example (3), simply for (2), partly for (2),
for instant (1), for tho" they (1), for contrast (1)
from
5
apart from (2), not from (2), even from (1), from the same time (1), 10 months from (1)
given
2
especially given (6), particularly given (2)
if
36
if than (59), not if (29), if not (12), at least if (7), regardless if (5), if and only if (3), either if (2), exspecially if (2), and if (2), perhaps if (2), maybe if (2), simply if (2), even it if (2), just if
(2), certainly if (2), least not if (2), irrespective if (1), for if (1), despite if (1), instead if (1), whether if (1), rather if (1), or rather if (1), than if (1), crucial if (1), almost if (1), like if (1),
primarily if (1). or if (1), now if (1), particuarly if (1), badly if (1), old if (1), /f (1), if anything
(1), sorry if (1)
if only
4
if only because (43), if only by (3), if only so (2), now if only (1)
if then
1
especially if then (1)
in
in conclusion (738), in summary (32), in essence (11), in parallel (10), in between (8), the other hand (7), in hrief (6), in the same way (6). in total (5), in consequence (4), in retrospect (4), as in (4), in the interim (4), in stark contrast (4), in effect (4), in opposite (3), in example (3), in result (3), in word (3), even in (3). in the alternative (3). in the same breath (3), in reality (2), in other word (2), in a word (2), in the same time (2), in spite of (2), in sharp contrast (2), in words (2). in away (1). in specific (1). in vivid contrast (1). in compensation
(1), in the contrary (1), in
(1). in truth (1). in the mean time (1), in sense (I), in closing (1), in the same manner (1), in the same token (1)
which case (1), in of (1), in future (1). only in
in addition
I
addition to (2)
in case
1
especially in case (1)
in order
2
just in order (2), in order for (1)

---

in particular
5
particular when (7), in particular by (4), in particular because (3), in particular as (3), particular since (2)
in the meantime
1
in the meantime (1)
instead
1
if instead (2)
later
72
a later (48), a short time later (14), two later (12), three later (11), a year later (10), moments later (10), only later (10), a minute later (9), hours later (8), an hour later (7), two years later (5), two minutes later (5), month later (5), minutes later (5), four years later (4), several later (4), five later (3), six months later (3), an later (3), two hours later (3), two months later (3), a years later (3), a time later (2), a few minutes later (2), a few days later (2), nine later (2), minutes later (2), four months later (2), three months later (2), several hours later (2), five minutes later (2), about an hour later (2), four later (2), a moment later (2), three years later (2), a months later
(2), few hours later (2), short later (1), less than half an hour later (1), less than minute later
(I), minute later (I), much later (1), 50 minutes later (1), few weeks later (1), seven later (l), a
few months later (1). years later (1), several minutes later (1), bit later (1), a day later (1), [...]
like
10
just like (137), much like (19), almost like (8), kind of like (2). kind like (2), bit like (2), rather like (1), somewhat like (1), pretty much like (1), or like (1)
meantime
1
meantime (1)
not just but
not just because but (1)
not only
13
not only because (65), not only and (7), not only when (4), not only by (4), not only while (2), not only if (2), not only or (1), not only it (1), not only then (1), not only that (1), not only indeed (I), not only since (1), not only as (1)
not only but
not only because but (1)
now that
3
especially now that (10), even now that (3), particularly now that (1)
on
on other (11), on the flip side (8), on the other side (7), on one hand on the other (5), further (3), the other end (3), the same note (2), other hand (2), on the other hands (2), based on (2), on another hand (I), on contrary (1), on the other had (I), on one hand
other (I),
on other extreme (1), on one on the other side (1). on upside (1). late on (I), on account (I), on one hand (1), immediately on (1), on the downside (1), on the completely other side (I),
on the opposite (1). on on the other (1), on side (1), earlier on (1)
the hand
2
the one hand (16), the hand the other (4)
on the other hand
1
on one hand on the other hand (6)
once
8
even once (7), only once (5), especially once (4), at once (3), particularly once (2), once again (1), at least once (1), all at once (1)
only
5
only that (2), only through (2), only cause (1), only order (I), only ifit (1)
6
whether or not (27)./ or (22), whether or (2), and or (I), cithers (1), the same (1)
otherwise
I
otherwise (1)
rather
I
rather elders (1)
rather than
2
rather than by (1), rather than because (1)
simultancously
and simultancouslly (1)
since
33
not since (16), at least since (9), almost since (5), years since (5), probably since (3), two years
since (2), months since (2), decades since (1), little over year since (1), even since (I), a century since (1), eight years since (1), before and since (1), only since (1), half since (1), all since (1). five years since (1). months since (1), roughly since (1), hours since (1), both since (1), half since (1), almost a year since (1), in since (1), ill since (1), more than two years since
(1), in weeks since (1), two days since (1), half years since (1), months since (1), almost two months since (1), ten years since (1), years since (1)
so
15 just (48), in part so (3). way (2), not so (2). mostly so (2), only so (2), all so (2), so the (1), precisely (I). so long (1), specifically so (1), mainly so (1), largely so (1), not just (1), especially so (1)
50 as
I
not so as (1)
so that
5
so that way (5), just so that (5), largely so that (l), presumably so that (I), often so that (1)
specifically
I
more specifically (56)
still
L
later still (1)
that is
3
that is say (3), that is to say (2), that is if (1)

---

then
then again (112). only then (72), unless then (14), once then (10), just then (10), rather then (7), since then (6), right then (6), then (4), until then (4), becuase then (3), because then (3), before then (2). then (1), then (1), due then (I), it then (1), from then (D, then (1), any then (1), anytime then (1), for then (1), other then (1), although then (1), at then (1)
thereafter
2
thereafter (9),
thereafter (1)
though
3
event though (1), what though (1), similar though (1)
thus
1
only thus (1)
2
at least till (2), not till (1)
unless
2
not unless (7), until and unless (1)
until
9
not until (22). unless and until (8), or until (2), up until (2), perhaps until (2), right until (2),
often until (1), unless until (I), supposedly until (1)
upon
3
immediately upon (2), almost upon (1), only upon (1)
when
62
not when (34), right when (13), perhaps when (5), mostly when (5), both when (4), immediately
when (4), starting when (3), specially when (3), mainly when (3), except when (3), where and
when (3), as and when (3), preferably when (2), expecially when (2), in when (2), apparently
when (2), not just when (2), at when (2), notably when (2), simply when (2), either when (2), sometimes when (2), now when (2), precisely when (2), exactly when (2), also when (1), specifically when (1), even sometimes when (1), exspecially when (1). primarily when (1), less than 24 hours when (I), espocially not when (1), most famously when (1), all when (1), especially now when (1), most commonly when (1), beginning when (1), around when (1), more when (I), a week when (1), ideal when (1), generally when (I), huilt when (1), most when (1), rarely when (I), often when (I), necessarily when (I), most recently when (I), initially when (I), certainly when (1), [...]
when and if
I
only when and (1)
whenever
wherever and whenever
where
5
particularly where (7), especially where (4), where (2), only where (1), when and where (1)
whether
I
whether not (1)
while
14
all the while (53), all while (28), only while (4), often while (3), both while (2), especially while
(2), at least while (2), not while (2), particularly while (1), apparently while (1), preferably while
(1), sometimes while (1), all this while (I). while (1)
with
what with (12), not with (9), only (6), albeit with (2), sometimes with (2), as with (1). with (1), often with (1), because
(1), not with (1), least with (1), literally with (1), perhaps with (1), especially now with (1)
without
even without (27), without (6), all without (6), with without (2), apparently without (2), preferably without (2), usually without only without (I), especially without (1), reportedly without (I), always without (I), again without (I), also without (I), simply without (I), almost without (I)

---

E Alternative Lexicalizations and their associated Senses
Below we present a selected number of alternative lexicalizations grouped by their predicted sense. We give exact occurrences in parentheses, while the second column presents the number of variants.
Comparison.
Comparison.Concession Concession
35
the only difference is that (5), it's as simple as that (2), but my point that (1), it's as easy as that (I, what am trying to say (I), with all that has been said (I) the biggest difference is that (1), the main difference is that (1), what is unusual is that (1). what has changed is that (1), what is different this time is that (I,, with the only difference being that (1), that of course was when (1), the one difference is that (1), one significant difference is that (1), the only difference being that (I), the only change the years is that (1), the difference that game was (1). the difference now is that (1), what is different is that (1), that's 's kind of the point (1) (1), but the point is that (I), all of this to say (1), all of which is to say (1), it's a combination of this (I), what this is actually saying is that (1). what's 's interesting is that (I), I think the point is that (1), what's happening here is (1), what was so funny about it was that )) the only difficulty is that (1), the thing about that was (I), what's 's different now is ), that shift in perspective is like (1), what is different today is that (1)
Comparison.Comm Contrast
41
this is in contrast to (5), that is significantly lower than (2), this was a significant jump (I), this is in stark contrast with (1), compared that new cost with (1), that's 30 percent higher than (1), that number is much higher than (I), the number is down from (1), the same is true for (1), likened the voting haiti to (1), that figure is comparable to (1), the figure was up from (1), at the other end of the spectrum (I), that is far higher than (1), the quarterly growth rate compared with (1), this "conversion rate" compares with (1), the inflation rate compares to (1), the difference this time was that (1), that is very different from (1), that is almost as many as (I), this is a change from (1), that is dramatic shift (I), that figure is slightly higher than (I), that total was significantly down from (I), that is in contrast to (I), the only difference is that (I), that is major increase from (I), that number is roughly the same as (1). that is in comparison to (I), major difference is that (1), these results are similar to (1), this is in strong contrast to (1), this behavior is in contrast to (I), this is in sharp contrast to (I), which is in stark contrast to (1), that's 's as opposed to (1), what you 'Il of all (1). it's little bit like (1), the research squared up with (1), what makes this claim different is (1), this is in line with (1)
Comparison.Simile
8 the same is true when (1), it's the same as (1), the whole effect is a little like (1), the same thing goes for (1), it's 's a bit like (I), this is kind of like (1), that's the same as (1). this is very similar to (1)
Contingency.Camd 1268 the purpose of this study was (20), the reason for that is (15), the reason for this is (12), part of the reason is (7), this is partly due to (6), this is the reason why (5), the purpose this study is (5), what this means is that (5), as a result of that (5), this is largely due to (4), that is the reason why (4), this may be due to (4), this is mainly due to (4), this is one reason why (3), that is one reason why (3), what I mean by that is (3), one reason for this is (3), what that means is that (3), that was the reason for (3). it's enough to make (3). the result of that is (3), one of the reasons is (3), the reason say this is because (2), that's 's one reason why (2), the goal of this task is (2), the key innovation is that (2). the goal of this paper is (2), this has given risc to (2), this situation has led to (2). the rise is due part to (2), popularity was behind the growth (2), the popularity of the net has meant that (2), the growth has heen fuelled by (2), part of the impetus comes from (2), crucial to this has been (2), putting the drop down to (2), was perfect example of this (2), these rumours were fuelled by (2). this is likely to mean (2), this is one of the reasons why (2), that is in part
because (2), that is one reason that (2), the delay was caused by (2), this is due in part to (2), some of that can be attributed to (2), that set the stage for (2), one way to do this is (2), the reason for it is (2). the no. 1 reason is (2), which may help explain why (2). [...]
Buntingency.Condiion
2
in order to do that (2). the more and more and more (1)
Contingency. Purpose
that is one simple way (1), this done in order (1), the plan intended to (1). the decision
appears to be an effort (1), as if to prove his point (1), the statement appeared to be an effort (1), what was an effort (1), the purpose of this study was (1), the purpose of this was (1), the aim of the study is (1), another way to look at this is (1)

---

Expansion. Expansion.Conjuuctior
143
that is addition (7), this is addition to (4), all adds up to (3), this not to mention (2), what worse is that (2), what I'm saying is (I), which leads me to my next point (I), that brings up the subject (1), a salient feature is that (1), addition to the es phenomenon (I), this aid comes on top of (1), the other key ingredient was (1), the most important of all is (1), another important point is that (1), one of the difficulties was (I), not to mention the fact (1), is in the same camp (1), what this about is that (1), putting aside the obvious point that (I), compounding the complexity is that (1), that's not to mention (1), are doing the same thing (I), the same downward trend applies to (1), what is more surprising is that (1), what is stranger is that (1), is a case in point (1), a common thread in these efforts is (I), leave aside the fact that (1), the same is true of (1), what's worse is that (1), included in that run was (1), in addition to the forest sale (1), the increase comes on top of (1), what made matters worse was that (1), but real bottom line was (1), more to the contemporary point (I), that is on top of (I), what was worse was that (1), the high point of that trip was (I), what is known is that (1), third benefit is that (I), the only real difference is that (1), that settlement was on top of (1), the only problem was that (1), it is as simple as that (I), what is true is that (1), all that could be said (I), that comes on top of (1), another factor in the fight (I), the issue of course is (1), [...]
Expansion.Instaann Instantiation
80
is case in point (1 1), a case in point is (4), one example of this is (3), one of these things is (2), to take just one example (2), as an example of this (2), the most recent example this was (2), it is as simple as that (1), that is just one example how (1), another bad example would be (1), this is one example of (1), good example of this is (1), is another example of this (1), one of those freedoms is (1). a prime example of this is (1). an example for this is (1), an example of this would be (1), a paradigmatic example of this situation is (1), another trend this year has been (1), another example of this is (1). as case in point (1), an obvious example of this happens (1), one such deal under consideration (1), among those he cited was (1), a case in point was (1), are just a few examples (1), the most notable example has been (1), one result of which was (I), one example of that is (I), the main one is that (I), an example of the first is (I), the best known example is (1), a recent case in point involves (1). the latest example of this is (1), a wonderful example here is (I), one of the best examples is (I), the most publicized example was (1), exhibit a has to be (1), at the top of that list (1), is the most literal example (1), one of the biggest is (1), among the strategies employed are (1). the single most salient fact is (1), be a case in point (1), the most frequent criticism has been (1), is only the most recent example (1). another way to look at is (1), the setup is follows (1), the most common one is (1), the most notorious examples were (I), [...]
Expansion Lovel-of-
42
what I am trying to say is (1), this is especially relevant for (1), a clear cut example that supports this idea is (1), is a case in point (I), what is implied is that (I), one of many cases in point involves (1), it was as simple as that (1), I my point is that (1), the main point is that (I), the big story is that (I), the list so far includes (I), what's happening is that (1), the whole idea here is (1), the larger view here is (1), what is unusual is that (1). what is new is that (1), what is clear is that (1), is the most dramatic case in point (1), the one noticeable difference was that (1), what's 's new is that (1), what it comes down to is (1). this is particularly relevant for (1), major theme to emerge (1), what I that is that (1). the main point here is (1), but the point here is (1), the whole point is that (1), what mean by that is (I). but the point is that (1), and what it was is (1), what's 's fantastic is that (1), what this is is that (I) what is striking about this is that (I), the general point is that (1), but the basic thing is (1), what was amazing is that (1), what I mean by this is (1), what she was telling me is that (I), examples of decisions taken include (I), the most vivid example of that is the fact (1), what is important that (1), is the most tangible proof of that (1)
detail
Cemporal.Asynchroooo
30
at the end of the day (4), in the years since that day (2), this is usually followed by (2), this was the beginning of (2), at the end of that (2), after you get done with that (1), the news comes week after (1). this was followed the same by (1), this was to be followed by (1), that meeting came day after (1), the deal comes days after (1), that came a week after (1), after that process was completed (1), was the beginning of (1), their removal came week after (I), it was at that point (1), their announcement came one day after (I), that was the start of (1), its statements came hours after (1), her comments came week after (1), his comments came day after (1), the expansion into asia follows (1), the move comes just weeks after (1), at the end of that process (1), once I put that in context (1). now that that's done (1), that's usually followed by (I), what happened next was that (1), that was the point that (1). after that has gone through (1)
Temporal.Synchronon Synchronous
13
he has done so (1), as I was doing so (1), this comes in the face (1), while I'm doing that (1). in nearly the same period (1), at the time of the sale (1), while you't 're about it (1). while this was all going on (1), in the course of this (1), the next step in the process that (1), in the process of this occupation (1), the the point is that (I), essential step in that regard (1)


Reviews:
[{"review_id": "V0IhMhcjdv", "reviewer": "Reviewer_n3Y1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments. We'd like to respond to the four reasons to reject:\n\n1) Connection between the two parts of the paper: Shallow discourse parsing is commonly seen as a series of distinct subtasks, one of them being the identification of senses of signals. We focus on this particular subtask and broaden its perspective by incorporating not only connectives but also „alternative lexicalization“, which so far has received very little attention. Then, based on this more comprehensive operationalization of the signal identification step, we show an application (sense flows, genre differences) for this particular subtask, which (i) can be done without the complete shallow discourse parsing machinery, and (ii) yields interesting results. - But we acknowledge that this link between the two parts could have been formulated more clearly and will do so in a revision.\n\n2) Merely modest enhancements of signal identification: We strongly believe that coherence relations that are signaled not by connectives but by other lexical means need to receive much more attention in the community if SDP as a whole is to improve (because connectives—at least in English—are basically exploited completely by now). The reported improvement appears marginal, but only as the high number of explicit connectives obscures the small number of alternative lexicalizations. Thus, the results of the latter have less weight on the changes in the final scores. A more precise comparison cannot be made because the referenced work did not distinguish these kinds of signals.\n\n3) Unexplored avenues for RST parsing: We think this is definitely a promising area for future work but beyond the scope of this paper, which for now concentrates on SDP/PDTB. Given the space limits, we don’t think that another paradigm for coherence modeling and parsing can be addressed. Actually, there may be a misunderstanding when the reviewer asks for a „comparative analysis against existing methods such as other RST parsing techniques“: We do not intend to compete with RST parsing; our goal is to enhance the signal identification performance only, and then apply that PDTB-style subtask, as sketched above.\n\n4) No thorough comparison between discourse sense flows and established methods for rhetorical style modeling: Yes, we acknowledge this comparison needs to be strengthened. In the submission, it was a matter of space, but if given another page for a final version, we will definitely expand our comparison to related work in this regard.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "see the main contributions", "reasons_to_reject": "The paper makes contributions to the field of discourse analysis, yet it also faces significant challenges in its current form. \n\nLack of Clear Structure: The organization of the paper poses a challenge, as it combines two distinct elements: signal extraction and discourse sense flows. Unfortunately, these components lack a clear connection and do not form a coherent objective or innovative methodology, as suggested by the paper's title.\n\nMethodological Limitations: While the paper's focus on examining discourse relations with lexical signals aligns with PDTB-style discourse analysis, the proposed methods seem to yield only modest enhancements in efficiency. Additionally, the method of discourse sense flows fails to effectively tackle challenges associated with Rhetorical Structure Theory (RST) parsing problems, leaving potential avenues unexplored.\n\nUntapped Potential in Parsing and Analysis: Although the idea of adapting discourse sense flow extraction to RST-style parsing is intriguing, the paper falls short in fully exploring this promising direction. The omission of an in-depth exploration of this concept creates a significant gap in the paper's coverage. Moreover, the absence of a robust comparative analysis against existing methods, such as other RST parsing techniques, hampers the paper's ability to underscore the uniqueness and advantages of its proposed discourse sense flows.\n\nLack of Comparative Framework: The paper does not provide a thorough comparison between its innovative discourse sense flows approach and established methodologies for rhetorical style modeling and genre/domain discrimination. This absence of comparative benchmarking diminishes the paper's ability to effectively highlight the distinctive contributions it brings to the field.\n\nIn conclusion, the paper introduces novel methodologies aimed at identifying and representing discourse sense, with a focus on capturing the inherent rhetorical style of documents through the concept of discourse sense flows. This approach holds promise for facilitating genre/domain discrimination. However, the paper's significance is overshadowed by identified shortcomings, spanning issues related to structural organization, methodological limitations, unexplored avenues, and unverified applications. As it stands, the paper falls short of meeting the rigorous standards expected for acceptance at the EMNLP conference.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper titled \"Discourse Sense Flows: Modelling the Rhetorical Style of Documents across Various Domains\" presents three contributions that advance the discourse analysis domain:\n\nAdvancements in Signal Extraction Models: The authors developed novel models for signal extraction from discourse relations, specifically concentrating on explicit connectives and alternative lexicalizations. These innovative models are meticulously tailored to classify the nuanced senses of these signals, harnessing the PDTB3 dataset.\n\nInception of Discourse Sense Flows: The paper proposed \"discourse sense flows,\" an ingenious technique that captures the rhetorical essence of a document by harnessing the sequential arrangement of coherence relations, as characterized by PDTB senses. \n\nEmpowering Genre/Domain Discrimination through Discourse Sense Flows: Through comprehensive comparative analyses spanning multifarious corpora, the paper meticulously examined sense flows. The authors leverage these findings to conduct experiments in automatic genre/domain discrimination.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4OKcYGGEgG", "reviewer": "Reviewer_jZkU", "summary": "", "questions": "A. The authors claim (lines 219-220) that they \"use our own predicted 219 discourse signals rather than the gold annotations [in the PDTB-3, which has the largest human-annotated set of annotated explicit and altlex relations]. The authors should tell us what labels from their set are assigned to the PDTB-3 annotations, and where this does and doesn't make sense.\n\nB. Shi and Demberg (2017) have shown the extent of variability in the Penn TreeBank and how this argues for using cross validation (and the error brackets this allows), rather than a held-out test set. Can the authors comment on how their use of the latter (rather than cross-validation) might call the own results into question?\n\n@inproceedings{shi-demberg-2017-need,\n  title = \"On the Need of Cross Validation for Discourse Relation Classification\",\n  author = \"Shi, Wei  and Demberg, Vera\",\n  booktitle = \"Proceedings of the 15th Conference of the {E}uropean \n  Chapter of the Association for Computational Linguistics: Short Papers\",\n     month = apr,\n     year = \"2017\",\n     address = \"Valencia, Spain\",\n     publisher = \"Association for Computational Linguistics\",\n     pages = \"150--156\",\n}\n\nC. The authors claim (lines 367-368) that \"Comparing both sense hierarchies, we notice an increased accuracy in our simplified hierarchy compared to the original one.\" This is disingenuous, given that (i) in terms of \"coarse\" senses, the model trained on the original sense labels does either better or equal to a model trained on the authors' simplified senses, and (ii) for the \"fine\" senses, they fail to indicate when any difference in performance is signficant.  Can they explain why this claim should not be corrected?\n\nD. The authors' Appendix D (\"Differences of Connectives and their Modifications\") implies that \"about\", \"in\", \"on\", \"only\" etc. are connectives when they are merely part of the modifiers of other connectives. For example, contrary to what the authors claim, \"in\" is not a connective in the phrase \"in conclusion\" or \"in summary\" or \"in contrast\". In these constructions, the whole prepositional phrase is a connective, not \"in\". This and the rest of the appendix should be corrected.\n\nE. Also with respect to Appendix D, the authors should also explain why the same modified forms appear multiple times in the same list of modifications -- e.g., \"not only because (51), not only because (6), not only because (6)\"; also \"in conclusion (305), in conclusion (269), in conclusion (164)\", etc.\n\nF. Figure 2 needs more explanation: There is a drop in both \"change flow\" and \"sense flow\" for N=4, which rises again for larger values of N.  This is not explained or even mentioned: all the authors to is claim that a combination of N=1,2 and 3 gives the best result. Is this an artifact? If so, of what? If not, can the authors explain it or even hypothesize an explanation?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for these extensive and constructive comments.\n\nThe questions revealed some lack of clarity in the text, which we would like to resolve:\n\n(A) This was misleading, as we here refer to the predicted signals used for the statistical evaluations (§4.2). We do not use the original annotations for comparison reasons (prediction errors and slightly different senses).\n\n(B) For each of the 10 iterations (§4.1), we use a different split for train/dev/test. However, we make sure that each model setting uses the exact same splitting in its i-th iteration.\n\n(C) We will reformulate the claim and make clear that it does not generalize to coarse senses, as the results are mixed, but for fine senses, the averaged results are all better, although we are missing tests to prove significance. This is mainly because our work is mostly interested in fine senses, while coarse senses are used more as an auxiliary task. For the mixed results on coarse senses, we hypothesize the model to focus on the easier task (classifying 4 types) rather than the more complex fine-sense classification, thus resulting in better coarse-sense scores. However, we can not provide strong evidence for this assumption.\n\n(D) The headline of the appendix is misleading and will be clarified. In particular, the list summarizes the predictions of our connective extraction model, and the list is analogously built to the PDTB3 manual (see below). We will describe the process of creating the list in more detail.\n\n(E) Thanks for pointing out his problem. We missed aggregating the individual results per corpus, thus leading to seemingly duplicate entries. We will correct this.\n\n(F) For our experiments we used n-grams of size up to 4, thus the mentioned drop is for the largest value and thus does not rise further for larger n. Instead, the table then presents results for three combinations of n-grams (1-2, 1-2-3, 1-2-3-4). Regarding the missing interpretation, we hypothesize that a major weak point of our modeling is the missing implicit relations, which account for half of the annotations. When linearizing the discourse senses, we have certain (implicit relation) gaps in the flow. The longer the n-gram we extract, the higher the chances that we end up with patterns with missing discourse information. Those seem more confusing than helpful for the classification model. Therefore, we mentioned accounting for implicit relations as an important step for future work.\n\nThanks for the missing references. We will mention Biran/McKeown.\nThe PDTB Annotation Manual indeed inspired the Table in the appendix, and we will cite it appropriately. In contrast to the table in the manual, our table summarizes automatically extracted explicit discourse signals and further groups them automatically by their connective head, thus alignment problems might occur. We also would like to highlight that these explicit signals in the table represent only a subset of all extracted signals, and they are disjoint from the ones found in the PDTB. We will clarify that in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "While most recent work on discourse relations has focussed on recognizing the sense of implicit relations and doing so for individual tokens, the current paper focusses on problems involving two types of explicit (albeit ambiguous) signals of discourse relations -- explicit connectives and alternative lexicalizations.", "reasons_to_reject": "While there are problems with the paper, I wouldn't consider any of them to be \"reasons to reject\".", "questions_to_authors": "A. The authors claim (lines 219-220) that they \"use our own predicted 219 discourse signals rather than the gold annotations [in the PDTB-3, which has the largest human-annotated set of annotated explicit and altlex relations]. The authors should tell us what labels from their set are assigned to the PDTB-3 annotations, and where this does and doesn't make sense.\n\nB. Shi and Demberg (2017) have shown the extent of variability in the Penn TreeBank and how this argues for using cross validation (and the error brackets this allows), rather than a held-out test set. Can the authors comment on how their use of the latter (rather than cross-validation) might call the own results into question?\n\n@inproceedings{shi-demberg-2017-need,\n  title = \"On the Need of Cross Validation for Discourse Relation Classification\",\n  author = \"Shi, Wei  and Demberg, Vera\",\n  booktitle = \"Proceedings of the 15th Conference of the {E}uropean \n  Chapter of the Association for Computational Linguistics: Short Papers\",\n     month = apr,\n     year = \"2017\",\n     address = \"Valencia, Spain\",\n     publisher = \"Association for Computational Linguistics\",\n     pages = \"150--156\",\n}\n\nC. The authors claim (lines 367-368) that \"Comparing both sense hierarchies, we notice an increased accuracy in our simplified hierarchy compared to the original one.\" This is disingenuous, given that (i) in terms of \"coarse\" senses, the model trained on the original sense labels does either better or equal to a model trained on the authors' simplified senses, and (ii) for the \"fine\" senses, they fail to indicate when any difference in performance is signficant.  Can they explain why this claim should not be corrected?\n\nD. The authors' Appendix D (\"Differences of Connectives and their Modifications\") implies that \"about\", \"in\", \"on\", \"only\" etc. are connectives when they are merely part of the modifiers of other connectives. For example, contrary to what the authors claim, \"in\" is not a connective in the phrase \"in conclusion\" or \"in summary\" or \"in contrast\". In these constructions, the whole prepositional phrase is a connective, not \"in\". This and the rest of the appendix should be corrected.\n\nE. Also with respect to Appendix D, the authors should also explain why the same modified forms appear multiple times in the same list of modifications -- e.g., \"not only because (51), not only because (6), not only because (6)\"; also \"in conclusion (305), in conclusion (269), in conclusion (164)\", etc.\n\nF. Figure 2 needs more explanation: There is a drop in both \"change flow\" and \"sense flow\" for N=4, which rises again for larger values of N.  This is not explained or even mentioned: all the authors to is claim that a combination of N=1,2 and 3 gives the best result. Is this an artifact? If so, of what? If not, can the authors explain it or even hypothesize an explanation?\n", "paper_topic_and_main_contributions": "The authors' claimed contributions are (i) developing novel models for classifying senses of explicit discourse connectives and AltLex phrases; (ii) characterizing patterns of discourse sense N-grams (for low N) across different genres and domains; and (iii) determining whether such sense n-grams allow them to distinguish the rhetorical style of a document.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "There has been earlier work on creating and using N-gram models of relations. The authors are refered to:\n\nOr Biran and Kathleen McKeown. 2015. Discourse Planning with an N-gram Model of Relations. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1973–1977, Lisbon, Portugal. Association for Computational Linguistics.\n\nThe authors should also refer to the The Penn Discourse Treebank 3.0 Annotation Manual, which documents connectives and their modifiers in Appendix B (\"Explicit Connectives and their Modifiers\"), rather than implying that they are the first to carry out such an analysis. They imply the same in Section 4.2, talking about modifiers of explicit connectives.  Both of these should be corrected.\nThe authors can get access to the annotation manual through documentation at the LDC. One does not need to have purchased the corpus to access the documentation at https://catalog.ldc.upenn.edu/docs/LDC2019T05/"}}, {"review_id": "aWALblbkEB", "reviewer": "Reviewer_zypT", "summary": "", "questions": "The paper seems to provide examples of one AltLex type; did the authors also deal with types that are syntactically and lexically flexible, e.g. \"that compares with, the increase was due mainly to, once triggered\", etc. Whichever type was targetted, it would be nice to see a sample of the extracted forms (particularly AltLexes) in an appendix.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the feedback and agree this is definitely an interesting question.\n\nWe neglected the list of AltLexes in the appendix due to their high variance, but we could easily provide some insides, e.g. grouped by its classified sense similar to the PDTB3 manual with an additional corpus source. However, we would prefer to focus on a subset, in particular, larger AltLexes to limit the used space. Also, these longer signals should be more interesting as they are usually more flexible and hard to catch with rule-based systems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is a nicely presented paper with several merits. Firstly, the authors use the PDTB labels to identify not only explicit signals but also alternative lexicalizations, which are known to have free forms. Secondly, the concept of sense flows (basically discourse senses that linearly follow one another) is introduced. Although this notion is not entirely novel, using them to identify rhetorical patterns of different genres/domains is quite new. Another novelty of the paper is that the authors use both the PDTB sense hierarchy and a simplified hierarchy, excarding less frequent labels involving belief and speech act. The models are applied to both hierarchies as well as their combination and compared. As most work on discourse, the authors target the first and second level of the PDTB sense hierachy and train both sets of senses jointly. \n\nThe background is well-described, and the pipeline is clearly explained, enabling the replication of this work.\n\nThe results are also good, especially regarding alternative lexicalizations, which are obviously more difficult.", "reasons_to_reject": "I don't have any reasons to reject this paper.", "questions_to_authors": "The paper seems to provide examples of one AltLex type; did the authors also deal with types that are syntactically and lexically flexible, e.g. \"that compares with, the increase was due mainly to, once triggered\", etc. Whichever type was targetted, it would be nice to see a sample of the extracted forms (particularly AltLexes) in an appendix.", "paper_topic_and_main_contributions": "This paper develops new models to extract explicit and alternatively lexicalized signals of discourse structure as well as their senses. These models are subsequently applied to raw corpora of various genres and domains. The notion of sense flows is introduced and compared across various genres/domains. One of the main findings is that n-gram patterns are stronger predictors than unigram distributions.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents models to detect explicit discourse connectives and alternative lexicalisations, as well as to classify their senses (based on PDTB3). The models are applied on texts of various genres and domains. The authors found out that n-gram patterns are stronger predictors than unigram ones.
The paper has several cons: (1) working not only on explicit connectives, but also alternative lexicalisations, which so far has received very little attention;  (2) using sense flows to identify rhetorical patterns of different genres and domains is novel; (3) using various levels of PDTB sense hierarchy; (4) in general a well-structured and a well-described paper enabling the replication of this work.
But it has also some disadvantages: the minor ones include some flaws (e.g. missing references or typos). The major ones include: (1) a lack of clarity in the subtasks or their connection; (2) comparison with the baseline and (3) a terminological confusion in terms of rhetorical analysis. 
However, the authors actively participated in the discussion and their rebuttal sounds convincing.r
Decision: Accept-Findings