Venue: NAACL 2019

# DiscoFuse: A Large-Scale Dataset for Discourse-Based

Sentence Fusion

Mor Geva 1

Tel Aviv University

morgeva@mail.tau.ac.il

Eric Malmi

Google AI

Indan Szpektor

Google AI

Szpektor@google.com

Jonathan Berant 2

Tel Aviv University,

Allen Institute for AI

joberant@cs.tau.ac.il

Footnote 1: Work done during internship at Google AI.

Footnote 2: Work done at Google AI.

###### Abstract

Sentence fusion is the task of joining several independent sentences into a single coherent text. Current datasets for sentence fusion are small and insufficient for training modern neural models. In this paper, we propose a method for automatically-generating fusion examples from raw text and present DiscoFuse, a large scale dataset for discourse-based sentence fusion. We author a set of rules for identifying a diverse set of discourse phenomena in raw text, and decomposing the text into two independent sentences. We apply our approach on two document collections: Wikipedia and Sports articles, yielding 60 million fusion examples annotated with discourse information required to reconstruct the fused text. We develop a sequence-to-sequence model on DiscoFuse and thoroughly analyze its strengths and weaknesses with respect to the various discourse phenomena, using both automatic as well as human evaluation. Finally, we conduct transfer learning experiments with WebSplit, a recent dataset for text simplification. We show that pretraining on DiscoFuse substantially improves performance on WebSplit when viewed as a sentence fusion task.

## 1 Introduction

Sentence fusion is the task of combining several independent sentences into a single coherent text Barzilay and McKeown (2005). Sentence fusion is important in many NLP applications, including retrieval-based dialogue Song et al. (2018); Yan and Zhao (2018), text summarization Barzilay and McKeown (2005); Bing et al. (2015) and question answering Li et al. (2018); Marsi and Krahmer (2005). Such systems retrieve multiple sentences from different sources, documents or paragraphs, and use them to construct a coherent text.

Sentence fusion is challenging because it requires understanding the discourse semantics between the input sentences. Consider the example in Figure 1: a coherent fusion of the sentences requires understanding that the second sentence _contexts_ the first one, in order to insert the discourse connective _"However"_. In addition, the gender and syntactic role of the entity _"Zeitler"_ needs to be inferred to insert the pronoun _"he"_.

Prior work on sentence fusion Barzilay and McKeown (2005); Turner and Charniak (2005); Filippova (2010); Elsner and Santhanam (2011); Thadani and McKeown (2013); Bing et al. (2015); Chali et al. (2017) utilized very small amounts of labeled data, which are insufficient to train modern neural models. In this work, we propose a method for automatically generating sentence fusion examples at scale from raw text corpora.

To this end, we go over sentences and contiguous pairs of sentences in a corpus, and apply a set of manually-constructed rules, which identify the occurrence of prevalent fusion operations. The rules specify how to modify the sentences such that they are "unfused" into two independent sentences. E.g., in Figure 1 one rule will delete the discourse connective _"However"_, and another will replace the pronoun _"he"_ with the named entity _"Zeitler"_.

Figure 1: Example for two independent sentences, and their fusion. The modifications applied are pronominalization (blue) and connective insertion (red).



In the generated examples, the original fused text becomes the target, and the unfused sentences (generated by rules) are the input. Importantly, sentence fusion models trained on our data cannot simply learn to invert rule application, because information is lost and can be recovered only by understanding the text semantics . As mentioned, learning to insert _"However"_ in Figure 1 requires inferring that the sentences contrast. We cover a wide range of fusion phenomena such as inserting discourse connectives in various positions of the sentences, anaphora and cataphora identification, and sentence merging through coordination, relative clauses and apposition.

We applied our method on two large document collections, Wikipedia and sports articles from the Web, resulting in two datasets of 16 million and 44 million examples respectively. We call the combined dataset DiscoFuse. We extensively analyze the quality of our dataset with crowdsourcing, and find that workers understand the text after splitting in 85% of the cases, and the other 15% are due to either the original text being unclear or errors in rule application.

We trained a state-of-the-art sequence-to-sequence model Vaswani et al. (2017) and analyzed the fusion phenomena in which the model struggles. We found that the model succeeds in fusing sentences through structural constructions such as apposition or relative clauses, but performs badly when fusion involves inserting a particular discourse connective, or selecting pronominals.

Last, we performed transfer learning by training on DiscoFuse and then fine-tuning on a smaller dataset from a different distribution. To this end, we utilize WebSplit, a recent dataset for sentence splitting Narayan et al. (2017); Aharoni and Goldberg (2018), viewing WebSplit as a sentence fusion task. We found that pre-training on DiscoFuse substantially improves the performance of a fusion model in this setup.

To conclude, our contributions are:

1. DiscoFuse: a dataset of 60 million sentence fusion examples from two different corpora.
2. A method for automatically generating sentence fusion examples from raw text.
3. Automatic and human evaluation of the Transformer model on the fusion task.
4. A transfer learning setting in which model performance improves when pre-trained with DiscoFuse. The DiscoFuse dataset is publicly available at: [https://discofuse.page.link/data](https://discofuse.page.link/data).

## 2 Background

Existing fusion datasets are small, which is perhaps why only few works have explored the application of supervised models to sentence fusion Elsner and Santhanam (2011); Thadani and McKeown (2013). McKeown et al. (2010) introduced a human-generated corpus of 3,000 examples. Elsner and Santhanam (2011) extracted around 300 fusion examples from pre- and post-editing news articles. Thadani and McKeown (2013) constructed 1,858 examples from summarization tasks. Such datasets are too small to train modern data-hungry neural models.

Related to sentence fusion is its "inverse" task of sentence splitting Collados (2013) automatically constructed a Spanish simplification dataset by splitting single sentences into several simpler ones. Recently, two larger datasets for text splitting were released Botha et al. (2018); Narayan et al. (2017); Aharoni and Goldberg (2018). However, using these datasets for the "mirror" task of sentence fusion is problematic. First, sentence splitting often involves removing content from the original sentence for simplification, and this content is impossible to recover in the fusion direction. Second, these datasets do not focus on discourse and thus prominent discourse phenomena may be missed. Last, our new dataset is more than an order of magnitude larger than the above sentence splitting datasets.

Another related line of recent work focused on predicting discourse connectives between sentences and automatically generating examples from raw text Liu et al. (2016); Malmi et al. (2018). We substantially expand over those works by handling more diverse linguistic phenomena, such as connectives in single sentences, generating anaphora and cataphora constructions, relative clauses, coordination and more, which are all represented in a single dataset. Moreover, our dataset is 20x larger compared to prior work, allowing us to examine in depth long-tail scenarios.

## 3 The DiscoFuse Dataset

We next describe our process for building DiscoFuse, which contains 60 million sentence fusion examples from two different document collections: Wikipedia and Web articles about sports 



### Example Generation

DiscoFuse contains union-fusion examples, _i.e._ fusing sentences without loss of content [25]. To automatically extract examples, we manually crafted a list of text splitting rules. Our rule-set covers \(9\) fusion phenomena, including handling discourse connectives, coordination and relative clauses, and entity resolution for anaphora and cataphora constructions. For entity resolution, both anaphoric pronouns ("_she"_, "_they"_, "_his"_) and anaphoric nominals ("_the team"_, "_then man"_) are considered, based on the output of a coreference system. The covered phenomena are summarized in Table 1 and a detailed description is given in Appendix A.1.

Given a text \(t\) consisting of one or two consecutive sentences, each of our rules addresses a specific discourse phenomenon and has two parts: (a) conditions for matching the phenomenon in \(t\), and (b) operations over a dependency tree annotated with coreference resolution. Applying the operations generates a fusion example \((x=(s_{1},s_{2}),t)\), in which \((s_{1},s_{2})\) are two independent sentences originating from \(t\), but stripped from the discourse phenomenon that tied them in \(t\).

Figure 2 gives an example of a rule for the apposition structure. The rule is applied to the sentence _"The Jacksonville Jazz Piano Competition, a 30 year tradition, takes place at the Florida Theatre"_. First, the input is matched to the rule's condition. In this case, the condition is a single clause surrounded by two commas, which has a determiner as its first token and includes an apposition with incoming edge from a preceding token to the clause. Once matched, an example is generated. For this rule, the first sentence is created by removing the apposition clause, and the second sentence by removing the part after the clause and inserting the appropriate _"be"_ verb ("is"). Generation examples for all 9 rule types are provided in Table 1.

As explained in Section 1, solving sentence fusion involves more than just reverse-engineering the generation rules. The model needs to decide whether to insert a discourse connective with the right semantics, whether to merge the input sentences, and what syntactic construction (relative clause, coordination, apposition) is most appropriate in the given context.

Last, often several discourse phenomena occur in a single text \(t\). Thus, we allow combining anaphora rules with one of the following rule types: discourse connective, inner connective and sentence coordination, which cover frequent combinations in our texts.

### Building the DiscoFuse Dataset

To create DiscoFuse we retrieved the latest Wikipedia release and crawled the Web for several million sports articles. Documents were annotated with dependency trees and coreference resolution using Google Cloud Natural Language.1

Footnote 1: [https://cloud.google.com/natural-language/](https://cloud.google.com/natural-language/)

We considered each sentence and pair of consecutive sentences in each document as candidates, applying the example generation process described in Section 3.1. Additionally, we added as examples sentence pairs from the original corpus that did not match any rule, that is \((s_{1},s_{2})=t\), so that a trained model would also learn when not to change the input. We filtered out examples with sentence length \(\leq 6\) tokens, and examples with non-ASCII characters. This process resulted in \(44,177,443\) sports examples and \(16,642,323\) Wikipedia examples. We randomly split these examples into 98% train, 1% dev, and 1% test sets,

Figure 2: Example generation rule for apposition. Given an input text and its dependency tree, we check for a match with the apposition pattern. We then use the dependency tree to split the sentence and create a new example.



making sure that each document contributes examples to only one of the split sets.

Like prior work [16], we observed a skewed distribution of discourse phenomena in the data. Specifically, examples with anaphora or the connectives _"and"_ and _"but"_ constitute 99.7% of Sports and 59% of Wikipedia examples. Such a skewed distribution is likely to bias models and will fail to elucidate the ability of models to capture a wide range of linguistic phenomena. Therefore, we constructed a version of DiscoFuse by down-sampling examples containing _"and"_ and _"but"_ or anaphora. The down-sampled dataset contains 12,080,513 Sports examples and 4,581,352 Wikipedia examples.

The resulting distributions of discourse types and most common connectives in the two parts of DiscoFuse are provided in Appendix A.2. We will release both the original and the down-sampled versions of DiscoFuse.

## 4 DiscoFuse Quality Evaluation

To assess the quality of the generated fusion examples in DiscoFuse, we randomly selected 500 examples from each of the development sets of the Wikipedia and the Sports parts. We then conducted a crowdsourcing experiment in which each example was rated by 5 proficient English speakers, limiting each rater to at most 6 items. Each rater was presented with the two independent sentences in the example and was asked to indicate whether the text is understandable. If the rater answered "yes", she was then asked to characterize the relation between the sentences and how she would fuse them. We next detail the results.

\begin{table}
\begin{tabular}{l l} \hline \hline
* 18 record. \\  & (a) **Rider** entered the weekend averaging 23.0 points , good for 10th in the league . \\  & (b) **Rider** sidu those numbers mean little because of the Hawks ’ 11
- 18 record. \\ \hline Forward & (A) **Although** the friendship somewhat heated years later , it was a devastating loss to Croly . \\ connective & (a) The friendship somewhat healed years later . \\  & (b) It was a devastating loss to Croly . \\ \hline Inner & (A) Open workouts are held every Sunday unless the gym is closed for a holiday or other special events . \\ connective & (a) Open workouts are held every Sunday . \\  & (b) The gym is closed for a holiday or other special events . \\ \hline Cataphora & (A) **Stating** that the proponents were unlikely to succeed in this appeal , \\  & **Walker** rejected the stay request on October 23 . \\  & (a) **Walker stated** that the proponents were unlikely to succeed in this appeal . \\  & (b) **Walker** rejected the stay request on October 23 . \\ \hline Sentence & (A) The time of the autumn floods came , and the hundred streams poured into the Yellow River . \\ coordination & (a) The time of the autumn floods came . \\  & (b) The hundred streams poured into the Yellow River . \\ \hline Verb phrase & (A) The Sharks started the year 0
- 4 , yet recovered to claim sixth spot . \\ coordination & (a) The Sharks started the year 0
- 4 . \\  & (b) The Sharks recovered to claim sixth spot . \\ \hline Relative & (A) Kubler , who retired from cycling in 1957 , remained a revered figure in the wealthy alpine nation . \\ clause & (a) Kubler remained a reverted figure in the wealthy alpine nation . \\  & (b) Kubler retired from cycling in 1957 . \\ \hline Apposition & (A) The frigidarium , the last stop in the bathhouse , was where guests would cool off in a large pool . \\  & (a) The frigidarium was where guests would cool off in a large pool . \\  & (b) The frigidarium is the last stop in the bathhouse . \\ \hline \hline \end{tabular}
\end{table}
Table 1: Generated fusion examples for different phenomena. The input text is marked in uppercase blue, and the generated sentence pair is marked in lowercase red. We show in **boldface** parts that allow us to detect the target phenomenon.

\begin{table}
\begin{tabular}{c|r|r} Rater Selection & Sports (\%) & Wikipedia (\%) \\ \hline Yes & 83.4 & 86.0 \\ \hline No majority & 10.9 & 8.9 \\ \hline No & 5.7 & 5.1 \\ \hline \end{tabular}
\end{table}
Table 2: Rater evaluation understandability of the text after splitting. For each example, the majority of 5 raters was taken as the final rater selection.

 



### Example Text Clarity

Raters were asked whether they can understand the text after the example is split. Table 2 summarizes this evaluation. Most examples were marked as understandable by the raters ("yes") - 86% of Wikipedia examples and 83.4% of Sports examples. The rest either had no majority of rater votes or were marked as not understandable.

To shed light on the possible reasons for obscurity, we analyzed 70 random examples that were not marked as understandable by the majority of raters. In 29 examples (41%) the original text was unclear and for 17 examples a broader context was needed (24%). In the remaining 24 examples (34%), our rules generated sentences with grammatical or semantic errors. Examples for these cases are in Table 3.

Additionally, we analyzed 100 random examples for grammatical errors, and found that our rules did not introduce any errors in 79 examples. For 15 examples, the errors did not modify the meaning of the text nor caused semantic errors. The detected grammatical errors include extra commas, missing determiniers and bad pronoun replacements, and are demonstrated in Table 4.

### Fusion Evaluation

Next, we evaluated agreement on the fusion task for the 847 examples marked as understandable in Section 4.1. Because there are many ways in which sentences can be fused, one cannot expect raters to produce the original text \(t\) verbatim. Instead, we analyzed three central decisions and estimated whether people agree on those: (a) whether to merge the two sentences into a single one or keep them separate; (b) whether there are entities in the text that should be replaced with nominal or pronominal anaphors or cataphors; and (c) which discourse connective to add (if any).

For the last question, we presented raters with one connective from each of the four coarse-grained senses for discourse connectives defined by the PDTB Prasad et al. (2008): _comparison_, _expansion_, _contingency_ and _temporal_, as well as a _no-connective_ option. If the original text in the example includes a connective, we provided it as one of the options.

We observed a strong bias among raters towards refraining from performing any changes. E.g., while only 38% of the examples did not contain a connective in \(t\), the raters chose not to add a connective in 69.2% of the cases. Similarly, only in 29.1% of the examples the two sentences were not merged into a single one, while the raters chose not to merge in 53.1% of the examples. Similar behavior was also observed by Malmi et al. (2018) and Rohde et al. (2016).

We further looked at the agreement between therater majority and the 'gold' fusion decision. This analysis is shown in Table 5. Agreement on merging the input sentences into one is almost random (52%), since usually both options are valid. Consensus on whether to add an anaphor is higher, but not very high (63%), especially in sentences when the anaphor in \(t\) is a nominal rather than a pronoun. Finally, there is higher agreement on selecting the connective category (57%), for which the random baseline is 20%.

As mentioned, raters tend to keep the sentences unchanged. But in cases where raters agree to add a connective, agreement figures increase substantially. Specifically, when it is clear that a connective is needed, there is also high agreement for picking the right one (76%), for deciding whether

\begin{table}
\begin{tabular}{p{42.7pt}|p{142.3pt}} Error & Generated sentence \\ \hline Extra comma & The space behind the fence in right field is blocked off \(\,,\,\) \\ \hline Missing determiner & \(My\) interism sent me for a mammogram and sonogram . \\ \hline Bad pronoun replacement & **Lions** have a 3 - 1 record overall against Lions . \\ \end{tabular}
\end{table}
Table 4: Examples of grammatical errors introduced by our rules. The **red boldface** text was incorrectly inserted and the _blue italic_ text was incorrectly removed.

\begin{table}
\begin{tabular}{p{42.7pt} p{142.3pt}} Reason & Example \\ \hline Original text & (A) UPDATE: Beat falls because footwork and quickness. \\ unclear & (a) UPDATE: Peat falls. \\  & (b) Footwork and quickness. \\ \hline Missing context & (A) We were right on the heels of Spurs, although Everton were closing in. \\  & (a) We were right on the heels of Spurs. \\  & (B) Everton were closing in. \\ \hline Bad rule generation & (A) He told reporters after the game his reaction was because he missed a wide-open Randall Cobb in the end zone. \\  & (a) He told reporters after the game his reaction was. \\  & (b) He missed a wide-open Randall Cobb in the end zone. \\ \end{tabular}
\end{table}
Table 3: Examples for three possible reasons for not understanding the text. In each example, (A) is the original text and (a) and (b) are the two sentences generated by our rules.



to add an anaphor (70%), and for deciding whether to merge the sentences or not (70%).

## 5 Supervised Neural Fusion Models

Using DiscoFuse, we trained a Transformer seq2seq model Vaswani et al. (2017) that reads the input sentence pair and generates a fused text. We report model performance on the test-set using automatic metrics as well as human evaluation. We also provide detailed analysis of the different phenomena captured by this model.

### Experimental Settings

We tokenized all texts using byte-pair-encoding Sennrich et al. (2015) and compared the following three Transformer models :

* trained on the sports portion of DiscoSFuse after down-sampling.
* trained on the Wikipedia portion of DiscoSFuse after down-sampling.
* trained on a 50%-50% mixture of the sports and Wikipedia portions of DiscoSFuse after down-sampling.

All models share the same network architecture, based on the best practices discussed by Popel and Bojar (2018). We tuned parameters to select the best learning and dropout rates for each model with respect to the Exact Match objective (described in Section 5.2). Network architecture and hyper-parameters are in Appendix A.3. As a baseline, we also tested a model called Copy, which simply concatenates the two input sentences.

### Automatic Evaluation Results

We evaluated model performance using two automatic metrics. The first is Exact Match (_Exact_) to see how often the model generates the exact same text as the gold fusion. The second is _SARI_Xu et al. (2016), which computes the set of added, removed, and kept n-grams in the model output, comparing the output both with the gold text and the input text. Then it computes the F\({}_{1}\) scores for these three sets and averages the scores. We compute _SARI_ on up to 4-grams, as in Xu et al. (2016). We refrained from using metrics like BLEU because in fusion there is large overlap between the input sentences and their fused version, and such metrics do not capture well fine-grained differences of only a single word.

We note that our definition of _SARI2_ slightly differs from the one given by Xu et al. (2016) in

\begin{table}
\begin{tabular}{|l|c c c|} \hline Decision & All Examples (\%) & Examples with \\ Type & & connectives (\%) \\ \hline Single/ pair & 52.0 & 70.1 \\ \hline Anaphor & 63.4 & 70.1 \\ \hline Connective category & 57.0 & 76.4 \\ \hline \hline \# Examples & 847 & 271 \\ \hline \end{tabular}
\end{table}
Table 5: Average agreement for each fusion decision between the gold annotation and rater majority on examples marked as understandable by the raters. The right column considers only examples in which both the ‘gold’ and rater majority agreed that a connective should be added.

 

two aspects: (\(i\)) We define \(\frac{0}{0}=1\) when computing precision and recall, otherwise _SARI_ could be less than 1 even if the output matches the gold text exactly. (\(ii\)) Instead of considering only the precision of deleted n-grams, we use F\({}_{1}\) for all three sets. Otherwise, _SARI_ will give high scores to models that merely copy everything in the input, without even trying to infer what to delete.

Table 6 summarizes the results. When training and testing on the same domain, either Sports or Wikipedia, _SARI_ score is a little above 80 points for the full dataset. Yet _Exact_ is not high, around 42% for Sports and 47% for Wikipedia, showing that in the majority of the examples the model's fusion differs from the gold. Tested on the downsampled test-set, performance increases significantly for _Exact_, especially on Sports, where discourse phenomena is more skewed.

We next turn to cross-domain evaluation. When applying a model trained on one domain to the other domain performance drops. This shows that the discourse phenomena distribution differs between the domains, indicating that transfer learning is not trivial even with these large datasets. This is especially evident when applying DfWiki to Sports, where _Exact_ falls from 42% to 32% on the full dataset and from 50% to 40% on the downsampled one. Interestingly, when learning on the mixed training set, performance on both domains is close to in-domain performance, showing that the model has the capacity to handle both domains.

Finally, we take advantage of the provided annotation of the different discourse phenomena within each example in DiscoFuse. We conducted a detailed analysis of in-domain model performance by discourse type, presented in Table 7. Results show that structural discourse types, such as apposition and relative clause, are easier to learn with both high exact match and _SARI_ scores. While differences with respect to _SARI_ scores are not large between phenomena, exact match varies more. Anaphora and verb phrase coordination are more challenging, but still require matching of the same noun (the named entity or the subject). On the other hand, discourse types that involve connective prediction, such as sentence coordination and discourse connective, require semantic understanding, and performance is significantly lower. In addition, when two discourse types are required for fusion, performance drops dramatically.

### Human Evaluation Results

As our second experiment we employed crowdsourcing to test how distinguishable the fusion model outputs are from the gold fused texts. Concretely, we present raters an independent sentence pair from DiscoFuse and two fused versions - the gold version and one generated by a model. Raters were asked to detect the gold version. For each example, we took the majority of \(5\) raters as the final choice. This experiment mitigates the difficulties of automatic text generation evaluation, where many outputs are valid for a single input.

We sampled \(1000\) random examples from each development set of the two domains and applied the in-domain model to both. The raters were presented only with examples where the model output was different from the gold fusion, and assumed 50% detection accuracy otherwise.

Table 8 depicts the results. Out of cases when model output differed from the gold, raters were able to identify the human version in 65% of Sports examples and 61% of Wikipedia examples. Looking at the entire set, humans were able to identify the human version in 57% (Sports) and 55% (Wikipedia) of the cases. This shows that our Transformer model, applied over a dataset of millions of examples, is able to learn good fusions in general. Nevertheless, models are still far from perfect - human accuracy is clearly better than random and this improvement is statistically significant at a level of \(p<10^{-5}\) for Sports and \(p<10^{-3}\) for Wikipedia.

### Alignment-based Analysis

We next present an analysis of the types of errors our models produce. To this end, we sampled 40K examples of DfSport and DfWiki outputs on Sports and Wikipedia development sets. We then automatically aligned predicted sequences to gold

\begin{table}
\begin{tabular}{l l|r r}  & & \# Examples & Detection \\ \hline \multirow{3}{*}{Sports} & output = gold & 525 & 50\% \\  & output != gold & 475 & 65\% \\  & total & 1000 & **57\%** \\ \hline \multirow{3}{*}{Wikipedia} & output = gold & 528 & 50\% \\  & output != gold & 472 & 61\% \\ \cline{1-1}  & total & 1000 & **55\%** \\ \end{tabular}
\end{table}
Table 8: Human detection (_Detection_) percentage for DfSport and DfWiki on 1000 samples from each of the Sports and Wikipedia development sets. We report _Detection_ for cases when model output differed from the gold, and cases when they were identical.

 

sequences and looked at the differences between aligned words. The trained models successfully learned to copy most of the input text, and thus errors due to alignment problems are rare.

We start by considering the semantic relation between the input sentences. Table 9 displays model accuracy in predicting the most common connectives in DiscoFuse, as well as the top connectives predicted in this slot. We observe that when the model predicts a wrong connective, that connective is often reasonable, e.g., predicting _"but"_ instead of _"and"_ or _"however"_. Moreover, a second source of error is not adding a connective at all. It is also clear that some connectives, like _"however"_, _"although"_ and _"for example"_, are harder to learn.

We also analyzed the models' ability to correctly infer pronoun anaphors including gender, possessive and plurality. Figure 3 shows the pronoun confusion matrix for DfWiki,3 where lines refer to gold pronouns and columns to the generated pronoun in the same position. The clear diagonal shows that in most cases, the model successfully outputs the correct pronoun. However, the \(\langle{\rm other}\rangle\) column indicates that occasionally the model tends not to replace the entity in the input with a pronoun anaphor. In addition, the model seems to struggle with possession and plural 3rd person (_"it"_, _"its"_, _"they"_, _"their"_, _"theirs"_).

Footnote 3: Results for DfSport are very similar.

## 6 Transfer Learning Experiment

With the DiscofFuse approach we can collect a large amount of examples automatically. Still, these examples only reflect the manual rules that identify discourse phenomena. We wanted to see if DiscoFuse covers enough cases such that a trained model would be helpful for testing on fusion datasets generated by different approaches.

### Experimental Settings

In this experiment, we looked at the recently released WebSplit dataset 1.0 Narayan et al. (2017). It consists of examples \((t,\{s_{i}\}^{n}_{i=1})\), where \(t\) is a sentence that verbalizes the same set of RDF triples as \(\{s_{i}\}^{n}_{i=1}\). We note that WebSplit was originally developed for sentence splitting, from \(t\) to \(\{s_{i}\}^{n}_{i=1}\), but here we view its examples for the reverse fusion task: from \(\{s_{i}\}^{n}_{i=1}\) to \(t\). We only considered examples where \(\{s_{i}\}^{n}_{i=1}\) corresponds to exactly two simpler sentences (\(n=2\)). This leaves us with 135K training, 8K validation, and 8K test samples.

We tokenized the data using byte-pair-encoding Sennrich et al. (2015) and compared three models: (\(i\)) The Copy baseline that concatenates the two input sentences, (\(ii\)) a model trained on WebSplit alone, and (\(iii\)) a model pre-trained on DfWiki and fine-tuned on WebSplit.

For the last two models, we use the CopyNet architecture Gu et al. (2016), which is similar

Figure 3: DfWiki outputs versus the gold pronouns. Rows refer to gold pronouns and columns refer to aligned model outputs at the gold pronoun position. Values in each row are normalized to 1. Column \(\langle{\rm other}\rangle\) refers to model outputs that are not pronouns.

 

to state-of-the-art models for the splitting task on WebSplit(Narayan et al., 2017; Botha et al., 2018). While the Transformer outperformed this model on our main experiments, here it overfit on the small training set of WebSplit. The training details are provided in Appendix A.3.

### Results

Table 10 shows the results of the experiment. Similarly to Section 5, we measured the model performance using _SARI_. Pre-training with DfWiK improves _SARI_ score by 9% compared to using WebSplit alone. In particular, the F1 of the 'kept' and 'added' n-grams is significantly higher, by 23% and 33% respectively. Specifically, 'added' tokens refer also to correctly choosing discourse connectives, to which the large-scale examples in DiscoFuse were likely helpful.

We note that even with pre-training, the SARI 'add' score is only 10.4. This is probably due to the large amount of paraphrasing done in WebSplit, which makes it problematic for fusion evaluation (see also Section 2). For example:

**Sentence 1:**_Bolt_ , a comic character AKA Larry Bolatinsky , was created by Paris Cullins and Ernie Colon_ .
**Sentence 2:**_Paris Cullins is a United States national_ .
**Gold:**_Larry Bolatinsky is the alternative name for the comic book character Bolt , which was created by Ernie Colon and the American Paris Cullins_ .

Correctly inferring the added terms (shown in red) requires paraphrasing knowledge that is outside the scope of DiscoFuse.

## 7 Conclusions

We presented DiscoFuse, a large-scale dataset for sentence fusion that was generated by applying a rule-based method. It contains millions of examples from two domains, annotated with multiple discourse phenomena.

We used DiscoFuse to build supervised neural models for sentence fusion and conducted fine-grained analyses of the results. Currently, our models fuse only two sentences together. We would like to expand them to more input sentences in future work.

We also demonstrated DiscoFuse's usefulness in a transfer learning setup on a different fusion test-set, hoping it would facilitate research on text fusion in data-scarce domains.

## Acknowledgments

We thank Marta Recasens, Jan Botha, and Sascha Rothe from Google AI for helpful discussions. This research was supported by the Yandex Initiative in Machine Learning and by The Israel Science Foundation grant 942/16. This work was completed in partial fulfillment for the Ph.D degree of the first author.

## References

* Aharoni and Goldberg (2018) Roee Aharoni and Yoav Goldberg. 2018. Split and rephrase: Better evaluation and stronger baselines. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 719-724. Association for Computational Linguistics.
* Barzilay and McKeown (2005) Regina Barzilay and Kathleen R McKeown. 2005. Sentence fusion for multidocument news summarization. _Computational Linguistics_, 31(3):297-328.
* Bing et al. (2015) Lidong Bing, Piji Li, Yi Liao, Wai Lam, Weiwei Guo, and Rebecca Passonneau. 2015. Abstractive multi-document summarization via phrase selection and merging. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1587-1597. Association for Computational Linguistics.
* Botha et al. (2018) Jan A Botha, Manaal Faruqui, John Alex, Jason Baldridge, and Dipanjan Das. 2018. Learning to split and rephrase from wikipedia edit history. In _Empirical Methods in Natural Language Processing (EMNLP)_.
* December 1, 2017, Volume 2: Short Papers_, pages 418-424.
* Collados (2013) Jose Camacho Collados. 2013. Splitting complex sentences for natural language processing applications: Building a simplified spanish corpus. _Procedia-Social and Behavioral Sciences_, 95:464-472.
* Collobert et al. (2017)

\begin{table}
\begin{tabular}{l c c c c} \hline Training data & _SARI_ & Keep & Add & Delete \\ \hline Copy & 18.1 & 52.9 & 0.5 & 0.9 \\ WebSplit & 40.5 & 44.6 & 7.8 & **69.3** \\DfWiki+WebSplit & **44.2** & **54.8** & **10.4** & 67.5 \\ \hline \end{tabular}
\end{table}
Table 10: Fusion results on WebSplit, measured by _SARI_ and the F1 scores that compose it.

 

Micha Elsner and Deepak Santhanam. 2011. Learning to fuse disparate sentences. In _Proceedings of the Workshop on Monolingual Text-To-Text Generation_, pages 54-63. Association for Computational Linguistics.
* Filippova (2010) Katja Filippova. 2010. Multi-sentence compression: Finding shortest paths in word graphs. In _Proceedings of the 23rd International Conference on Computational Linguistics_, pages 322-330. Association for Computational Linguistics.
* Gu et al. (2016) J. Gu, Z. Lu, H. Li, and V. O. Li. 2016. Incorporating copying mechanism in sequence-to-sequence learning. In _Association for Computational Linguistics (ACL)_.
* Li et al. (2018) Yutong Li, Nicholas Gekakis, Qiuze Wu, Boyue Li, Khyathi Chandu, and Eric Nyberg. 2018. Extraction meets abstraction: Ideal answer generation for biomedical questions. In _Proceedings of the 6th BioASQ Workshop A challenge on large-scale biomedical semantic indexing and question answering_, pages 57-65.
* Liu et al. (2016) Yang Liu, Sujian Li, Xiaodong Zhang, and Zhifang Sui. 2016. Implicit discourse relation classification via multi-task neural networks. In _AAAI_, pages 2750-2756.
* Malmi et al. (2018) Eric Malmi, Daniele Pighin, Sebastian Krause, and Mikhail Kozhevnikov. 2018. Automatic Prediction of Discourse Connectives. In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_, Miyazaki, Japan. European Language Resources Association (ELRA).
* Marsi and Krahmer (2005) Erwin Marsi and Emiel Krahmer. 2005. Explorations in sentence fusion. In _Proceedings of the Tenth European Workshop on Natural Language Generation (ENLG-05)_.
* McKeown et al. (2010) Kathleen McKeown, Sara Rosenthal, Kapil Thadani, and Coleman Moore. 2010. Time-efficient creation of an accurate sentence fusion corpus. In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_, pages 317-320. Association for Computational Linguistics.
* Narayan et al. (2017) Shashi Narayan, Claire Gardent, Shay B. Cohen, and Anastasia Shimorina. 2017. Split and rephrase. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_.
* Popel and Bojar (2018) Martin Popel and Onofrej Bojar. 2018. Training tips for the transformer model. _The Prague Bulletin of Mathematical Linguistics_, 110(1):43-70.
* Prasad et al. (2008) R. Prasad, N. Dinesh, A. Lee, E. Miltsakaki, L. Robaldo, A. K. Joshi, and B. L. Webber. 2008. The Penn discourse treebank 2.0. In _LREC_.
* Rohde et al. (2016) Hannah Rohde, Anna Dickinson, Nathan Schneider, Christopher NL Clark, Annie Louis, and Bonnie Webber. 2016. Filling in the blanks in understanding discourse adverbialis: Consistency, conflict, and context-dependence in a crowdsourced elicitation task. In _Proceedings of the 10th Linguistic Annotation Workshop held in conjunction with ACL 2016 (LAW-X 2016)_, pages 49-58.
* Sennrich et al. (2015) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural machine translation of rare words with subword units. _arXiv preprint arXiv:1508.07909_.
* Song et al. (2018) Yiping Song, Rui Yan, Cheng-Te Li, Jian-Yun Nie, Ming Zhang, and Dongyan Zhao. 2018. An ensemble of retrieval-based and generation-based human-computer conversation systems. In _Proceedings of IJCAI-ECAI, 2018_.
* Thadani and McKeown (2013) Kapil Thadani and Kathleen McKeown. 2013. Supervised sentence fusion with single-stage inference. In _Proceedings of the Sixth International Joint Conference on Natural Language Processing_, pages 1410-1418.
* Turner and Charniak (2005) Jenine Turner and Eugene Charniak. 2005. Supervised and unsupervised learning for sentence compression. In _Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics_, pages 290-297. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, pages 5998-6008.
* Xu et al. (2016) Wei Xu, Courtney Napoles, Ellie Pavlick, Quaze Chen, and Chris Callison-Burch. 2016. Optimizing statistical machine translation for text simplification. _Transactions of the Association for Computational Linguistics_, 4:401-415.
* Yan and Zhao (2018) Rui Yan and Dongyan Zhao. 2018. Coupled context modeling for deep chit-chat: towards conversations between human and computer. In _Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 2574-2583. ACM.

## Appendix A Supplemental Material

### Generation Rules

In this section we provide technical details of the generation rules used to create DiscoFuse. For the sake of clarity, we provide a simplified version of the rules, that does not include edge cases and minor implementation details. The discourse connectives we considered in the rules were selected from the Penn Discourse Treebank (PDTB) Prasad et al. (2008) and are listed in Table 12 

Given an input text, it is encoded with 3 lists: \(Z\) is the token list, \(Z_{t}\) is a list of POS tags, \(Z_{l}\) is a list of dependency labels (see Table 11). In addition, all entities mentioned in the text are extracted and stored such that for two token lists \(Z,R\), the set \(m(Z,R)\) holds all the mention pairs of the same entity in the two lists. Each rule is designed for a specific discourse phenomenon and contains two parts. First, a set of conditions is applied to the input lists to detect whether the phenomenon occurs in the text. If a discourse pattern has been identified, a short sequence of simple operations is applied to the input, yielding a new sentence pair. Table 13 summarizes the operations in use, which allow insertion and deletion of tokens and splitting of the input text.

Table 14 provides the technicalities of each rule, i.e. the detection conditions of the discourse structure, and the sequence of operations for generating a new sentence pair from it. A detailed example for two-rule execution process is given in Table 15.

As mentioned, the rules are simplified for clarity. However, we note two special cases where morphological modifications are required to produce text without grammatical errors. First, in some cases of forward connective and cataphora, the tense change of a verb is required when splitting the input sentence. For instance, in the cataphora example in Table 1, we change the verb "_stating"_ to have a past tense - _"stated"_. Likewise, occasionally a _"be"_ verb needs to be inserted when splitting a single sentence, as demonstrated in Figure 2. In our rules, we choose which _"be"_ verb to insert based on the tense and perspective of the rest of the sentence.

### DiscoFuse Data Distribution

Figure 4 and Table 16 show the distributions of discourse types and most common connectives in the two parts of DiscoFuse.

Analyzing the dataset reveals significant differences in discourse phenomena between the two types of documents (Figure 4). E.g., coordination is very common in Wikipedia while anaphora is dominant in Sports. Likewise, the distribution of discourse connectives is quite different (Table 16).

### Neural Models Parameters

The models DfSport,DfWiki,DfS+W share the same Transformer network architecture, that was originally proposed by Vaswani et al. (2017). During training, we split the samples to buckets

\begin{table}
\begin{tabular}{p{85.4pt} p{199.2pt}}
**Set** & **Values** \\ \hline \(C_{b}\) & “accordingly”, “additionally”, “afterward”, “alternatively”, “although”, ”and”, as a result ”, because of that”, ‘because of this”, ’besides”, “but”,’ typifier ”, “consopenty”, “cisely”, “else”, “hally”, “for example”, “In contrast ”, ”furtherrence”, “hence”, “however”, “in context ”, ”in fact ”, in other words”, “in particular ”, “in short ”, “in the sum ”, “in the end ”,“, “indeed”, “instead”, “Test”, likewise”, “mean”, “in the meantime ”, “nomething”, “noneover nevertheless”, “next “otherwise”, “overall”, “plus” rather “regardless ”similar”, “simultaneous”, “specifically”, “still ”, “then ”, “thereafter”, “herehereto”, “theroot”, “thus “thimultaneous”, “yet”, “you”, “second”, “what”, “usually “coventially”, “obviously” again “i Fortranely”, “meaning”, “increasing ”, “interesting”, “inetically”, “anyway”, “Clearly”, “negative”, “necessary”, “neveracy”, “theirs “, “because”, “because”, “hence”, “other”, “while “whereas”, “about”, “s, “about”, “s, “not “margin”, meaning” \\ \hline \(C_{f}\) & although, since, in addition to, aside from \(C_{f}\), \(c_{a}\) and, but, not, no way, so far \\ \hline \(P_{e}\) & who, which, whose, whom \\ \hline \(V\) & VB, VBD, VBG, VBN, VBF, VBZ \\ \end{tabular}
\end{table}
Table 11: Notation and definitions for Table 14 of generation rules. \(Z,R,S_{Z},S_{R}\) are token lists and \(1\leq i,j\leq|Z|\) are indices. The full lists of connectives and POS tags are provided in Table 16.

\begin{table}
\begin{tabular}{p{142.3pt} p{199.2pt}}
**Operation** & **Description** \\ \hline Delete\((X,i,n)\) & Delete a sequence of \(n\) tokens from \(X\), starting from index \(i\) \\ Preferred\((X,Y)\) & Attach the list \(Y^{\prime}\) at the beginning of \(X\). Replace every occurrence of \(Y\) in \(X\) with \(Z\), in a non-overlapping manner. \\ Split\((X,i)\) & Split \(X\) into two token lists \(V=\{x_{1},...,x_{i-1}\},W=\{x_{i},...,x_{i,X}\}\). \\ Trim\((X)\) & Delete all tokens in \(X\) after the first punctuation taken, e.g. period, commas, etc. \\ \hline \end{tabular}
\end{table}
Table 12: Connectives and POS tags used in our detection rules. A preceding comma is allowed for connections in \(\mathcal{C}_{c}\). For the connectives “_although”_ and _“since”_ in \(\mathcal{C}_{f}\), we do not allow a following comma.

 

by their text length, and use different batch size between 60-100 for each bucket. Further configuration details and the hyperparameters used for training of each model are provided in Table 17.

In our transfer learning experiment, we trained two CopyNet models Gu et al. (2016): a model

\begin{table}
\begin{tabular}{|p{45.5pt}|l|l|} \hline
**L. Input** & \\ \(Z=\{\text{Ruzi ordered his first shot to be retaken because Brazilian players entered the penalty area before his kick.}\}\) & \\ \cline{3-3}  & \multicolumn{1}{c}{} & \\
**2. Inner connective** & \\ _Detection_ & For \(S="\text{because}"\) it holds that \(S\in\mathcal{C}_{s}\) and \(\mathcal{S}\sim Z\). \\ _Generation_ & \\ Split\((Z,i)\) & \(A=\{\text{Ruiz ordered his first shot to be retaken.}\}\) \\ \(B=\{\text{Because Brazilian players entered the penalty area before his kick.}\}\) & \\ Delete\((B,1,|S)\) & \(B=\{\text{Brazil players entered the penalty area before his kick.}\}\) \\ Trim(B) & _no effect this case_ \\ \hline
**3. Anaphora** & \\ _Detection_ & \\ Replace\((B,h;\text{Ruiz})\) & \(B=\{\text{Brazil players entered the penalty area before Ruiz’s kick.}\}\) \\ \hline
**4. Output sentence pair** & \\ \(A=\{\text{Ruiz ordered his first shot to be retaken .}\}\) & \\ \(B=\{\text{Brazil players entered the penalty area before Ruiz ’s kick}\}\) & \\ \hline \end{tabular}
\end{table}
Table 15: Detailed two-rule execution example. We show in red parts of the input that are used for detection or modified during execution. The input token list \(Z\) is of a single sentence. First, the rule for inner connective is applied, splitting \(Z\) into two sentences \(A,B\), without the connective _“because”_. Then, applying the anaphora rule, the pronoun _“his”_ in \(B\) is replaced with the entity it refers to in \(A\), to obtain a new sentence pair.

Figure 4: Discourse type distribution of the sports and Wikipedia portions of DiscoFuse after downsampling.

\begin{table}
\begin{tabular}{|p{85.4pt}|p{99.6pt}|p{113.8pt}|p{113.8pt}|} \hline
**Phenomenon** & **Input** & **Detection** & **Generation** \\ \hline Discourse connective & (A,B) & \(\exists S\in\mathcal{C}_{s}\), \(i\in[1,5]\cdot S\subseteq\{b^{(i)},...,b^{(|H|D)}\}\) & Delete\((B,i,|S|)\) \\ \hline Anaphora & (A,B) & \(m(B,A)\neq\emptyset\) & RefACE\((B,g_{B},SA)\) \\ \cline{3-3}  & \multirow{2}{*}{Z} & \(\forall(SB,SA)\in m(B,A)\) \\ Forward connective & & \(\exists S\in\mathcal{C}_{f}:S\subset Z\) & \(A,B+\text{Split}(Z,i)\) \\  & & \(\exists i:|S|+1<i<|Z|\wedge z^{(i)}=""\) & DefETE\((A,1,|S|)\) \\  & & & Deletion\((B,1,1)\) \\ \hline Inter connective & \multirow{2}{*}{Z} & \(\exists S\in\mathcal{C}_{s}:S\prec Z\) & \(A,B+\text{Split}(Z,i)\) \\  & & & DetE\((B,1,|S|)\) \\  & & Term\((B)\) \\ \hline Cataphora & \multirow{2}{*}{Z} & \(\exists i::i<i<|Z|\wedge z^{(i)}=""\) & \(A,B+\text{Split}(Z,i)\) \\  & & \(Z^{(i)}_{t}=\text{VBG}\wedge Z^{(i)}_{t}=\text{vmod}\wedge Z^{(i+2)}_{t}\in\mathcal{ V}\) & Prefenda\((A,b_{1})\) \\  & & & Delete\((B,1,1)\) \\ \hline Sentence coordination & \multirow{2}{*}{Z} & \(\exists i<j\leq i+5:Z^{(i)}_{t}=\text{coir}\wedge Z^{(j)}_{t}=\text{coir}\) & \(A,B\leftarrow\text{Split}(Z,i)\) \\  & & \(\exists i<k<j:Z^{(i)}_{t}=\text{coir}\implies\text{coir}\implies\text{coir} \implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir} \implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{ coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coircoir}\implies\text{coir}\implies\text{coir}\implies\text{coir}\implies\text{coircoir}\implies\text{coir}\implies\text{coir}\implies\text{coircoir}\implies\text{coir}\implies\text{coircoir}\implies\text{coir}\implies\text{coircoir}\implies\text{coir}\implies\text{coircoir}\implies\text{coir}\implies\text{coircoir}\implies\text{coir}\implies\text{coircoir}\implies\text{coir}\implies\text{coircoir}\implies\text{coircoir}\implies\text{coir}\implies\text{coircoir}\ 

trained on WebSplit alone, and a model pretrained on DwWiki and finetuned on WebSplit. The first model was trained for 200,000 steps on WebSplit, whereas the second model was pretrained for 1 million steps on DfWiki and then finetuned for 100,000 steps on WebSplit. Again, the samples were split to buckets by their text length, with batch sizes between 25-125 for each bucket. The final test scores were computed with the parameters that maximize the validation _SARI_ score during training. The network architecture and hyperparameters were shared between the models and not optimized during training. They are listed in Table 18.

\begin{table} Sports & Wikipedia \\  & \% & \% \\ \hline and & 12.0 & and 12.5 \\ \hline but & 10.9 & but & 10.7 \\ \hline because & 8.1 & although & 8.4 \\ \hline although & 5.4 & however & 8.2 \\ \hline so & 4.9 & because & 7.7 \\ \hline or & 4.8 & so that & 2.1 \\ \hline however & 3.5 & while & 2.0 \\ \hline while & 2.4 & or & 1.8 \\ \hline so that & 2.2 & so & 1.2 \\ \hline unless & 2.2 & for example & 1.0 \\ \end{tabular}
\end{table}
Table 16: Most common connectives in DiscoFuse after down-sampling. Percentages are with respect to the entire dataset, including examples without a connective.

\begin{table}
\begin{tabular}{l c c}  & **DsPSoft** & **DfWiKi** & **DFS+W** \\ \hline number of hidden layers & 7 & 7 & 7 \\ hidden dimension & 1024 & 1024 & 1024 \\ filter size & 2048 & 2048 & 2048 \\ number of heads & 16 & 16 & 16 \\ beam width & 4 & 4 & 4 \\ attention dropout rate & 0.1 & 0.2 & 0.2 \\ ReLU dropout rate & 0.4 & 0.3 & 0.2 \\ learning rate & 0.14 & 0.07 & 0.11 \\ \hline \end{tabular}
\end{table}
Table 17: Parameters and hyperparameters of the models DfSport, DefWiki, DFs+W.

\begin{table}
\begin{tabular}{l r} \hline number of encoder layers & 3 \\ number of decoder layers & 1 \\ hidden dimension & 128 \\ beam width & 20 \\ scheduled sampling probability & 0.2 \\ dropout rate & 0.2 \\ learning rate & 0.001 \\ learning rate decay & 0.98 \\ \hline \end{tabular}
\end{table}
Table 18: Parameters and hyperparameters of the CopyNet models used for transfer learning.