Plain English Summarization of Contracts

Abstract: Unilateral contracts, such as terms of service, play a substantial role in modern digital life. However, few users read these documents before accepting the terms within, as they are too long and the language too complicated. We propose the task of summarizing such legal documents in plain English, which would enable users to have a better understanding of the terms they are accepting. We propose an initial dataset of legal text snippets paired with summaries written in plain English. We verify the quality of these summaries manually and show that they involve heavy abstraction, compression, and simplification. Initial experiments show that unsupervised extractive summarization methods do not perform well on this task due to the level of abstraction and style differences. We conclude with a call for resource and technique development for simplification and style transfer for legal language.

Although internet users accept unilateral contracts such as terms of service on a regular basis, it is well known that these users rarely read them. Nonetheless, these are binding contractual agreements. A recent study suggests that up to 98% of users do not fully read the terms of service before accepting them BIBREF0 . Additionally, they find that two of the top three factors users reported for not reading these documents were that they are perceived as too long (`information overload') and too complicated (`difficult to understand'). This can be seen in Table TABREF3 , where a section of the terms of service for a popular phone app includes a 78-word paragraph that can be distilled down to a 19-word summary.
The European Union's BIBREF1 , the United States' BIBREF2 , and New York State's BIBREF3 show that many levels of government have recognized the need to make legal information more accessible to non-legal communities. Additionally, due to recent social movements demanding accessible and transparent policies on the use of personal data on the internet BIBREF4 , multiple online communities have formed that are dedicated to manually annotating various unilateral contracts.
We propose the task of the automatic summarization of legal documents in plain English for a non-legal audience. We hope that such a technological advancement would enable a greater number of people to enter into everyday contracts with a better understanding of what they are agreeing to. Automatic summarization is often used to reduce information overload, especially in the news domain BIBREF5 . Summarization has been largely missing in the legal genre, with notable exceptions of judicial judgments BIBREF6 , BIBREF7 and case reports BIBREF8 , as well as information extraction on patents BIBREF9 , BIBREF10 . While some companies have conducted proprietary research in the summarization of contracts, this information sits behind a large pay-wall and is geared toward law professionals rather than the general public.
In an attempt to motivate advancement in this area, we have collected 446 sets of contract sections and corresponding reference summaries which can be used as a test set for such a task. We have compiled these sets from two websites dedicated to explaining complicated legal documents in plain English.
Rather than attempt to summarize an entire document, these sources summarize each document at the section level. In this way, the reader can reference the more detailed text if need be. The summaries in this dataset are reviewed for quality by the first author, who has 3 years of professional contract drafting experience.
The dataset we propose contains 446 sets of parallel text. We show the level of abstraction through the number of novel words in the reference summaries, which is significantly higher than the abstractive single-document summaries created for the shared tasks of the Document Understanding Conference (DUC) in 2002 BIBREF11 , a standard dataset used for single document news summarization. Additionally, we utilize several common readability metrics to show that there is an average of a 6 year reading level difference between the original documents and the reference summaries in our legal dataset.
In initial experimentation using this dataset, we employ popular unsupervised extractive summarization models such as TextRank BIBREF12 and Greedy KL BIBREF13 , as well as lead baselines. We show that such methods do not perform well on this dataset when compared to the same methods on DUC 2002. These results highlight the fact that this is a very challenging task. As there is not currently a dataset in this domain large enough for supervised methods, we suggest the use of methods developed for simplification and/or style transfer.
In this paper, we begin by discussing how this task relates to the current state of text summarization and similar tasks in Section SECREF2 . We then introduce the novel dataset and provide details on the level of abstraction, compression, and readability in Section SECREF3 . Next, we provide results and analysis on the performance of extractive summarization baselines on our data in Section SECREF5 . Finally, we discuss the potential for unsupervised systems in this genre in Section SECREF6 .
Given a document, the goal of single document summarization is to produce a shortened summary of the document that captures its main semantic content BIBREF5 . Existing research extends over several genres, including news BIBREF11 , BIBREF14 , BIBREF15 , scientific writing BIBREF16 , BIBREF17 , BIBREF18 , legal case reports BIBREF8 , etc. A critical factor in successful summarization research is the availability of a dataset with parallel document/human-summary pairs for system evaluation. However, no such publicly available resource for summarization of contracts exists to date. We present the first dataset in this genre. Note that unlike other genres where human summaries paired with original documents can be found at scale, e.g., the CNN/DailyMail dataset BIBREF14 , resources of this kind are yet to be curated/created for contracts. As traditional supervised summarization systems require these types of large datasets, the resources released here are intended for evaluation, rather than training. Additionally, as a first step, we restrict our initial experiments to unsupervised baselines which do not require training on large datasets.
The dataset we present summarizes contracts in plain English. While there is no precise definition of plain English, the general philosophy is to make a text readily accessible for as many English speakers as possible. BIBREF19 , BIBREF20 . Guidelines for plain English often suggest a preference for words with Saxon etymologies rather than a Latin/Romance etymologies, the use of short words, sentences, and paragraphs, etc. BIBREF20 , BIBREF21 . In this respect, the proposed task involves some level of text simplification, as we will discuss in Section SECREF16 . However, existing resources for text simplification target literacy/reading levels BIBREF22 or learners of English as a second language BIBREF23 . Additionally, these models are trained using Wikipedia or news articles, which are quite different from legal documents. These systems are trained without access to sentence-aligned parallel corpora; they only require semantically similar texts BIBREF24 , BIBREF25 , BIBREF26 . To the best of our knowledge, however, there is no existing dataset to facilitate the transfer of legal language to plain English.
This section introduces a dataset compiled from two websites dedicated to explaining unilateral contracts in plain English: TL;DRLegal and TOS;DR. These websites clarify language within legal documents by providing summaries for specific sections of the original documents. The data was collected using Scrapy and a JSON interface provided by each website's API. Summaries are submitted and maintained by members of the website community; neither website requires community members to be law professionals.
TL;DRLegal focuses mostly on software licenses, however, we only scraped documents related to specific companies rather than generic licenses (i.e. Creative Commons, etc). The scraped data consists of 84 sets sourced from 9 documents: Pokemon GO Terms of Service, TLDRLegal Terms of Service, Minecraft End User Licence Agreement, YouTube Terms of Service, Android SDK License Agreement (June 2014), Google Play Game Services (May 15th, 2013), Facebook Terms of Service (Statement of Rights and Responsibilities), Dropbox Terms of Service, and Apple Website Terms of Service.
Each set consists of a portion from the original agreement text and a summary written in plain English. Examples of the original text and the summary are shown in Table TABREF10 .
TOS;DR tends to focus on topics related to user data and privacy. We scraped 421 sets of parallel text sourced from 166 documents by 122 companies. Each set consists of a portion of an agreement text (e.g., Terms of Use, Privacy Policy, Terms of Service) and 1-3 human-written summaries.
While the multiple references can be useful for system development and evaluation, the qualities of these summaries varied greatly. Therefore, each text was examined by the first author, who has three years of professional experience in contract drafting for a software company. A total of 361 sets had at least one quality summary in the set. For each, the annotator selected the most informative summary to be used in this paper.
Of the 361 accepted summaries, more than two-thirds of them (152) are `templatic' summaries. A summary deemed templatic if it could be found in more than one summary set, either word-for-word or with just the service name changed. However, of the 152 templatic summaries which were selected as the best of their set, there were 111 unique summaries. This indicates that the templatic summaries which were selected for the final dataset are relatively unique.
A total of 369 summaries were outright rejected for a variety of reasons, including summaries that: were a repetition of another summary for the same source snippet (291), were an exact quote of the original text (63), included opinionated language that could not be inferred from the original text (24), or only described the topic of the quote but not the content (20). We also rejected any summaries that are longer than the original texts they summarize. Annotated examples from TOS;DR can be found in Table TABREF12 .
To understand the level of abstraction of the proposed dataset, we first calculate the number of n-grams that appear only in the reference summaries and not in the original texts they summarize BIBREF14 , BIBREF27 . As shown in Figure FIGREF14 , 41.4% of words in the reference summaries did not appear in the original text. Additionally, 78.5%, 88.4%, and 92.3% of 2-, 3-, and 4-grams in the reference summaries did not appear in the original text. When compared to a standard abstractive news dataset also shown in the graph (DUC 2002), the legal dataset is significantly more abstractive.
Furthermore, as shown in Figure FIGREF15 , the dataset is very compressive, with a mean compression rate of 0.31 (std 0.23). The original texts have a mean of 3.6 (std 3.8) sentences per document and a mean of 105.6 (std 147.8) words per document. The reference summaries have a mean of 1.2 (std 0.6) sentences per document, and a mean of 17.2 (std 11.8) words per document.
To verify that the summaries more accessible to a wider audience, we also compare the readability of the reference summaries and the original texts.
We make a comparison between the original contract sections and respective summaries using four common readability metrics. All readability metrics were implemented using Wim Muskee's readability calculator library for Python. These measurements included:
Flesch-Kincaid formula (F-K): the weighted sum of the number of words in a sentence and the number of syllables per word BIBREF28 ,
Coleman-Liau index (CL): the weighted sum of the number of letters per 100 words and the average number of sentences per 100 words BIBREF29 ,
SMOG: the weighted square root of the number of polysyllable words per sentence BIBREF30 , and
Automated readability index (ARI): the weighted sum of the number of characters per word and number of words per sentence BIBREF31 .
Though these metrics were originally formulated based on US grade levels, we have adjusted the numbers to provide the equivalent age correlated with the respective US grade level.
We ran each measurement on the reference summaries and original texts. As shown in Table TABREF23 , the reference summaries scored lower than the original texts for each test by an average of 6 years.
We also seek to single out lexical difficulty, as legal text often contains vocabulary that is difficult for non-professionals. To do this, we obtain the top 50 words INLINEFORM0 most associated with summaries and top 50 words INLINEFORM1 most associated with the original snippets (described below) and consider the differences of ARI and F-K measures. We chose these two measures because they are a weighted sum of a word and sentential properties; as sentential information is kept the same (50 1-word “sentences”), the differences will reflect the change in readability of the words most associated with plain English summaries/original texts.
To collect INLINEFORM0 and INLINEFORM1 , we calculate the log odds ratio for each word, a measure used in prior work comparing summary text and original documents BIBREF32 . The log odds ratio compares the probability of a word INLINEFORM2 occurring in the set of all summaries INLINEFORM3 vs. original texts INLINEFORM4 : INLINEFORM5 
The list of words with the highest log odds ratios for the reference summaries ( INLINEFORM0 ) and original texts ( INLINEFORM1 ) can be found in Table TABREF25 .
We calculate the differences (in years) of ARI and F-K scores between INLINEFORM0 and INLINEFORM1 : INLINEFORM2 INLINEFORM3 
Hence, there is a INLINEFORM0 6-year reading level distinction between the two sets of words, an indication that lexical difficulty is paramount in legal text.
We present our legal dataset as a test set for contracts summarization. In this section, we report baseline performances of unsupervised, extractive methods as most recent supervised abstractive summarization methods, e.g., BIBREF33 , BIBREF14 , would not have enough training data in this domain. We chose to look at the following common baselines:
Our preliminary experiments and analysis show that summarizing legal contracts in plain English is challenging, and point to the potential usefulness of a simplification or style transfer system in the summarization pipeline. Yet this is challenging. First, there may be a substantial domain gap between legal documents and texts that existing simplification systems are trained on (e.g., Wikipedia, news). Second, popular supervised approaches such as treating sentence simplification as monolingual machine translation BIBREF35 , BIBREF23 , BIBREF36 , BIBREF37 , BIBREF38 would be difficult to apply due to the lack of sentence-aligned parallel corpora. Possible directions include unsupervised lexical simplification utilizing distributed representations of words BIBREF39 , BIBREF40 , unsupervised sentence simplification using rich semantic structure BIBREF41 , or unsupervised style transfer techniques BIBREF24 , BIBREF25 , BIBREF26 . However, there is not currently a dataset in this domain large enough for unsupervised methods, nor corpora unaligned but comparable in semantics across legal and plain English, which we see as a call for future research.
In this paper, we propose the task of summarizing legal documents in plain English and present an initial evaluation dataset for this task. We gather our dataset from online sources dedicated to explaining sections of contracts in plain English and manually verify the quality of the summaries. We show that our dataset is highly abstractive and that the summaries are much simpler to read. This task is challenging, as popular unsupervised extractive summarization methods do not perform well on this dataset and, as discussed in section SECREF6 , current methods that address the change in register are mostly supervised as well. We call for the development of resources for unsupervised simplification and style transfer in this domain.
We would like to personally thank Katrin Erk for her help in the conceptualization of this project. Additional thanks to May Helena Plumb, Barea Sinno, and David Beavers for their aid in the revision process. We are grateful for the anonymous reviewers and for the TLDRLegal and TOS;DR communities and their pursuit of transparency.

Figure and table captions:
Figure 1: Unique n-grams in the reference summary, contrasting our legal dataset with DUC 2002 single document summarization data.
Table 4: Average readability scores for the reference summaries (Ref) and the original texts (Orig). Descriptions of each measurement can be found in Section 4.2.
Figure 2: Ratio of words in the reference summary to words in the original text. The ratio was calculated by dividing the number of words in the reference summary by the number of words in the original text.
Table 5: The 50 words most associated with the original text or reference summary, as measured by the log odds ratio.
Table 6: Performance for each dataset on the baselines was measured using Rouge-1, Rouge-2, and Rouge-L.
Table 7: Examples of reference summaries and results from various extractive summarization techniques. The text shown here has been pre-processed. To conserve space, original texts were excluded from most examples.

Questions about this paper:
What is the extractive technique used for summarization?

Answers:
TextRank, KLSum, Lead-1, Lead-K, Random-K