What to Read in a Contract? Party-Specific Summarization of Legal Obligations, Entitlements, and Prohibitions
Authors: Abhilasha Sancheti
Aparna Garimella
Balaji Vasan Srinivasan
Rachel Rudinger
Venue: EMNLP 2023 Main (2023)
Keywords: Legal NLP
Rights and Obligation Extraction
Summarization
Importance Ranking
TL;DR: We introduce a task of party-specific summarization of important obligations, entitlements, and prohibitions in lease agreements, collect a legal expert annotated dataset of pairwise importance rankings, and build an extractive summarizer.

Abstract: Reviewing and comprehending key obligations, entitlements, and prohibitions in legal contracts can be a tedious task due to their length and domain-specificity. Furthermore, the key rights and duties requiring review vary for each contracting party. In this work, we propose a new task of \textit{party-specific} extractive summarization for legal contracts to facilitate faster reviewing and improved comprehension of rights and duties. To facilitate this, we curate a dataset comprising of party-specific pairwise importance comparisons annotated by legal experts, covering ~293K sentence pairs that include obligations, entitlements, and prohibitions extracted from lease agreements. Using this dataset, we train a pairwise importance ranker and propose a pipeline-based extractive summarization system that generates a party-specific contract summary. We establish the need for incorporating domain-specific notions of importance during summarization by comparing our system against various baselines using both automatic and human evaluation methods.

What to Read in a Contract? Party-Specific Summarization of Legal Obligations, Entitlements, and Prohibitions
Abhilasha Sanchetit, Aparna Garimella', Balaji Vasan Srinivasan', Rachel Rudinger University of Maryland, College Park
Adobe Research
(sancheti, rudinge oudinger)@ume
{sancheti, garimell, balsrini }adoob. com
Abstract
sartaccing-sarty
Categorizer
Importance Ranker BT
Tonant
Tenant
Summmary
Reviewing and comprehending key obligations, entitlements, and prohibitions in legal contracts can be a tedious task due to their length and speciinip Furthermore, the key rights and duties requiring review vary for each con- tracting party. In this work, we propose a new task of party-specific extractive summarization for legal contracts to facilitate faster reviewing and improved comprehension of rights and du- ties. To facilitate this, curate a dataset prising of party-specific pairwise importance comparisons annotated by legal experts, cover- ing ~293K sentence pairs that include obliga- tions, entitlements, and prohibitions extracted from lease agreements. Using this dataset, we train a pairwise importance ranker and pro- pose a pipeline-basee based extractive summarization system that generates party-specific contract summary. We establish the need for incorporat- ing domain-specific notion of importance dur- ing summarization by comparing against various baselines using both automatic and human evaluation methods!.
Imporianoe gnied
with
sttin
calggonko
oragator
Figure An illustration of input contract with a party (Tenant) and output from CONTRA It first iden- tifies sentences in contract that contains any of the obligations entitlements and prohibitions specific to the party. Then, ranks the identified sentences based on their importance level (darker the shade, higher the importance) to produce summary. BT: Bradley-Terry
et al., 2021), well risk-focused summaries of privacy policies (Keymanesh et al., 2020), they aim to generate single summary for a contract and do not focus on key rights and duties.
However, we argue that a single summary may not serve all the parties as they may have differ- ent rights and duties (Ash et al., 2020; Sancheti et al., 2022) (e.g., tenant is obligated to timely pay the rent to the landlord whereas, the landlord must ensure the safety and maintenance of rented property). Further, each right or duty may not be equally important (e.g., an obligation to pay rent on time (otherwise pay a penalty) is more important for tenant than maintaining a clean and organized premise) and this importance also varies for each party. To address this, we introduce a new task of party-specisic extractive summarization of impor- tant rights and duties in contract. Motivated by the different categories in the software license sum- maries available at Te:DRLegal describing what users must, can, and cannot do under a license, we define rights and duties in terms of deontic modali- ties (Matulewar kewkle 2010; Peters and Wyner, 2016; Sancheti et al., 2022): obligations ("must")), enti- tlements ("can"), and prohibitions ("cannot"). In this work, refer to these as (modal) categories.
system
1 Introduction
A contract is a legally binding agreement that de- fines and governs the rights, duties, and responsibil- ities of all parties involved in To sign a contract (e.g., lease agreements, terms of services, and pri- vacy policies), it is important for these parties to preciscly understand their rights and duties as de- scribed in the contract. However, understanding and reviewing contracts can be difficult and tedious due to their length and the complexity of legalese. Having an automated system that can provide an "at a glance" summary of rights and duties can be useful not only to the parties but also to legal pro- fessionals for reviewing contracts. While existing works generate section-wing wise summaries of unilat- eral contracts, such as terms of services (Manor and Li, 2019) employment agreements (Balachandar
Data available at http://bit.ly/ Sobiit.lylegee
1y/legal- aegelliaaaan aeraalimmorttrn importance
Pnttps://tldrlegall.om

---

Existing summarization systems perform poorly on legal contracts due large compression ratios and the unavailability of public datasets (Manor and Li, 2019) that are large enough to train neural summarization systems (See et al., 2017; Paulus et al., 2017). While previous studies have either de- vised unsupervised methods to identify obligations, entitlements, and prohibitions for a party in con- tract (Ash et al., 2020) or collected annotations for the same (Sancheti et al., 2022), they do not con- sider the relative importance of instances within each category. Although existing unsupervised summarization methods rely on sentence centrality- based importance measures (Mihalcea and Tarau, 2004; Erkan and Radev, 2004), capturing sentential meaning (Zheng and Lapata, 2019), or analyzing the relationship between sentences to capture the structure of document (Ozsoy al., 2011), these methods fail to capture the regal-domain-speccte notion of importance among sentences. To address these issues, we construct a exper- annotated dataset of party-specific pairwise importance com- parisons between sentences from lease agreements. We focus on lease agreements, which have clearly defined parties with asymmetric roles (tenant, land- lord), and for which modal category labels have previously been collected (Sancheti et al., 2022).
for sentence pairs (that include obligations. entitle- ments, or prohibitions) from leasc agreements; and (c) we train a pairwise importance ranker using the curated dataset to build a pipeline- extrac- tive summarization system ($5) for the proposed task, and show the effectiveness of the system as compared to several unsupervised ranking-basse -baeed summarization baselines, under both automatic and human evaluations ($8); underscoring the domain- sensitive nature of "importance" in legal settings.
Related Work
Summarization of Legal Text Existing works focus summarizing legal case reports (Galgani and Hoffmann, 2010; Galgani et al., 2012; Bhat- tacharya al., 2021; Agarwal et al., 2022: Elaraby and Litman, 2022), civil rights (Shen et al., 2022), terms of service agreements (Manor and Li, 2019), privacy policies (Tesfay et al., 2018; Zaeem et al., 2018; Keymanesh et al., 2020). Hybrid approaches combining different summarization methods (Gal- gani et al., 2012) or word frequency (Polsley et al.. 2016) with domain-specifie knowledge have been used for legal case summarization task. Most sim- ilar works either propose to identify sections of privacy policies with a high privacy risk factor to include in the summary by selecting the riskiest content (Keymanesh et al., 2020) or directly gener- ate section-wise summaries for employment agree- ments (Balachandar et al., 2021) using existing extractive summarization methods (Kullback and Leibler, 1951; Mihalcea and Tarau, 2004; Erkan and Radev, 2004; Ozsoy et al.. 2011). We differ from these works in that instead of generating a single summary, we generate party-specific sum- maries of key rights and duties for contracts by learning to rank sentences belonging to each of these categories based on their importance as de- cided by legal experts.
We break down the contract-level task of party- specific summarization into two sentence-level sub- tasks: (1) Content Categorization - identifying the modal categories expressed in the sentences of a contract for a specified party, and (2) Im- portance Ranking - ranking the sentences based on their importance for a specified party. This ap- proach has three benefits: (a) enabling to use an existing corpus for identifying dcontic modalities in contract, (b) cognitively simplifying the annota- tion task for experts who only need to compare a few sentences (resulting in higher agreement) at a time rather than read and summarize a full contract (spanning 10- - 100 pages), and (c) reducing the cost of annotation as contract-level end-to-end summarization system requires more data to train than does sentence-level categorizer and ranker.
Rights and Duties Extraction Existing works either propose rule-based methods (Wyner and Pe- ters, 2011 1; Peters and Wyner, 2016; 6; Matulewska, 2010), or use a combination of NLP approaches such as syntax and dependency parsing (Dragoni et al., 2016) for extracting rights and obligations from legal documents such as Federal code regula- tions. Others (Bracewell et al., 2014; Neill et al., 2017: Chalkidis al., 2018) use machine learning and deep learning approaches to predict rights and duties with the help of small datasets which are not publicly available. While rale-based unsupervised
This work makes the following contributions: (a) we introduce new task of spety-specific extrac- tive summarization of important obligations, enti- tlements, and prohibitions in legal contracts; (b) we are the first to curate novel legal expert annotated dataset ($4) (using best-word wosst scaling) consisting of party- serty-ppecffic pairwise importance comparisons

---

approaches exist (Ash al., 2020) identify rights and duties with respect a party, they are not flex- ible and robust to lexical or syntactic variations in the input. Funaki et al. (2020) curate an annotated corpus of contracts for recognizing rights and du- ties using LegalRulcML (Athan et al., 2013) but it is not publicly available. Sancheti et al. (2022) introduced dataset of lease contracts with annota- tions for party-specific deontic categories and cor- responding triggers mentioned in sentence. We use the proposed model as the content categorizer.
on SEC are public information and can be redis- tributed without further consent".
Annotation Task Rating the party-speciffc im- portance of a sentence in contract on an abso- lute scale requires well-defined importance levels to obtain reliable annotations. However, defining each importance level can be subjective and re- strictive. Moreover, rating scales are also prone to difficulty in maintaining inter- and intra-annotator consistency (Kiritchenko and Mohammad, 2017), which we observed in pilot annotation experiments. We ran pilots for obtaining importance annotations for each sentence in a contract, as well as a pair of sentences from prior works (Sakaguchi et al., 2014; Sak- aguchi and Van Durme, 2018) but found they had a poor agreement (see details in A. Thus, fol- lowing Abdalla et al. (2021), we use Best-Worst Scaling (BWS), a comparative annotation schema, which builds on pairwise comparisons and does not require (2) labels. Annotators are presented with n=4 sentences from contract and party, and are instructed to choose the best (i.e., most important) and worst (i.e., least important) sentence. Given N sentences, reliable scores are obtainable from (1.5N-2N) 4-tuples (Louviere and Woodworth, 1991; Kiritchenko and Mohammad, 2017). To our knowledge, this is the first work to apply BWS for importance annotation in legal contracts.
3 Task Definition
We formally define the new task as: given con- tract C consisting of sequence of sentences (c1. C22 .... Cx) and party P. the task is to generate an extractive summary S consisting of the most im- portant m. obligations, m, entitlements, and My prohibitions (where my < n Vi E {o, eo,e,)) for the specified party. As previously mentioned, we break this task into two sub-tasks: (1) Content Catego- rization to identify the categories expressed in sentence of a contract for a given party, and (2) Im- portance Ranking rank the sentences based on their importance to a given party.
scale of 0-5 taking inspiration
4 Dataset Curation
The LEXDEMOD dataset (Sancheti et al., 2022) contains sentence-level, party-specific annotations for the categories of obligations, entitlements, pro- hibitions, permissions, non-obligations, and non- entitlements in lease agreements. However, the data does not contain any importance annotations. To facilitate importance ranking, we collect party- specific pairwise importance comparison annota- tions for sentences in this dataset. We only con- sider the three major categories: obligations, enti- t and prohibitions.
From list of N 3, 300 sentences spanning 11 lease agreements from LEXDEMoD, we generate 1, 950 (1.5N) unique 1-tuples (each consisting of distinct sentences from an agreement) such that each sentence occurs at least six 4-tuples. We hire 3 legal experts (lawyers) from Upwork with >2 years of experience in contract drafting and reviewing to perform the task. We do not provide detailed technical definition for importance but brief them about the task of summarization from the perspective of review and compliance, and en- courage them rely on their intuition, experience. and expertise (see below section for annotation re- liability). Each tuple is annotated by experts.
Dataset Source We use a subset of contracts from the LEXDEMOD dataset to collect importance annotations as it enables us to create reference sum- maries for which need both the category and im- portance annotations. LEXDEMOD contains lease agreements crawled from Electronic Data Gath- ering, Analysis, and Retrieval (EDGAR) system which is maintained by the U.S. Securities and Ex- change Commission (SEC). The documents filed
Annotation Aggregation Annotation for each 4-tuple provides us 5 pairwise inequalities. E.g., if is marked as the most important and d the least important, then we know that > b, c,
We merge entitlement and permission categories as there were a small number of permission annotations per contract. Entitlement is defined as right to have/do something while permission refery to being allowed to haveldo something
"ht / http///wwweeeeeeeeeeeeeeee//oett www.:ooo/prrvieeee httppprvaaayttaaaaa.eepe stmitdissmminntton 'The tuples are generated using the BWS scripts pro- vided in Kiritchenko and Mohammad (2017): http:// sai fmohammad.

---

Parly
%Obl.
Pro
Obl.
Bitllments
CONTRACT AGREEVENT
Landlord
Tenant
52. .25
46.56
55.98
39.31
54.49
46.82
BradlggTTryy
Profiitions
Importance Rankor
Serience
Table I: Importance comparison statisties (per party) computed from subset of pairwise comparisons where the category of the sentences in a pair is different. Bold win l6s are significant (p < 0.05). >- more important.
Swerenss
Phaty-caddbiiingg
CONTRACT
AGREENENT
CONTRACT AGREEMENT
Content Categariooe
a > d. > d, and d. From all the annotations and their inequalities, we calculate real-valued im- portance scores for all the sentences using Bradley- Terry (BT) model (Bradley and Terry. 1952) and use these scores to obtain (2) pairwise importance comparisons. BT is a statistical technique used to convert set of paired comparisons (need not be full or consistent set) into full ranking.
Para [arart)
Puryy Pory-oodooongg
Serkance
Figure 2: CONTRASUM takes contract and a party to first identify all the sentences containing party-specific obligations, entitlements, and prohibitions using con- tent categorizer. Then, the identified sentences for each category are pairwise-ranker using an importance ranker. A ranked list of sentences is obtained using the Bradley-Terry model to obtain the final summary.
Annotation Reliability A commonly used mea- sure of quality and reliability for annotations pro- ducing real-valued scores split-half reliability (SHR) (Kuder and Richardson, 1937; Cronbach, 1951). It measures the degree to which repeat- ing the annotations would lead to similar relative rankings of the sentences. To measure SHR?, an- notations for each 4-tuple are split into two bins. These bins are used produce two different inde- pendent importance scores using a simple counting mechanism (Orme, 2009; Flynn and Marley, 2014): the fraction of times a sentence was chosen as the most important minus the fraction of times the sen- tence was chosen as the least important. Next, the Spearman correlation is computed between the two sets of scores to measure the closeness of the two rankings. If the annotations are reliable, then there should be high correlation. This process is re- peated 100 times and the correlation scores are av- eraged. Our dataset obtains SHR of 0.66 + 0.01 indicating moderate-hee high reliability.
perform duties such as "paying rent on time" may lead to late fees). For landlords, on the other hand, knowing entitlements is more important than obli- gations or prohibitions, possibly due to the nature of lease agreements where landlords face fewer prohibitions and obligations than tenants.
As we do not explicitly define the notion of "importance" during annotation. We learned from feedback and interaction with the annotators about several factors they considered when determining importance. These factors included the degree of liability involved (e.g., 'blanket indemnifications' were scored higher than 'costs incurred in alter- ations' by a tenant since blanket rights can have uncapped amount of liability), and liabilities incurred by a more probable event were rated more important than those caused by less probable events, among others. As is evident from these examples, the factors that can influence importance may be complex, multi-faceted, and difficult to ex- haustively identify, which is why our approach in this work was to allow our annotators to use their legal knowledge to inform a holistic judgment.
Dataset Analysis We obtain a total of 293, 368 paired comparisons after applying the BT model. Table shows the percentage of sentences con- taining category annotated more important than those from another. For tenants, prohibitions are annotated as more important than both obli- gations and entitlements, perhaps as prohibitions might involve penalties. Similarly, obligations are more important than entitlements, as knowing them might be more important for tenants (e.g., failure to
5 CONTRASUM: Methodology
We build a pipeline based extractive summariza- tion system, CONTRASUM (Figure 2), for the pro- posed task. It consists of two modules for the two sub-tasks: Content Categorizer ($5.1) and Impor- tance Ranker ($5.2). The Content Categorizer first identifies sentences from a contract that mention any of the obligations, entitlements, or prohibitions for a given party. Then, the identified sentences per
We use scripts available at https: sai Fmohammad. com/WebDocs//Bne BBetteebboc//Btttbeeeeeeee
Berr//essttttttttte Worst- Scallnn Scripts.zip.

---

category for a given party are pairwise ranked for importance using the Importance Ranker. From all the pairwise comparisons, a ranked list of sentences is obtained using the Bradley-Tere model (Bradley and Terry, 1952) to produce a final party-specific summary consisting of most important m, obliga- tions, m. entitlements, and my prohibitions.
Label/Split
Train
Dev
Test
Positive
85529
2538
71576
Negative 62857
2137
68731
Table Statistics pairwise importance annotations.
helps ensure some degree of diversity across cat- egories). As the ranker provides ranking at a sen- tence pair level, to obtain an importance-ranker list of sentences from all the pairwise predictions for each category, we use the Bradlcy-Terry (BT) model described in $4. We produce the final summary by selecting the ma, me. and mp most important sentences predicted obligations, enti- tlements, and prohibitions, respectively.
5.1 Content Categorizer
The Content Categorizer takes in a sentence oj from C and party to output all the categories (such as, obligations, entilements, prohibitions) mentioned in G. Such categorization helps in partitioning the final summary as per the categories of inter- est. We use the multi-label classifier introduced in (Sancheti al., 2022) as the categorizer. Since sentence can contain multiple categories, the same sentence can be part of different categories.
Experimental Setup
CONTRASUM is pipeline-based system consist- ing of two modules. The two modules are trained (and evaluated) separately and pipelined for gener- ating end-to-end summaries as described in $5.3.
5.2 Importance Ranker
A contract may contain large number of obliga- tions, entitlements, and prohibitions for each party; however, not all instances of each category may be equally important. Also, the importance of sen- tences within each category may vary for each party. Therefore, this module aims to rank the sentences belonging to each category based on their level of importance for the specified party. As indicated in $4, we do not define the notion of "importance"; instead, we rely on the annotations from legal ex- perts based on their understanding from contract review and compliance perspective build the ranker. This module (Figure 2) takes in pair of sentences (c,, eÿ) from and a party to predict if c, is more important to be a part of the summary than Cj for P. We model this as a binary classifier that orders the input pair of sentences based on their importance for the specified party.
Datasets We use the category annotations in LEXDEMOD dataset (Sancheti et al., 2022) to train and evaluate the Content Categorizer. Dataset statistics are shown in Table 10 ($A.2).
For training the Importance Ranker, we need sentence pairs from contracts ordered by relative importance to a particular party. We use the pair- wise importance comparison annotations collected in $4 to create a training dataset. If, for pair of sentences (a.b), a > b (a more important than b), then the label for binary classification is posi- tive, and negative otherwise. We retain the same train/dev/test splits as LEXDEMOD to avoid any data leakage. Table 2 present the data statistics. As mentioned earlier, the two modules are sepa- rately trained and evaluated at a sentence- tence leir-level. However, CONTRASUM gener- ates party-specific summaries Therefore, for evaluating the system for the end-to- end summarization task, we need reference sum- maries. To obtain reference summaries, we need ground- category and importance ranking be- tween sentences in contract. Since we collected importance comparison annotations for the same sentences for which LEXDEMOD provides cate- gory annotations, we group the sentences belong- ing to category (ground-truth) and then derive the ranking among sentences within a category using the gold importance annotations and BT model (as
5.3 End-to-end Summarization
contract-level.
Recall that the Content Categorizer and Importance Ranker work at the sentence- and sentence-per -aite- level, respectively. Therefore, to produce the de- sired party-speciffc category- bause summary for contract, we obtain (1) categories for each sentence in the contract (using the Content Categorizer), and (2) ranking of the sentence pairs within each cat- egory according their relative importance (using the Importance Ranker) with respect to the party. We do not explicitly account for the diversity within each category (although organization by category

---

described earlier) for each party. We obtain ref- erence summaries different compression ratios (CR) (5%, 10%, 15%) with a maximum number of sentences capped at 10 per category to evaluate the output summaries against different reference sum- maries. CR is defined as the % of total sentences included in the summary for a contract. Please note that the reference summaries are extractive.
sentence pair-level (~130K for Ranker) is large, it amounts to 5 contracts for the extrinsic evaluation of CONTRASUM for the end-to-end summariza- tion task at a contract-level. Since the number of contracts is small, we perform 3-fold validation of CONTRASUM. Each fold contains contracts (in the test set) sampled from 11 contracts (remain- ing contracts used for training the modules for each fold) for which both category and importance annotations are available ensure the availability of reference summaries. CONTRASUM uses the best Categorizer and the best Ranker (RoBERTa-L models) trained on each fold (see Table and for results on fold-1;A.5 for other folds).
Training Details We use the RoBERTa- large (Liu al., 2019) model with binary (multi-label) classification head and fine-tun it on the collected importance dataset (LEXDEMOD) dataset for the Importance Ranker (Content Categorizer) using binary-cross entropy loss.
Evaluation Measures We report macro- averaged Precision, Recall, and FI scores for predicting the correct categories or importance order for the Content Categorizer and Impor- tance Ranker. We also report the accuracy of the predicted labels. As both the reference and predicted summaries are extractive nature, we use metrics from information retrieval and ranking literature for the end-to-end evaluation of CONTRASUM. We report Precision @k, Recall Fl @k (following Keymanesh et al., 2020), Mean Average Precision (MAP), and Normalized Discounted Cumulative Gain (NDCG) (details formulae in A.3) computed at category-level for each party: these scores are averaged across categorics for each party and then across parties for contract. We believe these metrics give a more direct measure of whether any sentence is correctly selected and ranked in the final summary. For completeness, also report ROUGE-1/2/L scores in Table 9 in A.5.
Implementation Details We use HuggingFace's Transformers library (Wolf et al., 2019) to fine-tune PLM-based classifiers. The content categorizer is fine-tuned for 20 epochs, and the importance ranker for 1 epoch with gradient accumulation steps of size 4. We report the test set results for model(s) with the best F1 -l-ro score the development set. Both the category and importance classifiers are trained with maximum sequence length of 256 and batch size 8. For both the classifiers, party conditioning is done with the help of special to- kens prepended to the sentence, as this has been successfully used previously for controlled text generation tasks (Sennrich et al., 2016; Johnson et al., 2017). The input format for the Content Categorizer is [PARTY] SENTENCE and for the Importance Ranker is [PARTY] SENTENCE! (/s) SENTENCE2. More details are provided A.6. Dataset Pre-processing, We use lexnlp (Bommar- ito et al., 2021) to get sentences from a contract and filter the sentences that define any terms (using regular expressions such as, "means", "mean", and "shall mean")) or does not mention any of the parties (mentioned in the form of alias such as, "Tenant", "Landlord", "Lessor", and "Lessee").
Importance Ranker Baselines We compare the Ranker against various pre-trained language mod- els; BERT-BU (Devlin et al., 2018), contract BERT (C-BERT-BU) (Chalkidis et al., 2020), and RoBERTa-B (Liu et al., 2019), and majority.
7 Evaluation
Content Categorizer Baselines We do not com- pare the categorizer against other baselines, instead directly report the scores in Table 7, as we use the model with the best F1-score on the development described in (Sancheti et al., 2022).
We perform an extrinsic evaluation of CONTRA- SUM to assess the quality of the generated sum- maries as well as an intrinsic evaluation of the two modules separately to assess the quality of cate- gorization and ranking. Intrinsic evaluation of the Categorizer and the Ranker is done using the test sets of the datasets ($6) used train the models. Al- though the size of datasets used to evaluate the two modules at a sentence- (~1 1.5 for Categorizer) or
Extractive Summarization Baselines We com- pare CONTRAS AUM against several unsupervised baselines which use the same content categorizer but the following rankers.

---

Model
Accuracy
Precision
Recall
F1
Majority
51.38/50.35/51.00
25.79/25.17// 12
30.00/50.00/
04/33/3 19/33.75
BERT-BU
09.80/67.722 12/00/88.12///
69.77/67.777 01////773/660
69.75/67.31 /68.80
69. 20..6/66.11 71
RoBERT=B
70.50/68. 11 69..75
70. 68 15/68.52
70. 46/66 66666660000 09 668. 43
70.46: (0..66/6...0 08 688/68.. .11
C-BERT-BL
70.32/68.22
24/69.22 29
00..00668-00 25/ 25/622//6888
/68 23/69..00
/9/ /68. 22/68/26660
RoBERTs
70.35/68..0 40.55/68.54/000 56/89.60
70.53/ /0///6866.11 68.61/69.62
70.47/ /6. /6.51.1/6///// /69 54
70.48/65 /6../66..... 49/69.54
Table 3: Evaluation results for the Importance Ranker. Scores for Tenant/Landlord//S are averaged over 3 different seeds, BU, and I. denote base-uncased, base, and large, respectively.
1. Random baseline picks random sentences from the set of predicted sentences belonging to each category. We report average scores over 5 seeds.
CR-0.65
CR-11.10
CR+1).15
Model
MAP
NDCG
MAP
NDCG
MAP
NDCG
Kandom
0.115
0.169
0.115
0.109
0.341
0.228
KL-San
0.125
0.173
0.194
0.338
0.218
LSA
0.093
0.154
0.303
0.177
0.342
01225
2. KL-Sum (Kullback and Leibler, 1951) aims to minimize the KL-divergence between the imput contract and the produced summary by greedily selecting the sentences.
TextKank
PACSUM (BERT)
0.185
0.182
0.194
0.333
0.282
0.219
0.352
0.350
01240
LesRank
0.130
0.185
0.360
CUNTRASUM
0.351
0.202
Uppes bound
0.028
0.080
0.514
Randum
0.206
0.208
0.209
0.318
0.228
0.346
KL-Sam
0.174
0.252
0.181
0.2855
0.204
0.313
3. LSA (Latent Semantic Analysis) (Ozsoy et al., 2011) analyzes the relationship between doc- ument sentences by constructing a document- term matrix and performing singular value de- composition to reduce the number of sentences while capturing the structure of the document. 4. TentRank (Mihalcea and Tarau, 2004) uses the PageRank algorithm to compute an importance score for each sentence. High-scoring sentences are then extracted to build a summary.
8
1SA
TextXank
0.254
0.1.4
0.309
0.237
0.939
0.151
0.392
0.264
0.378
0.267
0.588
0.303
PACSUM (BERTT)
0.162
0.254
0.178
0.281
0.210
0.300
LexRank
0.108
0.278
0.219
0.317
0.236
0.353
CONTRASUM-CC
0.398
0.825
0.393
0.586
0.431
0.575
Table 4: Evaluation results of end-to-end summarization for different compression ratios (CR) averaged across 3 folds. GC (PC) denotes use of ground truth (predicted from CC) categories. CC: Content Categorizer.
less than m; sentences. We also compare the per- formance of summarization systems that utilize the ground-truth categories (GC) followed by the above ranking methods to produce the summary. These systems help us investigate the effect of error propagation from the categorizer to the ranker and the final summaries. We use the Sumy? python package for implementing these ranking methods.
5. LexRank (Erkan and Radev, 2004) is similar to TextRank as it models the document as a graph using sentences as its nodes. Unlike TextRank, where all weights LexRank utilizes the degrees of similarities be- tween words and phrases, then calculates an importance score for each sentence.
assumed unit weights,
6. PACSUM (Zheng and Lapata, 2019) another graph-based based ranking algorithm that employs BERT (Devlin et al., 2018) to better capture the sentential meaning and builds graphs with directed edges as the contribution of any two nodes to their respective centrality is influenced by their relative position in document.
Results and Analysis
Automatic Evaluation of the Categorizer and the Ranker We report the evaluation results for the Ranker in Table 3. Fine-tuned PLMs out- perform the majority baseline on Fl score as ex- pected. While C-BERT-BU, which is pre-trained on contracts, performs better than BERT-BU and RoBERTa-B, the overall F1 score is the highest for RoBERTa-L, suggesting increased model size and training compensate for lack of pretraining on contracts. For the Categorizer, we report results from (Sancheti et al., 2022) as-is in Table 7 in A.5.
7. Upper- picks sentences from a contract for category based on the ground-truth impor- tance ranking derived from human annotators. It indicates the performance upper-bound of an extractive method that uses our categorizer. We limit the summaries to contain m,=10 (cho- subjectively) sentences per category for exper- imental simplicity and that it is short enough to be processed quickly by user while capturing the most critical points but this number can be adjusted. Note that if less than my sentences are predicted to belong to a category then the summary will have
Automatic Evaluation of Summaries We report the automatic evaluation results for the end-to-end summarization of contracts Table 4, C SUM achieves the best scores for both MAP and

---

ContraSum
Ku-Sum
Preceoonk
F1.K 8
TeatRank
PACSUM
Random
bound
Reference
Figure 3: Precision @ K. Recall @ K. and F1@K (K (1, 10}) scores averaged across 3 folds for the end end-to-end summaries at CR=0.15. All the models (except Reference) use predicted categories for the final summary.
NDCG computed against the gold-references at different compression ratios establishing the need for Nomain--ppciife notion of importance which is not captured the other baselines. Surprisingly. Random baseline performs better than (LSA) or is comparable to other baselines (KL--Sm) when predicted categories are used. While PACSUM achieves better scores than LexRank at low com- pression ratios, using centrality does not help at CR=0.15, As expected, we observe consistent increase NDCG with the increase in the compres- sion ratio as the number of sentences per category increases the gold-re gold-references. While CONTRA- SUM outperforms all the baselines, it is still far away from the upper-bound performance which uses the gold importance ranking. This calls for more sophisticated and knowledge-driven approach to learning the importance of different sentences.
Model
Info.
Use
AoC1
Red.,
AoIR
Of
Rand
2.96
2.87
2.92
1.92
2.50
2.91
LR
3.00
2.79
3.01
2.04
2.46
3.06
Cs Ref
3.25
3.37
3.25
2.04
2.54
3.50
3.17
3.00
1.75
Table Results for human evaluation of model sum- maries. Mean scores of the 2 annotators are averaged across categories and parties. Overall (O) rating is scaled down half make comparable. Info., Use, AoC, Red., AolR denote informativeness, usefulness. accuracy of categorization, redundancy, and accuracy of importance ranking, resp. Rand: Random (PC), L.R: LexRank (PC), CS: CONTRASUM, Ref: References.
bound leaving scope for improvement in the future. Owing to the recent advancements and power of LLMs, we prompt ChatGPT (details in $A.5) to asses its performance on this task. We find that it is not straightforward for ChatGPT to perform the task with simple prompting due hallucinations in the generated output and token limit. Further work is needed to look into how to best use LLMs for such tasks in domains such as legal.
As CONTRASUM is a pipeline-based based system where the sentences predicted as containing each of the categories are input to the importance ranker, erroneous category predictions may affect the final summaries. Thus, we present scores (last block in Table 4) from different systems that use ground- truth (GC) categories and various ranking methods, While the performance of all the systems improves over using predicted categories, the random base- line with GC outperforms all other baselines with GC, except for LSA, suggesting off- off-the-shelf sum- marizers are not well-suited for this task. Neverthe- less, CONTRASUM beats all the systems without the Categorizer indicating the effectiveness of the Importance Ranker in comparing the importance of sentences. We report party-wise results in A.5 (Ta- ble 11) and an output summary in Figure 4 and Figure shows the Precision @k, Rocall @k, and F1 @Kk (1,3,5, 5, 10) trends. As expected, preci- sion decreases with while recall and Fl increase. Similar to Table 4, CONTRASUM outperforms the baselines in each plot while there is a huge gap between our model's performance and the upper-
Human Evaluation of Summaries addition to automatic evaluation, we also give contracts along with their party-specific summaries from CONTRASUM, gold-references (CR=0.15), the best baseline-LexRank (PC), and Random (PC) baseline legal experts for human evaluation. 16 summaries, each consisting of maximum of 10 sentences per category, for contracts are provided to 2 experts. They are asked to rate the summaries for each category on a 5-point scale (1 least; 5 most) per: (1) informativeness: (2) usefulness; (3) accuracy of categorization; (4) redundancy, and (5) accuracy of importance ranking. In addition, ask them to rate the overall quality of a sum- mary on 10-point scale. The average scores are presented Table CONTRASUM produces the best summaries overall but lacks in the diversity of sentences within each category. Interestingly,

---

experts found summaries from CONTRASUM be more informative, useful, and correctly categorized than the gold-references. This may happen as both predicted and reference summaries are capped at 10 sentences per category however, because the cat- egory labels themselves were predicted, the system and reference summaries did not always contain the same number of sentences per category if one was below the allowed limit (and in principle it is not possible to enforce minimum number of sentences per category). It is surprising that possi- ble miscategorizations led to higher human ratings of overall summary outputs; this is an unexpected finding that highlights the potential challenges of evaluating complex, structured summarization out- puts such as in this task (see $A.4 for detailed dis- cussion). Furthermore, experts also considered the importance of a category expressed in a sentence a party. Hence, they penalize for the accuracy of categorization if the category is indirectly (although correctly) expressed in a sentence for a party (see B for an example, category-wiss evaluation results, and more details).
contract. We leave further simplification of extrac- tive summaries future work due the unavail- ability and difficulty in collecting abstractive sum- maries of long contracts and issues of hallucination and factual errors associated with the existing sum- marization and simplification systems (Garimella et al., 2022). (2) CONTRASUM may not be able to include sentences (containing obligations, en- titlements, or prohibitions) that do not explicitly mention either of the parties (e.g., (a)"On termina- tion of the lease, the apartment has to be handed over freshly renovated." " and (b) tenant can vacate the place at any point in time without any consequences. ") in the extracted summaries. While these sentences contain important obligations for tenant and information that is important for land- lord to know, were constrained by the cases that the prior work (Sancheti et al., 2022) covers as our dataset is built atop that dataset. A more sophisticated system could handle example (a) by identifying the tenant as an implicit participant in the "handing over" event, (and the landlord in ex- ample (b) even arguably plays an oblique role as the owner of "the place"); the omission of these cases is arguably due to our method rather than fundamental limitations of the task formulation. Follow-up work could explore these aspects either through the identification of implicit participants, or the expansion of categories. (3) Our data col- lection and model evaluation is limited to lease agreements; studying the generalization of Impor- tance Ranker and CONTRASUM to other types of contracts can be valuable follow-up work. (4) We believe that the most linguistic and semantic at- tributes necessary for category classification and importance ranking are captured at the sentence level. Therefore, to improve the heterogeneity of data under resource-limited scenario in in-terms of number of contracts that we could annotate, we split each contract at a sentence level. However, it is possible that splitting at sentence level may result in an out-of-context annotation. Due to the complexity of involving the whole context and the cognitively challenging nature of the task, we leave the study of document-level generalizability of the annotations for future work. (5) As a sentence may contain multiple categories, same sentence can be a part of summaries for different categories. We leave further segregation of categories within sen- tence and explicit modeling of diversity for future.
9 Conclusions
We introduced a new task to extract party-specific summaries of important obligations, entitlements, and prohibitions in legal contracts. Obtaining ab- solute importance scores for contract sentences can be particularly challenging, as we noted in pilot studies, thus indicating the difficulty of this task. Instead, we collected a novel dataset of le- gal expert-annotated pairwise importance compar- isons for 293K sentence pairs from lease agree- ments to guide the Importance Ranker of CON- TRASUM built for the task. Automatic and human evaluations showed that our system that models domain notion of "importance" produces good-quality summaries as compared to several baselines. However, there is a large gap between the performance of CONTRASUM and the upper- bound, leaving scope for future work including the generation of abstractive summaries.
10 Limitations and Future Work
We note the following limitations of this work: (1) We take a non-trivial step of generating party- specific extractive summaries of contract case contract reviewing and compliance. However, a simplified abstractive summary of key points will be valuable in improving the understanding of a

---

11 Ethical Considerations
Paheli Bhattacharya, Soham Poddar, Koustav Rudra, Kripabandhu Ghosh, and Saptarshi Ghosh. 2021. In- corporating domain knowledge for extractive summa- rization of legal case documents. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Law, pages 22-31.
We are committed to ethical practices and protect- ing the anonymity and privacy of the annotators who have contributed. We paid annotators at an hourly rate of > 12.5 USD for their annotations.
Michael J Bommarito II, Daniel Martin Katz, and Eric M Detterman. 2021. Lexnlp: Natural language processing and information extraction for legal and regulatory texts. In Research Handbook on Big Data Law. Edward Elgar Publishing.
Societal Impact Advances in MI contract under- standing and review, including agreement summa- rization, can reduce the costs of and increase the availability of legal services to small businesses and individuals. We believe that legal professionals would likely benefit from having auxiliary analysis provided by ML models in the coming years. How- ever, we recognize and acknowledge that our work carries possibility of misuse including malicious adulteration of summaries generated by our model and adversarial use Categorizer and Ranker to mislead users. Such kind of misuse is common to any predictive model therefore, we strongly recom- mend coupling any such technology with external expert validation. The purpose of this work is to provide aid legal professionals or laypersons dealing with legal contracts for a better understand- ing of them, and not to replace any experts. As contracts are long documents, party-specific sum- mary of key obligations, entitlements, and prohibi- tions can help significantly reduce the time spent on reading and understanding the contracts.
David Bracewell, David Hinote, and Sean Monahan. 2014. The author perspective model for classifying deontic modality in events. In The Twenty-Seventh International Flairs Conference.
Ralph Allan Bradley and Milton E Terry. 1952. Rank analysis of incomplete block designs: the method of paired comparisons. Biometrika, 39(3/4):324- 345.
Ilias Chalkidis, lon Androutsopoulos, and Achilleas Michos. 2018. Obligation and prohibition traction using hierarchical rnns. arXiv preprint arXiv:1805.088
Ilias Chalkidis, Manos Fergadiotis, Prodromos Malaka- siotis, Nikolaos Aletras, and Ion Androutsopoulos. 2020. LEGAL-BERT' The muppets straight of law school. In Findings of the Association for Com- putational Linguistics: EMNLP 2020, pages 2898- 2904, Online. Association for Computational Lin- guistics.
Lee J Cronbach. 1951. Coefficient alpha and the internal structure of tests. psychometrika, 16(3):297-334.
References
Mohamed Abdalla, Krishnapriya Vishnubhotla, and Saif M Mohammad. 2021. What makes sentences se- mantically related: A textual relatedness dataset and empirical study. arXiv preprint arXiv:21 10. 04845.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and
Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing, arXiv preprint 1rXiv.1810.04803
Abhishek Agarwal, Shanshan Xu, and Matthias Grab- mair. 2022. Extractive summarization of legal deci- sions using multi-task learning and maximal marginal relevance. arXiv preprini 2xXi:22100.14333
Mauro Dragoni, Serena Villata, Williams Rizzi, and Guido Governatori. 2016. Combining nip approaches for rule extraction from legal documents. In 1st Workshop on MIning and REasoning with Legal texts (MIREL 2016).
Elliott Ash, Jeff Jacobs, Bentley MacLeod, Suresh Naidu, and Dominik Stammbach. 2020. Unsuper- vised extraction workplace rights and duties from collective bargaining agreements. In 2020 Inter- national Conference on Data Mining Workshops (ICDMW), pages 766-774. IEFE
Mohamed Elaraby and Diane Litman. 2022. Arglegal- summ: Improving abstractive summarization of legal documents with argument mining. arXiv preprint arXiv:2209.01650.
Tara Athan, Harold Bolcy, Guido Governatori, Monica Palmirani, Adrian Paschke, and Adam Wyner. 2013. Oasis legalruleml. In proceedings of the fourteenth international conference on artificial intelligence and pages 1-.
Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph- basph lexical centrality as salience text sum- marization. Journal of artificial intelligence research, 22:457-479.
Terry N Flynn and Anthony AJ Marley. 2014. Best- worst scaling: theory and methods. In Handbook of choice modelling, pages 178-201. Edward Elgar Publishing.
Keshav Balachandar, Anam Saatvik Reddy, A Shahina, and Nayeemulla Khan. 2021. Summarization of com- mercial contracts.

---

Ruka Funaki, Yusuke Nagata, Kohei Suenaga, and Shin- suke Mori. 2020. A contract corpus for recogniz- ing rights and obligations. In Proceedings of the 12th Language Resources and Evaluation Confer- ence, pages 2045-2053.
Aleksandra Matulewska. 2010. Deontic modality and modals in the language of contracts.
Rada Mihalcea and Paul Tarau. 2004. Textrank: Bring- ing order into text. In Proceedings of the 2004 con- ference on empirical methods in natural language processing, pages 404-411.
Filippo Galgani, Paul Compton, and Achim Hoffmann. 2012. Combining different summarization tech- niques for legal text. In Proceedings of the workshop on innovative hybrid approaches to the processing of textual data, pages 113-123.
James O' Neill, Paul Buitelaar, Cecile Robin, and Leona O' Brien. 2017. Classifying sentential modal- ity in legal language: use case in financial regula- tions, acts and directives. In Proceedings of the 16th edition of the International Conference on Articial Intelligence and Law, pages 159-168.
Filippo Galgani and Achim Hoffmann. 2010. Lexa:
Towards automatic legal citation classification. In Australasian Joint Conference on Artificial Intelli- gence, pages 445-454. Springer.
Bryan Orme. 2009. Maxdiff analysis: Simple counting. individual-lever logit, and hb, Sawtooth Software.
Aparna Garimella, Abhilasha Sancheti, Vinay Aggar- wal, Ananya Ganesh, Niyati Chhaya, and Nandak- ishore Kambhatla. 2022. Text simplification for legal domain: Insights and challenges. In Proceedings of the Natural Legal Language Processing Workshop 2022, pages 296-304, Abu Dhabi, United Arab Emi- rates (Hybrid). Association for Computational Lin- guistics.
Makbule Gulcin Ozsoy, Ferda Nur Alpaslan, and Ilyas Cicekli. 2011. Text summarization using latent se- mantic analysis. Journal of Information Science, 37(4):405-417,
Romain Paulus, Caiming Xiong, and Richard Socher. 2017. A deep reinforced model for abstractive sum- marization. arXiv preprint arXiv: 1 003.04
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, et al. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351.
Wim Peters and Adam Wyner. 2016. Legal text inter- pretation: identifying hohfeldian relations from text. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC' 16), pages 379-384.
Seth Polsley, Pooja Jhunjhunwala, and Ruihong Huang. 2016. Casesummarizer: a system for automated sum- marization of legal texts. In Proceedings of COLING 2016, the 26th international conference on Compu- tational Linguistics: System Demonstrations, pages 258-262.
Moniba Keymanesh, Micha Elsner, and Srinivasan Sarthasarathy. 2020. Toward domain-guided control- lable summarization of privacy policies. In NLLP@ KDD, pages 18 24.
Svetlana Kiritchenko and Saif M Mohammad. 2017. Best-worst scaling more reliable than rating scales: A case study on sentiment intensity annotation. arXiv preprint ArXiv:1712.01700
Keisuke Sakaguchi, Matt Post, and Benjamin Van Durme. 2014. Efficient elicitation of annota- tions for human evaluation of machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 1-11.
G Frederic Kuder and Marion W Richardson. 1937, The theory of the estimation of test reliability. Psychome- Irika, 111):15
Keisuke Sakaguchi and Benjamin Van Durme. 2018. Er- ficient online scalar annotation with bounded support. arXiv preprint 1006::1806.0111111
Solomon Kullback and Richard Leibler. 1951 On information and sufficiency. The annals of mathe- matical statistics, 22(1):79-86 86.
Abhilasha Sancheti, Aparna Garimella, Balaji Vasan Srinivasan, and Rachel Rudinger. 2022. Agent- specific deontic modality detection in legal language. In Proceedings of the 2022 Conference Empiri- cal Methods in Natural Language Processing, pages 11563-1153 11579, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907.1 11622.
Abigail See, Peter Liu, and Christopher D Man- ning. 2017. Get to the point: Summarization with pointer-generator networks. arXiv preprint 1:04v:/704.044
Jordan J Louviere and George G Woodworth. 1991. Best-worst scaling: A model the largest difference judgments. Technical report, Working paper.
Laura Manor and Junyi Jessy Li. 2019. Plain en- glish summarization of contracts. arXiv preprint arXiv:/906.0044
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Controlling politeness in neural machine trans- lation via side constraints. In Proceedings of the

---

2016 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 35-40.
Zejiang Shen, Kyle Lo, Lauren Yu, Nathan Dahlberg, Margo Schlanger, and Doug Downey, 2022. Multi- lexsum: Real-world summaries of civil rights law- suits at multiple granularities. arXiv preprint arXiv:2206. 200.222306.088....
Welderufael Tesfay, Peter Hofmann, Toru Nakamura, Shinsaku Kiyomoto, and Jetzabel Serna. 2018. Priva- cyguide: towards an implementation of the gdpr on internet privacy policy evaluation. Proceed- ings of the Fourth ACM International Workshop on Security and Privacy Analytics, pages 15-21.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. 2019, Huggingface's transformers: State-of- the-art art natural language processing, arXiv preprint arXiv: 1910.03777.
Adam Wyner and Wim Peters. 2011. On rule extraction from regulations. In Legal Knowledge and Informa- tion Systems, pages 113-122. IOS Press.
Razieh Nokhbeh 7aeem, Rachel German, and K Suzanne Barber. 2018. Privacycheck: Automatic summarization of privacy policies using data mining. ACM Transactions on Internet Technology (TOIT), 18(4):1-18.
Hao Zheng and Mirella Lapata. 2019. Sentence cen- trality revisited for unsupervised summarization. In Proceedings of the 57th Annual Meeting of Asso- ciation for Computational Linguistics, pages 6236- 6247, Florence, Italy. Association for Computational Linguistics.

---

A Appendix
Party
Obligation
Rniidement
Probibitao
Verant
A.1 More details on Importance Dataset
Landlord
17.60/
88/121
6.98/21
We hire 3 lawyers from Upwork (1 female, 2 males) based in India. We instructed the annotators rate the importance of sentences with the vision of im- portant sentences being part of summary that can be used for contract review and complaince purposes. We also mentioned to them that these annotations will be used to train a machine learning model to produce a summary of given contract with respect to each contracting party. We also mentioned that their annotations will be shared anonymously for research purposes. We observed increase in annotation reliability with the in- crease in the number of annotations done by the annotators. As the task is subjective, we also ob- served that the perception of importance changes with years of experience.
Table 6: Average(#: Std)/Maxmmum number of sen- tences per contract for each party.
counting-based method did not consider any transi- tivity relations, Therefore, used Bradely-Terry model to get real-valued scores from the compari- son annotations.
Party-wise statistics of the frequency of sen- tences in each category per contract are presented in Table 6. As we focus on lease agreements, there more obligations or prohibitions and fewer en- titlements for tenant compared to the landlord.
A.2 More details on LEXDEMOD
We present the dataset statistics Table 10.
Challenges faced during data collection. We ran below pilot studies with different annotation task designs to obtain reliable annotations.
A.3 Evaluation Metries
The formula for each of the automatic measures is given below. We compute these scores for each category, for each party of a contract.
Rating scale importance for each sentence: We provided sentences from contract to the annotators and asked them to the impor- tance level of each sentence with respect given party on scale of 0-5 where denotes 'not at all important', least important, and most important. We asked the annotators to rate sentences from the preamble as least important as our focus is on a scenario where contract is already been signed. However, faced inter- and intra-rating consistency issues with this task design.
Precision@k true positive@k
true positive@k
Recollak
true positive@k + false negative@k
2 Precision @k Recall @k
Fl
(Precision
k @k Recall@k)
Q
n
MAP
P@k (Rak Ra - 1))
Q 4=1
k=1
Rating scale importance for pair of sen- tences: We provided a pair of sentences and party to the annotator and asked them to rate each sentence's importance level. Providing a pair of sentences provides information on the relative importance. However, we observed inconsistencies in terms of the same sentence being rated with different scores (> +2).
reli
DCG
logg (i + 1)
1
relfrue
IDCG = 1=1
logz (i + +1)
DCG
nDCG IDCG
Combining annotations. For combining the an- notations, we also experimented with simple counting-based method (Orme, 2009; Flynn and Marley, 2014); the fraction of times sentence was chosen as the best (i.e., most important) mi- nus the fraction of times the item was chosen as the worst (i.c., least important). However, there were many ties the resulting scores as well as the
where rel: is the relevance (1 or 0) for the predicted sentences and reffrue is the relevance of the ideal ordering of sentences. We take min(1, (1, number of predicted sentences) for NDCG, is the number of parties or contracts on which the measure averaged.
The average score is computed by using the fol- lowing formula for each metric m, N contracts

---

Model
Accuracy
Precision
Recall
F1
Majority
more sentences (capped at 10) than reference sum- maries (e.g., number of prohibitions is < 10 at 99.53/28.66 20/22266663444 66/34.38
6.49/5.23/11.77
1129//11.9//1.00
4.92/T.66/155
Rule-based
04552/50.54/566
51.81/75.211/8 /80
46.66 4.66//55.544000
50.13/ 46. 16.14/8..7
BERT-BU
T3.68/74.11/76./4
75.84/71.02/77.17
78.81 13.8177.18/75..8
RoBERTa-B
75.53/T1.42/
T3.54/72.17/T4..bB
78.39/T2.88/78..
T4.90/T1.991 /75666
C-BERT-BU 77.50/T3.25, /3/////555/5555 25/55448
76.63/76.22
80.47 80.47/77.54/78.81
77.95/ /2.31//7..66
RoBERTa-I.
33.28/75.003/T1 76.74
7 (005/77.69/777
(4.59/75.21/799
7..1/76.00/77.88
Table 7: Evaluation results for the Content Categorizer. Scores averaged across 3 different seeds are reported for Tenant/Landlord/Bott from Sancheti et al. (2022). BU, B, and I. denote base-uncased, base, and large, respectively.
Module
Accuracy
Precision
Recall
F1
Categorizer
77.80
77.19
Content Categorizer-3
Importance Ranker-2
68.74
76 .29
68.34
78.88
79.50
68.39
79.
68.36
CR = 0.15 for a party): (1) when categorizer makes false positive predictions (2) when predic- tions are correct and capped at 10 but the overall number of sentences belonging to category are less resulting in fewer than 10 sentences in the ref- erences for compression ratio. We believe that the choice of m; be kept fixed justified as there can be false positives or false negatives dur- ing category prediction resulting in more or fewer sentences belonging to a category. Also, setting my to be the same as reference summaries is not a realistic choice as in a real-world setting reference summaries are not available. However, we also provide results (see Table 12) of the systems when my is kept the same as the reference summaries demonstrate that CONTRASUM outperforms others in this setting as well.
Importance Ranker-3
69.42
09:23
69.31
60.26
Table 8: Evaluation results of content categorizer (-fold) and importance ranker (-fold) for remaining two folds. RoBERTa-L is fine-tuned for both the modules as it was the berf-performing model for fold-1.
Clbmuucs
CR-i
CleiL11
Modal
ROUGGE-11OLL
BOUGBA1/2
KL-San
12,32/17 203/20 TL
16.57 MAST722 (2/2223312
LSA
12.75/15 415/2a1.300
41 al) 18/26
02 13.33/18.000 26.22
41 751-2
PACSUM (BERT)
LeaBank
32.22 78:188188 121/2.10) 58
42 58 07722555 55
4%, 19 20/23 38
Cow mASu
22.49 222
99/93 00
76,
87/87 0333833333110
Upper
37 07/04/000000
Raadont
40/03401 61 /26
29/31 40/224
KL. San
38 28:2222 99/ O7
L3 56 (111188.00 LG
U.A
36 44//44441
uG (S) 40/30.50
TT/TT 26/43
PACSUM (BEICT)
39. 11/222
21
CS 14.50 10/200 29-15
Lesllank
25. 16/225
13/224.00
53 077 291 22/22/220000
$2.20
12/25.15
CON CONTRASING-CC
47.45/22 24/018
49.37 15.37/00..3
Table 9: Rouge scores for end-to-end summarization for different compression ratios (CR) averaged across folds. GC (PC) denotes of ground-truth (predicted from the content categorizer) categories. CC: content categorizer.
A.5 Additional Results
Evaluation results of content categorizer against baselines. We report the results in Table 7.
Split
#Sent,
4Spans
Obl
Eat
Pro
Per
Nobi
Nent
Nome
Evaluation results of content categorizer and importance ranker for remaining two folds. We create three splits of the data for the 3 fold cross- validation of CONTRASUM. Each fold consists of 5 contracts in the test set, 1 in the dev set, and the remaining in the train set. The results presented in the main paper for the content categorizer and the importance ranker are for fold-1 and the results for the other two folds of the dataset are presented in PrategorTable 8. As the number of annotated contracts in the:LeDDDMoD dataset is more than that of the importance comparison dataset, we always keep the sentences from non-overlapping contracts in the train set for the content categorizer. We run each experiment with 3 seed values (123, 111, 11) and use the predictions from the model which gives the best result on the dev set. The dev set is kept the same across the three folds for both the content
Train 1262
5279
1811
1281
3:3
265
239
1071
Dev 330
421
176
56
18
22
78
Test
1777
1952
573
418
61
167
101
539
Table 10: Dataset Statisties.
the test set with number of parties as N Nparty, and my metric value for the ktth category.
Aug(m)=
1
N Σ j=1
1 Neategooy
N party
party
A.4 Discussion on Choice of m.
Since my is capped at 10 for predicted summaries and one predicted summary compared against references with different compression ratios, there are two cases when a predicted summary can have

---

Treant
Landlord
CR=0.05
CR=0.10
CR-0.15
CR=0.05
CR=D.10 NDCG
CR=0.15
Model
MAP
NDCG
MAP
NDCG
NDCG
MAP
NDCG
NDCG
Random
0.102
0.162
0,116
0.195
0.139
0.228
0.124
0.178
0,130
0.203
0.144
0.228
KL-Sum
0.076
0.137
0.089
0.178
0.210
0.175
LSA
0:089
0:130
0.102
0.176
0.140
0.221
0.097
0:157
0.103
0.178
0.145
0.229
TextRank
0.095
0.156
0.111
0.199
0.125
0.224
0. 0.148
0. 0.208
0.155
0.236
0.180
0.265
PACSUM (RFRT)
0.151
0.208
0.167
0.257
0.168
0.268
0.119
0.180
0.129
0.206
0.133
0.229
LexRank
0.144
0.200
0.117
0.172
0.206
0.148
236
CONTRASUM
0.207
0.317
0.209
0.236
0.378
0.240
0.321
0.258
0.361
0.289
0.406
Upper-bound
0.579
0:630
0.614
0.688
0.613
0.094
I|
0:579
0.600
0.671
0.015
0.702
Random
0.195
0.298
0,196
0.313
0.221
0.349
0.217
0.258
0,221
0.319
0.236
0.343
0.119
0.190
0.236
0.368
LSA
0:150
0:241
0.173
0.267
0.267
0.366
0:310
0:377
0.306
0.397
0.307
0.41.3
TextRank
0.111
0.209
0.125
0.212
0.146
0.275
0.176
0.265
0,180
0.285
0.210
0.330
PACSUM (RFRT)
0.142
0.256
0.161
0.289
0.182
0.255
0.794
0.203
0.228
0.338
LexRank
0.319
0.181
CONTRASUM-CC
0:355
0.499
0.342
0.495
0.549
0:441
0.551
0.444
0.574
0.467
0.601
Table 11: Evaluation results of end-to-end summarization with respect to Tenant and Landlord for different compression ratios (CR) averaged across 3 folds. GC (PC) denotes use of ground-trui (predicted from the content categorizer) categories; CC: content categorizer.
CR=00
CR=B..0
"lessor" under Landlord for presenting the results. CRe6.15
Model
MAP
NDCG
MAP
NDCG
MAP
NDCG
0.042
0.064
0.080
0.118
We observe similar trends in the performance of dif- ferent models the overall performance presented
KT-Samn
0.077
0.082
0.063
0.124
0.021
0.119
TestRink
0.000
(BBRIT
0.661
0.300
0.338
in the main paper. CONTRASUM outperforms each of the baselines across both the measures and for both the parties. As the number of obligations and entitlements more than prohibitions and since we cap the maximum number of predictions at 10, as
0.125
CONTRASUM
0.136
0.152
0.219
Uppar-oond
0.540
0.686
0.573
0.642
0.580
0.671
Random
0.107
0.125
0.117
0.167
0.132
0.188
KI-Samm
0.072
0.078
0.100
0.145
0.100
0.165
LSA.
0.161
0.163
0.168
0.282
0.229
0.251
TestRonk
0.057
0.054
0.622
0.077
0.000
0.128
(BFRT)
0.096
0.067
0.069
0.104
0.107
0.160
LeaRank
0.089
0.102
0.117
0.164
0.150
0.212
the compression ratio inereases the % improvement
CONTRASUM-OC
0.248
0.275
0.360
0.535
0.541
0.456
will be small.
Table 12: Results of end-to-end summarization for dif-
Qualitative outputs for the end-to-end summa- rization task from ChatGPT Since contracts are much longer than the token limit of chatGPT (3.5- turbo), we cannot prompt it to generate the sum- mary of all the obligations for a tenant in one go. Instead, we segment the contract at a page-level and first prompt (Can you exiract sentences that mention any obligations for the Tenant from this text. Ensure that sentences are present in the pro- vided contract.) ChatGPT to extract all the obliga- tions mentioned for the tenant in the provided text from this contract Since the output of ChatGPT is not deterministic, after running the same prompt for 5 times, we observed that the output sentences were extractive only 50% of times. Also, generated obligations were sometimes hallucinated; generic but not present in the text of the page provided as context. For e.g., it generated "Section 13.1 of the lease mentions that it is the Tenant's responsibility to maintain the Leased Premises in good condition and repair." obligation for the first page in the contract.
ferent compression ratios (CR) averaged across 3 folds when the maximum number of sentences per category in the output summary is set same the reference sum- maries. GC (PC) denotes use of ground-truth (predicted from CC) categories. CC: Content Categorizer.
categorizer and the importance ranker.
ROUGE scores for end-to-end summarization task. We report ROUGE- 1/2/L. for the summariza- tion task in Table 9. We observe similar results as in Table CONTRASUM outperforms all other baselines on ROUGE-2 and ROUGE-L for all com- pression ratios. A minor difference in ROUGE-1 scores among different baselines might be because of the limited vocabulary used in legal documents resulting in similar 1-grams. There is huge gap between CONTRASUM and bound scores showing great room for improvement. This gap increases with the increase in compression ratios. End-to-end summarization results for each party. We present the summarization results with respect to each party averaged over the folds in Table 11 for Tenant and Landlord. We club "ten- ant", and "lessee" under Tenant, and "landlord" and
*We experiment for one contract https: /tww///wwwww///// sec. gec. Archives/ (addgaardaaaa17776////e/ 14428419013746/ tv516010_er 10-1 htm

---

Full sample summary. We provide the full sample summary Figure 4 and Figure 5.
for the Tenant and an indirect entitlement for the Landlord. So, in case this sentence present in the summary for both tenant and landlord, then catego- rization score with respect to tenant will more than for landlord. Usefulness is scored with respect to a summary that the experts have in mind after reading the contract, whereas informativeness is scored only the basis of the produced summary. We present the category-wise human evaluation scores Table 13. CONTRASUM obtains good scores most of the measures and even better than references in a few cases. However, it is rated low on diversity which is possible because we do not ex- plicitly account for diversity during our modeling.
A.6 Implementation Details
We use Adam optimizer with a linear scheduler for learning rate having an initial learning rate of 2e-5, and warm-up ratio set at 0.05. All the models are trained and tested NVIDIA Tesla V100 SXM2 16GB 904 GPU machine. We exper- iment with batch size {2, 4, 8) (E {8, 16, 32}), number of epochs {3,5, 10, 20, 30) (E (1,3)), learning rate E {le-8, 2e 2, 5e 5] (E {1e-5, f1e-B 2e-5, 3c 35-5, 1.5e- 5e- "), and warm-up ratio E
{0.05,0. 0.10} (0.05) for the content selector (impor- tance predictor). BERT-base (110M parameters) and Roberta-base (125M parameters) models took 46 (120) mins, and RoBERTa-large (355M param- eters) took (6) hrs to train for content selector (importance predictor). We use choix python pack- age for Bradley-Terry model's implementation. We use official implementation and released models of PACSUM 9
B Human Evaluation
We perform human evaluation of 16 summaries, 8 each for 2 contracts; 1 for Tenant and 1 for Land- lord. Annotators are provided with contract and the output summaries to rate them several crite- ria. Different criteria are defined as:
Informativeness: How informative is the mary for the given party from the review or compliance perspective?
2. Usefulness: How useful is the summary for the given party from the review or compliance per- spective?
3. Accuracy of Categorization: How correct is the partitioning of the sentences in each category?
4. Redundancy: How much of the content of the summary is repetitive or about the same topic?
5. Accuracy of importance ranking: How correctly are the sentences within category ranked?
6. Overall: How good is the overall quality of the whole summary?
The annotators were legal experts from India (1 fe- male, 1 male); they went through the contract, and then scored the summaries. For the correctness of categorization, they also considered if the sentence belongs to category directly or indirectly with respect to a party. For e.g., "Tenant shall pay the rent to the landlord", represents a direct obligation
""The contract available https://wwa http://wwwwwwc.....ec.. sec. gov/
Archi trchives/edgadd /ete:///aaa///aaa///////////// 77777777777801114442281117777 10516600_ex1x1111
I "The contract is available at https: aennive//edgar/dtt////7//// 144 14428419013766/ tv516010_ee 10-1.htm

---

Model
Acc. of Categorization!
Informnativeneset
Redundancy]
Uscfalnesst
Acc. of Importance Ranking"
Random (PC)
3.25/2.75/2.75
3.00/2.65/3.0 25
1.50/2.37/L.8
2.73/2.75/3.12
2.25/2.50/2.75
LecRank (PC)
3.00/3.25/2.87
2.75/3.37/22
2 50 50/2.25
2.50/'. 87/2
2.37 /2.
CONTRASUU
3.62/3.12/3.00
3.37/3.50 /2..7
2.25/1.87, /28/000
3.37/3.37/ 33.37
2.50/2.12/3..2
Reference
3.25/3.62/2265
3.37/3.50/2.62
187/2.00/1.37
2.87/3.50/2.62
2.25/3.12/3322
Table 13: Category-wise (Obigaiio/Enniieennn//hhhbbootoo results for human evaluation of model summaries. Mean scores of the 2 annotators are averaged for the parties and then averaged across contracts.
Obligations
1. any Installment Rent due from Tenant not received by Landlord within three (3) days after the date such payment is due,
Tenant shall to Landlord (4) an additional sum of tan percent (10%) of the overdue Rant as late charge plus (b) interest at
lesser of (a) fifteon percent (15%) and (b) the highest permitted by Applicable Laws. 2. Tenant shall pay to Landlord as Additional Rent all sums so paid or incurred by Landlord, together with interest at the Default Rate, date such sums were pald Incurred.
annual rate (the "Default Rato") equal
computed from
Tonant shall make all arrangements for and for all water, sewar, gas, heat, light, power. telephone service and any other service utility Tenant requires
Premises.
4. If Tenant refuses or neglects to repair or maintain (or commence and pursue the process of repairing or maintaining) the Premises as required hereunder the reasonable satisfaction Landlord, Landlord, at any time following (10) business days from date on which Landlord shall make written demand on Terant to affect such repair maintonance, may, but shall not have the obligation msintenance (without liabiiity Tensnt for any loss damage which may occur Tenant merchandise, fixtures or other personal property, or to Tenant's business by reason thereoff and upon completion thereof, Tenant shall pay to Landlord as Landlord Additional Rent Landlord's costs making such repairs, plus interest at Default Rate date of expenditure by Landlord upon demand therefor.
to, make such repair
Tenant shall indemnifly, save, defend (at Landlord's option Landlord Indemnitees harmlees from and against any Claims arising from any such liens, including any administrative, court or other legel proceedings related to
counsel reasonably acceptable Landlord) hold the
6. In the event that any utiliiees are furnished by Landlord, Tenant shall pay to Landlord the cost thereof as an Operating Expense. cost expense, provide eliminators other devices (such fliters, cleaners, scrubbers and whatever other equipment may in Lardlord 's judgment be nacessary or appropriate from time to time) to abate any odors, substances in Tenant's exhaust stream that emanate from consistent with the Permitted Use.
Tenant shall,
Premises commercially reasonable level
permitted by Tenant indemnify, protect, defend and hold Landlord (and employees and agents) harmless from and against any and all claims, costs, expenses, suits, judgments, actions, Investigations, proceedings and liabilities connection with Tenant's obligations under this Section 20, including, without limitation, any acts, omissions negligence in the making or performance of any such repairs or replacements. The following conditions procadent indemnify, save, defend (at Landlord's 's option and with counsal reasonably acceptable to Landlord) and hold the Landlord Indemnitees harmless from and against any and Claims any kind or nature, real or alleged, arising from (a) injury to death of any person or damage to any property accurrig within or about the Premises arising directly or indirectly out of the presence at or acoupancy of the Premises Project by Tenant Party, (b) act omission the of any Tanant Party, (c) breach default by Tenant the performance of any of its obligations hereunder, including with respect to compliance with the obligations of Landlord under the Oil and Gas Lease; or injury death of persons or damage to loss of any property, real alleged, arising from the serving of any intoxicating substances at the Premises or Project, except to the extent directly caused by Landlord gross negligence willful misconduct.
To the fullest
arising
Transfor
Landlord considering request Tonant Transfer: Tanant agrees
10.Tenant shall reimburee Landlord for all third-perly costs actually incurred by Landlord connection with any Alterations, insluding Landlord's third-party costs plan review, engineering review, coordination, scheduling and supervision thereof.
Prohibitions
1. Except for odors and fumes that are consistent with the Prior Course of Dealing or that are typical and customary in connection with operations permitted under the Oil and Gas Loase, which, any event, not otherwise violation of any Applicable Laws CC&Rs. Tenant shall not cause or permit (or conduct any activities that would cause) any release of any odors or tumes of any kind from the Premises.
Each Landlord and Terant shall keep the terms and conditions of this Lease confidential and shall not (a) disclose to any third party any terms conditions this Lease any other Lease-related document (including subleases, assignments, work letters, construction contracts, letters of credit, subordination agreements, non-disturbance agreements, brokerage agreements or estoppelc) (b) provide to any third party original copy of this Lease (or any Lease-related document).
The following are conditions precedent to a Tranefer or to Landlord considering a request by Tenant to a Transfer: Whenever consent approval of either party required, that party shall not unreasonably withhold, condition delay such consent or approval, except as may be expressly set forth to the contrary.
Figure 4: Obligations and permissions of full sample summary produced by CONTRASUM with respect to Tenant for a contact 10 The sentences appear in decreasing order of importance in each category. Entitlements in Figure 5

---

Entitlements
Notwithstanding anything to the contrary set forth in this Lease, in the event that (a) a Notice of Demand filed with the AAA in connection with a Disbursement Claim (es such terms are defined In the Development Agroement), (b) the Final Arbitration Decialon (as defined in Development Agreement) requires Landlord to pay any Required Arbitration Construction Payments or any other monetary damages Tenant, and (c) Landlord fails to pay such monetary damages to Tenant within thirty (90) days of the issuance of such decision, then Tenant shall be entitled to offeet payments of Rent due under this Lease required to be made by Tenant under from Tenant under this Lease until such time as Tonant has been relmbursed full for the unpaid amount of such award. The following conditions precedent to Transfer or to Landlord considering request Tanant to Transfer: Netwithstanding anything to the contrary contained herein, Tenant may, at any time and from time to time, without the consent of Landlord, aesign this Lease or any interest hereunder to, or sublesse or Icense the Premises or any part thereof to (each of the following is referred to horein a "Pormitted Transfor"); (a) any successor entity Tenant resulting from merger, reorganization, consolidation with Tenant (provided that such merger, reorganization or consolidation is undertaken primarily for independent business purposes, and not primarily for purposes of transferring this Leese or any interest in the Premises); (b) any initial public offering by Tenant or any or affilliates, (c) any entity succeeding to all or substantially all of the business and assets Tenant (provided that such transaction is undertaken primarily for independent business purposes, and not primarily for purposes of transferring this Lease in the Premisesi; (4) any entity that, as of the date of deterinaation, an Affiliate of Tenant; or (e) any entity that, concurrently with such Transter, acquiring all substantially of the business being conducted at the Premises by Tenant or affiliates, provided that Tenant shall notify Landlord in writing at least twenty (20) days prior to the effectivaness of such Permitted Transfer, (1) Tenant shall provide Landlord with evidence reasonably satisfactory to Landlord that the Transfer quaífies as a Permitted Transfer and shall otherwise comply with the requirements of this Lease regarding such Tranefer, (li) the transferee has a worth that is equal or greater than the net worth of the transferring Tenant, and (M Tenant and esch Guarantor (in accordance with such Guarantor's Guaranty) shall remain fully liable for the payment of all Rent and other sums due or to become due hereunder, and for the full performance all other torms, conditions and covenants be kept and performed by Tonant.
interest
3. if (0 the estimated cost of restoration is less than One Million Dollars ($1,0C0,000.00 (1) prior commencement of restoration, Default or event which, with the passage of time, would give rise to a Default shall exist and no mechanics' or materislmen's 's liens shall have been filed and remain undischarged, (ii) the architects, contracts, contractors, plans and specifications for the restoration shall have been approved Landlord (which approval shall not be unreasonably withheld delayed], (V) Landlord shall be provided with reasonable assurance against mechanics' liens, acorued or incurred, as Landlord or its lenders may reasonably require and such other documents and instruments as Landlord or its lenders may reasonably require, and (v) Tenant shall have procured acceptable performance and payment bonds ressonably acceptable to Landlord in amount and form, and from sursty, reasonably acceptable to Landlord, and naming Landlord as an additional obligee; then Landlord shall make available that portion of the Insurance Proceeds to Tenant for application to pay the costs of restoration Incurred by Tenant and Tenant shall promptly complete such restoration.
4. If the estimated cost of restoration is equal to or exceeds One Million Dollars $1,00000000000 and if Tenant provides evidence satiefactory to Landlord that sufficient funds are available to restore the Premises, Landlord shall make disbursements from the available Insurance Proceeds from time time an amount not exceeding cost of the work completed since the dale covered by the last disbursement, upon receipt of ()) satisfactory evidence, including architect's certificates, of the stage of completion, of the estimated cost of completion and of performance of the work to date in good and workmanlike manner in accordance with the contracts, plans and specifications, (i) reasonable assurance against mechanics materialmen's 's liens, accrued incurred, satisfactory bring-to
Landlord or its lenders may ressonably require, (li) contractors' and subcontractors' sworn statements, date of title insurance, (M) performance and payment bonds reasonably acceptable to Landlord in an amount and form, and from a surety, reasonably accoptable to Landlord, and naming Landlord as an adcitional obligoe, (vi) such other documents and instruments Landlord or its lenders may reasonably require, and (vii) other evidence of cost and payment so that Landlerd can verity that the amounts disbursed from time to time are represented by work that is completed, in place and free and clear of mechanics' lien claims
Tenant shall keep the Premises free from any lians arizing out of work services performed, materials furnished to obligations incurred by Tenant, os, in the alternative, Tenant may bond over any liens to the reasonable salisfaction of Landlord.
The following are conditions precedent to a Transter or to Landlord considering a request by Tenant to a Transfer: Tenant sublets the Promises or any portion thereof, Tenant horeby immediately and irevocably assigns to Landlord, security for Tanant's obligations under this Lease, all rent from any such subletting, and appoints Landlord as assignee and attoreey-nn-fact for Tenant, and Landlord (or receiver for Tenant appointed on Landlord's 's applicttion may collect such rent and apply toward Tenant's Default (as defined below) by Tenent, Tenant shall have the right
obligations under this Lease; provided that, until the occurrence to collect such ront.
Subject to the terms of the Oil and Gas Lease and except for emergency maintenance or repairs, the right of entry contained in this paragraph shall exercisable during business hours with not less than 24 hours prior notice, and subject Tenant's authorized personnel accompanying Landlord's agents in all areas of the Premises.
8. Any Insurance Proceeds remaining upon completion of restoration shall be refunded to Tonant up to the amount of Tenant's payments pursuant the immedialely preceding sentence.
9. The following conditions precedent to Transfer or to Landlord considering a request by Tenant Transfer: Tenant shall be entitled to any award that is specifically awarded as compensation for (a) the taking of Tonant's personal property or trade fixtures that were purchased installed at Tenant's 's expense, including without fimitation, any crops or inventory at the Property and (b) costs of Tenant moving to a new location.
10. The following are conditions precedent to a Transfer or to Landlord considering a request by Tenant to a Transfer: In the event of any default by Landiord, Tenant shall give notice by registered or certified mail to any (8) beneficlary of a deed of trust or (b) mortgagee under mortgage covering the Premises any portion thereof and to any landlord of any lease of land upon within which the Premizes are located, and shall offer such beneficiary, mortgagee or landlord a reasonable opportunity to cure the default, including time to obtain possession of the Premises by power of sale or judicial action if such should prove necessary to effect cure; provided that Landlord shall furnish to Tenant in writing, upon writton request by Tenant, the names and addresses of such persons who are to receive such notices.
Figure 5: Entitlements of full sample summary produced by CONTRASUM with respect to Tenant for a contact" !1 The sentences appear in decreasing order of importance in each category. Obligations and Permissions in Figure 4


Reviews:
[{"review_id": "9QA0K7TTcH", "reviewer": "Reviewer_at5K", "summary": "", "questions": "There are some questions in my review, the answers will not particularly impact my scoring of the paper. I mostly pose questions in reviews to expose areas I found confusing as a reader or help prompt others' thinking.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging feedback and constructive suggestions. We will incorporate the writing, presentation, and missing references-related suggestions in the camera-ready. \n\n- Not clear that the data is reliable: We screened annotators based on their prior experience with US contract drafting and reviewing (annotators had worked with US-based clients). We agree that importance ranking is a complex task and there will be some level of subjectivity. However, we observed good annotation reliability scores (SHR=0.66; L236-237) with best-worst scaling annotation design. As mentioned in L400-413, the annotated dataset for the two sub-tasks at a sentence (5K) and sentence-pair (148k) level are large enough to train high capacity models via fine-tuning. However, training a supervised system for generating category-wise summaries would require a large amount of agreement-summary paired dataset which is quite challenging and expensive to obtain as legal experts are required. This is one of the benefits of our approach which breaks down the difficult task of document-level end-to-end summarization into two sentence-level sub-tasks.\n\n- Not clear that (or how) this extends to more complicated legal frameworks: This is a good point, we will add the below discussion in the camera-ready. Our annotation framework could be extended to include each pair of contracting parties in a multi-party contract. As one party can have obligations towards any of the other parties, Best-worst scaling could be done per pair of parties. For  e.g., the annotation task could be defined to find the most and least important obligation of Party A towards party B. Similarly from modeling perspective, both parties could be specified in the input.\n\n- Expand on several concepts: \n1. We agree that the lease-specific findings were not interesting but we mentioned it to reinforce the properties of the annotated dataset. We will shorten the findings as suggested.\n2. As mentioned in L167-178, we ran multiple pilot studies and obtained low agreement when sentences were rated independently, as annotators found the task difficult due to high subjective nature of the task. We found BWS to be the most suitable for this task achieving good annotator reliability scores  (SHR=0.66; L236-237). Detailed discussion about challenges faced during annotation process is provided in A.1.\n3. This is a good suggestion. We will add that the best-performing model was chosen on the basis of the best F1-score on the dev set and will add a few lines describing the baselines for content categorizer which were different transformer-based models.\n4. Entitlement is defined as a right to have/do something while permission refers to being allowed to have/do something. We will clarify this difference in the paper.\n\n- ChatGPT : what the hallucinations look like: We have provided a discussion and an example of hallucination in A.5 (L1085-1108) due to limited space. For e.g., ChatGPT generated “Section 13.1 of the 1102 lease mentions that it is the Tenant’s responsibility to maintain the Leased Premises in good condition and repair.\" as an obligation for the first page in the contract (link in the paper). However, that information was not mentioned in that page.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Overall, I am firmly on the fence for this paper. Overall I liked it and found it informative, but am not sure if the underlying annotation reliability issue is a showstopper at this time.\n\n- Problem is well chosen, but motivation has room for improvement\n\nLegal documents touch a wide variety of people, many of whom are ill-prepared to understand and execute contracts correctly. Improvements in allowing people to do so has obvious societal benefit\n\n- Paper describes the pipeline well\n\nI think I could work with a student and stand up something similar", "reasons_to_reject": "Overall, while I am on the fence and this section is longer than the \"reasons to accept,\" I think all but the first concern (annotation reliability) are fixable within this round of review.\n\n- Not clear that the data annotations are reliable\n\nL142-163 describe how the dataset is composed of US contracts, but Appendix A.1 mentions that the lawyers annotating these contracts are based in India. While this experimental design decision makes sense as a cost-control mechanism, there are substantial differences in the legal systems of these two jurisdictions. This means it bears some discussion about how this potential mismatch between annotator and data corpus might affect the labelling and downstream systems that learn from this labelling. Similarly, it isn't clear that this data set size is sufficient to train a high-capacity model.\n\n- Not clear that (or how) this extends to more complicated legal frameworks.\n\nThe paper restricts itself to lease agreements, which is fine (around L90). However, the paper should contain some discussion about generalization questions beyond what we find around L680 in the limitations section. As I read the paper, I was brainstorming a bit about examples beyond \"employment agreements,\" because those share the 2-party asymmetric role structure of a lease agreement. In particular, I thought about multilateral trade agreements, which are multi-party and receive far more scrutiny than housing rental contracts due to the higher stakes. As is, I don't envision many prospective renters or landlords using such a system beyond curiosity. While I recognize that expanding the universe of contracts studied will be laborious, this concern can be dealt with more cheaply by painting a vision early in the paper of the future where diverse types of legal documents have tools to assist laypersons parse said documents, then present this paper as a necessary step along that path.\n\n- Expand on several concepts...\n\n1. ContraSUM methodology: Most of the novelty in this paper lies in the methodology to handle 2-party contracts with content categorization and so on. Parts of the document focusing on results that are peculiar to housing leases are less interesting and are candidates to shorten (e.g., around L240).\n2. Why the BWS scheme described around line 180 is important: I recognize the savings that comes from providing a partial ordering in this way, but it is not clear why this ordering is important in the first place. It seems like Appendix figures 4 and 5 list the sentences in an order based on this importance scores. However, it does not appear to be used for filtration or anything further. Given the design decision that the user will observe all the prohibitions/entitlements/obligations anyway, it seems plausible just list the detected sentences in the order that they appear in the document. In so doing, the labelling effort would be substantially lighter.\n3. Content categorizer baselines: Providing 1-2 sentences saying what the model is and by what measure the prior work deemed it \"best performing\" around L446 would be helpful to a reader. \n4. Footnote 3: It isn't clear to me as a reader what the conceptual separation between entitlement and permission categories are. This whole section would benefit from greater clarity on the definition of all of these categories, as they are critical to understanding the paper.", "questions_to_authors": "There are some questions in my review, the answers will not particularly impact my scoring of the paper. I mostly pose questions in reviews to expose areas I found confusing as a reader or help prompt others' thinking.", "paper_topic_and_main_contributions": "The paper describes a party-specific legal document summarization system, moving from a newly-annotated dataset to a system that seems useful to help understand leases.\n\nOverall, I found this paper to be pretty strong, with my main concern being regarding the reliability of the annotation process.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[Wilson] Wilson et al. The Creation and Analysis of a Website Privacy Policy Corpus. ACL 2016. https://aclanthology.org/P16-1126/\n\n...has also curated a dataset based on legal expert annotations.\n\n[Fok] Fok et al. Scim: Intelligent Skimming Support for Scientific Papers. ACM IUI 2023. https://dl.acm.org/doi/abs/10.1145/3581641.3584034\n\n...presents an extractive summarization system for helping humans process academic documents using a slightly different taxonomy but a similar overall approach.", "Typos_Grammar_Style_and_Presentation_Improvements": "- The bit about LLMs around L560 felt a bit hollow\n\nAs written, the paper seems to insinuate that ChatGPT DID produce output on this problem (or perhaps output on the meta-problem of being asked if it is capable of doing this problem), and that this answer indicated unsatisfactory performance for various reasons. If that is the case, the paper should say more about what the \"hallucinations\" looked like and so on. If that is not the case, the paper should clarify the non-pursuit of that line of inquiry for the provided reasons, as opposed to the pursuit of that line of inquiry and discovery that it was not promising. \n\n- Consider reorganizing Introduction and related work.\n\nWhile I know placing related work before conclusion is conventional by some standards, it is not my preferred formatting. In this case, it has caused background to blend into both introduction and related work. Further, the last line of section 8 is moving into methodology, suggesting that the paragraph contains a nice segue from this section into what is currently section 4."}}, {"review_id": "xPzX7TTJzM", "reviewer": "Reviewer_CFoT", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive feedback and helpful suggestions. We will incorporate the writing and presentation-related suggestions in the camera-ready. \n\n- Comparison against a supervised baseline: We chose not to train a supervised system because it requires a large amount of annotated (document, summary) pairs. We had 11 document-summary pairs at a contract-level which is not enough for training an end-to-end supervised system such as in [1]. Furthermore, obtaining a large number of summary annotations at a contract-level will be expensive as well as cognitively demanding. This is one of the benefits (L107-110) of our approach which breaks the complex task of party-specific category-wise summarization into sub-tasks (multi-label classification, and pairwise comparison at sentence-level) for which annotations are collected at a sentence or sentence-level pair at a large-scale with lesser cognitive demand. This enables us to use existing 5K sentence-level annotations at a sentence-level for the categorizer and collect 148K labeled sentence-pair level samples (from the 11 contracts) that can be used for training the importance ranker (as opposed to 11 contract-summary pairs to do an end-to-end summary generation). Furthermore, summaries are generated specific to each party in our case. Therefore, it is not straightforward to use [1] as a baseline for these two reasons.\n\n- Surprising human evaluation results: For both the predictive and reference summaries, we established a maximum number of sentences allowed in each category (10 sentences). This maximum was the same for both model output and reference summaries. However, because the category labels themselves were predicted, the system and reference summaries did not always contain the same number of sentences per category if one was below the allowed limit (and in principle it is not possible to enforce a minimum number of sentences per category). One way to control for this length mismatch would have been to perform the human evaluation instead on system outputs conditioned on gold categorizations; however, because we did not have the resources to perform a human evaluation for all settings, we instead reserved the human evaluation for comparing the full end-to-end system outputs against the reference summaries. We agree it is suprising that possible miscategorizations led to higher human ratings of overall summary outputs; this is an unexpected finding that highlights the potential challenges of evaluating complex, structured summarization outputs such as in this task. We will include a discussion of this challenge and possible approaches for follow-up work in the final version. We also provide details on what rationales experts used during the evaluation in L1148-1168.\n \n- We will add qualitative comments by the annotators on why a certain decision was made in the camera-ready. The different factors used by the annotators include the degree of liability involved (e.g., ‘blanket indemnification’ was scored higher than ‘costs incurred in alterations’ by a tenant since blanket rights can have an uncapped amount of liability), and liabilities incurred by a more probable event were rated as more important than those caused by less probable events, among others.\n\n- Reproducibility: We have submitted the dataset along with this submission and will publicly release it upon acceptance. The contracts used are publicly available and the LexDeMod dataset can be accessed by directly contacting the authors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I believe this paper introduces an important task, and the introduction does a good job of describing the advantages of a system that can summarize contracts party-wise. The dataset that will be released by the authors seems like a great resource not just for improving legal document summarization, but summarization systems overall, since datasets that focus on this type of entity or aspect based summarizations are rare. I also think the annotation process described here is interesting, since it doesn't require annotation guidelines specific to the domain, that lay out what makes a sentence important. However, the fact that this information is implicit in the rankings provided by the legal experts makes the data and corresponding systems slightly opaque -- I believe it might be useful to see more analysis or interpretation of these scores apart from what is described in Table 2, such as qualitative comments by the annotators on why a certain decision was made, or cluster analysis for high-ranking or poorly ranking sentences. Apart from this, the dataset curation process seems very sound, and the methods described here could influence the development of further summarization datasets. ", "reasons_to_reject": "I do not think this paper should be rejected, but I am not sure if the chosen baselines are strong enough to illustrate the strength of the system. All the baselines are unsupervised, and the only system (apart from the proposed model) that makes use of the annotated data is the upper bound. I would be curious to see how the contra-sum model performs against a supervised system such as in [1]. I'm also surprised by the human evaluation results that rate the system outputs higher than the gold reference for multiple aspects. While the authors correlate this to the size of the summary, I believe at least for the purposes of the human evaluation that the system output length should have been made comparable to the reference. \n[1] Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018b. Ranking sentences for extractive summarization with reinforcement learning. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1747–1759, New Orleans, Louisiana", "questions_to_authors": "", "paper_topic_and_main_contributions": "Summary: This paper investigates how summarization of legal contracts can be improved by providing individual summaries to each party involved in a contract. Towards this goal, they introduce a dataset and model for identifying important sentences in lease agreements, focusing on rights and obligations of each party in the contract. \n\nKey contributions:\n\n1. A dataset of 11 contracts (3000 sentences) annotated by experts for party-wise importance.\n2. An extractive summarization model that identifies sentences describing prohibitions, entitlements, and obligations, ranks them by importance and generates party-specific summaries\n3. Empirical and human evaluations demonstrating the value of the system.", "excitement": 4, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "The paper was well-written overall, but I believe Section 4, and in particular 4.2 could be re-written with illustrative examples to make the process clearer. The current text also repeats content from the introduction and Section 3, which could be removed. Please use algebraic notations consistently: such as using a variable for party and categories as well and mathematically describing the classifier's objective, or only use plaintext descriptions throughout and avoid the Ci, Cj notation.\n\nTypos: Bradley misspelled in line 315."}}, {"review_id": "4uOTklOBve", "reviewer": "Reviewer_EMMa", "summary": "", "questions": "A: Related to my comment above, what is the use case for / value of a summary that only considers obligations, entitlements, and prohibitions of one party?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for constructive feedback and great points to add as discussion. Thanks for pointing out the missing references, we will add them in the camera-ready.\n\n- During annotation, the party perspective is not taken into account: We would like to clarify that we did consider party-specificity during importance annotation (L182-185) as well as modeling of the Importance Ranker (L384). The experts were asked to choose the most and least important sentences (out of 4 sentences) from the perspective of a given party.\n\n- limitations that are not addressed in the paper: We agree that such cases may not be covered by our system. However, we were constrained by the cases that the prior work (Sancheti et. al, 2022) covers as we build our dataset atop the dataset introduced in that work. Including such cases during the importance annotations alone would not have helped as our categorizer is developed using prior dataset; such sentences may be categorized as none and thus may not become part of the summary since ContaSum is a pipeline-based system. In principle, a more sophisticated system could handle example (1) by identifying the tenant as an implicit participant in the “handing over” event,  (and the landlord in example 2 even arguably plays an oblique role as the owner of “the place”); the omission of these cases in which parties play an implicit or oblique role is arguably due to our method rather than fundamental limitations of the task formulation. However, this is an important point which could be explored further in follow-up work, either through the identification of implicit participants, or the expansion of categories. We will clarify this in the limitations section. We would also like to point out that the deontic modality dataset does cover instances such as  “Tenant agrees to indemnify the Landlord from all costs and charges for the Utility Services consumed on the Leased Premises” wherein the obligation of the tenant is also considered an entitlement for the Landlord and therefore can be included in Landlord’s entitlement summary. However, we agree that in case Landlord is not mentioned in the sentence, it may not be included in the summary as described above with the current methods, and this is a noteworthy limitation.\n\n- Is legal importance (which is what the experts annotate) equal to importance for a summary?: This is an excellent point. In our annotation process, annotators were made aware of the end-purpose of their annotations (L200-203), so we believe that the annotators were sensitive to factors like “paying the rent on time” being expected/unsurprising. We did learn from interaction and feedback with our annotators about several factors they considered when determining importance. These factors included the degree of liability involved (e.g., ‘blanket indemnification’ were scored higher than ‘costs incurred in alterations’ by a tenant since blanket rights can have an uncapped amount of liability), and liabilities incurred by a more probable event were rated as more important than those caused by less probable events, among others. As is evident from these examples, the factors that can influence importance may be complex, multi-faceted, and difficult to exhaustively identify, which is why our approach in this work was to allow our annotators to use their legal knowledge to inform a holistic judgment. However, we fully agree that a discussion of the factors involved here is well warranted and we will include a discussion of these factors based on annotator feedback in the final camera-ready.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper presents an interesting application case for tailored summaries\n* A data set annotated by legal experts in a domain where very little annotated data exists so far will be published alongside the paper\n* The authors conducted a human evaluation of their system", "reasons_to_reject": "The main limitation of the paper in my view is that it has no notion or discussion of what is important to the parties, which should be the basis for a party-specific summarization. The approach only considers sentences that explicitly mention one of the parties. That introduces two very important limitations that are not addressed in the paper: (1) If a sentence does not explicitly mention either of the parties, it is not considered. The sentence \"On termination of the lease, the apartment has to be handed over freshly renovated.\" clearly contains an (arguably important) obligation of the tenant, but will be ignored in the summary. (2) If the agreement would contain the sentence (excuse the slight exaggeration) \"The tenant can burn the place down at any point in time without any consequences.\" This would not show up in the landlord summary. For me, this raises the question: In which constellation would it be relevant to have a party-specific summary in the way \"party-specific\" is used in the paper? As a landlord I would certainly want to see that information in the summary, as a lawyer reviewing the agreement for a landlord, I would also certainly want to see that. While a party-specific summary that is tailored to the information needs of one party seems like a good idea to me, it is hard for me to imagine the value of a party-specific summary that only contains the obligations, entitlements, and prohibitions of one party.\n\nAnother aspect of importance that is not investigated or discussed by the paper: Is legal importance (which is what the experts annotate) equal to importance for a summary? The authors use the example of \"paying rent on time\" as an important obligation of tenants. From a legal perspective that's agreeable. For a summary, where brevity is important, it is questionable if this information carries any value to the recipient of the summary because that is information that is probably to be found in every rental agreement and therefore its absence would be more noteworthy than its presence. Some of the existing work suggests e.g. lawfulness (Braun et al., 2017) or risk (Keymanesh et al., 2020) as criteria for importance in a summary. A discussion of the different aspects of importance would have significantly enriched the paper.", "questions_to_authors": "A: Related to my comment above, what is the use case for / value of a summary that only considers obligations, entitlements, and prohibitions of one party?", "paper_topic_and_main_contributions": "The paper presents an approach to generate extractive summaries of the obligations, entitlements, and prohibitions in tenancy agreements separately for tenants and landlords (i.e. the contracting parties), based on a two-staged approach of a content categoriser and an importance ranker. Additionally, the paper provides a new, annotated, data set for summary generation.", "excitement": 3, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "*Summary of Terms and Condtions*\n\nIn addition to the legal document types mentioned in the paper, automated summaries also exist for terms and conditions \n\nT. Perera and T. Perera, \"Barrister-Processing and Summarization of Terms & Conditions / Privacy Policies,\" 2021 6th International Conference for Convergence in Technology (I2CT), Maharashtra, India, 2021, pp. 1-7, doi: 10.1109/I2CT51068.2021.9418090.\n\nDaniel Braun, Elena Scepankova, Patrick Holl, and Florian Matthes. 2017. SaToS: Assessing and Summarising Terms of Services from German Webshops. In Proceedings of the 10th International Conference on Natural Language Generation, pages 223–227, Santiago de Compostela, Spain. Association for Computational Linguistics.\n\n*Question answering *\n\nThere is a number of works on question answering on contracts (mostly privacy policies), which are also relevant in the context of personalised summaries.\n\nWasi Ahmad, Jianfeng Chi, Yuan Tian, and Kai-Wei Chang. 2020. PolicyQA: A Reading Comprehension Dataset for Privacy Policies. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 743–749, Online. Association for Computational Linguistics."}}]

Author rebuttal:

Meta-review: The paper introduces a new dataset and pipeline model for the task of party-specific legal document summarization, focused on leases. The goal is to produce summaries for each party containing specific information regarding obligations, entitlements, and prohibitions.

Reviewers agree that this an interesting use case of tailored summaries. They also agree that the description provided for the pipeline apporach is adequate. The data annotation procress is interesting and, while some concerns were raised regarding the suitability of the annotators, these were clarified in the authors' rebuttal. Authors are encouraged to include this additional information in the paper.

Some concerns were raised regarding the suitabilty of the dataset for training models due to its size. The authors' rebuttal clarified that, indeed, the size is a limitation for training end-to-end models, and that's why they proposed a different pipeline-based approach. This is also the reason why no fully-supervised baselines were included in the comparison. Finally, other concerns were raised regarding relevant  statements that could be missing, either due to not explicitly mentioning the party involved, or because "importance" is subjective. Interesting clarifications on these details were provided, as well as acknoledgments of limitations of the paper. Authors are encouraged to include them in the paper.
Decision: Accept-Main