Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper first introduces and describes Multi-LexSum: a benchmark for multi-document summarization and three levels of granularity (L, S, T). Multi-LexSum consists of the documents comprising the docket for cases. The task is to summarize the facts and procedural history of the case. The paper then evaluates a range of models on Multi-LexSum. They explore summary-summary summarization (i.e. going from a long summary to a short summary) and multi-task summarization. Finally, they measure human-assess quality of the summaries using domain experts, and discuss how such tools can be incorporated into real world pipelines.

strengths: __Significance of contributions__

I think the paper has two strengths. 

First, from the perspective of the legal NLP community, the dataset that the authors introduce is incredibly valuable for several reasons: 

1. Legal NLP benchmarks are *extremely* difficult to build, given how expensive legal annotations are. This is a very high quality dataset, on US case law (also rare!). An open question in legal NLP is whether pretraining on legal texts actually helps performance. A large difficulty in answering this question is the lack of datasets corresponding to “hard” tasks. This seems like a task/dataset which could really push our understanding forward.
2. Summarizing legal texts is a non-trivial task which is a core task that many lawyers do. Law students are required to summarize cases every day in class. Judges summarize cases in legal opinions. Lawyers summarize cases when constructing arguments in court and in legal briefs. Thus, this task could enable tools which have a meaningful impact.  

From the perspective of the machine learning/NLP community, the task that the authors identify–multi-granularity summarization–seems understudied (based on my skim online). Thus, this datasets would be of value to the broader NLP community (and not just those focused on legal applications). 


__Rigor__
The experiments are well done and reasonable in light of the claims made by the paper. 
weaknesses: I don’t really see any major weaknesses!
correctness: Yes, all seems good. 
clarity: Yes, the paper is well written. I have a background in law though, so I’m not sure if some of the legal jargon would be difficult for a non-law CS person to understand.  
relation to prior work: I really like the positioning between (1) prior work on legal NLP, and (2) summarization datasets. This is very well done.
additional feedback: One thing I’m curious about–do the authors have a sense of which documents are most important for producing a summary? My intuition tells me that complaints and opinions would be most valuable, in part because they already summarize the important aspects of a case (e.g. the facts section, the procedural posture section, etc).
review_title: Great dataset

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: The authors present a new dataset of 9,280 expert-authored summaries from large-scale civil rights lawsuits that include three types of summary (tiny, short, and long). They experiment with state-of-the-art models for abstractive summarization (BART, Pegasus, LED, and PRIMERA) on their new dataset and show that these models perform poorly on lengthy summaries.
strengths: * Since labels are difficult to acquire in law, datasets with expert annotations (such as this one) are immensely valuable.
* The task of generating a summary from multiple lengthy documents is mainly untouched but extremely important. As the authors point out, such an automatic summarization would help legal experts in drafting lawsuit descriptions more efficiently, which would benefit citizens in accessing and understanding more legal disputes and resolutions.
* The authors perform an extensive analysis of existing summarizers on their dataset.
* The paper is a joy to read, very clear and well structured.
* The dataset documentation is thorough.
weaknesses: The extent of novelty, technical or scientific depth is limited, yet, appropriate for a resource paper.
correctness: The dataset construction and experimental results seem sound.
clarity: The paper is very well written and structured. Also, it is easy to follow, understand, and good to read.
relation to prior work: The related works are accurately described in the second section.
additional feedback: All in all, the paper is very well written, and the dataset is unique and valuable.
review_title: A well-crafted resource with excellent documentation 

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: Multi-LexSum is a law-specific domain multi-document text summarization dataset. The summaries are claimed to be abstractive. The summaries in this dataset vary from long, short, and tiny and derived from multiple documents. The paper conducts series of experiments to show how challenging the dataset is on the current state-of-the-art text summarization models. 
The contribution of the paper is as follows: 
1 – The summaries have multiple-granularities which means that there are long, short and tiny summaries. 
2 – The summaries are written by law experts. 
3 – The dataset includes a fair number of multiple documents and their summaries 

strengths:  1 – The dataset creates new challenges for the current text summarization models. 
2 – The paper conducted a good series of experiments to support their claim. 
3 – The dataset can enhance the summarization models for the law and civil rights domain

weaknesses: Although the paper has conducted a series of good experiments, but it misses many crucial points that weakens the paper’s claim.  In the paper, there are many ambiguities and unclear statements that hinder the delivery of data construction, experiment settings and data evaluation. 
For data collection and construction, in page two the authors mention that there are about “40,000 source documents and 9,000 expert-written summaries”. Then, in page 4 the number of the samples are less than 4600. There is a great confusion here. Besides, it is not clear that whether each document has multiple summaries or multiple documents that have one or more summary. For example, in table 2 the number of samples for tiny summaries for documents/multi-documents is 1603, 3138 for short summaries, and 4534  for long summaries. The average documents for those summaries vary between 8-10 documents. It is better to show that how many documents that only have one tiny, short, or long summary or one document can have many summaries.  Is each summary derived from more than one document? How many summaries that are only derived from one single document? In addition, there is a misleading claim regarding the size of the dataset. The paper compares the law-specific domain summaries (which is acceptable) to Newsroom and other large text summarization datasets. Newsroom, CNN/Daily Mail and Xsum are far larger in size. How can you claim that your data is large compared to those datasets? 
There are other ambiguities in the data evaluation regarding the characteristics of the summaries. The paper claims that their summaries are abstractive because they are written by authors which is undeniable. However, when looking at the Coverage scores of their analysis to the characteristics of their data the numbers are concerning. The coverage score quantifies how much the summary borrows words from the original document. Multi-Lex Sum average scores range between 0.92-0.96 which indicates a very high number of words borrowed from the document. It is true the sentences are not extracted directly from the document but there are not new words that could represent the summary. Almost all the words are present in the original document which makes them extractive but not necessarily organized in the same order. The Density measure could support the author’s claim because the numbers are not very high. It is better to include figures to show the scores of coverages and density. 
The other concerning point is the human evaluation of dataset. The first attempt did not succeed in page 8, so the authors designed an alternative system to score their human evaluation, but the system is not clearly described, and the results are briefly described which casts some doubts and unclarity on the human evaluation method. What is the specific objective of the human evaluation? Is there a small example of the evaluation in the paper? The detailed examples can be added in the appendix, but the main explanation and short examples should be present in the paper. The intended measurements of Human evaluation are missing.  
For the experiment setting: The paper tries to generate more succinct version (tiny or short summaries) of the long summaries and the documents.  What is the objective of this experiment? Multi-LexSum has already multiple granularities. The experiment of generating summaries from summaries casts doubts on the evaluation. Are the summaries generated only from the original document, from the original document + long summary, or only from the long summary? Each scenario has its own follow-up questions for evaluation. 

correctness: Yes, the dataset is built in a correct and sound method. The evaluation methods can to be revised and the claim of abstractive summaries can be reviewed. 
clarity: Yes, the paper is well organized and well written. 
relation to prior work: Yes, the paper mentions most available works, but it is confusing that sometimes the paper compares to BookSum as the closest work to theirs but the paper does not mention how much it is different from it. The conducted comparison to other relevant works are the same as BookSum even though BookSum is the most closely related to their work. More comparison is expected from the most similar works. 
additional feedback: No
review_title: Multi-LexSum dataset releases 4500 U.S. federal civil rights lawsuits and 9000 summaries

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces Multi-LexSum, a multi-doc summarization dataset based on civil rights lawsuits from U.S. federal courts.

Compared to existing benchmarks, Multi-LexSum provides three granularities (tiny, short, and long) of summaries for source documents. It enables the studies of multi-target summaries at these three granularities. The paper also conducted a series of experiments on Multi-LexSum. The results indicate that exiting popular summarization models cannot achieve satisfying performances.

strengths: 1.	From the perspective of document summarization, Multi-LexSum firstly provides summaries at multiple levels of granularity.
2.	From the perspective of legal AI, the proposed Multi-LexSum consists of legal expert-authored summaries, and would encourage legal AI studies, especially the applications such as legal case summarization etc.
3.	The experiments were conducted by considering both the automatic evaluations and the human evaluations. 
weaknesses: 1.	Some details on data pre-processing, especially the data selection, are not introduced clearly. The dataset is drawn from the ongoing CRLC writing. What features are used for selecting/filtering the data records from the whole posts? Are they selected based on the charge, year, or other features? Sometimes, the inappropriate data selection criteria may result in a dataset with potential social biases. 

2.	Multi-LexSum is based on large-scale civil rights lawsuits. It depends on the lawyers and law students to produce high-quality summaries of key events and outcomes. Is there any critical difference between the general-purpose long document summarization and the summarization in Multi-LexSum? Are there any legal-domain specific designs in Multi-LexSum due to the special properties of lawsuits? 

3.	The details on the annotation are not provided. For example, how many annotators are spent on one document? How about the inter-annotator agreement and how to measure the agreement? 

correctness: The dataset is constructed in a sound way. 
clarity: the paper is well-organized and easy to follow.


relation to prior work: The paper discussed natural language processing for legal documents and summarization datasets in other domains. 
additional feedback: There are some typos. For example, in Line-197, “Test(30%, 908)” --> “Test(20%, 908)"
review_title: A good multi-doc summarization dataset based on civil rights lawsuits 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: Thanks for the opportunity to review this paper. I recommend accepting. 

The authors introduce a new set of summaries of long legal documents concerning civil rights cases. To my knowledge, a set of expert-written summaries of this magnitude is unique for a legal application, especially because many of the previous projects cited rely on data that is subject to extremely restrictive licensing regimes (CanLII and AusLII). And this data is focused on a subject area with high potential for social impact. The existence of multiple levels of granularity is a very nice feature. This is not only useful for legal applications,; many people are increasingly aware of the need to handle long sequences, and this will be a nice task to push models in that direction. The authors' point on the length of output sequences from SOTA models speaks to this point; this data will be really useful in future efforts to solve that problem. Also, I was especially impressed by the HCI contribution, which is all too rare in studies like this one. The paper is clearly written.

Some issues to be aware of. My #1 question is about data quality. In the example notebook, the snippet of the source document suggests serious parsing errors ("DIscTcquR FOR THE MIDDLE DISTRICTLO ALAAMA"). These are not atypical for OCR projects but I wonder what the prevalence of errors like these is. **Please help clarify this in your responses.** Pervasive parsing issues would make learning the summaries difficult and reduce the value of the data. As for the benchmark results, I wasn't sure why the authors chose to finetune for only 6 epochs or the choices that were made to handle other specific elements (e.g. unique legal vocab, citations, etc.) that might have impeded performance. More detail on the HCI summarization project would have been nice to include in the main paper as that's a unique and important element. Overall, I think these problems can probably be solved. 

strengths: The dataset is to my knowledge a unique collection of expert-written summaries of very long documents. The summaries are presented at varying levels of granularity. The data addresses an important social problem. The experiments demonstrate the value of the data. The authors demonstrate a real-world application with feedback from human subjects. 
weaknesses: I am uncertain about the data quality of the source documents. I was not confident that the experiments displayed the best possible versions of the models. 
correctness: With the sole exception of the data quality question for the source documents, everything else seemed to be in great shape. The code repo looks good. I think putting the data on HF would be good.
clarity: Yes, the paper is well written and packs a lot of content into a short space.
relation to prior work: Yes. 
additional feedback: Congrats on really nice work!
review_title: Good dataset on a policy-relevant problem that fills multiple gaps 

Meta-review: metareview: This is a very high-quality dataset, which is especially noteworthy since legal NLP benchmarks are very difficult to build. The authors approached every aspect of the process with extreme care. It will likely have an impact on law practice and start interesting discussions about the use of ML in these settings. The paper is also very well written and enjoyable to read.

Most reviewers are heavily in favor of acceptance. Reviewer Aeqk brought up some weaknesses, but at least from the AC's perspective, these seem to be answered well by the authors.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept