MILDSum: A Novel Benchmark Dataset for Multilingual Summarization of Indian Legal Case Judgments
Authors: Debtanu Datta
Shubham Soni
Rajdeep Mukherjee
Saptarshi Ghosh
Venue: EMNLP 2023 Main (2023)
Keywords: Cross-Lingual Summarization
Multilingual corpus for Summarization
Summarization-Translation Pipeline
Legal NLP
TL;DR: A Dataset Creation and Benchmarking Study for Multilingual (English and Hindi) Summarization of Legal Case Judgments in India.

Abstract: Automatic summarization of legal case judgments is a practically important problem that has attracted substantial research efforts in many countries. In the context of the Indian judiciary, there is an additional complexity -- Indian legal case judgments are mostly written in complex English, but a significant portion of India's population lacks command of the English language. Hence, it is crucial to summarize the legal documents in Indian languages to ensure equitable access to justice. While prior research primarily focuses on summarizing legal case judgments in their source languages, this study presents a pioneering effort toward cross-lingual summarization of English legal documents into Hindi, the most frequently spoken Indian language. We construct the first high-quality legal corpus comprising of 3,122 case judgments from prominent Indian courts in English, along with their summaries in both English and Hindi, drafted by legal practitioners. We benchmark the performance of several diverse summarization approaches on our corpus and demonstrate the need for further research in cross-lingual summarization in the legal domain.

MILDSum: A Novel Benchmark Dataset for Multilingual Summarization of Indian Legal Case Judgments
Debtanu Datta", Shubham Soni", Rajdeep Mukherjee, Saptarshi Ghosh
Indian Institute of Technology Kharagpur
debtanumathcc@@kpiiin litg.....c. ac. shubhamsonkkp@@@ail.comm rajdeepl 989ei i tkgg.. wt..pa................m......... ac. in, saptarshi@cse. iitkgp. ac. ac.in in
Abstract
There exist few Indian legal case judgment summarization datasets, e.g., the datasets devel- oped by Shukla et al. (2022), but all of them contain English case documents and summaries only. In this work, we introduce MILDSum (Multilingual Indian Legal Document Summarization), dataset consisting of 3,122 case judgments from multiple High Courts and the Supreme Court of India in English, along with the summaries in both English and Hindi, drafted by legal practitioners.
Automatic summarization of legal case judg- ments a practically important problem that has attracted substantial research efforts in many countries. In the context of the Indian judiciary, there is an additional complexity - In- dian legal case judgments are mostly written in complex English, but significant portion of In- dia's population lacks command of the English language. Hence, crucial summarize the legal documents in Indian languages to en- sure equitable access to justice. While prior research primarily focuses on summarizing le- gal case judgments in their source languages, this study presents a pioneering effort toward cross-lingual summarization of English legal documents into Hindi, the most frequently spo- ken Indian language. We construct first high-quality legal corpus comprising of 3,122 case judgments from prominent Indian courts in English, along with their summaries in both English and Hindi, drafted by legal practition- ers. We benchmark the performance of sev- eral diverse summarization approaches on our corpus and demonstrate the need for further research cross-lingual summarization legal domain.
There have been previous efforts in compiling document-summary pairs in the legal domain, such as the Eur-Lex (Aumiller et al., 2022) dataset con- taining 1,500 document-summary pairs per lan- guage, for several European languages. However, to our knowledge, there has not been any similar effort in the Indian legal domain. Thus, MILDSum is the first dataset to enable cross-lingual summa- rization of Indian case judgments.
To construct MILDSum, we utilize the website LiveLaw (https: //www. 1 live velaw. in/), a popu- lar site among Indian Law practitioners, that pub- lishes articles in both English and Hindi summa- rizing the important case judgments pronounced by the Supreme Court and High Courts of India. According to the site, these articles are written by qualified Law practitioners; hence, they can be considered as high-quality summaries of the case judgments. We carefully extract the case judg- ments (in English) from this website, along with the English and Hindi articles summarizing each judgment. A major challenge that we had to ad- dress in this process is link the English article and the Hindi article corresponding to the same judgment. The MILDSum dataset is available at https: //gi thub.
1 Introduction
Legal judgment summarization is an important and challenging task, especially considering the lengthy and complex nature of case judgments (Shukla et al., 2022; Bhattacharya et al., 2019; Aumiller al., 2022). In the context of the Indian judiciary, there is an additional requirement Indian legal case judgments are mostly written in complex En- glish due to historical reasons, but a significant por- tion of India's 's population lacks a strong command of the English language. Hence, it is important to summarize case judgements in Indian languages to ensure equitable access to justice.
We also benchmark wide variety of summa- rization models on the MILDSum dataset. We con- sider two broad approaches - (1) a summarize-the translate pipeline, where a summarization model is used over a case judgment (in English) to generate
"Equal contribution by the first two authors

---

English summary, and then the English sum- mary translated into Hindi, and (2) direct cross- lingual summarization approach, where a state-of- the-art cross-lingual summarization model (Bhat- tacharjee et al., 2023) is finetuned over our dataset to directly generate Hindi summaries from the En- glish documents. We observe that the summarize- then-translate approach performs better (e.g.. best ROUGE-2 score of 32.27 for English summaries and 24.87 for Hindi summaries on average) than the cross-lingual summarization approach (best ROUGE-2 score 21.76 for the Hindi summaries), in spite of reduction in summary quality due to the translation stage. Thus, our experiments demon- strate the need for better cross-lingual summariza- tion models for the legal domain. To this end, we believe that the MILDSum dataset can be very use- ful in training and evaluating cross-lingual sum- marization models, for making legal judgments accessible to the common masses India.
Domain
#Doc
Avg. #tokens
Doc
EN Sum
HI
Court Judgments
3.122 4,696
724
695
Table I: Statistics of the MILDSum dataset
the Indian legal domain, and this work takes the first step to bridge this gap.
We refer the readers Appendix for more details about related work.
The MILDSum Dataset
In this work, we develop MILDSum (Multilingual Indian Legal Document Summarization), collec- tion of 3,122 Indian court judgments in English along with their summaries in both English and Hindi, drafted by legal practitioners. Statistics of the dataset are given in Table 1. This section de- scribes the data sources and methods applied to create MILDSum, as well as studies some impor- tant properties of the dataset.
Also note that, apart from training summarizing models, the MILDSum dataset can also be used to train evaluate Machine Translation (MT) models in the legal domain, given the paired English and Hindi summaries. MT in the Indian legal domain is another challenging problem that has not been explored much till date.
3.1 Data Sources
The data for this study was primarily collected from LiveLaw, reputed website known for its reliable coverage of court judgments in India. The site maintains one version English and another in Hindi². Both versions of LiveLaw provide articles summarizing recent case judgments pronounced in the Indian Supreme Court and High Courts. Ac- cording to the site, the articles (which consid- ered as the summaries) practitioners, which gives us confidence about the quality of the collected data.
2 Related Works
written by qualified Law
There has been limited prior work on cross- lingual summarization in Indian languages. A popular large-scale multi- lingual summarization dataset MassiveSumm was introduced by Varab and Schluter (2021) which includes some Indian lan- guages as well. The ILSUM dataset (Urlana et al., 2023) encompasses Hindi and Gujarati, in addition to English. Bhattacharjee et al. (2023) recently introduced a large-scale cross-lingual summariza- tion dataset CrossSum, while Ladhak et al. (2020) released WikiLingua cioss-lingual summarization. However, none of these datasets have specifically addressed cross- lingual summarization within the legal domain.
3.2 Data Collection and Alignment
We observed that the articles in the English and Hindi websites of LiveLaw do not link to the cor- responding articles in the other website. Hence the primary challenge is to pair the English and Hindi articles that are summaries of the same court judgment. We now describe how we address this challenge.
benchmark dataset for
We observed that most of the Hindi article pages contain some parts of the titles of the correspond- ing English articles, if not the whole, in the form of metadata. We leveraged this information to find the matching English article for particular Hindi arti- cle. To this end, we used the Bing Search API with specialized query - site: https:///ww:: wwt.::wwwelwiiii.... in/ (English ritle metadata) - to get the search results from http:://www.liellw..nn/ (the English site), and
There exist some cetti/cross-lingue summariza- tion datasets in the legal domain for non-in languages, such as Eur-Lex (Aumiller et al., 2022) and CLIDSUM (Wang al., 2022). In particular, Eur-Lex is multi- and cross-lingual corpus of legal texts originating from the European Union (EU); it covers all 24 official languages of the EU. But to our knowledge, no such effort has been made in
chttps://wwer livelaw. linelal.nn/ Phttps://hinds livelaw. inn

---

then we considered the first result returned by the Bing Search API.
3.4 Dataset statisties and analysis
MILDSum comprises total of 3,122 case doc- uments and their summaries in both English and Hindi. The average document length is around 4.7K tokens, the average English summary length is around 724 tokens, and the average Hindi sum- mary length is around 695 tokens. The Compres- sion Ratio in MILDSum - the ratio between the length of the English summaries to that of the full documents - is 1:6. The document length distribu- tions of judgments and summaries are reported in Appendix B.2.
To ensure the correctness of the matching (be- tween the Hindi and English articles), we computed the Jaccard similarity between the metadata title (obtained from the Hindi article) and the title on the first page fetched by the Bing Search API (the potentially matching English article). We then en- forced these 3 conditions for accepting this match data-poim (i) If Jaccard similarity is greater than 0.8, then we accept the match, (ii) If Jaccard similarity in between 0.5 and 0.8, then we go for manual checking (whether the two articles indeed summarize the same case judgment), and (iii) Jaccard similarity is lower than 0.5, then we dis- card this data-point. Additionally, we require that at least one of the articles must contain a link to the original judgment (usually a PDF).
We observed that the LiveLaw articles (which are the reference summaries in MILDSum) are a mixture of extractive and abstractive summaries they comprise of both verbatim quotations from the original case judgments (extractive parts) well paragraphs written in simplified and con- densed manner (abstractive parts). To quantify this extract/abstract-iveees of the summaries, we com- puted Extractive Fragment Coverage and Extrac- tive Fragment Density as defined by Grusky et al. (2018). We get a high value of 0.90 for Cover- age, which measures the percentage of words in the summary that are part of an extractive fragment from the document. We also observe high value of 24.42 for Density, which quantifies how well the word sequence of the summary can be described as a series of extractions. These high values are expected as the summaries in our dataset frequently contain sentences directly quoted from the original judgments.
If the conditions stated above are satisfied, then we include the English judgment, the correspond- ing English article, and the corresponding Hindi article, as data-point in MILDSum.
Dataset cleaning: The case judgments we ob- tained are in the form of PDF documents, while the English and Hindi articles are HTML pages. We employed various standard tools Python li- braries to extract the text from the downloaded HTML and PDF files. Particularly for the PDF doc- uments, we utilized the pdftotex/ tool that converts PDFs plain text while preserving text alignment. More details about the text extraction process and pre-processing of the documents are given in Ap- pendix B.1.
Experiments and Results
3.3 Quality of the dataset
We divided the MILDSum dataset into 3 parts in a 70:15:15 ratio - train ser (2185 data points), vali- dation set (469 data points), and test set (468 data points). Only the test split is used to benchmark the performance of several sate--f-ttheart summariza- tion models. We consider two broad approaches (i) mammarrze-then-traastie pipeline approach, and (ii) direct cross-lingual summarization ap- proach that are detailed next.
As described earlier, the MILDSum dataset was constructed by matching English and Hindi arti- cles summarizing the same judgment. Concerns may arise regarding the quality of this matching, as they were not cross-referencee on the websites. To address this concern, we took a random sample of 300 data-points and manually checked whether the English and Hindi summaries corresponded to the said judgment. Only one data-point was found erroneous, where the English and Hindi summaries corresponded different judgments. Hence, conclude that a very large majority (> 99%) of the data-points in MILDSum are correct.
4.1 Wemmaarze-then-Traasse approach
In this pipeline approach, we first use a summariza- tion model to generate an English summary from a given English case judgment, and then we translate the English summary to Hindi. In the first stage, we compare the performances of the following sum- marization models.
Phttps: /

---

Unsupervised Extractive methods: We use LexRank (Erkan and Radev, 2004) a graph- based method that uses eigenvector centrality to score and summarize the document sentences; LSA-based summarizer (Yeh et al., 2005) that uses Singular Value Decomposition to project the singular matrix from higher dimensional plane to lower dimensional plane to rank the important sentences in the document; and Luhn- summarizer (Nenkova al., 201 1) which a frequency-base -based method that uses TF-IDF vec- tors to rank the sentences in a document." 4
mechanism that scales linearly with sequence length, enabling the model to perform seq-to-seq tasks over long documents containing thousands of tokens.
Translation of Generated summaries: In the sec- ond stage of this pipeline approach, all the gener- ated English summaries are translated into Hindi using the Machine Translation (MT) model Indic- Trans (Ramesh al., 2022) which has been specif- ically trained for translation between English and Indian languages. We compared several MT mod- els over a sample of our dataset, and observed In- dic Trans to perform the best. More details on the translation experiments and the choice of Indic- Trans are given in Appendix C.2.
Supervised Extractive models: These models treat summarization as a binary classification task (whether to include a sentence in the summary), where sentence representations are learned us- ing hierarchical encoder. We use SummaRuV- Ner (Nallapati al., 2017) that uses two-layer bi-directional GRU-RNN. The first layer learns contextualized word representations which are then average-pooled pooled to obtain sentence represen- tations from the input document. Second, we use BERTS (Liu and Lapata. 2019) which takes pre-trained BERT (Devlin et al., 2019) as the sentence encoder and an additional Trans- former the document encoder.? Details of training these models on the training split of MILDSum are given in Appendix C.I.
4.2 Direct Lingull Summarization
In this approach, we directly generate a Hindi summary for a given English document (without needing any translation). To this end, we use the state-of-the-art cross-lingual summarization model CrossSun-nT75 (Bhattacharjee et al., 2023) which is a fine-tuned version of m75 (Xue et al., 2021). We specifically use the version that is fine-tumer over all cross-lingual pairs of the CrossSum dataset. where target summary is in Hindi. In other words, this model is meant for summarizing text written in any language Hindi.
We use Legal- Pegasus", fine-tuned version of Pegasus (Zhang et al., 2019) model, that specifically designed for summarization in the legal domain by fine- tuning over the aec-litigation--elle releases dataset consisting of 2,700 US litigation releases & com- plaints. We also use two Long Document Sum- marizers. First, we use LongT5 (Guo et al., 2022), enhanced version of the T5 (Raffel et al., 2019) model with Local and Transient Global attention mechanism than can handle long inputs up to 16,384 tokens. Next, we use LED (Longformer Encoder Decoder) (Beltagy et al., 2020), a Longformer variant with an attention
Pretrained Abstractive models:
4.3 Experimental setup
Chunking of long documents: Abstractive mod- els like Legal-Pegasus and CrossSum-niTS have an input capacity of 1024 tokens and 512 tokens re- spectively; hence a case judgement often cannot be input fully into such a model. So, to deal with the lengthy legal documents, we chunked the docu- ments into m small chunks, where the size of each chunk is the maximum number of tokens (say, n) that the model is designed to accept without trun- cating (e.g., n 512 for CrossSum-m73), Then, we asked the model to provide summary of k tokens for each chunk, and append the chunk-wise summaries in the same order in which the chunks appear in the document, such that the combined summary (of total m k tokens) is almost equal in length to the reference summary.
"We used the implementations of these unsupervised meth- ods from https: com/ mi co-belice ca/ sumy. / wwwteehuu..co/hhhthteeee //ttps:///tthub wwtp.//aww././ummmaaeao Suurraoo/uumRRuNWeerrrr
proceprocessing step called Trigram Blocking that excludes candidate sen- tence if has a significant amount of trigram overlap with the already generated summary to minimize redundancy in the summary. But, we observed that this step leads to too short summaries, as also observed by Sotudeh al. (2020). Hence, we ignore this step in our work.
"In the original BERTSumFXr, there
Long document summarizers such LED and LongT5 have the input capacity of 16,384 tokens; hence, these models successfully handled almost
"nttps://huggingff acce. ndi_ crs_ccossSum
Phttps: / /h/ggguugggacccceeeeer acee. cw/nsinn/nn/eeeee/eccccccoo 1etle//9/llllaaee -aggal--pegaus

---

English Summary
Hindi Summary
Model
ROUGE-2
ROUGE-L BERT Score
ROUGE-2
ROUGE-L BERT Score
Extractive Unsupervised
LexRank
30.49
29.24
83.97
23.83
23.73
74,36
LSA-based
49.46
28.54
83.71
23.09
23.27
74.07
Lathn
28.76
28.12
83.58
22.46
22.93
73.80
Extractive Supervised
SummakuNNer
32.27
30.34
84.13
24.87
24.55
74.30
BERTSumEXt
28.71
29.11
83.77
21.11
22.50
73.36
Abstractive
Legal- Peasssu-tuu---eeeeeedd
31.84
28.36
84.14
24.40
22.58
74.62
Abstractive Long Document Summarizer
LongT5 fifeetuued
23.25
23.47
83,90
21.32
20.34
75.70
L.D L.DD-finetneed
25.51
27.81
84.20
19.56
22.51
74.73
Direct cross-lingual summarization (without translation)
CrussSum-mi15
8.75
13.86
70.55
CrossSum-milS--fnettur
21.76
20.68
75.05
Table 2:
Benchmarking different types of summarization methods over the MIL.Suu dataset. scores are averaged over the test split of MILDSum. The best value of each metric is boldfaced.
all (~96%) the case judgments in MILDSum with- out truncation chunking. The few documents that longer were truncated at 16,384 tokens.
ments (as stated earlier in Section 3.4). However, the abstractive long document summarizer methods (LongT5-finetuned and LED-finetuned) perform slightly better than extractive methods in terms BERTScore.
We finetune the abstractive models Legal-Pegasus, Long75, LED, and CrossSum-m775 using the train and vali- dation split of MILDSum. The method for generat- ing finetuming data is explained in Appendix C.3. dlpeepaarametes: For all models, have used the default hyperparameters, since we wanted to benchmark their off-the-shelf performances over our dataset. We used a default seed value of 42 to initialize the model parameters before fine- tuning/training, to ensure that the same results can be reproduced with the same settings." 10
Fine-tuning of abstractive models:
The direct cross-lingual summarizer, Cross- Summ, performed poorly when used off-the-shelf (without fine-tuning). But, CrossSumm--fnetuuee (fine-tuned over the train split of our MILDSum corpus) outperforms the off-the-shelf CrossSumm by significant margin. This clearly shows the sig- nificance of our MILDSum corpus in direct cross- lingual summarization.
Overall, based on our experiments, the "Summarize-thee Translate pipeline approach' per- forms better than the DDirect Cross-Lingus Sum- marization' over MILDSum. Note that, in the pipeline approach, the translation stage usually in- troduces some additional errors, shown by the consistently lower scores for Hindi summaries than the corresponding English summaries. In spite of the additional translation errors, the Summarize- then-Transie Translate approach achieves higher scores over the MILDSum dataset. This result shows the need for improved cross-lingual summarization models for the legal domain in future.
Evaluation metries: To evaluate the quality of the summarics generated by the models, we considered these standard metrics ROUGE (Lin, 2004), and BERTScore (Zhang et al., 2020). Specifically, we reported the FI scores of ROUGE-2, ROUGE-L, and BERTScore. More details on how the metrics are computed are reported in Appendix C.4.
4.4 Main Results
Table reports the performance of all methods stated above, where all metric values are aver- aged over the test split of MILDSum. We see that the extractive method SummaRumner achieved the highest ROUGE scores, closely followed by Legal- gggasus-finetuncee This is possibly because the reference summaries in MILDSum have mix of extractive and abstractive features, and they fre- quently quote exact lines from the original judg-
Conclusion
This study develops the first multi- and cross- lingual summarization dataset for Indian languages in the legal domain (available at sttps://g/t ub com/Law- AI AI/MILDSum). We also benchmark variety of summarization models on our dataset. Our findings emphasize the need for better cross- lingual summarization models in the Indian legal domain.
10 The experiments were carried out on a system having a NVIDIA V100 16GB GPU. The total GPU time needed for all fine-tuning experiments was approximately 48 hours.

---

Limitations
Iz Beltagy, Matthew E. Peters, and Arman Cohan, 2020. Longformer: The gongddoumentt transformer. ArXiv, abs/2004.05
One limitation of the MILDSum corpus is that it is developed entirely from one source (the LiveLaw site). However, sources that provide cross-lingual summaries in the Indian legal domain (i.e., En- glish judgments and corresponding summaries in some Indian language) are very rare. Also note that our dataset covers a wide range of cases from diverse courts such as the Supreme Court of India, and High Courts in Delhi, Bombay, Calcutta, etc. Also, the different articles (summaries) were writ- ten by different Law practitioners according to the Livelaw website. Therefore, we believe that the dataset is quite diverse with respect to topics as well as in writers of the gold standard summaries.
Abhik Bhattacharjee, Tahmid Hasan, Wasi Uddin Ah- mad, Yuan-Fang Li, Yong-Bin Kang, and Rifat Shahriyar. 2023, CrossSum: Beyond English-centric tross-lingual summarization for 1,500+ language pairs. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Vol- ume 1: Long Papers), pages 1541-2564, Toronto, Canada. Association for Computational Linguistics.
Paheli Bhattacharya, Kaustubh Hiware, Subham Raj- garia, Nilay Pochhi, Kripabandhu Ghosh, and Sap- tarshi Ghosh. 2019. A comparative study of summa- rization algorithms applied to legal case judgments. In Advances in Information Retrieval, pages 413-428, Cham. Springer International Publishing.
Also, a good way of evaluating model-generater legal summaries is through domain experts. While an expert evaluation has not been carried out on the MILDSum dataset till date, we plan to con- duct such evaluation of the summaries generated by different methods as future work.
Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Another limitation of the dataset is the lack of summaries in Indian languages other than Hindi. practically important future work is to enhance the dataset with summaries in other Indian languages.
Günes Erkan and Dragomir R. Radev, 2004. Lexrank: Graph-based lexical centrality as salience in text sum- marization. J. Artif. Int. Res., 22(1 22(1):457-479
Ethics Statement
Fangxiaoyu Feng. Yinfei Yang. Daniel Cer, Naveen Ari- vazhagan, and Wei Wang. 2022. agnguac--agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume I: Long Papers), pages 878-891, Dublin, Ireland. Association for Computa- tional Linguistics.
As stated earlier, the dataset used in this work is constructed from the Livelaw website, using con- tent that is publicly available on the Web. Accord- ing to the terms and conditions outlined on this web- site, content such as judgments, orders, laws, regu- lations, or articles can be copied and downloaded for personal and non-commereial use. We believe our use of this data adheres to these terms, as we are utilizing the content (mainly the judgments and the articles from which the summaries are obtained) for non-commercial research purposes. Note that the MILDSumm dataset created in this work is meant be used only for non-commercial purposes such as academic research.
Max Grusky, Mor Naaman, and Yoav Artzi. 2018. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume 1 (Long Pa- pers), pages 708-719, New Orleans, Louisiana. As- sociation for Computational Linguistics.
Mandy Guo, Joshua Ainslie, David Uthus, Santiago On- tanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. 2022. Long'T5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguisties: NAACL 2022, pages 724- 736, Seattle, United States. Association for Compu- tational Linguistics.
References
Dennis Aumiller, Ashish Chouhan, and Michael Gertz. 2022. EUR-Iex-sum: A multi- and cross-lingual dataset for long- form summarization in the legal do- main. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 7626-7639, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Tahmid Hasan, Abhik Bhattacharjee. Md. Saiful Is- lam, Kazi Mubasshir, Yuan-Fang Li, Yong-Bin Kang. M. Sohel Rahman, and Rifat Shahriyar. 2021. XI. sum: Large-scale multilingual abstractive summariza- tion for 44 languages. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021,

---

pages 4693-4703, Online, Association for Computa- tional Linguistics.
2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 4512 1522-4525, Online. Association for Computational Linguistics.
Faisal Ladhak, Esin Durmus, Claire Cardie, and Kath- leen McKeown. 2020. Wikil.inuaz new bench- mark dataset for cross-lingual abstractive summariza- tion. In Findings of the Association for Computa- tional Linguistics: EMNLP 2020, pages 4034 4048, Online. Association for Computational Linguistics.
Thomas Scialom, Paul-Alexis Dray, Sylvain Lamprier. Benjamin Piwowarski, and Jacopo Staiano. 2020. MLSUM: The multilingual summarization corpus, In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8051-8067, Online. Association for Computa- tional Linguistics.
Chin- Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Abhay Shukla. Paheli Bhattacharya, Soham Poddar, Ra- jdeep Mukherjee, Kripabandhu Ghosh, Pawan Goyal, and Saptarshi Ghosh. 2022. Legal case document summarization: Extractive and abstractive methods and their evaluation. In Proceedings of the 2nd Con- ference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th Interna- tional Joint Conference on Natural Language Pro- cexsing (Volume 1: Long Papers), J, Online only. Association for Computational Linguis- tics.
Yang Liu and Mirella Lapata. 2019. Text summariza- tion with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing pages 3730-37400 Hong Kong, China. Association for Computational Linguistics.
pages 1048-1064,
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of doc- uments. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI'17, 17, page 3075-3081. AAAI Press.
Sajad Sotudeh, Arman Cohan, and Nazli Goharian. 2020. On generating extended summaries of long documents. CoRR, abs/2012.
Y. Tang, C. Tran, Xian Li, Peng-Jen Chen, Naman Goyal, Vishrav Chaudhary. Jiatao Gu, and Angela Fan. 2020. Multilingual translation with extensi- ble multilingual pretraining and finetuning. ArXiv, has/2008.004011.
Ani Nenkova, Sameer Maskey, and Yang Liu. 2011. Au- tomatic summarization. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguisticx: Tutorial Abstracts, page 3, Portland, Ore- gon. Association for Computational Linguistics.
NIlb team, Marta Ruiz Costa-jussà, James Cross, Onur cCelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang. Guillaume Wenzek, Alison Youngblood, Bapi Akula, Loic Bar- rault, Gabriel Mejia Gonzalez. Prangthip Hansanti. John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon L. Spruit, C. Tran, Pierre Yves Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzm' 'n, Philipp Koehn, Alexandre Mourachko, Christophe Rop- ers, Safiyyah Saleem, Holger Sehwenk, and Jeff Wang. 2022. No language left behind: Seal- ing centered machine translation. ArXiv, abs/2207. 0bb/2207.04462..
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with unified text-to-text trans- former. CORR, 0ts//990.11088.
Gowtham Ramesh, Sumaanth Doddapaneni, Aravinth Bheemaraj. Mayank Jobamputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Ma- halakshmi J, Divyanshu Kakwani, Navneet Kurnar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Ku- mar, and Mitesh Shantadevi Khapra. 2022. Samanan- tar: The largest publicly available parallel corpora collection for 11 Indic languages. Transactions of the A Association for Computational Linguistics, 10:145- 162.
Jörg Tiedemann and Santhosh Thottingal. 2020. OPUS- MT building open translation services for the world. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 479-480, Lishoa, Portugal. European Associa- tion for Machine Translation.
Ashok Urlana, Sahil Manoj Bhatt, Nirmal Surange, and Manish Shrivastava. 2023. Indian language sum- marization using pretrained sequence-to-sequeee models. In Fire.
Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual us- ing knowledge distillation. In Proceedings of the
Daniel Varab and Natalie Schluter. 2021. Mas- siveSumm: a very large-scale, very multilingual,

---

news summarisation dataset. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10150-10161, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Jiaan Wang, Fandong Meng, Ziyao Lu, Duo Zheng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2022. ClidSum: benchmark dataset for cross lingual dialogue sum- marization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 7716-7729, Abu Dhabi, United Arab Emirates. Association for Computational Lin- guistics.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou. Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 483-496 On- line. Association for Computational Linguistics.
Jen-Yuan Yuan Yeh, Hao-Ren Ke, Wei-Pang Yang. and Heng Meng. 2005. Text summarization using a train- able summarizer and latent semantic analysis. Infor- mation Processing Management, 41(():75 An Asian Digital Libraries Perspective.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Pe- ter J. Liu. 2019. Pegasus: Pre-training with extracted gap-sentences for ahstractive summarization. ArXiv, abs/191..08777.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Eval- uating text generation with bert. International Conference on Learning Representations.

---

Appendix
the pdf2tex/ tool. We randomly selected 5 pages from various documents and then manually com- pared the text present the PDF with the text extracted using the p4f2text tool. We find that the tool's output closely matches the content present in the PDFs, with only minor instances of punc- tuation errors. Also, these judgment PDFs were of high quality, contributing to the tool's accurate performance.
A Details of Related work
Datasets for summarization in Indian languages: Some recent work on tross-lingue summarization has been done by Bhattacharjee et al. (2023) who developed large-scale cross-lingual summariza- tion dataset CrossSum comprising 1.7M article- summary samples in more than 1.5K language- pairs. Another dataset WikiLingua was provided by Ladhak et al. (2020), which comprises of Wiki- How articles and their summaries in multiple lan- guages, including a substantial collection of over 9K document- pairs in Hindi. However, note that the summaries in this dataset are rela- tively short, with an average length of only 39 tokens. This characteristic implies that the Wik- iLingua dataset not ideal for training legal sum- marization systems, since legal summarization of- ten necessitates longer summarics that encompass intricate legal concepts.
We also observed the need for data cleaning pre-processing while developing the MILDSum corpus. Case judgements usually contain some metadata additional text the beginning, which should not be part of the input for summarization. Figure shows an example of the first page of a court case judgment, where the unnecessary text portions are highlighted in pink. Such text portions should be discarded and not be part of the input to a summarization model.
To clean this unnecessary part, we developed an algorithm that removes lines with fewer non-space characters than the average for a document, since the said lines mostly contain very few words. The algorithm proceeds as follows: compute each doc- ument's average number of non-space characters per line. Then, iterate through cach line, deleting those lines with character counts below the average, unless the previous line was included. This ensures that the last line of each paragraph is retained. Us- ing this algorithm, we discard the metadata and retain the content of the main order/judgment from the judgment PDFs.
Notably, none of the datasets stated above are focused on cross-lingual summarization in the legal domain. This work is the first to develop dataset for cross-lingual summarization of legal text in Indian languages.
Datasets for cross-lingual summarization non- Indian languages: the case of non-Indian lan- guages, Scialom et al. (2020) proposed a dataset called MLSUM, which is compiled from online news sources. It comprises more than 1.5M pairs of articles and summaries in 5 languages: French, German, Spanish, Russian, and Turkish. Recently, Wang et al. (2022) released benchmark dataset called CLIDSUM for Cross- Lingual Dialogue Sum- marization.
B.2 Document length distributions of judgments and summaries in MILDSum
This section describes the document length distri- butions (with respect to the number of words con- tained the documents) of the case judgments and summaries in MILDSum. A document length dis- tribution graph visually represents the number of documents in a dataset that contain a certain num- ber of words. Each point on the graph represents bin of document lengths (in terms of number of words) and the number of documents in that bin. The shape of the graph can vary depending on the dataset's 's characteristics.
In the context of legal text summarization, Au- miller et al. (2022) recently introduced Eur-Lex, a multi- and cross-lingual corpus of legal texts and human-written summaries from the European Union (EU). This dataset covers all 24 official lan- guages of the EU. To our knowledge, no such effort has been made in the Indian legal domain; we are taking the first step bridging this gap.
B Additional details about the MILDSum dataset
Figure 2 shows the distribution of lengths of the case judgments in the MILDSum dataset. Similarly, Figure 3 and Figure show the document length distributions of the English and Hindi summaries respectively. For our MILDSum corpus, the dis-
B.1 Data pre-processing and cleaning
We used the pdf2text tool for extracting text from PDFs (the original case judgements). We con- ducted a quality check to ensure the reliability of

---

Selection of translation model for C.2
5:58
translating English summaries to Hindi
IN THE HIGH COURT OF DELHE NEW DELHI P.P/C/667//2222
For the second stage of the pipeline approach then-arrrie--ee---ransaaaeee
Throught Mr. Malesh Verns Adv. alonguit
pelifioner personi
tried out sev- eral state-of-the-art Machine Translation (MT) systems for English-to tindi translation, such as Google Cloud Trunslator' Microsoft Azure Translator IndicTrans (Ramesh et al., 2022), mBART-5004 (Tang et al., 2020), NLLB!S (team et al., 2022), and OPU516 (Tiedemann and Thottin- gal, 2020). To compare among these MT models, randomly selected 100 data-points from MILD- Sum, and then used these MT models for English- to-Hindi translation of the reference English sum- maries. We used the standard MT evaluation met- ric BLEU (Papineni et al., 2002) for comparing the quality of the translations, by matching a machine- translated Hindi summary with the reference Hindi summary for the same case judgment.
GF DELHI&
Kuran Bhardwaj ASC for R-Il and
Mr. Sidharth Panda, Advi for
CORAME
ORDER
28,85.2022
This wrih petition has becn peefereed socking the following relicfe-
Diecting repposdent
Kabwyy fater S6. Catak hingh withon cleaning adfintiation fioss Ioskaad
comsend
Sh. Galdb Singh only das dhe given date 03 06.3022 tre long""
2. The petiionne sodks donate her kidney her ailing father. Along with the witl pettion, the relevant medical records have also been placed and which saggost that tranepant recommended. The petitioner allegoe although the ready willing
organ father, application not being proceased since the respandent hospital imiisting on subriission o No Ofjecction Certificate from her hasband. futther alleged that the relationahip betwern the petitionce haaband presently cstranged coneqqucstly would
Among these MT systems, IndicTrans exhib- ited the best performance with BLEU score of 50.34 (averaged over the 100 data-points randomly selected for this evaluation). The scores for other models are as follows - MicrosaffTrans: 47.10, GoogleTrans: 43.26, NLLB: 42.56, mBART-50: 33.60, and OPUS: 10.65. Also, IndicTrans was found to be the best for English-to- Hindi transla- tion in the prior work (Ramesh et al., 2022). For these reasons, used IndicTrans for translation of the English summaries Hindi.
Figure I: Example of the first page of a court judgment. The metadata (that be removed) is highlighted.
tribution for court judgements (Figure 2) seems to follow an Inverse- Sairss--ChiSqquaee distribution, while for both English and Hindi summaries, the distribu- tions seem to follow bell-shaped positively skewed distribution curves.
More details about the experiments
C.3 Generating fine-tuning data for abstractive summarization models
C.1 Training supervised extractive summarization models
This section describes how we created fine-tuning data for the abstractive models Legal-Pegasus and CrossSum-nnT5. Fine-tuning of these models re- quires data in the form of document chunks and the corresponding summaries of the chunks. For this experiment, we fixed the chunk length to 512 tokens. As we are chunking a case judgment, we need to chunk the corresponding reference sum- mary as well to match the context of the judgment's
The supervised extractive summarization mod- els SummaRuNNer and BERTSumEXt have been trained over the training split of the MILDSum dataset. These supervised methods require labeled data for training, where every sentence in the doc- ument must be labeled as if this sentence is suit- able for inclusion in the summary, and labeled as 0 otherwise. Hence, we convert the reference En- glish summaries of MILDSum to purely extractive form to train these methods. To this end, we adopt the technique mentioned by Nallapati et al. (2017). Briefly, we assign label (suitable for inclusion in the extractive summary) to those sentences from the full document that greedily maximize the ROUGE- 2 (Lin, 2004) overlap with the human-written refer- ence summary. The rest of the sentences in the full document are assigned label O.
" http::::///llou.gooo//// ctm/ hmmm/rrrasateedcc/ssaamm les/ Lee/tttaesaatee---aaalalatt---e
12 "https://aaurremiccroofftt
com//nn-us//proouutss// http:///.urreectttttcccccc///c/s
"nttpss://gitubb.com// http://shhaab.cooonaleeaaarrrarr
-o-ooe-tto-mmn--mmttt
is http::///mngiiiigeeeeooo//oooooo
t-en-mul

---

length
100
Nunber diccueetss
10
1000
2000
4000
5000
6000
7000
8000
12000 13000 14000 15000 16000 17600 18000 15000 Number words
10000
31000
Figure 2: Document length distribution court judgments in MILDSum
Document length distribution of English Summaries
Number of summary documents
400
300
100
200
400
600
800
1000
1200
1400
1600
1800
2000
2200
2400
2600
Number words
Figure 3: Document length distribution of English summaries in MILDSum
Document length distribution of Hindi Summaries
of summary
200
100
400
600
1000
1200
1400
1600
1800
2000
2200
2400
2600
2800
Number of words
Figure 4: Document length distribution of Hindi Summaries in MILDSum

---

chunks, In order to generate parallel pairs (docu- ment chunk and corresponding summary chunk), we followed the method of Reimers and Gurevych (2020) i.e., bitext mining. We generate embeddings using LaBSE (Feng et al., 2022). Then, we use the following scoring function to compute a score for each pair of embeddings; we set the threshold to I for including given pair in the fine-tuning dataset.
scorela, v) -aargin cos( w):
cos 2)
cos(y 2)
2k
2k
(x))
Where z and are the embeddings of the judgment chunk and summary chunk respectively. N (2) denotes the k nearest neighbors of x. We have used ratio-based margin as the margin function, i.e., margin(a, b) a/b.
C.4 Evaluation metrics for summarization performance
The following standard metrics were used to evalu- ate the quality of the model-generated summaries, by comparing with the reference summaries.
ROUGE (Lin, 2004) stands for Oecall-Oriented Understudy for Gisting Evaluation. ROUGE- measures the textual overlap (bi-grams) be- tween the monel-generated summaries and the reference summaries. ROUGE-L measures the longest matching sequence of words using the Longest Common Subsequence (LCS).
To calculate multilingual ROUGE scores (En- glish and Hindi in this work), we used the mul- tilingual rouuerouuueeee mnuge_scoring library (provided by Hasan et al. (2021)) which uses the OpenNMT I8.
BERTScore (Zhang al., 2020) uses BERT to compute the similarity scores between the token- level embeddings of the model-generated and reference summaries. It is known to correlate better with human judgments as it considers the semantic part.
To calculate multilingual BERTScore, used the Huggingface evaluate library that incorpo- rates the official BERTScore 2roject
com//csebuetrllp/x1-ssmm Pttrosss/githooo.oc///eeeetrttts
"nttpss:/openntt net/
"ht h aat.ac///acccee//aaattteeeeteeee
c/bertscore
20 "ttpps:
http://guuuucmmmmigeeeeeee//eeee com/Tiiiggerbbertsccrr


Reviews:
[{"review_id": "zrgMktZmPi", "reviewer": "Reviewer_egHn", "summary": "", "questions": "A. I am concerned about the data collection and alignment processes described in Sections 3.1 and 3.2. You mentioned that LiveLaw's Hindi site is \"not frequently updated as its English counterpart\" and the sites do not link articles in the two languages. How did you ensure that this lack of direct linking did not bias your data set toward the judgments published in English?\n\nB. The \"Summarize-then-Translate pipeline approach\" and the \"Direct Cross-Lingual Summarization\" method were both introduced in the paper. Based on your experiments, can you clarify each method's specific advantages and disadvantages?\n\nC. Your abstract states that you \"benchmark the performance of several competitive summarization approaches on our corpus.\" However, the provided content does not go into detail about these benchmarks. What specific metrics did you use to evaluate the quality and accuracy of the summaries? Were there any human evaluators involved in this process?\n\nD. I noticed that you've trained various models on the MILDSum dataset. How does the MILDSum dataset compare to other similar datasets (like Eur-Lex) in terms of challenges and opportunities for model training?\n\nE. In Section 3.4, you mention that the dataset error rate is less than one percent. While this is commendable, can you shed light on the potential impact of this error on the validity of the research? Would there be any specific biases introduced by this error rate?\n\nF. In terms of future applications: How can the MILDSum dataset aid in improving access to legal information for the general public in India? What are the potential implications of this research in broader legal tech domains?\n\nG. The paper states that \"extractive methods showcased superior performance compared to existing abstractive methods.\" Can you provide some insights into why extractive methods might be more suited for legal document summarization, especially in the Indian context?\n\nH. You've mentioned the use of Bing search API for data collection. Were there any considerations or challenges you faced in ensuring data privacy and security during this data collection process?\n\nI. Finally, while you have done an extensive job in curating this dataset, how do you plan on updating and maintaining MILDSum in the future to ensure its relevance and utility?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for giving us valuable feedback and for appreciating our work. You have pointed out 8 Weaknesses (1-8) and gave 9 questions (A-I). We are trying to address them below. To keep the rebuttal brief, we have combined some related weaknesses and questions.\n\n[1] We understand the concern about using just one source (\"LiveLaw\") for our dataset, MILDSum. However, sources that provide cross-lingual summaries in the legal domain (English judgments and corresponding Hindi summaries) are very rare in India, and we did not find any other website of this type. Also please note that our dataset covers a wide range of cases from diverse courts such as the Supreme Court of India, and High Courts in Delhi, Bombay, Calcutta, etc. Also, different summaries were written by different legal experts. Therefore, we believe that the dataset is diverse with respect to topics as well as in writers of the gold standard summaries.\n\n[2, A, E] We agree that the pipeline for matching articles from Hindi and English versions of LiveLaw are intricate. This is one of the biggest challenges in developing the dataset, and we designed the pipeline very carefully (as described in Section 3.2 and 3.3 in the paper). As reported in Section 3.4, to check the correctness of pairing Hindi and English summaries, we conducted a careful quality check on 300 random data points of MILDSum and found just one mismatched instance. So we believe our matching method is very accurate. Also, given that we found only one error, it is difficult to comment on any particular biases introduced by this very small error rate. \n\n[3] Regarding English-to-Hindi translation, we tried out several Machine Translation (MT) systems such as Google-Translator, Microsoft-Translator, IndicTrans, mBART, NLLB, and OPUS over randomly selected 100 data points from MILDSum. We used the standard MT evaluation metric BLEU for comparing the quality of translations. Among these MT systems, IndicTrans exhibited the highest performance with a BLEU score of 50.34 [scores for other models are as follows -- MicrosoftTrans: 47.10, GoogleTrans: 43.26, NLLB: 42.56]. Also, IndicTrans was found to be the best MT system for English-to-Hindi translation in the prior work https://aclanthology.org/2022.tacl-1.9/. We mentioned briefly that we tried out these MT systems in Appendix C. We will add the BLEU scores and these details in the final version of the paper. \n\n[4, 7] Since MILDSum is the first cross-lingual summarization dataset for legal domain in Indian languages, we could not compare model performances over MILDSum with that over any other dataset in the legal domain. But we acknowledge the importance of comparative analysis using other non-legal datasets such as CrossSum (https://aclanthology.org/2023.acl-long.143/). We will include this comparative analysis in the final version.\n\n[5, 6, D, I] We fully acknowledge the value of a multilingual summarization dataset in the Indian legal domain. But please note that legal domain-specific data sources with cross-lingual nature are very rare. We found only one source - “LiveLaw” - from where we were able to obtain 3,122 judgment-summary pairs till date. Note that, with 3,122 judgment-summary pairs, MILDSum is large enough for training and fine-tuning summarization systems to improve their performances. Additionally, MILDSum is large enough compared to other cross-lingual legal datasets (non-Indian) like Eur-Lex (https://aclanthology.org/2022.emnlp-main.519/). Eur-Lex contains 1,500 doc-summ pairs per language. Although language diversity is less in MILDSum compared to Eur-Lex, MILDSum contains 3,122 cross-lingually aligned doc-summ pairs that is much higher than Eur-Lex. Similar to Eur-Lex, one of the most obvious challenges for this dataset is the extreme document lengths, and also length disparity between documents. Finally, we are committed to enriching MILDSum's impact by including summaries in other Indian languages in the future, as well as increasing the number of document-summary pairs by periodic crawling of the Livelaw websites. \n\n[8] We agree that chunking might lead to the loss of context. But, please note we have also used LongT5 and LegalLED, which can handle most of our documents in full; so chunking was not required for these models. We already stated this in lines 263 to 269 of the paper.\n\n[B] Based on our experiments, \"Summarize-then-Translate pipeline approach\" performs better than \"Direct Cross-Lingual Summarization\" over MILDSum. However the pipeline approach has the problem that errors in the summarization stage (e.g., omission of important information) are propagated to the second stage (translated). On the other hand, \"Direct Cross-Lingual Summarization\" models are free from such risks of error propagation; also these methods can be improved in the future.\n\n[C] For automatic evaluation, we have used the standard metrics ROUGE-2, ROUGE-L and BERTscores (as reported in Section 4.3). \"Expert Evaluation\" could not be done in this work due to unavailability of Law experts with knowledge of Hindi. We plan to conduct an  \"Expert Evaluation\" of the summaries generated by different methods as a future work.\n\n[F] As stated in the Introduction of the paper, most legal text in the Indian judiciary is written in complex English. But, a significant portion of India's population lacks a strong command in English. Hence, it is crucial to make legal knowledge available in Indian languages, of which Hindi is the most commonly spoken/understood language. While prior research primarily focuses on summarizing legal case judgments in their source languages, this study presents a pioneering effort toward cross-lingual summarization of English legal documents into Hindi. We believe that models trained on the MILDSum dataset can be very useful in future in making legal judgements accessible to the common masses in India. \n\n[G] As per our experiments over the MILDSum dataset, extractive methods showed superior performance compared to abstractive methods. In lines 310-313 of the paper, we already mentioned that the summaries in MILDSum have a mix of extractive and abstractive features, as they directly quote several lines from the original judgment text. Also, we have now computed coverage and density [https://aclanthology.org/N18-1065/] to measure the extractiveness /abstractiveness of the dataset. We find Extractive Fragment Coverage as 0.90, and Extractive Fragment Density as 24.42. So we get a high value for Coverage which measures the percentage of words in the summary that are part of an extractive fragment from the document/judgment. We also get a high value for Density which quantifies how well the word sequence of a summary can be described as a series of extractions. Since the summaries in our dataset contain directly quoted lines from original judgment, so Density is very high compared to other datasets (e.g., 3.94 for Arxiv/PubMed, 2.38 for BigPatent) [https://aclanthology.org/2022.findings-emnlp.488/]. This explains why Extractive summarizers outperform abstractive ones over our dataset. We will include this explanation in the final version. \n\n[H] We did not encounter any difficulties related to data privacy and security during the data collection process. The main challenge was for alignment between English and Hindi Summaries, which was reported in Section 3.2. \n\nAlso, thank you for pointing out the typos and giving us suggestions regarding the improvement in Grammar Style and Presentation. We will definitely incorporate these in the final version (if accepted).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The introduction of the MILDSum dataset is a groundbreaking effort, given that no such dataset previously existed for multilingual summarization in the Indian legal domain. The meticulous curation of 3,122 judgment-summary pairs in both English and Hindi by legal practitioners ensures a high-quality resource for future research.\n\n2. With a significant portion of India's population lacking proficient English skills but needing access to legal judgments, the paper addresses a pressing real-world problem. This has immense societal and practical implications, especially in a multilingual country like India.\n\n3. The paper doesn’t just introduce a dataset; it delves deep into its characteristics, highlighting the unique features of Indian legal documents. This offers insights into domain-specific challenges and opportunities.\n\n4. By testing various summarization models, the paper sets a benchmark for future research. The finding that extractive methods outperformed abstractive ones on this dataset provides a crucial insight for model selection and development in this domain.\n\nBenefits to the NLP Community:\n\n1. Given the dearth of resources and research on Indian legal summarization, this paper offers a fresh avenue for NLP researchers to explore, innovate, and develop applications.\n\n2. Emphasizing the importance of cross-lingual summarization, this paper can act as a catalyst for similar initiatives in other multilingual countries, fostering global research collaboration.\n\n3. With the rise of legal tech solutions globally, the insights and dataset from this paper can propel the development of tools that make legal judgments more accessible to a broader audience.\n\n4. The methodology and considerations presented in this paper can serve as a blueprint for researchers looking to develop similar datasets in other specialized domains, further enriching the NLP research landscape.", "reasons_to_reject": "Weaknesses:\n\n1. The dataset, MILDSum, is primarily constructed from the website \"LiveLaw\". Depending entirely on one source may limit the diversity and generalizability of the dataset.\n  \n2. The described pipelines, especially when matching articles from Hindi and English versions of LiveLaw, are intricate and may be prone to errors.\n\n3. While the paper mentions translating English summaries into Hindi, the quality of these translations isn't explicitly discussed. Given the intricacies of legal language, ensuring high-quality translations is paramount.\n\n4. The paper benchmarks performance using MILDSum, but a comparative analysis using other datasets would make the results more robust. This could help establish the advantages of MILDSum over existing resources.\n\n5. While Hindi is one of the most spoken languages in India, India is home to several other languages. A true multilingual dataset would encompass multiple Indian languages, broadening its applicability.\n\n6. While 3,122 judgment-summary pairs can be considered a reasonable size, it might still be limited when training sophisticated deep learning models. \n\n7. The models have been trained and tested only on MILDSum. Their performance on external datasets or real-world applications isn't discussed.\n\n8. The approach to handle long legal documents by chunking might lead to the loss of context or crucial information.\n\nRisks for Conference Presentation:\n\n1. Even a less than 1% error rate in a legal dataset can have serious implications. Any inaccuracies in the legal domain can be consequential, and attendees might raise concerns.\n\n2. Given the complexities in data collection and alignment, presenters should be well-prepared to address detailed technical questions regarding the methodology.\n\n3. Audience members familiar with the Indian legal system or multilingual summarization might question the dataset's applicability across various domains or other Indian languages.\n\n4. Any data collection activity, especially from a legal domain, should uphold ethical considerations. If these aren't explicitly mentioned in the paper, the presenters might face related queries.\n\n5. Attendees might want to know how this research can be used practically, especially in real-world legal settings in India.", "questions_to_authors": "A. I am concerned about the data collection and alignment processes described in Sections 3.1 and 3.2. You mentioned that LiveLaw's Hindi site is \"not frequently updated as its English counterpart\" and the sites do not link articles in the two languages. How did you ensure that this lack of direct linking did not bias your data set toward the judgments published in English?\n\nB. The \"Summarize-then-Translate pipeline approach\" and the \"Direct Cross-Lingual Summarization\" method were both introduced in the paper. Based on your experiments, can you clarify each method's specific advantages and disadvantages?\n\nC. Your abstract states that you \"benchmark the performance of several competitive summarization approaches on our corpus.\" However, the provided content does not go into detail about these benchmarks. What specific metrics did you use to evaluate the quality and accuracy of the summaries? Were there any human evaluators involved in this process?\n\nD. I noticed that you've trained various models on the MILDSum dataset. How does the MILDSum dataset compare to other similar datasets (like Eur-Lex) in terms of challenges and opportunities for model training?\n\nE. In Section 3.4, you mention that the dataset error rate is less than one percent. While this is commendable, can you shed light on the potential impact of this error on the validity of the research? Would there be any specific biases introduced by this error rate?\n\nF. In terms of future applications: How can the MILDSum dataset aid in improving access to legal information for the general public in India? What are the potential implications of this research in broader legal tech domains?\n\nG. The paper states that \"extractive methods showcased superior performance compared to existing abstractive methods.\" Can you provide some insights into why extractive methods might be more suited for legal document summarization, especially in the Indian context?\n\nH. You've mentioned the use of Bing search API for data collection. Were there any considerations or challenges you faced in ensuring data privacy and security during this data collection process?\n\nI. Finally, while you have done an extensive job in curating this dataset, how do you plan on updating and maintaining MILDSum in the future to ensure its relevance and utility?", "paper_topic_and_main_contributions": "The paper presents a study on the automatic summarization of legal case judgments in India, particularly focusing on translating these summaries from English, the language in which most judgments are written, to Hindi to benefit a broader audience. The core contributions of the paper are:\n\n1. The introduction of MILDSum (Multilingual Indian Legal Document Summarization), a first-of-its-kind dataset that consists of 3,122 judgment-summary pairs in both English and Hindi. This data is derived from several high-profile courts, including the Supreme Court of India. The summaries have been meticulously curated by legal practitioners, ensuring both their linguistic quality and legal accuracy.\n\n2. Unlike previous works that predominantly focused on summarizing in the source language, this paper emphasizes the importance of cross-lingual summarization. This responds to the linguistic reality of India, where many citizens do not have proficient English skills but need access to legal judgments for various reasons.\n\n3. The paper sets its dataset apart from existing European datasets like Eur-Lex, emphasizing the unique characteristics of Indian legal documents, including length, specialized terminologies, and referencing patterns.\n\n4. The authors benchmarked various summarization models on the MILDSum dataset. Interestingly, they found that extractive methods outperformed abstractive ones, a finding they attribute to the specific nature of their dataset.\n\nIn conclusion, the paper addresses an important gap in the field of automatic summarization, especially in the context of the Indian legal system. By offering a new dataset and shedding light on the unique requirements of cross-lingual summarization in this domain, the study makes a significant contribution that holds potential for both academic and practical applications.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Sometimes you use \"English & Hindi\" and other times \"English and Hindi\". It's better to maintain consistency.\n2. \"Regarding non-indian langaues\" has typos. It should be \"Regarding non-Indian languages\".\n3. \"The data for this study was primarily collected from\" should be \"The data for this study were primarily collected from\".\n4. \"Here we focuses on fetching\" should be \"Here we focus on fetching\".\n5. \"Judgment\" should have a consistent spelling.\n6.  In the data collection and alignment sections, more context might help the reader. For instance, explaining why the Jaccard similarity measure was used and its significance.\n7. Some sections like \"Experiments and Results\" could benefit from further subdivision to make it easier for readers to follow the sequence of activities and understand the methodology and findings."}}, {"review_id": "kFamSALrxg", "reviewer": "Reviewer_XjyQ", "summary": "", "questions": "- title: The title describes the dataset as multilingual, however, isn't it a cross-lingual dataset since the languages are aligned?\n- L039: \"our dataset includes detailed quotations from the ...\" What does this mean?\n- L069: \"the heterogenous nature nature of our dataset\" What does this mean?\n- L235: SummaRuNNer is a little bit dated? Why not use something more performant like BERSUMExt? (https://aclanthology.org/D19-1387/)\n- L278: The paper mentions that IndicTrans is best according to BLEU. What does this refer to? Did you compute BLEU yourself?\n- L281: \"Indic is known to be the best English-to-Hindi translation\". Such a claim lacks a citation or supporting evidence.\n- L299: ROUGE and BERTScore are parametric measures. What tokens are ROUGE computed over? What model is BERTScore computed. over? The standard is English, but your results are on Hindi.\n-", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for giving us valuable feedback and for appreciating the importance of our dataset. We denote your comments by “C” and our answers with “A”.\n\nC: “very little information about the documents and summaries…”\n\nA: Please note that several statistics about the dataset, that you suggested to include, are already reported in Appendix A.2. Please refer to Figures 2, 3 & 4 of the paper. Figure 2 shows the distribution of document lengths, Fig 3 and Fig 4 show the distributions of length of English and Hindi summaries respectively. \n\nC: “paper would benefit from some quantitative measures … E.g. extract/abstract-iveness”\n\nA: Following your suggestions, we computed the coverage, density, and Compression ratio to measure extract/abstract-iveness of the dataset. We find Extractive Fragment Coverage as 0.90, Extractive Fragment Density as 24.42, and Compression ratio as 5.84 for our dataset. The high values of Coverage and especially Density indicate high extractiveness of the summaries with respect to the source documents. This is expected, since summaries in our dataset contain several quoted lines from the original judgments (as already stated in lines 310-313). These values also justify why extractive summarization models tend to perform better than abstractive summarization models on this dataset. We will include these details in the final version. \n\nC: “The evaluation measures ROUGE and BERTScore … how they are adapted to Hindi”\n\nA: For multilingual ROUGE scores (in our case English and Hindi), we have used this library https://github.com/csebuetnlp/xl-sum [provided by https://aclanthology.org/2021.findings-acl.413/] which uses the OpenNMT tokenizer (https://opennmt.net/). To calculate multilingual BERTScore, we have used the 'evaluate' library (https://huggingface.co/spaces/evaluate-metric/bertscore) that incorporates the official BERTScore project (https://github.com/Tiiiger/bert_score). We will add these details in the final version. \n\nC: “Details about how the best model (SummaRuNNer) is not described.”\n\nA: We trained the vanilla SummaRuNNer model, obtained from https://github.com/hpzhao/SummaRuNNer, on our training data. We will include more details about the summarization models used, in the final version. \n\nC: “Choosing fewer, more motivated models might improve the message of the experiment.”\n\nA: We applied a few representative summarization models from different families - unsupervised extractive, supervised extractive (where we will add BERTSumExt too), abstractive (pre-trained and fine-tuned), and abstractive long document summarizers. Most of the supervised models have been trained / fine-tuned on the MILDSumm training set. The experiments show extractive models to perform slightly better than abstractive models, since the gold standard summaries often contain quotes / sentences extracted from the source documents. Another take-away is that the summarize-then-translate pipeline performs better than the cross-lingual summarization model. We will explain these takeaways more clearly in the final version. \n\nC: Query about paper title\n\nA: MILDSum contains 3,122 Indian court judgments in English along with their summaries in both English and Hindi. Since the dataset contains summaries in two languages, we considered this dataset as Multilingual. \n\nC: “L039: our dataset includes detailed quotations ..” \n\nA: We meant to say that MILDSum contains the actual judgments in their original form as provided by the respective courts. We will rephrase the sentence and clarify what we meant.\n\nC: “L069: the heterogeneous nature of our dataset\" \n\nA: By the “heterogeneous nature of our dataset”, we meant that our gold standard summaries contain a mixture of directly quoted lines from the original judgments (extractive parts) as well as abstractive parts. Hence, the summaries are a heterogeneous mixture of extractive and abstractive parts. We will rephrase the sentence and clarify what we meant.\n\nC: “L235: Why not use something more performant like BERSUMExt?” \n\nA: Thank you for this suggestion. We assessed the performance of BERTSumExt as well. We trained it using the default settings on the training split of MILDSum and then tested it on the test split of MILDSum. The scores for the automatic metrics are as follows (F1 scores):\nScores for English Summary - ROUGE-2: 32.08 , ROUGE-L: 31.12 and BERTScore: 86.78\nScores for Hindi Summary - ROUGE-2: 24.34 , ROUGE-L: 25.62 and BERTScore: 74.59\n\n\nC: L278, L281: Queries about IndicTrans\n\nA: Yes, IndicTrans was observed to be the best translator from English to Hindi according to our internal experiments (which are briefly described in Appendix C). We tried out several Machine Translation (MT) systems such as Google-Translator, Microsoft-Translator, IndicTrans, mBART, NLLB, and OPUS over randomly selected 100 data points from MILDSum. We used the standard MT evaluation metric BLEU for comparing the quality of translations. Among these MT systems, IndicTrans exhibited the highest performance with a BLEU score of 50.34 [scores for other models are as follows -- MicrosoftTrans: 47.10, GoogleTrans: 43.26, NLLB: 42.56]. Also, IndicTrans was found to be the best MT system for English-to-Hindi translation in the prior work https://aclanthology.org/2022.tacl-1.9/. We mentioned briefly that we tried out these MT systems in Appendix C. We will add the BLEU scores and these details in the final version of the paper. \n\nThank you for suggesting the additional references and typos. We will definitely incorporate these in the final version. Overall, we are happy to address your concerns/suggestions. Also, we would like to answer any follow-up questions that you may have.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Summarization datasets with high-quality reference summaries are highly desirable as existing datasets are often automatically collected from online sources where it is not always clear whether reference summaries are in fact satisfying summaries. This paper presents a dataset that builds on a reputable source where summaries appear to be of high quality, in addition to being distributed in two languages, and aligned across these languages. One of the languages, Hindi, has poor coverage in existing summarization resources and is the first dataset to cover the legal Hindi domain, providing a novel and valuable resource for Hindi and legal NLP.", "reasons_to_reject": "The dataset is valuable and will undoubtedly be a much-welcomed resource that will benefit the community, however, I do find that the paper, in its current state, lacks some rigor and depth. In particular, very little information about the documents and summaries in the dataset is described. Outside the lengths of the average length of documents and summaries and brief mentions of the dataset being \"extractive\" due to quotations, it is unclear what the dataset looks like. I think the paper would benefit from some quantitative measures to help the reader understand the shape of the dataset. E.g. extract/abstract-iveness using measures like coverage/density/compression (grusky et al, 2018), what is the variation in document/summary length, vocabulary size, etc. Similarly, the included baselines experiment is confusing and light on the details. Some models are trained on a training split of the dataset, while others are not. Details about how the best model (SummaRuNNer) is not described. The evaluation measures ROUGE and BERTScore can be used in other languages but it is crucial to specify how they are adapted to Hindi (e.g. what tokens are ROUGE scores computed over? and which model is used to compute BERTScores?). The baseline results are very similar across all models, thus not telling the reader very much about what type of model is suitable for the data. Choosing fewer, more motivated models might improve the message of the experiment.", "questions_to_authors": "- title: The title describes the dataset as multilingual, however, isn't it a cross-lingual dataset since the languages are aligned?\n- L039: \"our dataset includes detailed quotations from the ...\" What does this mean?\n- L069: \"the heterogenous nature nature of our dataset\" What does this mean?\n- L235: SummaRuNNer is a little bit dated? Why not use something more performant like BERSUMExt? (https://aclanthology.org/D19-1387/)\n- L278: The paper mentions that IndicTrans is best according to BLEU. What does this refer to? Did you compute BLEU yourself?\n- L281: \"Indic is known to be the best English-to-Hindi translation\". Such a claim lacks a citation or supporting evidence.\n- L299: ROUGE and BERTScore are parametric measures. What tokens are ROUGE computed over? What model is BERTScore computed. over? The standard is English, but your results are on Hindi.\n-", "paper_topic_and_main_contributions": "This paper introduces the dataset MILDSum (Multilingual Indian Legal Document Summarization), a summarization dataset comprised of 3122 Indian legal case judgments. The dataset is comprised of aligned document-summary pairs in both English and Hindi (thus a total of 6.2k document-summary pairs). Documents and summaries are collected from the website LiveLaw, a reputable site for coverage of court judgments in India.\n\nThe paper includes a description of the data collection methodology which includes two pipelines to collect data and approaches to align and clean documents and summaries. 300 documents were sampled and manually checked to ensure cross-lingual alignment - only 1 document was found to be misaligned. The paper includes baseline results using 9 models, including both unsupervised, supervised, extractive, abstractive, and cross-lingual systems. The monolingual abstractive systems are fine-tuned on a training split of the proposed dataset, the cross-lingual system is an off-the-shelf system trained on newswire data (CrossSumm), and details about the monolingual extractive systems are not included. Baseline outputs on a held-out test set are evaluated using ROUGE and BERTScore. All models give similar scores except the cross-lingual model which performs very poorly.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "abstract/L024: interesting inferences -> interesting insights?"}, "extra_text": {"Missing_References": "In the related work section it may be worth mentioning the few existing summarization datasets that do contain Indian languages:\n- ILSUM (Urlana et al, 2023 - https://arxiv.org/abs/2303.14461)\n- TeSum (Urlana et al, 2022 - https://aclanthology.org/2022.lrec-1.614)\n- MassiveSumm (Varab and Schluter, 2021 - https://aclanthology.org/2021.emnlp-main.797/)"}}, {"review_id": "qxDRFcJO2x", "reviewer": "Reviewer_eZYq", "summary": "", "questions": "1) What is the datasets' license?\n2) Did you validate the quality and error rate of the pdf2text tool?\n3) According to line 294, it seems like you didn't finetune CrossSum-mT5 on your dataset. Is this correct? If so, why?\n4) Did you fine tune several models with several seeds? what is the confidence score of your evaluation?\n", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and effort in giving us valuable feedback and for recognizing our efforts in this work. We refer to your “reasons to reject” as “RR”, and your questions as “Q” below.\n\nRR1 (Related Work): Yes, we could not give details in the main paper due to lack of space. We will expand the section in the final version. \n\nRR2 (Writing issues): We apologize for the issues in writing. We will proofread the paper thoroughly and correct the issues.\n\nRR3: While we acknowledge your perspective, we would like to clarify a few points. It is important to highlight the specific target audience and the broader impact of our work. This study is intended for researchers who are interested in Summarization and Legal-NLP. In a linguistically diverse country like India, where legal texts are written in complex English due to historical reasons, it is crucial to summarize them in Indian languages to ensure equitable access to justice. So, recognizing India's linguistic diversity and the need for improving access to justice, multilingual summarization datasets are very much required in the Indian legal domain. This study presents a pioneering effort towards cross-lingual summarization of English legal documents into Hindi, the most frequently spoken Indian language. Also, we are committed to enriching MILDSum's impact by including summaries in other Indian languages. \n\nAdditionally, please note that MILDSum is large enough compared to other cross-lingual legal datasets (non-Indian) like Eur-Lex [https://aclanthology.org/2022.emnlp-main.519/]. Eur-Lex contains 1,500 doc-summ pairs per language. Although language diversity is less in MILDSum compared to Eur-Lex, MILDSum contains 3,122 cross-lingually aligned doc-summ pairs that is much higher than Eur-Lex. So, we believe MILDSum is a valuable resource for researchers in this domain. We also envision to evaluate the performances of more recent Large Language Models (LLMs) such as LLaMA and GPT-4 on our proposed legal domain-specific dataset in the future.\n\nAlso note that, apart from training summarizing models, this dataset can also be used to train / evaluate Machine Translation (MT) models, given the paired English and Hindi summaries. MT in the Indian legal domain is another challenging problem which has not been explored much. We will point out this additional utility of the dataset in the final version. \n\n[Q1] The dataset used in this work is primarily constructed from the \"LiveLaw\" website. According to the terms and conditions outlined on the website, content such as judgments, orders, laws, regulations, or articles can be copied and downloaded for personal and non-commercial use. Our use of this data adheres to these terms, as we are utilizing the content (mainly the judgements and the articles from which the summaries are obtained) for non-commercial research purposes.\n\n[Q2] We have now conducted a quality check to ensure the reliability of the pdf2text tool. We randomly selected 5 pages from various documents and then manually compared the text present in the PDF with the text that was extracted using the pdf2text tool. We find that the tool's output closely matches the content present in the PDFs, with only minor instances of punctuation errors. Also, these judgment-PDFs were of high quality, contributing to the tool's accurate performance. We will add this quality check in the camera-ready draft.\n\n[Q3] We acknowledge that we did not report the results of fine-tuned CrossSum on our dataset. We attempted to fine-tune the CrossSum-mT5 model (source code and English-to-Hindi checkpoint obtained from https://github.com/csebuetnlp/CrossSum/ and https://huggingface.co/csebuetnlp/mT5_m2o_hindi_crossSum). We obtained fairly low scores even after fine-tuning the model for a few epochs (using default hyperparameters). Upon careful observation, we found that the fine-tuned model was frequently hallucinating facts and producing repetitive summaries. Hence, we avoided reporting the scores in the paper. The scores we got:\n[Scores of off-the-shelf version (reported in paper) - Rouge-2: 6.54, Rouge-L: 14.45, BERTScore: 69.49]\n[Scores of fine-tuned version - Rouge-2: 4.15, Rouge-L: 14.85, BERTScore: 62.81]\nWe will further investigate the issue, and try to add the results of fine-tuned CrossSum-mT5 in the final version.\n\n[Q4] We used a default seed value of 42 to initialize the model parameters before fine-tuning/training begins, to ensure that the same results can be reproduced every time with the same settings. We mentioned in L248 of the paper that we have used the default hyperparameters for all models. As suggested, we will run the models with different seed values and report the average values with confidence scores in the final version.\n\nAlso, thank you for highlighting the presentation and writing issues. We will definitely rectify these issues in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The authors tackle a real-world important problem, that doesn't have existing good solutions. The authors collected a decently sized dataset and will release it.\n2) The authors manually evaluated the quality of a large partition of the collected dataset.\n3) The authors benchmark the dataset using popular and reasonable models.\n", "reasons_to_reject": "1) Related work section is too short. But I expect this issue can be easily fixed given the extra page of the camera ready, as more relevant info is provided in the appendix.\n2) There are many presentation and writing issues. The paper must be better proofread.\n3) While the authors tackle a real-life problem that affects many people, it is still quite limited, and not many researchers will benefit from the release of the dataset.", "questions_to_authors": "1) What is the datasets' license?\n2) Did you validate the quality and error rate of the pdf2text tool?\n3) According to line 294, it seems like you didn't finetune CrossSum-mT5 on your dataset. Is this correct? If so, why?\n4) Did you fine tune several models with several seeds? what is the confidence score of your evaluation?\n", "paper_topic_and_main_contributions": "The authors address in their work an important problem - making legal case judgments given by Indian courts accessible to the Indian populace. While the case studies are usually written in complex English, the English proficiency of many of the country's people is not high enough to be able to read and understand them. Therefore, the authors wish to develop automatic methods to summarise the English cases in Hindi. \n\nFor this reason, the authors collect a cross-lingual dataset of legal cases and their summaries, and benchmark it. They find that first summarising and then translating is the best approach in practice.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- line 24) write what the interesting inferences that you found are, otherwise the abstract is incomplete\n- line 132) what date mismatch? In general, this entire subsection is confusing and unclear\n- The paper clearly requires more proofreading, there are many mistakes throughout\n- The introduction contains too many superlatives in my opinion\n- The underlined and bold text in section 4.1 is a bit much\n"}}]

Author rebuttal:

Meta-review: This study introduces a multilingual legal document summarization dataset, comprising 3,122 judgment-summary pairs in both English and Hindi. In conjunction with the dataset, the authors presented benchmark results using various models.

All reviewers recognized the value of the dataset for this novel domain. Reviewer egHn raised several concerns, many of which have been addressed. However, the ethics statement remains unclear. To enhance clarity, the authors should elaborate (i) the implications of the models’ output, and (ii) the consent required for such data collection (as also mentioned by Reviewer eZYq). When releasing the dataset, it's imperative to ensure license terms accompany the released package. Concerning Q2 from Reviewer eZYq, providing a quantitative value would be more informative for readers. 

Addressing Reviewer XjyQ's point on "some quantitative measures" is crucial to verify the dataset's quality. Although this concern was responded to in the rebuttal, such details should be incorporated into the paper.
Decision: Accept-Main