ODSUM: Benchmarking Open Domain Multi-Document Summarization
Authors: Yijie Zhou, Kejian Shi, WENCAI ZHANG, Yixin Liu, Yilun Zhao, Arman Cohan


Reviewer paper summaries: This paper introduces two new benchmarks for open-domain MDS via utilizing existing query-focused MDS datasets. They conducted extensive experiments using various approaches and found that open-domain MDS remains challenging, with significant performance drops due to retrieval errors. Human evaluations also showed that current automated metrics do not accurately reflect model performance.
Summary of strengths: 1. Proposed some new benchmarks for the evaluation of MDS for the open-domain scenarios. 
2. Conducted extensive experiments with various methods and identified limitations of existing models. 
3. Also demonstrated the limitations of various evaluation metrics for this task.
Summary of weaknesses: 1. The proposed dataset is a bit synthetic since it was developed via utilizing existing datasets, although an important new problem is studied. 
2. Also, it would have been nice to present a real-world example in the introduction section to better explain the ODSUM task.
3. The proposed methodology also lacks novelty. While it is not mandatory to propose a novel method since a new benchmark is proposed, the authors did not cite related work that utilized some of their methodologies, e.g., truncation, map-reduce, etc. [Ref1, Ref2, Ref3, Ref4]:

Ref 1: Travis Goodwin, Max Savery, and Dina Demner-Fushman. 2020. Flight of the PEGASUS? Comparing Transformers on Few-shot and Zero-shot Multi-document Abstractive Summarization. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5640–5646, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Ref 2: Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2022. Domain Adaptation with Pre-trained Transformers for Query-Focused Abstractive Text Summarization. Computational Linguistics, 48(2):279–320.

Ref 3: Md Tahmid Rahman Laskar, Xue-Yong Fu, Cheng Chen, and Shashi Bhushan TN. 2023. Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspective. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 343–352, Singapore. Association for Computational Linguistics.

Ref 4: Baumel, Tal, Matan Eyal, and Michael Elhadad. "Query focused abstractive summarization: Incorporating query relevance, multi-document coverage, and summary length constraints into seq2seq models." arXiv preprint arXiv:1801.07704 (2018).
Comments, suggestions and typos: Would have been nice to see how long context LLMs, e.g., LLM with 128K or longer context perform in this dataset. 

Relevant papers should also be cited:

1. Travis Goodwin, Max Savery, and Dina Demner-Fushman. 2020. Flight of the PEGASUS? Comparing Transformers on Few-shot and Zero-shot Multi-document Abstractive Summarization. In Proceedings of the 28th International Conference on Computational Linguistics, pages 5640–5646, Barcelona, Spain (Online). International Committee on Computational Linguistics.

2. Md Tahmid Rahman Laskar, Enamul Hoque, and Jimmy Xiangji Huang. 2022. Domain Adaptation with Pre-trained Transformers for Query-Focused Abstractive Text Summarization. Computational Linguistics, 48(2):279–320.

3. Md Tahmid Rahman Laskar, Xue-Yong Fu, Cheng Chen, and Shashi Bhushan TN. 2023. Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspective. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: Industry Track, pages 343–352, Singapore. Association for Computational Linguistics.

4. Baumel, Tal, Matan Eyal, and Michael Elhadad. "Query focused abstractive summarization: Incorporating query relevance, multi-document coverage, and summary length constraints into seq2seq models." arXiv preprint arXiv:1801.07704 (2018).
Overall assessment scores: 3