Title: GlobeSumm: A Challenging Benchmark Towards Unifying Multi-lingual, Cross-lingual and Multi-document News Summarization
Benchmark: GLOBESUMM
Summary: Propose GLOBESUMM and introduce prompting method for silver summary annotation. Validate the quality and difficulty of the dataset.
Phenomenon type: Specific Application (A single use case, where the benchmark is likely to be examples of that use case)
Target phenomenon: Text summarization
Phenomenon defined by authors: Yes
Phenomenon definition: The goal of Multi-lingual, Cross-lingual and Multi- document Summarization (MCMS) is to succinctly capture the key information from a collection of documents written in various languages and present a cohesive summary in the target language. Notably, the MCMS task has three distinctive features: (1) the input consists of multi- ple documents, (2) the multiple documents are in different languages, and (3) the multiple documents revolve around the same event. 
Definition scope: Subset
Definition integrity: Composite phenomenon
Definition integrity detail: Yes
Phenomenon contested: Widely-agreed
Task definition: (a) Single- turn Summarization summarizes a document set within a single-turn generation; (b) Chronological Recurrent Summarization iteratively summarizes two documents at a time in a time-ordered manner 
Task item definition: The model is given a set of articles and asked to summarize them in one or multiple turns.
Task source: Real task examples (e.g. GitHub issues)
Task ecology: Complete real task (e.g. providing medical advice to real people interactively)
Task ecology detail: People would use chatbots to summarize news articles, in my opinion.
Dataset size: 74 events  942 documents  868 summaries
Dataset size notes: Training Set: 222 events, 2,848 documents, and 2,626 summaries​  Validation Set: 74 events, 897 documents, and 823 summaries
Dataset metadata available: Yes
Dataset metadata detail: language
Sampling method: Targeted items (creators defined a task space and chose tasks within it strategically)
Split reviewed: Test, Train, Validation
Response format: Free response (e.g. summary paragarph)
Metric definition: n-gram (BLEU, ROUGE, chrF), LLM post-processing (extracting answers, reformatting for automated scoring)
Metric definition detail: On top of ROUGE, authors also use Red (Chen et al., 2021) for redundancy, Normalized Inverse of Coverage (NIC) for Omission, and Conflict Resolution Effectiveness (CRE) for conflict
Metric statistics: simple mean
Metric subscores: Yes
Metric subscores detail: for different languages
Metric access: Outputs alone
Authorship: Mix (multiple authors from industry and academia)
Benchmark availability: Yes
Task face validity: Yes
Metric face validity: Yes
Results interpreted by authors: Yes
Results compared: No
Results comparison explanation: No comparisons made
Results realism: No
Human baseline: No
Authors discuss validity: Yes
Author validity detail: the authors conduct extensive human validation in the annotation process. They also validated their annotation method against other benchmark (XQuAD specifically).
Taxonomy: NLP / Summarization
Alternate taxonomy: Multilinguality
New taxonomy: General Application / Summarization