# Solving Multi-Document Summarization as an Orienteering Problem
## 6 Experiments
### Corpora  
Since 1998, a number of conferences and workshops have been organized to address automatic summarization problems. In this study, two corpora were selected to evaluate the performance of MDS-OP. The first one is DUC 2004 (Task 2). DUCs ([[http://www.nlpir.nist.gov/projects/duc/index.html](http://www.nlpir.nist.gov/projects/duc/index.html)]([http://www.nlpir.nist.gov/projects/duc/index.html](http://www.nlpir.nist.gov/projects/duc/index.html))) were an important series of conferences that addressed issues of automatic summarization. They were held yearly between 2001 and 2007. Competitions were organized to compare summarization systems on different corpora related to different tasks. The corpus of Task 2 from DUC 2004 consists of 50 English clusters where each one of them contains around 10 documents, and the main task is to create a short summary with a length up to 665 bytes for each cluster. The published results include the scores of eight human summarizers, a baseline, and 16 participating systems (see Table 2).  
The second corpus used to evaluate the algorithm is the corpus of the MMS task at MultiLing 2015 ([[http://multiling.iit.demokritos.gr/pagesview/1516/multiling-2015](http://multiling.iit.demokritos.gr/pagesview/1516/multiling-2015)]([http://multiling.iit.demokritos.gr/pagesview/1516/multiling-2015](http://multiling.iit.demokritos.gr/pagesview/1516/multiling-2015))) [67], which is a special session at SIGdial 2015 ([[http://www.sigdial.org/workshops/conference16/](http://www.sigdial.org/workshops/conference16/)]([http://www.sigdial.org/workshops/conference16/](http://www.sigdial.org/workshops/conference16/))). It was built upon the corpus of the MultiLing 2013 ([[http://www.nist.gov/tac/2011/Summarization/index.html](http://www.nist.gov/tac/2011/Summarization/index.html)]([http://www.nist.gov/tac/2011/Summarization/index.html](http://www.nist.gov/tac/2011/Summarization/index.html))) workshop at ACL 2013, which in turn is based on the TAC 2011 MultiLing Pilot ([[http://multiling.iit.demokritos.gr/pages/view/662/multiling-2013](http://multiling.iit.demokritos.gr/pages/view/662/multiling-2013)]([http://multiling.iit.demokritos.gr/pages/view/662/multiling-2013](http://multiling.iit.demokritos.gr/pages/view/662/multiling-2013))) corpus. This corpus contains sets of documents written in 10 languages. MDS-OP was evaluated on the English version which consists of 15 sets, each including 10 documents. The participants were asked to provide partially or fully language-independent multi-document summarizers, which produce at most 250 words summaries. Each summarizer was applied to at least two different languages. Ten teams participated to MMS task in MultiLing 2015 (see Table 3).  
\begin{table}
\begin{tabular}{l l l} \hline \hline
**System ID** & **Research Group (Participant)** & **Reference** \\ \hline CCSNA04 & NSA & [22] \\ MEDLAB\_Fudan & Fudan University & * \\ CL & CL Research & [14] \\ LARIS.2004 & Laris Labs & [48] \\ ULeth2004 & University of Lethbridge & [43] \\ columbia1 & Columbia University & [28] \\ CLaCDUCTape2 & Concordia University & [44] \\ webcl2004 & ISI & * \\ kul.2004 & KU Leuven & [31] \\ lcc.duc04 & LCC & [15] \\ uofo & University of Ottawa & * \\ msr-nlp.duc2004 & Microsoft & [37] \\ crl\_nyu.duc04 & CRL/NYU & [16] \\ nttcslab.duc2004 & NTT & [20] \\ shef2004.saggion & University of Sheffield & [30] \\ UofM-MEAD & University of Michigan & [33] \\ \hline \hline \end{tabular}  
* The official DUC website does not contain any details on these participants.  
\end{table}
Table 2: Systems that participated at DUC 2004 (Task 2).