## 4 Evaluation  
### Evaluation Methodology  
According to [PERSON] [13], the clusters produced by a document clustering system can be evaluated in two ways, namely by using qualitative evaluation where the actual clusters are evaluated and quantitative evaluation by demonstrating the benefit of clustering for a particular retrieval task. For our clustering system, we performed qualitative evaluation since we evaluated the quality of the actual clusters. We utilised three different datasets of clustered news reports, and we compared how close the clusters produced by our system are to the clusters in these datasets. We then repeated the evaluation using a baseline system. The metrics generally used for clustering evaluation are the recall, precision and F-measure--metrics that have been adapted from Information Retrieval. In Information Retrieval, recall is taken to be the fraction of the relevant documents that have been retrieved. Conversely, precision is the fraction of the retrieved documents that are relevant to the query. The F-measure metric is calculated as a weighted average of recall and precision. The use of these metrics in Document Clustering is discussed in [7,10,13,17,19]. We also utilised the recall, precision and F-measure metrics. In our evaluation, the F-measure metric gave equal weight to recall and precision. For the evaluation of single-label text clustering, where a document may be a member of only one cluster, [PERSON] [3] modifies the Reuters-21578 collection by discarding documents with multiple category labels, and removing clusters with less than 40 documents. Since our clustering system also performs single-label clustering, we discarded those news reports which belonged to more than one category. We utilised three different corpora for our evaluation. The first corpus is the _Reuters-RCV1_ collection--this collection has been used in [13,25] and is the successor of the _Reuters-21578_ collection that was used in [3,4,7,17]. The original corpus contains about 810,000 Reuters English Language News stories published between 20 th August 1996 and 19 th August 1997. The filtered corpus (where the reports belonging to more than one category have been removed) contains 26,310 news reports classified into 78 categories. The second corpus is the _Yahoo! News_ collection--a collection of news reports downloaded via the RSS feeds provided by the _Yahoo! News_. _Yahoo! News_ provides RSS feeds for a number of categories that are not so general. Examples of such categories are \"_Apple Macintosh_\", \"_India_\" and \"_Democratic Party_\". This corpus contains 7342 news reports classified into 68 categories. These reports were downloaded in December 2010 and January 2011. The third corpus is the _Google News_ collection--this is a collection of news reports download via the RSS feeds provided by _Google News_. These news reports are clustered according to the event that they describe. Therefore, the clustering in this dataset is much finer than the categories in the other datasets. This dataset is considered the most appropriate for the evaluation of our system, since the aim of our clustering component is to cluster together those news reports that are describing the same event. This dataset contains 1561 news reports downloaded in January 2011 that are covering 205 different news events. [PERSON] [6], [PERSON] [7] and [PERSON] [11] evaluate their clustering methods by comparing their automatic clustering results with manual clustering results. Others perform comparisons with other baselines. For example, [PERSON] [26] compares ranked documents in a best cluster to an equivalent cutoff in the original ranked retrieval results. On the other hand, [PERSON] [15] compares the presented results with a basic First Story Detection system that uses the traditional vector space model to compute syntactic similarity between documents and clusters. In our evaluation, we compared the clusters obtained by our system with a baseline system that uses a standard k-means algorithm as described in [23].