Topic Detection and Tracking using idf-Weighted Cosine Coefficient

The goal of TDT Topic Detection and Tracking is to develop automatic methods of identifying topically related stories within a stream of news media. We describe approaches for both detection and tracking based on the well-known idf -weighted cosine coefficient similarity metric. The surprising outcome of this research is that we achieved very competitive results for tracking using a very simple method of feature selection, without word stemming and without a score normalization scheme. The detection task results were not as encouraging though we attribute this more to the clustering algorithm than the underlying similarity metric. 1. The Tracking Task The goal of the topic tracking task for TDT2 is to identify news stories on a particular event defined by a small number ( Nt) of positive training examples and a greater number of negative examples. All stories in the news stream subsequent to the final positive example are to be classified as on-topic if they pertain to the event or off-topic if they do not. Although the task is similar to IR routing and filtering tasks, the definition of event leads to at least one significant difference. An event is defined as an occurrence at a given place and time covered by the news media. Stories are on-topic if they cover the event itself or any outcome (strictly-defined in [2]) of the event. By this definition, all stories prior to the occurrence are off-topic, which contrary to the IR tasks mentioned, theoretically provides for unlimited off-topic training material (assuming retrospective corpora are available). We expected to be able to take advantage of these unlimited negative examples but in our final implementation did so only to the extent that we used a retrospective corpus to improve term statistics of our database. 1.1. idf-Weighted Cosine Coefficient As the basis for our approach we used the idf -weighted cosine coefficient described in [1] often referred to as tf idf . Using this metric, the tracking task becomes two-fold. Firstly, choosing an optimal set of features to represent topics, i.e. feature selection. The approach must choose features from a single story as well as from multiple stories (forNt > 1). Secondly, determining a threshold (potentially one per topic) which optimizes the miss and false alarm probabilities for a particular cost function, effectively normalizing the similarity scores across topics. The cosine coefficient is a document similarity metric which has been investigated extensively. Here documents (and queries) are represented as vectors in an n-dimensional space, where n is the number of unique terms in the database. The coefficients of the vector for a given document are the term frequencies ( tf ) for that dimension. The resulting vectors are extremely sparse and typically high frequency words (mostly closed class) are ignored. The cosine of the angle between two vectors is an indication of vector similarity and is equal to the dot-product of the vectors normalized by the product of the vector lengths. cos( ) = ~ A ~ B k ~ Akk ~ Bk tf idf (term frequency times inverse document frequency) weighting is an ad-hoc modification to the cosine coefficient calculation which weights words according to their usefulnessin discriminating documents. Words that appear in few documents are more useful than words that appear in many documents. This is captured in the equation for the inverse document frequency of a word: