Completed implementation a range of document clustering techniques operating on both traditional features and features utilizing shallow linguistic analysis. Detailed comparison experiments on a collection of over 40,000 news articles establish that the linguistic features are useful for document clustering. This clustering tool will be used for organizing incoming information into sets of related documents to be processed by our summarizer. Current speed of summary generation is slow requiring from 28 to 55 minutes to generate a summary, depending on the size and number of input documents. Optimizations are currently being implemented which will reduce running time by a factor of 10.
Current performance degrades as the articles become less similar.
Extend the similarity detection tool so that it can run on input documents in languages other than English. The first step will be to handle input documents in a language other than English and the second will be to handle input documents in different languages. Implement terminology finder for Japanese and Chinese and tools for translating identified tools into English. Currently, we have a number of domain independent tools for text analysis and generation that are available for dissemination. These tools are building blocks for the larger tasks of detecting similarity across documents, identifying contradictions, partial translation, and summary generation.