Evaluating Text Understanding Systems


The objectives of this project are to advance our understanding of the merits of current text analysis techniques, as applied to the performance of realistic text analysis tasks, and to achieve this understanding by means of a sound performance evaluation methodology. The performance data can be interpreted in light of information known about the text interpretation techniques for the various systems to yield qualitative insight into the relative validity of those techniques for the text analysis task. The data can also be used as a means for determining which research areas are most critical to the successful performance of the task. The most recent performance evaluation was conducted in May 1991, on systems contributed by 15 R&amp;D sites. The evaluation task was intended to yield insight into text analysis technology, including the use of information retrieval technology (document retrieval and categorization) instead of or in concert with language understanding technology. The evaluation concluded with the Third Message Understanding Conference (MUC-3).