The MUC proceedings contain the results of these performance evaluations as well as system descriptions for each participating system. This article was originally published in AI Magazine and is a good introduction to both information extraction and the MUC evaluations. The Department of Defense has been an important advocate and resource for research in information extraction. Two important metrics for assessing the performance of an IE system are recall and precision . Recall refers to how much of the information that should have been extracted was correctly extracted.
Precision refers to the reliability of the information extracted.
These are independent measures and they paint a useful picture of system performance. Your precision would depend on how many questions you actually tried to answer. When we look at the test results of various IE systems that have been subjected to rigourous tests, it is important to consider the baseline of human performance as a key comparison point. When people are asked to perform information extraction tasks, they tend to fall short of perfect performance for a number of reasons. Human factors associated with tedium, throughput pressures, and imperfect command of the domain guidelines can all contribute to less than perfect recall and precision.