User-Oriented Machine Learning Strategies for Information Extraction: Putting the Human Back in the Loop

Efforts in information extraction (IE) have concentrated on fundamental issues concerning the viability of the technology. Two of these important issues are scalability and portability. To illustrate, consider the AutoSlog system for learning a dictionary of extraction patterns [Riloff, 1993]. AutoSlog automatically acquires extraction patterns based on training documents annotated with extraction templates, thus eliminating the need for a human programmer to encode patterns or rules for extraction. Even further, more recent versions of AutoSlog acquire these patterns without explicit training data, using either relevancy signatures [Riloff, 1996] or mutual bootstrapping with a semantic lexicon [Riloff and Jones, 1999], eliminating the need for a human annotator to produce training data from raw documents. In short, to address the issues of scalability and portability, it has been important to find ways to get the human out of the loop, both as a programmer encoding patterns, and as an annotator producing training data. Scalability and portability have been useful and important issues for driving research in information extraction. However, as IE technology matures, it becomes increasingly relevant to consider issues concerning the deployment of IE applications for real users. Real users will not be experts in machine learning or text processing, nor will they care how extraction patterns are encoded or acquired. They will, however, be uniquely qualified experts in the identification of the specific information they wish to locate and extract. Real IE applications must allow users to specify the nature of the information structures they desire while shielding them from the details of how the system locates new structures. Assuming users specify their information needs by providing examples, there is a natural tension between two criteria of coverage and responsiveness. Coverage is the system’s ability to extract all desired information for the user, i.e. to completely cover the task; coverage encourages the system to demand training examples from the user. In contrast, responsiveness is the system’s ability to achieve a reasonable level of performance without undue burden upon the user; responsiveness thus discourages the system from demanding training examples. This position paper outlines a user-oriented metalearning strategy that attempts to balance coverage and responsiveness when training examples are solicited from a real user. Following a high-level description of the strategy, we discuss some related experiments that indicate the promise of this learning paradigm. The key idea behind user-oriented learning is the recognition of the complementary strengths of the human user and the IE system. On one hand, the human is proficient at judging an information structure as desirable or undesirable; on the other, the machine is proficient at rapidly locating similar examples from large quantities of text. Both human and machine are allowed to exercise their strengths in a scenario where the human begins by providing training examples, the machine attempts to locate additional instances based on the training examples, the human responds by confirming the desirability (or undesirability) of the new instances, the machine adds them to the training examples and continues to search for more instances, and so on. Figure 1 sketches this process as three interleaving procedures. The procedures access a collection of documents D, and sets of positive, negative, and unconfirmed examples E, E −, and E, respectively. In the Annotate procedure, the user explicitly provides training data, while in the Confirm procedure, the user judges the desirability of instances located by the system. Finally, in the Locate procedure, the system finds new instances on the basis of all confirmed training data. From the point of view of the user, user-oriented learning manages the tension between accuracy, coverage, and responsiveness very naturally. By interleaving the Annotate and Confirm procedures, we allow the user to decide what priority is given to either criterion. If better coverage is desired, the user may wish to annotate more examples. If better accuracy is desired, the user may wish to review and confirm more instances. The perceived responsiveness, of course, depends upon the patience of the user, but we note that user-oriented learning—having the system locate candidate instances and require only a simple yes/no response from the user—is a vast improvement over annotating all the training data by hand.