Learning Discourse Relations with Active Data Selection

The paper presents a new approach to identifying discourse relations, which makes use of a particular sampling method called committeebased sampling (CBS). In the committee-based sampling, multiple learning models are generated to measure the utility of an input example in classification; if it is judged as not useful, then the example will be ignored. The method has the effect of reducing the amount of data required for training. In the paper, we extend CBS for decision tree classifiers. With an additional extension called error feedback, it is found that the method achieves an increased accuracy as well as a substantial reduction in the amount of data for training classifiers. 1 I n t r o d u c t i o n The success of corpus-based approaches to discourse ultimately depends on whether one is able to acquire a large volume of data annotated for discourse-level information. However, to acquire merely a few hundred texts annotated for discourse information is often impossible due to the enormity of the haman labor required. This paper presents a novel method for reducing the amount of data for training a decision tree classifier, while not compromising the accuracy. While there has been some work exploring the use of machine leaning techniques for discourse and dialogue (Marcu, 1997; Samuel et al., 1998), to our knowledge, no computational research on discourse or dialogue so far has addressed the problem of reducing or minimizing the amount of data for training a learning algorithm. * The work reported here was conducted while the first author was with Advanced Research Lab. , Hitachi Ltd, 2520 Hatoyama Saitama 350-0395 Japan. A particular method proposed here is built on the committee-based sampling, initially proposed for probabilistic classifiers by Dagan and Engelson (1995), where an example is selected from the corpus according to its utility in improving statistics. We extend the method for decision tree classifiers using a statistical technique called bootstrapping (Cohen, 1995). With an additional extension, which we call error . feedback, it is found that the method achieves an increased accuracy as well as a significant reduction of training data. The method proposed here should be of use in domains other than discourse, where a decision tree strategy is found applicable. 2 Tagging a corpus w i t h d i scourse re lat ions In tagging a corpus, we adopted Ichikawa (1990)'s scheme for organizing discourse relations (Table 1). The advantage of Ichikawa (1990)'s scheme is that it directly associates discourse relations with explicit surface cues (eg. sentential connectives), so it is possible for the coder to determine a discourse relation by figuring a most natural cue that goes with a sentence he/she is working on. Another feature is that, unlike Rhetorical Structure Theory (Mann and Thompson, 1987), the scheme assumes a discourse relation to be a local one, which is defined strictly over two consecutive sentences. 1 We expected that these features would make a tagging task less laborious for a human coder than it would be with RST. Further, our earlier study indicated a very low agreement rate with 1This does not mean to say that all of the discourse relations are local. There could be some relations that involve sentences separated far apart. However we did not consider non-local relations, as our preliminary study found that they are rarely agreed upon by coders.