Evaluation Metrics for Generation

Certain generation applications may profit from the use of stochastic methods. In developing stochastic methods, it is crucial to be able to quickly assess the relative merits of different approaches or models. In this paper, we present several types of intrinsic (system internal) metrics which we have used for baseline quantitative assessment. This quantitative assessment should then be augmented to a fuller evaluation that examines qualitative aspects. To this end, we describe an experiment that tests correlation between the quanti tat ive metrics and human qualitative judgment. The experiment confirms that intrinsic metrics cannot replace human evaluation, but some correlate significantly with human judgments of quality and understandabili ty and can be used for evaluation during development. 1 I n t r o d u c t i o n For many applications in natural language generation (NLG), the range of linguistic expressions that must be generated is quite restricted, and a grammar for a surface realization component can be fully specified by hand. Moreover, iLL inany cases it is very important not to deviate from very specific output in generation (e.g., mari t ime weather reports), in which case hand-crafted grammars give excellent control. In these cases, evaluations of the generator that rely on human judgments (Lester and Porter , I997) or on human annotat ion of the test corpora (Kukich, 1983) are quite sufficient . . . . However. in other NLG applications the variety of the output is much larger, and the demands on the quality of the output are solnewhat less stringent. A typical example is NLG in the context of (interlinguaor transfer-based) inachine translation. Another reason for relaxing the quality of the output may be that not enough time is available to develop a full gramnlar for a new target, language in NLG. ILL all these cases, stochastic methods provide an alternative to hand-crafted approaches to NLG. 1 To our knowledge, the first to use stochastic techniques in an NLG realization module were Langkilde and Knight (1998a) and (~998b) (see also (Langkilde, 2000)). As is the case for stochastic approaches in natural language understanding, the research and development itself requires an effective intrinsic metric in order to be able to evaluate progress. In this paper, we discuss several evaluation metrics that we are using during the development of FERGUS (Flexible Empiricist/Rationalist Generation Using Syntax). FERCUS, a realization module, follows Knight and Langkilde's seminal work in using an n-gram language model, but we augment it with a tree-based stochastic model and a lexicalized syntactic grammar. The metrics are useful to us as relative quanti tat ive assessments of different models we experiment with; however, we do not pretend tha t these metrics in themselves have any validity. Instead, we follow work done in dialog systems (Walker et al., 1997) and at tempt to find metrics which on tim one hand can be computed easily but on the other hand correlate with empirically verified human judgments in qualitative categories such as readability. The structure of the paper is as follows. In Section 2, we briefly describe the architecture of FEacUS, and some of the modules. In Section 3 we present four metrics and some results obtained with these metrics. In Section 4 we discuss the for experimental validation of the metrics using human judgments, and present a new metric based on the results of these experiments. In Section 5 we discuss some of the 'many problematic issues related t o the u s e Of metrics and our metrics in particular, and discuss on-going work. 2 S y s t e m O v e r v i e w FERGUS is composed of three mQdules: .the Tree Chooser, tile Unraveler, and the Linear Precedence (LP) Chooser . Tile input to the system is a dependency tree as shown in Figure 2. t Note that the nodes are unordered and are labeled only with lexemes, not with any sort of syntactic annotations. 2 The Tree Chooser uses a stochastic tree model to choose syntactic properties (expressed as trees in a Tree Adjoining Grammar) for the nodes in the input structure. This step can be seen as analogous to "supertagging" -(Bangalore-und doshh 1:999);. except that now supertags (i.e., names of trees which encode the syntactic properties of a lexical head) must be found for words in a tree rather than for words in a linear sequence. The Tree Chooser makes the siinplifying assumptions that the choice of a tree for a node depends only on its daughter nodes, thus allowing for a top-down algorithm. The Tree Chooser draws on a tree model, which is a analysis in terms of syntactic dependency for 1,000,000 words of the Wall Street Journal (WSJ). 3 The supertagged tree which is output from the Tree Chooser still does not fully determine the surface string, because there typically are different ways to at tach a daughter node to her mother (for example, an adverb can be placed in different positions with respect to its verbal head). The Unraveler therefore uses the XTAG grammar of English (XTAG-Group, 1999) to produce a lattice of all possible linearizations tha t are compatible with the supertagged tree. Specifically, the daughter nodes are ordered with respect to the head at each level of the derivation tree. In cases where the XTAG grammar allows a daughter node to be attached at more than one place in the mother supertag (as is the case in our example for was and for; generaUy, such underspecification occurs with adjuncts and with arguments if their syntactic role is not specified), a disjunction of all these positions is assigned to the daughter node. A bot tom-up algorithm then constructs a lattice that encodes the strings represented by each level of the derivation tree. The lattice at the root of the derivation tree is the result of the Unraveler. Finally. the LP Chooser chooses the most likely traversal of this lattice, given a linear language 1The sentence generated by this tree is a predica t ivenoun construction. The XTAG g ram m ar analyzes these as being headed by the noun , ra the r than by . the copula, and we follow the XTAG analysis. However, it would of course also be possible to use a graminar that allows for the copula-headed analysis. 21n the system that we used in the experiments described in Section 3. all words (including function words) need to be present in the input representation, fully inflected. Furthermore, there is no indication of syntactic role at all. This is of course unrealistic f~r applications see ,Section 5 for further renlarks. :3This wa~s constructed from the Penn Tree Bank using some heuristics, sirice the. l)enn Tree Bank does not contain full head-dependerit information; as a result of the tlse of heuristics, the Tree Model is tint fully correct. 2 I TAG Derivation Tree without Supertags