Keynote: Traditions in the Evaluation of MT

It has been a cliche in the field for years that machine translation evaluation is a better founded subject than machine translation. The reason why one could say this is that over a substantial period there would seem to have been less fundamental dispute in machine translation evaluation. There has certainly been fundamental dispute in machine translation. That is very evident at the moment in MT efforts in the United States, and it’s a very interesting situation for empiricists. However, there also is some dispute, as you know, in machine translation evaluation, and one of the great virtues and interests of this workshop is that we are going to make it explicit. Machine translation evaluation shares some features with MT, particularly what I like to think of as the easy start-up feature. A slide I’ve shown for some time says that “Any theory will give you some MT.” It’s an unfair slide, but there’s a great deal of truth in it. We all know, if we’ve been around for some time, that there is no theory, however silly, upon which you cannot base some machine translation. That’s what makes MT quite different from, say, space and flight, where not just any theory gets an object off the ground. And of course, MT evaluation shares this easy start-up feature, because it is not hard to do a naive machine translation evaluation: you just look at it. Do the two say the same thing or not? But we know, don’t we, that that’s not quite enough. Since U.S. machine translation has had something of a revival in the last few years, this is a very timely moment for evaluation. I’m not sure who currently holds the record for the world’s largest MT project. It was EUROTRA for a long time, but I suspect that at the moment it is the DARPA machine translation project, where, because of the U.S. empiricist tradition, evaluation was built in right from the beginning. Also, evaluation has been very successful with speech research here in the United States in recent years, and many believe that if it has been so successful with speech, it ought to be equally successful in MT. One may doubt that argument, but there can be no question of the value of evaluation. One cannot be in machine translation and say “We know we’re right, we know we’re doing good stuff, but we’d rather not be evaluated”—although that feeling does linger in some corners of the field. I put it down to the linguistic inheritance in the mixed gene pool of MT. The linguistic tradition has been antiempirical, at least in the last 40 or so years, in this country. I suggest that there are four traditions of MT evaluation, of which the first and last are the most interesting, though not ones we shall be discussing here. For those of you who don’t know the story of PIMA, this was a restricted form of English created during the Vietnam war for the maintenance of helicopters. As I recall, there was a very simple test developed for the adequacy of PIMA that measured the number of helicopters that crashed which had been repaired and maintained using PIMA versus those that had been repaired and maintained using either full English or a (not then available) full Vietnamese manual. That’s what I would call a radical behavioral test. In theory, some such methodology could be developed for MT. It would have to be a wholly domain-orientated methodology. If it was financial mergers you were interested in translating, for example, there could be a financial test based on it. It’s a test staged entirely in terms of the subject matter. The last type of evaluation is one I was once involved with many years ago—namely, an evaluation of SYSTRAN for the Air Force. Its purpose was not to test SYSTRAN—it was assumed that SYSTRAN