The dynamic process of the user-system interaction must be taken into account in the evaluation design. New media affect evaluation One complication in evaluating retrieval of multi-media material is that even basic questions, for example, what retrieval tasks people actually want to carry out are highly dependent on the particular media represented in the system and on special skills of the users. Technically, we can offer users tools which allow for these sorts of tasks to be carried out but we seem less than confident that users are interested in actually carrying them out. At present, the crucial questions in multimedia evaluation may be upstream, when requirements are formulated. Progress in multimedia development may be best fostered by an emphasis on the evaluation of scenarios of use, rather than by the evaluation of finished systems.
Documents are usually richly structured and linked in multi-media repositories.
Queries are often a mix of querying, navigation, consulting displays, note-taking, and reformulation. Another crucial feature of multimedia is that normally any one unit of material plays a rhetorical role in a larger context of exposition. An example is where a user failed to solve a programming problem efficiently because, although a search got him to the right spot, he failed to read necessary prerequisite material. The system failed to reveal the rhetorical relations between one unit of material and the others within a larger substructure. So perhaps an important kind of evaluation is not how fast a retrieval engine can take a person to a unit of relevant material, but rather how well a retrieval engine can reveal the relations between one unit of relevant material and its context or how well can a retrieval engine present an alternative view of some set of potentially relevant units.