Grading Diagrams Automatically

This paper describes a feasibility study into the marking (grading) of diagrams automatically. The diagrams were produced by students during an on-line examination using a simple drawing tool. The students’ examination answers, which included a diagram, were submitted over the Internet to an automatic marking tool for grading. This paper describes the methodology adopted by the diagram marking tool and comments on the results achieved. Whilst only a small number of diagrams were available for marking, the grades awarded by the automatic marker were sufficiently close to the marks awarded by four independent human markers to suggest that marking diagrams automatically, for small diagrams at least, is feasible and should be pursued further. Introduction For several years we have been investigating the automatic grading of examination papers in which student answers are all free-form text. That is, the questions require answers that are akin to essays and are not multiple choice questions [Thomas, 2001]. The students have typed their answers into a web form using a PC at home and submitted their answers via the Internet to a server for grading. This electronic examination system gives immediate feedback by providing a grade and textual feedback for each answer. The results have been encouraging [Thomas, 2002]. However, until recently, the system would only deal with textual answers; there was no provision for the creation or marking of diagrams. We felt that this was a limitation of the tool. In addition, the introduction of a diagrammatic marking component raises a number of interesting research questions about the correspondences between diagrammatic and sentential forms. Both grammatical and statistical techniques have been applied successfully to understanding and manipulating sentential language [Gazdar & Mellish, 1989; Manning & Schütze, 1999]. Many automatic marking systems for text generally take a statistical approach: our own work looks for appropriate keywords in answers [Thomas, 2001], and the commercial Intelligent Essay Assessor system [Foltz et al., 2000] uses latent semantic analysis for marking. In contrast, the existing literature on reasoning with diagrams [Mariott et al., 1998; Blackwell & Engelhardt, 2002] concentrates on rulebased and grammatical representations of diagrams and we have not seen this work applied to the grading process. In this work, we take some first steps towards applying statistical techniques to diagram understanding. Therefore, this work represents an early step in the automatic marking of diagrams, and seems to be the first application of statistical techniques to diagram understanding. The remainder of this paper describes how our initial diagrammatic marking tool was developed, and some early results of its use. In the next section we briefly describe the specialised drawing tool that was developed for this experiment. The remainder of the paper takes a more detailed look at how the diagram is represented, how a marking scheme is represented, and how the diagram is combined with the marking scheme to generate a mark for the student. The drawing tool Given that the student volunteers did not require the use of a drawing tool of any description for their course, we felt that a simple tool that would require minimal familiarisation would be appropriate. The tool should also facilitate the drawing of diagrams appropriate to the examination – in order to minimise the time of use during the examination. In addition, we are primarily interested in diagrams that commonly occur in the teaching of computing (such as E-R and UML diagrams). Therefore, we adopted a very simple approach in which the drawing tool supported the creation of diagrams that are limited to two simple geometric objects that we have named boxes and links. A box has a rectangular shape and can contain text. A link is a directed line (arrow) and can also be associated with some text. The text associated with each object can be used for labelling and explanation purposes. Furthermore, the role of a link is limited to joining two boxes and can be used to indicate a relationship between them. For example, in a UML class diagram a box labelled with a name would represent a class, and a link, also labelled, would represent an inheritance relationship between two classes as illustrated in Figure 1. road transport Bus Vehicle Figure 1 A simple inheritance relationship. Objects are placed on the drawing canvas using a drag-and-drop technique. Boxes can be placed anywhere on the canvas, but links must join two boxes. A link is constructed with the mouse by leftclicking inside one box, dragging the cursor to another box and releasing mouse button. A range of editing and file handling facilities are also provided. However, if the user wishes to delete a box from the diagram, any associated links will also be removed (but not before the user has been prompted with an appropriate warning). A link can be removed at any time. Thus, a link is associated with two boxes and has a direction and text. A box has text and can be associated with zero, one or more other boxes (but not itself, in the current implementation) through a set of links. The location and size of each box and each link on the drawing canvas is also recorded. Hence, a box has the following attributes: •text, which may be empty, •a location on the drawing canvas, consisting of the (x,y) co-ordinates of its top-left-hand corner, •a size, consisting of its width and height, •a set, possibly empty, of references to its associated links. A link has the following attributes: •text, which may be empty, •a location on the drawing canvas, consisting of the (x,y) co-ordinates of the top-left-hand corner of its bounding rectangle, •a size, consisting of the width and height of its bounding rectangle, •references to its two associated boxes, with the start box reference given first. Example Figure 2 shows an example of a drawing constructed with the drawing tool that will be used throughout the remainder of the paper. The drawing represents a solution to the examination question given in Figure 3. The diagram illustrates a computer processor architecture known as a 4-stage pipeline. The boxes represent the four stages of execution of a machine code instruction (fetch, decode, execute and write) and the links shows the order in which the stages occur. In this example, there are two pipelines and an association labelled “forward” between the two execution stages. Figure 2 A drawing of two pipelines with a link between them.