Interaction, External Representation and Sense Making

Interaction, External Representation and Sense Making David Kirsh (kirsh@ucsd.edu) Dept of Cognitive Science, UCSD La Jolla, CA 92093-0515 ABSTRACT the property. Why? If the sentence were “The soup is boiling over” or “A square measuring 4 inches by 4 inches is larger than one measuring 3 inches by 3 inches” virtually no on would bother. Why do people create extra representations to help them make sense of situations, diagrams, illustrations, instructions and problems? The obvious explanation – external representations save internal memory and computation – is only part of the story. I discuss eight ways external representations enhance cognitive power: they provide a structure that can serve as a shareable object of thought; they create persistent referents; they change the cost structure of the inferential landscape; they facilitate re-representation; they are often a more natural representation of structure than mental representations; they facilitate the computation of more explicit encoding of information; they enable the construction of arbitrarily complex structure; and they lower the cost of controlling thought – they help coordinate thought. Figure 1. By drawing an example of a right angle triangle and median it is easier to understand the claim ‘in a right-angled triangle the median of the hypotenuse is equal in length to half the hypotenuse’. The illustration does not carry the generality of the linguistic claim but it is easier to convince ourselves of its truth. In 1b the equalities are explicitly marked and the claim is even easier to read and helps hint at problem solving approaches. Keywords External representations, interactivity, sense making, cost structure. Introduction Here is a basic puzzle about sense making. In a closed world, consisting of a person and a representation – a diagram, illustration, spoken instruction or written problem statement – why do people so often perform actions to help them understand? If we assume there is no one to ask, no tool to generate new results, no clock to provide chronometric input, no process to run and then observe the outcome, then nothing changes in the environment other than what that person changes. If all the information needed for full understanding is logically present in mind and initial representation, then in principle, the environment contains no additional information after a person’s actions than before. Yet people make marks, they gesture, point, mutter, manipulate the inert representation, they write notes, annotate, rearrange things and so on. Why not just ‘think’? Why interact? Figure 1a illustrates a simple case where interaction is almost inevitable. A subject is given the sentence “A basic property of right-angled triangles is that the length of a median extending from the right angle to the hypotenuse is itself one half the length of the hypotenuse”. What do people do to make sure they understand? After re-reading the sentence a few times, if they have an excellent imagination and some knowledge of geometry, they just think about the sentence and come to believe they know what it means. They know how to make sense of it without interacting with anything external. Most of us, though, reach for pencil and paper, and sketch a simple diagram, such as in figure 1a or 1b, to better understand the truth of This essay is an inquiry into why we interact with the world when we try to make sense of things. There are, I believe, two major types of interaction concerned with external representations. The first, and most familiar type – the only one I will examine in this article – concerns our reliance on tools, representations and techniques for solving problems and externalizing thought. In the right-angled triangle case, for example, we make an illustration to facilitate understanding. We then perform a variety of operations, mental or physical, on that external representation. My discussion of this resource-oriented sort of interaction focuses on the power of physical sorts of operations – ways sense makers interact to change the terrain of cognition. The second, and less well-documented type of interaction concerns those things we do to prepare ourselves to use external representations, things we do to help us project cognitive structure. They are activities that help us tie external representations to their referents. For example, before we use a map to wayfind, we typically orient or ‘register’ the map with our surroundings to put it into a usable correspondence with the world. Many of us also gesture, point, talk aloud, and so on. These sorts of ‘extra’ actions are pervasive when people try to understand and follow instructions. They are not incidental and quite often vitally important to sense making, though rarely studied. The theme unifying both the first and second types of interaction is their connection with our ability to project structure onto things and then modify the world to materialize or reify our projection. This core interactive process – project then materialize – underlies much of our