Semantics of Simple Arrow Diagrams
Yohei Kurata and Max J. Egenhofer
National Center for Geographic Information and Analysis
andDepartment of Spatial Information Science and Engineering
Boardman Hall, University of Maine
Orono, ME 04469-5711, USA{yohei, max}@spatial.maine.edu

Abstract
Arrows illustrate a large variety of semantics in diagrams.
An automated interpretation of arrows would be highly
desirable in pen-based interfaces. This paper formalizes thestructural patterns of arrows, and identifies three structuralproperties that contribute to the interpretation of arrows: (1)the assignment of components to three slots, (2) thesemantic types of the components, and (3) the orientation of
the components.

Introduction
Arrows are a major component of diagrams. Despite theirsimple shapes, arrows capture a large variety of semantics,
such as directions, movements, changes, temporal orders,interactions, and binary relations. Accordingly, arrows
frequently appear in various graphic representations, suchas traffic signs, illustrations, route maps, and flowcharts(Horn 1998; Wildbur and Burke 1998). Pen-based
interactions with computers are also expected to supportthe use of diagrams with arrows. In the current pen-based
systems, however, people still cannot use arrows naturally,
due to the restriction of the arrows to a small set ofmeanings (Kurtoglu and Stahovich 2002; Landay andMyers 2001; Alvardo and Davis, 2001) or the cumbersome
requirement to specify the meaning of every arrow in a
diagram (Forbus and Usher, 2002). In this way, an
automatic interpretation of arrows in diagrams remains achallenging problem.
Arrows have a remarkable ability to facilitate thecommunication of dynamic processes and complex
mechanisms. The existence of arrows encourages people tointerpret causal and functional aspects in a diagram(Tversky et al. 2000). For example, Figure 1 shows adiagram with a sequence of arrows, which captures a
dynamic process that the El Niño Effect (i.e., the sea
temperature rise in the Southeastern Pacific) indirectly
influences the rise of tofu prices in Japan, due to fewer fishcaught in South America and an implied growth in theconsumption of soybeans in North America. Such aninterpretation, however, requires an intricate reasoning
process based on our commonsense. For example, an

upward arrow symbol is known to conventionally express
rise or increase, and a price is known to rise; therefore,
people deduce that the tofu price accompanied by an
upward arrow symbol may illustrate an event where thetofu price rises. Similarly, it is deduced that the soybeansconsumption accompanied by an upward arrow symbol
indicates another event where the soybeans consumption
increases. These two events are connected by anotherarrow symbol. Such an arrow symbol connecting twoevents is typically interpreted as a causal relation. Thus,people interpret that the decrease of the soybeansconsumption may cause the rise of the tofu price.
This example indicates that the structural properties ofarrows, such as accompaniments and connections,contribute to the interpretations of the arrows. The goal ofthis paper is to identify such structural properties inherent
in an arrow that contribute to its interpretation.

Tofu Price

Soybeans Consumption(as feed grains)

Fish Catches
El Niño

Figure 1: An arrow diagram depicting the influence of theEl Niño Effect on the rise of the tofu price.
The appearance of an arrow symbol is subject to avariety of visual variables, such as length, width, shape,
color, direction, orientation, and pattern (Bertin 1983). The
arrow symbol alone, however, does not determine any
specific meanings (Figure 2). The meaning of an arrow iscomposed when the arrow symbol refers to othersurrounding elements. Thus, this paper focuses on thesemantics associated with the surrounding elements, whilethe visual variables are kept invariant.

Mr. K

Figure 2: Variations of the visual variables of an arrowsymbol (Horn 1998).
The combination of an arrow symbol and thesurrounding elements to which the arrow symbol refers isconsidered a unit of meaning, and called an arrow diagram.
Then, these surrounding elements are called thecomponents of the arrow diagram. The interpretation of anarrow diagram is not a simple symbol recognition processof individual components, but requires the consideration of
the possible role and behavior of each component in thecombination of the components, which are aligned in acertain meaningful way.
The remainder of this paper is structured as follows:First, we introduce the slots of arrow diagrams that capturethe alignments of components, and observe how thosealignments influence the interpretations of the arrow
diagram. Then, we classify the components into five types,and demonstrate their influence on the interpretations.
Based on these two settings, we distinguish arrow
diagrams structurally. In addition, we demonstrate that theorientation of the components may provide a critical clue
for determining a unique interpretation. Finally,
conclusions and future works are shown.

5:50pm
5:30pmFigure 4: Three slots of an arrow diagram populated withcomponents.
It is necessary to distinguish among these component
slots, because the meaning of an arrow diagram changes if
its components are placed in different slots. For example,
Figures 5a and 5b show two arrow diagrams in which thetail and the head components have been exchanged,
essentially reversing the meaning of the diagram frommounting a wheel to a car to removing a wheel from a car.
Figures 5c and 5d show another pair of arrow diagrams
where an icon of Maine State has been moved from thehead slot to the body slot, such that the meaning changes
from a traveler heads for Maine to a traveler passes
through Maine.

(a)

(b)

(c)

(d)

Three Slots of an Arrow Diagram
An arrow symbol is a deictic reference frame, identifying
three different areas that contain the components of thearrow diagram. These three areas are referred to as the tailslot, the head slot, and the body slot (Figure 3).

Tail slot

Body slot

Head slot

Figure 3: Three conceptual slots associated with an arrowsymbol.
Each component of an arrow diagram is uniquely
assigned to one of these three slots, thereby making thedistinction of tail components, body components, and headcomponents. Each slot may contain zero, one, or morecomponents (Figure 4). These three slots are conceptual
rather than physical areas, although we regard that therespective components are spatially contained in these slots.

Figure 5: Two pairs of arrow diagrams, each with the same
components in different slots.
The importance of the three component slots leads tothe first postulate of parsing arrow diagrams:• The assignment of the components to the three slots (tail,body, and head slot) of an arrow diagram captures
critical information about the meaning of the arrowdiagram.

Five Types of Components
A component of an arrow diagram may be mentioned by
an icon, a text, or a specific position in the backgrounddrawing. In semantic level, however, the following fivedifferent types of components are distinguished:• An object takes an action. A person, a car, and a houseare examples of objects (A house seems inactive butpeople may regard that a house takes such an action as
sheltering a person).
• An event occurs in time, and characterized by a set ofchanges. An event occurs at an instance (e.g., a trafficaccident) or over an interval (e.g., a conference).

• A location is a point or a homogeneous region in space.
Examples of locations are a mountain’s peak (a point)
and a mountain (a region).
• A moment is an instant or a homogeneous interval intime. Examples of moments are a departure time of aflight (an instant), or hours on an airplane (an interval).
• A note is a description that supplements anothercomponent, usually in the form of text. Some notes arelocated close to the corresponding components, whileothers are connected to the corresponding components
by a line or an arrow.
The influence of these component types on theinterpretation of an arrow diagram is highlighted in thefollowing examples. Figures 6a-c show three
configurations in which an object (a traveler) fills the tailslot, whereas the head slot is filled with an object (a bag),an event (AAAI spring symposium), and a location(Maine), respectively. These different types of headcomponents lead to different interpretations of the arrow
diagrams: the traveler leaves his bag (Figure 6a), thetraveler participates in the AAAI spring symposium(Figure 6b), and the traveler heads for Maine (Figure 6c).
On the other hand, arrow diagrams with identicalalignments of component types often lead to a same classof interpretation. For example, Figures 6c and 6d, bothshowing the configurations in which an object fills the tailslot and a location fills the head slot, illustrate themovement of the object heading for the location.
AAAI
SpringSymposium(a)(b)(c)(d)Among 216 types of simple arrow diagrams, some lead
automatically to a unique class of interpretation. For
example, (O, –, L) is always interpreted as a movement of
the object heading to the location (Figure 6c and 6d). On
the other hand, some arrow diagrams follow more than one
class of possible interpretations. An example is (O, –, O),
which is discussed in the next section.
The distinction that arises from the component types
leads to the second postulate of parsing arrow diagrams:• The
distinctionof
thecomponenttypesc ∈ {M , E,L,O,N ,−} is essential for parsing themeaning of an arrow diagram.

Intrinsic Orientation of Objects
If an arrow diagram has more than one object, its
interpretation depends on which object is supposed tomove. For example, Figure 7a has two possible
interpretations: a car approaches a person and a person
leaves from a car. In such a case, an important aspect for a
successful interpretation is the object’s orientation withrespect to the arrow symbol. Since the arrow symbol is theframework for the diagram’s deictic reference system, theparticipating objects are evaluated with respect to thearrow symbol’s orientation. An object expressed by anicon often has an intrinsic orientation toward which theobject usually moves. For example, in Figure 7b, theintrinsic orientation of a car is identical to the arrowsymbol’s orientation (i.e., both point to the right), whereasthat of a traveler is opposite to the arrow symbol’sorientation. Such intrinsic orientations of objects are often
critical to determine a unique interpretation. For example,in Figure 7b, since only the intrinsic orientation of the car
is identical to the arrow symbol’s orientation, only the caris supposed to move. Thus, although Figures 7a and 7b
refer to the same set of objects, only Figure 7b is uniquelyinterpreted as a car approaches to a person.

Figure 6: Arrow diagrams with various types of headcomponents: (a) object, (b) event, (c) location,and (d) location.
With five types of components and three slots, 53 typesof arrow diagrams with exactly one component in each slot
could be distinguished. Since arrow diagrams may have
empty slots as well, a sixth component type, the empty
component, is introduced. Together with the five nonempty components, a total of 63 = 216 types arrow
diagrams are distinguished. They are called simple arrow
diagrams, since they are related to exactly one arrowsymbol and each slot has at most one component. Withc ∈ {M , E,L,O,N ,−} , referring to moment, event, location,
object, note, and empty component, respectively, a simple
arrow diagram will be denoted symbolically as (c, c, c).

Figure

(a)

(b)

(c)

(d)

7: Arrow diagrams with different object
orientations: (a) (+, ,+), (b) (+, ,–), (c) (–, ,–),and (d) (*, ,0).

If the orientation of an object is same as the arrowsymbol’s orientation, the object is called positivelyoriented (+). Conversely, if the orientation of an object isdifferent from that of the arrow symbol, the object is called
negatively oriented (–). An object that does not move, likea car without a wheel in Figure 7d, is called static (0).

Conversely, an object that may move in any direction, likea wheel in Figure 7d, is called neutral (*). If an illustrated
scenario premises an object to move, this object must beeither + or *. Therefore, if an arrow diagram refers to only
one + or * object, the valid interpretation is uniquelydetermined (Figures 7b and 7d). If all objects are – or 0,then the arrow diagram may illustrate no movement or
imply a movement of something that is not explicitly
drawn in the diagram. For example, Figure 7c may
illustrate a static relation between a car and a traveler or a
scenario that a person brings out something from a car.
The above discussion leads to the third postulate ofparsing arrow diagrams:• The intrinsic orientation of an object is criticalinformation to determine a unique interpretation of a
scenario that accompanies a movement.

Conclusions
To understand the meaning of every arrow in a diagram isoften a fundamental first step for the correct understandingof the whole process or mechanism illustrated in thediagram. This paper identified three structural properties ofarrow diagrams, which are available as the postulates ofparsing arrow diagrams. Based on these postulates, we are
now developing a formal method for interpreting simple
arrow diagrams. In this method, the candidates of theinterpretation are derived from the distinction of simple
arrow diagrams, which was introduced in this paper. Then,the valid interpretation is determined from these candidates
with the aid of various clues, which includes the objectorientations. Since arrows are major components ofdiagrams, this method is expected to enhance the usabilityof pen-based systems such that their users can freely
represent various semantics with arrows.

Acknowledgements
This work is supported by the National GeospatialIntelligence Agency under grant number NMA201-01-12003. Yohei Kurata is further supported by a University ofMaine International Tuition Scholarship. Max Egenhofer’sresearch is further supported by the National Science
Foundation under grant numbers EPS-9983432 and IIS9970123; the National Geospatial-Intelligence Agency
under grant numbers NMA201-00-1-2009, and NMA40102-1-2009, and the National Institute of Environmental
Health Sciences, NIH, under grant number 1 R 01
ES09816-01.

References
Alvardo, C. J. and Davis, R. 2001. Resolving Ambiguities
to Create a Natural Sketch Based Interface. In Proceedingsof the 17th International Joint Conference on Artificial
Intelligence (IJCAI-01), 1365-1374. Seattle, Wash.

Bertin, J. 1983. Semiology of Graphics: Diagrams,Networks, Maps. Madison, Wis.: University of Wisconsin
PressForbus, K. D. and Usher, J. M. 2002. Sketching forKnowledge Capture: A Progress Report. In Proceedings ofthe 7th International Conference on Intelligent User
Interfaces, 71-77. San Francisco, Calif: ACM Press.
Horn, R. E. 1998. Visual Language: Global
Communication for the 21st Century. Bainbridge Island,Wash: MacroVu, Inc.
Kurtoglu, T., and Stahovich, T. F. 2002. Interpreting
Schematic Sketches Using Physical Reasoning. In AAAI
Technical Report SS-02-08 (Proceedings of AAAI Spring
Symposium, Sketch Understanding, 2002), 78-85. Melon
Park, Calif.: AAAI Press.
Landay, J. A. and Myers, B. A. 2001. Sketching Interfaces:
Toward More Human Interface Design. Computer 34(3):56-64.
Tversky, B.; Zacks, J.; Lee, P.; and Heiser, J. 2000. Lines,Blobs,Crosses
andArrows:
DiagrammaticCommunication with Schematic Figures. In Anderson, M.,
Cheng, P., and Haarslev, V. eds. Theory and Applicationof Diagrams: 221-230. Berlin: Springer.
Wildbur, P. and Burke, M. 1998. Information Graphics:
Innovative Solutions in Contemporary Design, London:Thames & Hudson.