# Presentation: 664759

## Modeling the spatial relations among objects

- Laura Carlson
- University of Notre Dame

**Notes:**
Consider the following scenerio:

## Where is the phone booth?  (Geosims Virtual Philadelphia)

**Where is the phone booth?**  (Geosims Virtual Philadelphia)

**Notes:**
You are visiting a new city, and you need to make a phone call. You stop a doorman in a highrise complex, and ask him to indicate the nearest phone booth. He walks you to the sidewalk, points down the street, and tells you where it is.  The critical question of interest is what he says. 
He could say many different things, and he needs to pick the one that is the most helpful and informative to you, following Gricean maxims of conversation
The question is, how does he determine what the best spatial description would be?

## Where is the phone booth?  (Geosims Virtual Philadelphia)

**Where is the phone booth?**  (Geosims Virtual Philadelphia)

- The phone booth is south of the Liberty Bell.
- 2.   The phone booth is on your left, by the newspaper box.
- 3.   The phone booth is in front of the State Street Building.
- 4.   The phone booth is to the left of the Food Market.

**Notes:**
Each of these descriptions contains two critical components:
A reference object – an object from which to specify the location of the phone booth
A spatial term – a means of indicating the spatial relation between the reference object and the target (phone booth).

The first description uses a famous landmark (the Liberty Bell) as the reference object, based on the speaker’s assumption that as a tourist you will know its location. Indeed, Isaacs & Clark (1987) showed that speakers assess the expertise and familiarity of an addressee with a particular environment, and adjust their descriptions accordingly.  The spatial term in this description uses an allocentric/environmental frame with cardinal directions

The second utterance uses you (the addressee) as the reference object, and the spatial term uses an egocentric frame based on your movement through the space. It also uses the newspaper box which is salient in the speaker’s mental representation of the environment, given that he purchased a newspaper from it earlier that morning.

The third utterance uses the State Street Building as the reference object, and and uses an intrinsic frame with specification of an object-object relation. This object is closest to the phone; however, it is unmarked and thus the speaker may not know which building is being referred to.

Finally, the fourth utterance uses a marked building (the Food Market) as a reference object, and a spatial term that is potentially ambiguous – left with respect to the deli or left with respect to one’s perspective walking past the Food Market.

## Components of a spatial description

- To formulate a spatial description, speaker must select:
- Target
- 	For this type of relation judgment task (“Where is the phone booth?”), the target is specified in advance
- 		(Logan & Sadler, 1996).
- Reference Object
- On what basis?
- Spatial term
- On what basis?

## Goals:

*1. Define Reference Object Selection*
*	*Provide theoretical and computational bases* *
*2. Define Spatial term Selection*
*	*Provide theoretical and computational bases
*3. Examine the relationship between these processes, testing among three models*
*	*Provide empirical data that tests among these models.

## Selecting a Reference Object:Where is the calculator?

- Speaker picks a salient reference object that is known or can easily be found by addressee (Clark, 1996; Sperber & Wilson, 1995; Talmy, 1983).

**Notes:**
Consider now a new scene, consisting of a desk with numerous objects, that we will use in our empirical work.
The target is the calculator. With respect to reference object selection, all of the objects on the desk including the book, the ruler, the stapler, the laptop, the bookshelf, are viable reference objects.
The assumption underlying reference object selection is that the speaker picks a salient reference object that is known or can easily be found by the addressee. 
Note that salience for the listener may or may not be the same as what is salient within the speaker’s mental representation of the environment.

## Theoretical basis

- What defines *salience*?
- Perceptual attributes such as size, permanence of location, geometric complexity (Talmy, 1983)
- Conceptual attributes such as an object recently mentioned, of shared importance, animacy (de Vega, Rodrigo, Ato & Dehn, 2001; Sperber & Wilson, 1995; Talmy, 1983)
- Semantic features such as association with target (Carlson-Radvansky & Tang, 2000)
- Research in referential communication similarly suggests salient features that distinguish an object from its neighbors are selected for expression (Clark, 1996; Deutsch & Pechmann, 1982; Mangold & Pobel, 1988; Olson, 1970).

**Notes:**
Theoretically, this assumption raises the important question as to what defines salience. There are many possibilities:

In our empirical work, we will examine perceptual salience and conceptual salience.

## Computational basis

- _Logan (1996) Code Theory of Visual Attention (CTVA)_
- Objects compete to be classified as members of a set of response categories.
- Attentional selection = an object in display assigned to a category in the response set.
- Attention = choice
- Object and its categorization are chosen simultaneously, with different categorizations racing against each other for selection.

**Notes:**
Computationally, the idea that objects can vary in salience with respect to numerous dimensions has been captured in Logan’s 1996 Code theory of visual attention.  Some principles of this model are as follows:

- Parameters affecting rate at which a categorization races:
-  1)  (for evidence) = similarity between the object in the display and the representation of the response category, Determined by stimulus properties (bottom up)
-  2)  (for bias) = bias toward selecting objects in a particular response category. Determined by top-down factors.
- 3)   (for priority) = importance of selecting objects that contain one of the properties in the stimulus set; impacts amount of attention (w) allocated to a particular object:

- Logan (1996): CTVA

## Logan (1996): CTVA

- The probability of selecting a given object as a particular type of categorization is influenced by the evidence, bias and priority for categorizing that object into other categories, and the evidence and bias for categorizing other objects into other categories, as in:

## CHOOSE “book” as reference object:

Bias () to categorize “book” objects 

Priority for book features (i.e., color; shape; size) 

Evidence () for “book” greater than the evidence for the presence of other objects (i.e., “stapler”)

- Bias () to categorize “book” objects
- Priority for book features (i.e., color; shape; size)
- Evidence () for “book” greater than the evidence for the presence of other objects (i.e., “stapler”)

- “Calculator is behind the book”

- CTVA applied to reference object selection

- _More generally:_
- Set of response categories within CTVA =
-  Dimensions that define salience.
- Determine bias and priority associated with each dimension
- 		 Empirical work

## Selecting a Spatial Term

- Speaker prefer  “easy to describe/understand” spatial relations between targets and reference objects (e.g., Franklin, Henkel & Zangas, 1995; Hayward & Tarr, 1995; Mainwaring, et al., 2002; Schober, 1993)

**Notes:**
With respect to spatial term selection, again there are numerous possibilities:
Behind relation, front of relation, right relation, left relation, back (environmental) relation

Selection is based on the assumption that speakers prefer easy to describe/understand spatial relations

## Theoretical basis

- What defines “*easy to define” *?
- For any given candidate reference object, spatial terms parse space around the object into regions, with some regions considered prototypical or canonical (Logan & Sadler, 1996; Hayward & Tarr, 1995; Herskovits, 1986).

- Objects falling in the
- prototypical regions can be
- described by simple spatial terms;
- Objects falling outside the
- prototypical regions need to be
- hedged or modified
- (Franklin, Henkel,& Zangas, 1995;
- Hayward & Tarr, 1995; Lakoff, 1972)

- “good”
- above

- “acceptable”
- above

- “bad”
- above

- ratings

**Notes:**
Theoretically, this raises the important question: what is easy to define?

## Computational basis

- _Regier & Carlson (2001) Attention Vector-Sum Model_
- Computes the acceptability of a given spatial relation between a given reference object and target
- 		Two components:
- 			1. Attention
- 			2. Vector Sum
- As applied to Spatial Term Selection, one could run AVS on reference object/target pairs for a range of spatial terms; select term corresponding to a given threshold.
- 		Specify the criteria for establishing thresholds

**Notes:**
Computationally, spatial term selection can be computed using the attention vector sum model.

## Computational basis

## Three possible models of selection

- What is relationship between reference object selection and spatial term selection?
- _Three possible models:_
- 1. Reference Object First Selection
- 2. Spatial Term First Selection
- 3. Joint Object-Term Selection

## Reference Object First Selection

- Speaker picks the salient reference object first, thereby restricting the selection of a spatial term to those that convey the relation between the target and the reference object.

*The calculator is behind and*
*to the right of the book.** *

## Spatial Term First Selection

- Speaker picks an “easy to define” spatial relation first, thereby restricting the selection of a reference object to those that stand in that relation to the target.

- Select
- reference
- object(CTVA)

- Select
- spatial
- term
- (AVS)

*2. The calculator is behind *
*the stapler.*

## Joint Object/Term Selection

- Speaker evaluates object/term pairs to select the most salient object in the most easily described location.

- Select
- reference
- object(CTVA)

- Select
- spatial
- term
- (AVS)

*3. The calculator is somewhat in front of the bookshelf.*
*4. The calculator is somewhat to the left of the whiteout.*

## Empirically testing among the three possible models

- Elicit spatial descriptions in response to queries “Where is the target?”
- Manipulate salience of candidate reference objects, and their spatial relations to target
- Test across variety of dialog contexts and response modalities (written, oral, memory)
- _Critical data_:
- 	WHICH REFERENCE OBJECT IS MENTIONED FIRST?
- Other measures (converge):
- # of reference objects mentioned
- # of spatial terms used

## Study 1: Hill and Carlson (in prep.)

- Target

- Where is the mustard?

- 8 object sets

- N = 24; each provides 128 written descriptions

- Ref Obj-1st: Hamburger
- Term-1st: Pesticide
- Joint:   Mix

- Ref Obj-1st: Hamburger
- Term-1st: Pesticide
- Joint:   Mix

- Ref Obj-1st: Hamburger
- Term-1st:    Hamburger
- Joint:          Hamburger

## Study 1: Hill and Carlson (in prep.)

- Hamburger  		             .39
- Pesticide		             .44
- Environment/Other  		.17
*	**	**Supports Joint*

- Hamburger    	 		.03
- Pesticide 	 		.89
- Environment/Other	 	.08
- 		*Supports Term/Joint*

- Where is the mustard?

## Study 2: Carlson, Hill, Watson, & Ashley (in prep.)

*“**Mary needs to borrow Aaron’s calculator. Tell Mary where it is:”*

- Book is salient reference object (perceptually)

- Figure 3

- N = 64 written; N = 36 oral; each participant gives one description

## Study 2: Carlson, Hill, Watson, & Ashley (in prep.)

*“**Mary needs to borrow Aaron’s calculator. Tell Mary where it is:”*

- Figure 3

## Study 3: Carlson, Hill, Watson, & Ashley (in prep.)

*“**Can you tell me where I left my keys?*

- Notebook is salient (perceptually); manipulate conceptual salience – context
- N = 114; each = 1 oral response and then written description from memory

## Study 3: Carlson, Hill, Watson, & Ashley (in prep.)

## Implications of empirical results

- Across 3 studies with diverse dialog contexts/modalities:
- Support for Term 1st when there is a good spatial relation that is preferred (above vs. left)
- Support for Joint Object/Term consideration when acceptable spatial relation or non-preferred term
- Salience may be perceptually driven
- No initial evidence for conceptual salience
- With respect to interaction between objects (mustard/hamburger)
-  With respect to setting (notebook/stapler manipulations)

## Next empirical steps:

**Identify perceptual dimensions (size, color, position)**
**Further test conceptual dimensions**
**Further contrast dialog and memory responses**
**Many dialog = deictic: pointing; “here”; or they simply picked up the keys and jangled them**
**But memory included reference objects**
**	**** ****(Bangarter, 2004: pointing replaces specifying location)**
**4. Test memory representation without language**
**	****	****Forced choice RT paradigm**
**5. Examine self as reference object**
**	****	****egocentric vs. intrinsic vs. allocentric**

## Next modeling steps: Implement joint object/term model

- Select
- reference
- object(CTVA)

- Select
- spatial
- term
- (AVS)

**Start with spatial term selection (iterate through terms for candidate objects, etc).**
**If find a “good” relation, select corresponding reference object with CTVA**
**If less preferred relation (like “left”) or only  an “acceptable”, pick among object/term pairs based on perceptual salience**
_**CHALLENGES**_**:**
**Specify sequence of terms for iteration**
**	****Term and frame (egocentric/allocentric) preferences**
**Specify threshold for selection**
**Specify biases for object dimensions**