LMSC-D246461
Part II

Part IIDESCRIBING AND ABSTRACTING
PICTORIAL DATALOCKHEED PALO ALTO RESEARCH LABORATORYLMSOD246461
P a r t II

Section 1
INTRODUCTION

To index and a b s t r a c t pictorial data, much of the information concerning m e t r i c
information must be discarded and only the objects, their p r o p e r t i e s , their r e l a t i o n ships to other objects, and their locations a r e retained.

Then, by using only a limited

number of object n a m e s , modifiers, and r e l a t i o n s , a wide c l a s s of pictures can ber e p r e s e n t e d for future r e t r i e v a l in response to g e n e r a l q u e r i e s .

Thus, one important

purpose of picture description i s that of generalization, and in our investigation ther e t r i e v a l p r o c e s s consists of trying to match the query s t r u c t u r e to general description
structures.
It is possible to approach the study of pictorial data in either a formal or a nonformalmanner.

That i s , one can attempt to p r e s e n t the descriptions in a formal, rule-bound,and analytic manner, o r one can approach the problem in an analogical, intuitive
manner.

Much r e c e n t work h a s tended to be formal in nature (Refs. 1, 2) and h a s

produced disappointingly few r e s u l t s relevant to our i n t e r e s t s .

In our study, we havetaken a nonformal approach, studying pictorial description from the point of view oflanguage in u s e .
Over the p a s t s e v e r a l y e a r s , we have examined the variety of ways in which humano b s e r v e r s d e s c r i b e and a b s t r a c t pictorial s t r u c t u r e s .

After a review of the field(Ref. 3), we c a r r i e d out experiments in which descriptions of pictorial data p r e p a r e dby human o b s e r v e r s were examined.

These e x p e r i m e n t s , r e p o r t e d in Ref. 4, gave

u s a data b a s e of descriptive m a t e r i a l and allowed u s to examine the terminology ands t r u c t u r e of such d e s c r i p t i o n s .

In the first p a r t of the p r e s e n t contract, we examined

the problem of description of e a r t h r e s o u r c e s imagery* and indicated in Ref. 5 how a"dynamic 11 data entry s y s t e m , i . e . , one which allows for the entry of m o r e information
concerning a given picture obtained from later o b s e r v e r s , could be used to obtain an*See Table 1-1 for typical descriptions of the various e a r t h r e s o u r c e s disciplines.
1-1
LOCKHEED PALO ALTO RESEARCH LABORATORYL O C K H E E D
A

GROUP

M I S S I L E S

D I V I S I O N

OF&S P A C E

C O M P A N Y

LOCKHEEDAIRCRAFTCORPORATIONLMSC-D246461
Part II

Table 1-1
TYPICAL DESCRIPTIONS FOR THE VARIOUS EARTHRESOURCES DISCIPLINES
Geography
The Nile River and its distributaries within the delta appear as sinuous buff-colored ribbons.
The main stream l i e s against the east (right) side of the valley to the south. The Rosetta
Distributary is seen winding along the west margin of the delta, while the Damietta Distributary roughly bisects the irrigated area. Meanders and islands are everywhere in evidence.
The major canal system appears as a s e r i e s of straight buff-colored lines radiating outward
from the settlement nodes. Such straightline segments are nearly always canals rather
than roads.
Agriculture
Infrared photo taken from an airplane at Weslaco,Texas, shows crop vigor, growth, and soil
salinity: (1) healthy cotton, (2) unhealthy cotton, (3) bare soil, (4) pig weeds in wet area,(5) pig weeds above short sorghum, (6) dry topsoil between rows of sorghum, and (7) baresoil between sorghum in moist a r e a s .
Forestry
Figure 13 i s an Ektachrome infrared picture taken by the Apollo 9 astronauts of the test sitenear Vicksburgh, M i s s i s s i p p i . . . . The dark blotches within this area are uncut patches of hardwood, and a large area of hardwood forest can be seen toward the lower right corner of thephoto. Although the seasonal state of deciduous forest affects the appearance of such areas,
little difficulty is encountered in distinguishing between cultivated land and forests in spacephotography.
Geology(A Gemini 4 photograph of southwest Saudi Arabia.) Most of the upper half of the photographis occupied by a range of rugged mountains composed of Igneous rocks. Toward the base is an
expanse of sedimentary rocks that dip very gently toward the upper right of the photograph.
The distinction between igneous and sedimentary rocks is made by noting the presence of stratification, or layering, in sedimentary rocks. These two units appear to be separated by amajor fault.
Hydrology
On the far side of the Tigris River shown in Fig. 22 is one end of a large lake that, judging
from the pattern of dark tones around it, varies in size and is now at a low stage. Between thelake and the river are distributaries — subsidiary streams that carry water away from the mainstream and which branch downstream. Such streams deposit large amounts of fine-grained
sediment.
Oceanography(A Gemini photo of Campeche, Yucatan.) Some of the prominent features here are longshore
and offshore currents and a sandbar development. An easterly longshore current i s madevisible by the sediments discharged into the Gulf of Mexico's clear water by a river hnmen lately
to the left of the photo margin. The sediments are light-colored marls, silts, and sands. To
the right of the laguna, the sandbar development is clearly visible.
Cartography
The valley of the Rio Grand cuts a c r o s s the picture from the upper left to the bottom center.
The Franklin and Organ Mountains are near the center. Below them are El Paso, Texas, andJaurez, Mexico. These cities are in a strategic location in a pass in the range of mountains.
There are irrigated areas on both sides of the international borderline. The Rio Grande Valleyi s slightly depressed and our highways pass through El Paso.

1-2
LOCKHEED PALO ALTO RESEARCH LABORATORYLMSC-D246461
Part II"encyclopedic" data base. The use of a relational net for representing the content
of picture description was described, and methods for merging and searching such
structures were indicated. A summary of this paper is given in Appendix A. The
second portion of this contract period was devoted to the study of representations of
the content of descriptions of terrain photography written in natural language. This
report is devoted to that problem.
There are three basic problem areas in representing the content of a description
expressed in natural language:(1) The problem of finding canonical forms, i . e . , how to represent paraphrasesof the same concept(2) How to translate from natural language to this representation automatically(2) How to organize the data base so as to keep the representations simple, andstill capture the full conceptual intent of the description• The representation problem. Given a set of sentences such a s :Many wharves line the river.
On the sides of the river are a lot of wharves.
Along the river are a number of wharves.
There are many variations of these sentences using "docks, M "river, " and "many"in place of "wharves, " "stream, " and "lot of, " respectively, and it is important thatthese variations be given the same conceptual representation. The question of selectionof the primitive terms used in the conceptual representation arises, e . g . , should "sideof" be considered a primitive, or should it be represented as "on the outside edge ofan object, "using the primitives "on, " "outside, " and "edge. " Finally, the conceptual
representation must indicate the semantic role played by the primitives.
• The translation problem. The problem of translating from natural language
expressions to a conceptual representation is very difficult.

Because there are often

many allowable parsings of a given sentence if only syntactic criteria are used, it is

1-3
LOCKHEED PALO ALTO RESEARCH LABORATORYL O C K H E E D
A

GROUP

M I S S I L E S

D I V I S I O N

OF&S P A C E

C O M P A N Y

LOCKHEEDA I K R A F T

CORPORATIONLMSC-D246461
Part II

necessary to use additional aids. As indicated in Part I, section 2 . 1 , one possible toolthat can be used is the word government table, developed earlier in this study by
Robison (Ref. 6), a dictionary of words and allowable adjacent syntactic structures
for each word meaning. These associated syntactic structures pinpoint a given word's
semantic meaning, and can therefore be used as an aid in deriving the conceptual
representation of the sentence.
• Data base organization. If we have the description, "an airport is to theleft of the freeway, " and represent this as left of (freeway, airport), then a query suchas n what is to the right of the airport?" cannot be answered unless the data base has
stored the property of the relationship, "left of (a,b) implies right of (b, a). " Similarly,
the system must contain axioms which indicate that if an object is inside another object,then it cannot also be outside that object, and that if an object is far from an object,then it is outside that object.

If such properties of relations are not included in thesystem as general rules, then either the representation of each description becomes
encumbered with a large number of relationships, or, if such relationships are not present
in the representation, then the ability to respond to queries is diminished.

1-4
LOCKHEED PALO ALTO RESEARCH LABORATORYLMSOD246461
Part II

Section 2
REPRESENTATION OF MEANING

A parsing of a sentence indicates how the words interact, and which words or groups
of words are related on various levels of aggregation.

However, to represent thecontent or meaning of a sentence, one must indicate the conceptual roles played by
the words or word groupings.

For example, a representation of meaning should

indicate whether a term relates to another so as to indicate quantity (Mthree lakes M ),location (Mat the river t f ), or attribute of an object ("waterway has docks 1 '), etc. It istherefore important in any investigation ot representation of meaning to identify thesec l a s s e s , to study natural language expressions representative of their use, and to
determine effective techniques for representing content as a function of these c l a s s e s .
2.1 CONCEPTUAL CLASSES FOR PICTURE DESCRIPTION
Consider the description of an aerial photograph given in Table 2 - 1 . In the table below
the description, we have indicated the n concept kernels 11 — phrases which capture abasic idea or concept.

On the right, we have noted the general concept class (orc l a s s e s ) of each kernel. If we represent the meaning of the description by a networkin which terms are shown as nodes and the relationships by links, then we obtain adiagram such as is shown in Fig. 2 - 1 . In that figure, the concept c l a s s e s have beenused to label the links with the conceptual role played by that link.
Such conceptual c l a s s e s serve several r o l e s .

First of all, they help to organize thestudy of terms and expressions used in description, since we can examine terms andexpressions representative of each c l a s s , rather than having to consider all terms
and expressions.

Second, they can be used in the conceptual net to indicate thesemantic role played by relationships and thus simplify the search and analysis
tasks when extracting information from the net.

2-1
LOCKHEED PALO ALTO RESEARCH LABORATORYL O C K H E E D
A

GROUP

M I S S I L E S

D I V I S I O N

OF&S P A C E

C O M P A N Y

LOCKHEEDAIRCRAFTCORPORATIONLMSOD246461
Part II

Table 2-1
BASIC CONCEPT CLASSES OF A DESCRIPTION
Description by human observer
It is an aerial photograph, whose most arresting feature is a waterway, possibly an
estuary or part of a harbor, with many ship docks. The waterway stretches across theupper part of the photograph with a single stream on the left breaking into three arms
at not quite midphoto. The upper arm divides again into inlets cut off from view; atthe top, the middle arm is a small, straight-sided, closed inlet or docking area in apeninsula or island; the lower arm passes out of the picture on the right, possibly
dividing again at the right edge. Across the waterway just to the left of the division
is a bridge.
Concept Kernels of the Description

Concept Classes

It is an aerial photographPhotograph as physical object

Most arresting feature is a waterway
Waterway is possibly an estuary or part of a harbor

Attribute of object
Classification of object

Waterway has many ship docks

Set membership (has)Waterway stretches across upper part of thephotographOperative (stretches across)

Waterway breaks into three arms
Break is at not quite midphoto
The upper arm divides again into inlets

Operative (partitions)
Location of object[Inlets are cut off from viewAt the top of the middle arm is a docking area or
inlet
Inlet/docking area is straight-sided and closed

Existence

Operative (partitions)
Existence location
Attribute of object
Location of object

Inlet/docking area is in a peninsula or island
The lower arm passes out of the picture on the right
The lower arm possibly divides again at the right side

Operative (partitions)j Across the waterway is a bridge

Operative (crosees)

J Bridge is located just to the left of the division of thewaterway

Location of object

Existence (viewability)

2-2
LOCKHEED PALO ALTO RESEARCH LABORATORYLMSC-D246461
Partn

§
U

o
CA

a
i
o

ss

s
1
CD

1
CM

bio

2-3
LOCKHEED PALO ALTO RESEARCH LABORATORYL O C K H E E D
A

GROUP

M I S S I L E S

D I V I S I O N

OF&S P A C E

C O M P A N Y

LOCKHEEDAIRCRAFTCORPORATIONLMSC-D246461
P a r t II

Finally, many v e r b s a r e associated with certain conceptual c l a s s e s and allowable
word forms, and such c l a s s e s lead to constraints on the s t r u c t u r e of the sentence.
These constraints can be analyzed and tabulated for use as a guide in performing
both the syntactic and semantic a n a l y s i s . Recently, s e v e r a l approaches to linguistic
analysis based on concept c l a s s e s or " c a s e s " have been presented by investigatorssuch as F i l l m o r e , Schank, and T e s l e r .

We have examined c a s e g r a m m a r s which

identify the underlying syntactic-semantic relationships, and give a brief s u m m a r yof this work in Appendix A.
In examining descriptions of a e r i a l photographs, we have chosen the following s e t ofconcept c l a s s e s for r e p r e s e n t i n g meaning of picture description.
•

1-8 Case r e l a t i o n s . Relations such as (1) location, (2) localizing,(3) qualifying(4) quantifying, (6) operative, (7) use, (8) existence, which a r e often strongly
related to the v e r b of the sentence.

•

9 Attributes.

Attributes of an object, such as color, texture, shape, s i z e .

Many of the attributes have a value, e . g . , "red, " "rough, " e t c .
•10-Comparison. Comparisons can be made on the values of attributes, e . g . ,"redder than, " "rougher than, " "bigger t h a n . "•11-Logical linking. Linking of concepts using logical relationships such as"and, " "or, " "not, " "if

•

then"

12-Set m e m b e r s h i p . Indication that an object is a subset or p a r t of anotherobject.

•13-Doubt, approximation, or uncertainty.

Indications of doubt, approximation,or uncertainty on the p a r t of the o b s e r v e r .
•14-Picture properties.

Expressions which deal with the physical p r o p e r t i e sof the picture, such as size, resolution, and picture quality.
Some of t h e s e conceptual c l a s s e s a r e described m o r e completely below.

2-4
LOCKHEED PALO ALTO RESEARCH LABORATORYLMSOD246461

Part n2 . 1 . 1 Case Relations
The first eight conceptual c l a s s e s grouped together under the term n case relations' 1
are listed in Table 2-2
each c a s e .

which shows the role played by each case and examples of

It is expected that the list will be modified as the analysis proceeds.

Because location tends to figure strongly in observer descriptions of pictures, weinvestigated expressions for location in some detail.

In Table 2-3 (a tabulation of

expressions used to denote location), expressions appearing in the same box haveroughly the same meaning, while those appearing in adjacent columns are either
antonymns, or express somewhat related ideas.

Similar concepts have been grouped

together, and the column on the left indicates the name of the grouping whenever a
suitable general name can be found.

The question of suitable primitives for these

expressions will be discussed in section 2 . 2 . 2 .
2 . 1 . 2 The Remaining Conceptual Classes
The remaining six conceptual c l a s s e s are described briefly below.
•

Attributes.

Various attributes that appear in pictorial description are indicatedin Table 2-4.

Each attribute has a value, e . g . , the value of "color" may be "red, "and often the values of the attributes can be compared, as described below.
•

Comparison.

Comparison of attribute values can be indicated in the conceptual

net as follows.

Suppose we wish to indicate, "object A is longer than object B . "We use the net,

91

(B)

ATTRIBUTE

ATTRIBUTELENGTHLENGTH

1

I^

Value (LENGTH) - ©

I— Value (LENGTH)

to indicate that the value of the LENGTH attribute of A is bigger than that ofB.
2-5
LOCKHEED PALO ALTO RESEARCH LABORATORYt O C K H E f O
A

OPOUP

M I S S H E S

D I V I S I O N

OF&S P A C E

C O M P A N YI O C K H E E D

AIUCPAFTC O P P O P A T I O NLMSC-D246461
Part II

Table 2-2
CASE RELATIONS USED IN PICTURE DESCRIPTION

Case

Role Played

Examples1.

Location/extent

Location of object with respect
to frame of picture, or withrespect to another object. Also
extent of object

X is near Y, X is westof Y (see Table 2-4)

2.

Localizing

Top of X, edge of X
middle of X

3.

Qualifying

Portions of objects (can bethought of as location withrespect to object itself)
Which of an object is meant

1 4.

Quantifying

How much or how many

Three arms, few, many

5.

Classificational

An entity in the photo is
identified as to class or name6.

Operative

An entity does something toanother entity (usually intrusion,
crossing, or cutting)7.

Use or application

Purpose of object

8.

Existence

An entity can be seen

X is a Y
X appears to be a Y
Y seems to be a Y
X is similar to a Y
X resembles a Y
X is classified as a Y
X is related to a Y
X crosses Y
X cuts Y
X disrupts Y
X intertwines Y
X invades Y
X divides Y
X stretches across Y
X developed for Y
X used by Y
There is an X
an X can be seen
an X is visible
an X can be foundan X is observed
an X will be noted
an X appears

The upper arm (todistinguish this armfrom other arms)

2-6
LOCKHEED PALO ALTO RESEARCH LABORATORYLMSC-D246461

Part nTable 2-3
EXPRESSIONS FOR LOCATION
Concept

Expressions Used

Distanceto

X i s far from Y
X i s at Y
X i s in the vicinity of Y X is away from Y
X is distant fromX i s by Y
X is with Y
X i s close to Y
X i s near Y
X i s toward Y
X i s in place Y

X i s (distance) from Y

X i s on Y
X i s above Y
X i s over Y

X i s under Y
X i s underneath Y
X i s below Y ;X i s beneath Y

X i s off Y

X i s in Y
X i s inside Y
X i s within Y

X i s out of Y
X i s outside of Y

X i s along Y

X i s here

X i s here

X i s everywhereX i s to the left of Y

X i s to the right of Y

to the side ofX i s East of etc.

X i s West of etc.

X i s in front of Y

X i s in back of Y
X i s behind Y

X i s between Y and Z
X i s around Y
X i s alone Y
X i s among Y
Relative
position

Orientation

X i s perpendicular to Y X i s parallel to Y
X i s a diagonal
X runs north(south....)X makes an angle of
Q with Y

2-7
LOCKHEED PALO ALTO RESEARCH LABORATORYL O C K H E E D
A

GROUP

M I S S I L E S

D I V I S I O N

OF&S P A C E

C O M P A N Y

LOCKHEEDA I K I A F T

CORPORATIONLMSC-D246461
Part II

Table 2-4
ATTRIBUTES OF OBJECTS
Values of Attributes

Attributes•

Comparatives of Values

Shapecurvedstraight
round

more curved, curvier,straighterroundersame shape as

Size
LengthAreabig, little
short, long, 5 in.
big, small, one acregreater than, smaller than,
equal

Color

1 red, blue

redder than, bluer than,same color as

Grey Level

light, darkbright, dull

lighter than, darker than,same grey level as,
brighter, duller

Texturegrainy, sandy, mottled
smoothsprinkled with
scattering ofclumps ofrougher than, grainier than,rough assmoother thansmooth as

Logical linking. Objects are often linked using logical relationships such as Mand, n"or, " Mnot, M "since x, then y, " etc. Such linking can be indicated using the standard
logical notation.

Some of the expressions used are shown below.
Natural Language Formand

Symbol Used
*

or

+

oil or water

not

t>j

does not crossLogic Expression

if X then Y

X —Y

roads and mountains

X shows that Y
X indicates that Y
X confirms that Y
X gives evidence that Y
judging from X, then Y
because of X, then Y

2-8
LOCKHEED PALO ALTO RESEARCH LABORATORYLMSC-D246461
Part II•

Set membership. Indications that an entity is a subset of another entity are given

by such natural language expressions a s :A is part of B
A i s an element of B
A belongs to B•

Doubt/uncertainty/approximation.

It is important that the links of the conceptual

net retain an indication of doubt, uncertainty, or approximation concerning observations, s i z e , and quantity.
mately,tf

Thus, a value of an attribute should be labeled as •'approxi-

and that of an observation u possibly, " when so indicated by the observer.

It may be desirable to use a scale of approximation or of possibility, but this should
offer no difficulty.
•

Picture properties. Because of the importance of such picture properties as clarity,

resolution, scale, e t c . , we have separated this topic area out as a conceptual c l a s s .
2.2

PICTURE PRIMITIVES

The terms appearing in the relational net should be selections from a basic set ofprimitives to enable matching to be performed between queries in canonical form andthe data base.

Primitives appropriate for the various conceptual c l a s s e s are described

below.
2 . 2 . 1 Attribute Primitives
Some of the attributes concerning size, quantity, and grey level (given previously in
Table 2-3) permit representation in terms of a small s e t of primitives.

For example,

if one defines BIG as a general primitive denoting a large value of an attribute, then

2-9
LOCKHEED PALO ALTO RESEARCH LABORATORYL O C K H E E DAO l O U f

M I S S I L E S

D I V I S I O N

OF&S P A C E

C O M P A N Y

l O C K H I f D

A I I C I A M

CORPORATIONLMSC-D246461
Pari II

we can define
LENGTH:QUANTITY:
few= ~ BIGshort= ~ BIGa lot, many

= BIG

long

= BIG

not long and not short= ~ (BIG* ~ BIG)

several, some - ~ (BIG* - B I G )SIZE:GREY LEVEL:

little, small» ~ BIGlight= ~ BIGlarge, big

= BIG

dark

- BIG

greyish, not dark andnot light= ~ (BIG* ~ BIG)

moderate sized = ~ (BIG*-BIG)

However, the primitives for shape, direction, color, and texture are not as straightforward. In the case of shape, for example, one could define as primitives a few
selected geometric shapes. Human description of shape, however, often uses thenames of objects to indicate shape, e . g . , "S-shaped, " "hook-shaped, " and it is notclear whether any set of primitives of reasonable size can be found to cover the
spectrum of such descriptions.
For color, one can use the primary colors, indicating the other colors in terms ofprimaries, e . g . , "orange" would be lf red/yellow, " and shades could be indicatedusing BIG. Thus, an orange that was on the reddish side could be indicated "BIG-red/yellow. M Texture is a difficult class to deal with in terms of a small set of primitives,and it is not clear as to what an acceptable set of such primitives might be.
2.2.2 Location Primitives,\ tentative set of 15 to 20 location primitives chosen from Table 2-3 is currently
being evaluated. The number of such location primitives can be reduced by

2-10
LOCKHEED PALO ALTO RESEARCH LABORATORYLMSC-D246461

Partn

defining a s e t of primitives based on the idea of an object being inside or outsidea suitably defined space o r region.

T e s l e r (Ref. 8) has used such a s e t of primitives,

as follows:space(Y)= the space defined by the boundaries of object Y

nearspace(Y)= the space near object Y, where "near 11 would dependon the particular problem a r e a

betweenspace(Y, Z)» the space between Y and Z

amongspace(Y, Z, . . . ) = tho space in a region of a s e t of objects
Each of the spaces or regions i s assumed to have an edge, a s does an object. One
then can define a s primitives "inside, " "outside, " and "on, " having their usual
meanings.

We can then define many of the location t e r m s using these primitives

as follows:
Expression

In T e r m s of Primitives

P a r a p h r a s e of the Notation

X is inside Y

inside(X, space(Y))

X i s inside the space of Y

X i s far from Y

outside(X, nearspace(Y))

X i s outside the nearspace of Y

X is along Y

on(X, edge (nearspace(Y))

X is on the edge of the n e a r s p a c eof Y

X i s between y and z

inside(X, betweenspace(Y, Z)) X i s inside the betweenspace ofY and Z

Localizing t e r m s such a s "top of," "bottom of, " "right side of, " which indicate portionsof an object a r e thought of a s defining a region.

Thus, the outside of "top" is "above, "while i t s inside i s " b e l o w . "2,2.3

Localizing P r i m i t i v e s

Localizing t e r m s such a s "top, " "bottom, " and " s i d e " will be expressed in t e r m s ofthe following p r i m i t i v e s .

2-11
LOCKHEED PALO ALTO RESEARCH LABORATORYL O C K H E E D
A

GROUP

M I S S I L E S

D I V I S I O N

OF&S P A C E

C O M P A N Y

LOCKHEEDAIRCRAFTCORPORATIONLMSC-D246461
Part II

Primitives

Examples of Use of Primitives

topbottom • ~ top; center = 1/2 (top)rightleft =~ right| front

back =~ front

edge

edge

J possible combining into one term
portion '2.2.4 Qualifying Primitives
Qualifying terms, such as Mthe upper arm, " are used in natural language expressions
to denote which of several objects is being referred to. In terms of the conceptualnet, they indicate the proper link connections, but do not appear as a high level
identifier of a link.
2.2.5 Classificational Primitives
The classificational primitives depend on the subject area or discipline for which thedescription is being prepared. A set of such primitives would therefore have to beprepared for each such discipline.
2.2.6 Operative Primitives
To deal with terms such as "cuts, " "crosses, " etc., our initial selection is one term
which indicates a crossing over, one term which indicates a cutting off, and one term
which indicates entry into, as shown below:
cuts across • crosses, spans, divides, partitions
cuts off= disrupts
cuts into» invades, enters

2-12
LOCKHEED PALO ALTO RESEARCH LABORATORYLMSOD246461
Part II

2.3 STRUCTURING THE DATA BASE
We have considered several strategies for structuring the data base s o as to allow
efficient search of the pictorial descriptions.
M

Two approaches are the use of themetanet, " and the use of relationship properties.

2 . 3 . 1 The Metanet
The metanet i s a high-level network which uses only conceptual c l a s s e s and not therelationships themselves to label links. In addition, only the T?upperM levels of theoriginal conceptual net are retained.

Search in response to a query i s first to bemade on the ordered s e t of metanets, each of which indicates the general theme or
structure of the actual conceptual net.
2 . 3 . 2 Relationship Properties
To keep the number of links appearing in a conceptual net to a minimum, it i s desirable
to make available to the retrieval system certain "world knowledge M concerning
relationship properties.

For example, instead of using the net* is to the right of -v©

©

M s to the left oi~S
it i s preferrable to use the net (a)— i s to the right of —(b), and to store the rule:
right of (a,b) = left of (b, a).
Additional properties of relationships are required because mismatches can occur
between query and data base even when relationships are reduced to a primitive form.
For example, "at" and "near" cannot be expressed by the same primitive, yetsomething which i s "at11 a location is also "near" that location, and therefore the

2-13
LOCKHEED PALO ALTO RESEARCH LABORATORYL O C K H E E D
A

GROUP

M I S S I L E S

D I V I S I O N

OF&S P A C E

C O M P A N Y

lOCKHEEOA I K R A F T

CORPORATIONLMSC-D24G461
Part H

property, at(m,n) implies near(m, n) should be part of the system.

An example ofproperties of relations concerned with location is given in Table 2-5, derived withthe aid of Ref. 9.
If an extensive body of world knowledge is to be provided to the system, some efficientmethod of searching this collection of facts must be provided.

That i s , the system

must somehow locate relationship properties that are relevant to a query without
performing an exhaustive search of the collection of its world knowledge.

Althoughit may be possible to develop a suitable organization and indexing scheme for thiscollection, the alternative of portraying this information using a two-dimensional
map s e e m s more attractive.

Using this approach, various concepts are defined

individually in this space, and the interrelationships are determined by performing"experiments" in the space.

For example, if the relationship, MX is between Y

and Z" is depicted in this space, (where Y and Z are parallel lines and X is a point),
then, as X approaches Z so that MX is nearer to Z, M the system can note
experimentally that MX is farther from Y . "We plan to explore the two-dimensional portrayal of two-dimensional relationships
during the next contract period.

2-14
LOCKHEED PALO ALTO RESEARCH LABORATORYLMSC-D246461
P a r t II

Table 2-5
PROPERTIES OF RELATIONS CONCERNED WITH LOCATIONS
Reciprocal relations
Near(m, n)
F a r from(m, n)
Beside(m, n)= Near(n, m)= F a r from(n, m)= Beside(n, m)

Opposite relationships= Under ( n, m)= Under( n , m )= Below ( n,m)Upon(m, n)On(m, n)Above (m, n)Mutually exclusive relationships
Inside(m, n)Near(m, n)
Below(m, n)
Below (m, n)
Above (m, n)At(m, n)Inside(m, n)

excludesexcludesexcludesexcludesexcludesexcludesexcludesOutside(m, n)
F a r from(m, n)
Above (m, n)
Beside(m, n)
Beside(m, n)
Between(k, m, n ) W
Between(k, m, n)

implies
implies
implies
implies
implies
implies

Outside(m, n)
Outside (m, n)Near(m, n)At(m, n)Above (m, n)
Below (m, n)

Implications
Near(m, n)
F a r from (m, n)At(m, n)Upon(m, n)Upon(m, n)
Under (m, n)( a ) i . e . , k cannot be between m and n.

2-15
LOCKHEED PALO ALTO RESEARCH LABORATORYL O C K H E E DAG I G U P

M I S S I L E S

D I V I S I O N

OF&S P A C E

C O M P A N Y

LOCKHEEDAIRCRAFTCORPORATIONLMSC-D246461

Partn

Section 3
NATURAL LANGUAGE ASPECTS OF CONCEPTUAL MAPPINGThe previous section dealt with problems of representation of meaning; this section
deals with problems in determination of meaning from natural language expressions.
3 . 1 SEMANTIC AMBIGUITY
To indicate the problems of ambiguity in picture description, let us examine threedescriptions from Table 1-1; often sentences that offer no problem to the humanare quite difficult for an automated process to handle. Consider the following sentence:(1) The Nile River and its tributaries within the delta appear a s sinuous buffcolored ribbons.
Note that "appear11 can be appear± - arc represented a s or appear^ = come to sight.
The program must, from a government table, identify the appear a s form a s indicating appear! , and the parsing program must then be able to identify the two subjects the Nile River, and i t s tributaries and that within the delta i s a prepositional
phrase modifying tributaries. If we use the primitive terms,ainside ofb are represented a scconsisting ofe i s visible incapproximately inthen the conceptual net for sentence 1 would be mapped a s ,Nile River ^s~\jancflare represented a s —*sinuous buff-colored ribbons

tributaries ^^^( a } inside of
W

1
delta

3-1
LOCKHEED PALO ALTO RESEARCH LABORATORYL O C K H E E D
AGIIOUPM I S S I L E S

D I V I S I O N

OF&S P A C E

C O M P A N Y

l O C K H f f O

A I K I A F T

CORPORATIONLMSC-D246461
Part II(2) Such straightline segments are nearly always canals rather than roads.
Here we are faced with the ambiguity of a r e . = identity, and are = represent.
The program would have to identify the are meaning and the rather than would haveto be identified as not to obtain the representation,straightline segments «— are represented as — canals
straightline segments — not (are represented as) — roads(3) The dark blotches within this area are uncut patches of hardwood, and alarge area of hardwood forest can be seen toward the lower right cornerof the photo.
The can be seen has the alternative meanings, can be seen. = is visible, and can beseen^ = are represented a s . The program would have to handle the ambiguity of botha r e , and can be seen to obtain the conceptual net,©
dark blotches *(a) inside ofare represented as

uncut patches(d) consisting ofarea

hardwood©
large area^

is visible in(d) consisting of'

1(c) approximately inlower right corneri(a) inside ofihardwood forest

photo
The importance of the government tables in resolving ambiguity can be seen from thisbrief examination of mapping from natural language to concepts. (It should also be
noted that not all ambiguities are resolved by such tables.) A word government table

3-2
LOCKHEED PALO ALTO RESEARCH LABORATORYLMSOD246461

Part nsuitable for pictorial description is being prepared by extracting and expanding upon
the entries from the Robison tables • An example of the entries for the letter a. as
extracted from the Robison tables is shown in Table 3 - 1 . The tables to be constructed
will be in the form described in Part I, section 2.2 (which includes word meanings),
and is expected to have l e s s than 1000 entries.

3 . 2 PARSING PICTORIAL DESCRIPTIONSThe PHRASE parsing program of Lois Earl was used to analyze four descriptions asfollows: (1) an earth resources geological description, (2) a description of an aerial
photography by a lay person, (3) description of a surgical scene by a surgeon, and(4) a botanical description.

The resultant parsing are shown in Fig. 3 - 1 . It will

be noted that there were nine sentences in the descriptions, and there were sevenphrase-constructions whose function was incorrectly labeled.

Two of these errors

were due to incorrect part-of-speech determination of technical words, and two were
due to incorrect interpretation of conjunctions.
These results were obtained using both Level I and Level II parsing, as defined in
Section 1.

The Level II parsing is necessary for the identification of word government

patterns in text, while for full semantic analysis, all four levels of parsing will beneeded.

During the next contract period, the Level III parser will be available forparsing these descriptions, and it will be interesting to see what improvement inparsing results.

3-3
LOCKHEED PALO ALTO RESEARCH LABORATORYL O C K H E E D
A

CROUP

M I S S I L E S

D I V I S I O N

OF&S P A C E

C O M P A N Y

LOCKHEEDAIRCRAFTCORPORATIONLMSC-D246461
Part II

Table 3-1
ENTRIES FOR LETTER "A" FROM GOVERNMENT TABLES PERTINENT
TO PICTURE DESCRIPTION

absence12nnof S/from S
of S/in S

vl2
aj

I*/from S
from S| accessnof S/to S/by S

accessibleaj

to S/by S

1
2

vt
vt

S/to S
S/with S

1
1
2
3

vt
vt
vi
vl

S/onto S
S/to Sto S
up to S

1 adjacent

aj

to S

i adjoinvt

S/to S| adjunctn

to S

1 advance

vt
vt
vt
vt
vt
vt
n
n
n
n
n
n

S/toS
S/toward S
S/into S
S/on S/from S
S/upon S
S/from S/to S/over S
of S/to S
of S/toward S
of S/lnto S
of S/on S/from S
of S/u)>on S
of S/to S/ovor S

vt
vt
vt

S/U) S
S/onto S
S/u|M>n S

1 aheadaj

of S/by S/in S

1 aim

vi
vi

for S
at S

vt

S/with S

absentnof S/wlth S

1 amountnof S/in S

angle

vt
vt
n
n

S/at S
S/toward S
of S/with S
between S/and S

anterior

aj

to S

apart

av

from S

appendvt
vt

S/to S
S/onto S

appendage

n
n

to S
of S

1
1
2

n
n
n

2

n

of S/into S
of S/to S
of S/between S/and S/
according to S
of S/among S/according to S

vi
n

to S
of S/to S

vi
vi
vi

from Sout of S
at S

arm

n

of S

arrivevi

at S/by Salignment|
I accompany
add

affix

align

apportionment

approach
arise

1
1
2

association

1
1

n
n

of S/wlth S
between S/and S

attachment

1
1
2
2

n
n
n
n

of S/to S/with S
of S/to S/by S
between S/and S
of S/for S

attributenof S

augmentationnnof S/with S
of S/by S

3-4
LOCKHEED PALO ALTO RESEARCH LABORATORYLMSC-D246461
Part II
Description No

1, KM i lit HCHOUII IH,•cologjcul DeHi Ilotion
I'HI'

NAP
Vlbll.l.

The muni oltvioUM leuture
NAI»

I,

VIII'

NAP

the s p a c e d all m r

the lar^e

NAI'

NAP

CJ

r~
drilling sand durua that completely cover tin Udrock.
PHP
I
CJ

PHP
NKPP
I
\ I
NAP NKP
NKP
NAP
NA-VH Vlll
I
\
I
\ /
\ I
\
v«ry
thin
wuati nl recent u>
quaternary alluvium

\
NAP

NAP

AV

—\
u

or bedrock sreaa

and bruMd \x dlim nU

PHI
NAP

NAP

CJ

VHP

NAP

AJ

InvelberKN and •mall ruck outcrops In the pedlnn ntn are

numerous.

Description No. 2, Description ol Aerial Holograph by I .ay Observer
PH P
NAP

VHP

The picture

lit

NAP

NAP

PV-PP

VH-AU PK AH

u color aerial photograph of u land area Invadedulion shaped

Ly

CJ
VBP
NAP
r~\ i
\ i
»
or three-pronged waterway
PHP
NAP

VBP

Wharves

line

NAP

PHP

NAP

NAP

AJ-AV

NAP

the sides ol Um waterway , a bridge, probably lor auto tralllePHP

AV I'll CJ

AJ-AV

iHjt

possibly

Inscription No

PHP
VHP

NAP

misseslor rail trallle

NAP

NAP

the handle ol the Irllon

J. Description ol a Surgical Scene by Medical Expert
PHP
/

\

PHI
I
NAP
NAP NAP~\ f~
1M roughly (uHltorin In shape; tla
VHP

NAP

NAP

N

An appendix.

6 2 centimeters

In length
PHP

NAP

r

VBP
NAP
\ r-\
i
\

midpoint

la

AV PH

r I I I -\

NAP
dime

I 6 cm.

PHP
NAP

NAP

VHP

NAP

PHP

At this point,

there

la

a tan excres< ence extending 1. 3 centimeters

above

PHP

PHP

NAP ~v

PHP
\

r

NAP

I
NAP VHP NAP

the level of the appendiceal wall that
PHP

In

PHP
\

I

1
NAP

NAP

tan

In contrast

to the blue-gray

PHP
NAP

NAP

of the organ as a whole.
Description No. 4. BoUnlcal Description

NAP

VBP

NAP

VBP

NAP

NAP

I

CJ

\ /—\

Moilugo are low glabrous much-branched annuals with whorled leaves andNAP
obsolete stipules.

NAP

VBP

NAP

-\ m
r
are axillary,

The flowers

on slender pedicels.

I
\t—\
I
\
There are five scarloua margined sepals

Fig. 3-1 Description Parsings Using BPHRAS
3-5
LOCKHEED PALO ALTO RESEARCH LABORATORYL O C K H E E D
A

GROUP

M I S S I L E S

D I V I S I O N

OF&S P A C E

C O M P A N Y

LOCKHEEDAIRCRAFTCORPORATIONLMSC-D246461

Partn

Section 4
REFERENCES

A. C. Shaw, " P a r s i n g of Graph-Representable P i c t u r e s , " ACM Journal, Vol. 17,
No. 3, 1970
M. B . Clowes, "Pictorial Relationship — A Syntactic Approach, " MachineIntelligence 4, B . Meltzer, and D. Michie, e d s . , New American Elsevier
Publishing C o . , I n c . , 1969, N. Y.
O. Firschein and M. A. F i s c h l e r , "Describing and Abstracting Pictorial Structures, n
P a t t e r n Recognition Journal, Vol. 3, No. 4, Nov 1971
O. F i r s c h e i n and M. A. F i s c h l e r ,Pictorial Data,f!

M

A Study in Descriptive Representation ofSecond Int. Joint Conf. on Artificial Intelligence, Sep 1971,

Iiondon, England
O. F i r s c h e i n and M. A. Fischler, "Descriptive Representation of RemotelySensed Image Data, M 1971 IEEE Systems, Man and Cybernetics Group, Annual
Symposium Record, IEEE Catalog No. 71 C46-SMC, Oct 1971, Institute of
E l e c t r i c a l and Electronic Engineers, N. Y.
H. R. Robison, "Computer-Detectable Semantic Structures, " Information
Storage and Retrieval, Vol. 6, Pergamon P r e s s , 1970, Oxford, New York,
London, and P a r i s
L. T e s l e r , "New Approaches to Conceptual Dependancy Analysis, " in Spinoza I I :Conceptual C a s e - B a s e d Natural Language Analysis, by R . C. Schank, L . T e s l e r ,and S. Weber, Stanford Artificial Intelligence Project Memo AIM-109, J a n 1970
E. Sandewall, "Representing Natural Language Information in Predicate Calculus, "in Machine-Intelligence 6, B . Meltzer and D. Michie, e d s . American Elsevier
Publishing C o . , I n c . , New York

4-1
LOCKHEED PALO ALTO RESEARCH LABORATORYL O C K H E E D
A

GROUP

M I S S I L E S

D I V I S I O N

OF&S P A C E

C O M P A N Y

LOCKHEEDAIRCRAFTCORPORATIONLMSOD246461

Partn

9.

C. J. Fillmore, MThe Case for Case, M in Universals in Linguistic Theory,
E. Bach and R. T, Harms, Holt, Rinehart and Winston, Inc., Chicago, 1968

10.

R. F. Simmons, "Some Relations Between Predicate Calculus and Semantic Net
Representations in Discourse, " Second Int. Joint Conf. on Artificial Intelligence,
Sep 1971, London, England

11.

R. C. Schank, L. Tesler, and S. Weber, "Spinoza II: Conceptual Case-BasedNatural Language Analysis, M Stanford Artificial Intelligence Project, Memo
AIM-109, Jan 1970

12.

R. C. Schank, "Finding the Conceptual Content and Intention in an Utterance in
Natural Language Conversation, " Second Int. Joint Conf. on Artificial Intelligence,
Sep 1971, London, England

LOCKHEED PALO ALTO RESEARCH LABORATORYLMSC-D246461
P a r t II

Appendix A
THE USE OF CASE STRUCTURES IN SEMANTIC MAPPINGThe "case 1 1 notions c o m p r i s e a s e t of universal concepts which identify certain typesof judgments human beings a r e capable of making about the events that a r e going onaround t h e m ; judgments about such m a t t e r s as who performed an action, who did it
happen to, with what was it done, and what was changed.

The p r e s e n t appendix

reviews some of the work in the use of case concepts to obtain a representation of
the meaning of a sentence.
Fillmore, Ref. 10, views the sentence as consisting of a v e r b and one or m o r e nounp h r a s e s , each associated with the v e r b in a p a r t i c u l a r case r e l a t i o n s h i p .
the c a s e s and their definitions a r e given in Table A - l .

Some ofHe gives a slight indicationof how one can go from the surface s t r u c t u r e of the sentence to the underlying c a s e s
by noting the c h a r a c t e r i s t i c s of the verbs and the prepositions.

For example, theAgentive preposition is b ^ , the Instrumental preposition is by if t h e r e is no Agentive,
otherwise it is with; the Objective and Facitive prepositions a r e typically deleted,and the Dative preposition is typically t o .
Simmons and Bruce, Ref. 11, use the Fillmore case concept to obtain an "attributevalue " r e p r e s e n t a t i o n for the meaning of a sentence.

For example, given the sentence,"John made c h a i r s with tools on October 20th in Austin, " they get a d i a g r a m based on
the c a s e s a s follows:

NPI

NP2

i

NP3

i

John

NP4

NP5

i
chairstools

i
Oct 20i
AustinA-l
LOCKHEED PALO ALTO RESEARCH LABORATORYL O C K H E E D
A

GROUP

M I S S I L E S

D I V I S I O N

OF&S P A C E

C O M P A N Y

LOCKHEEDAIRCRAFTCORPORATIONLMSC-D246461

Partn

Table A-l
CASES USED BY FILLMORE

Agentive (A). Instigator of the action identified by the verb.
Instrumental (I). The inanimate force or object causally involved in the action orstate identified by the verb.
Dative (D). The case of the animate being affected by the state or action identified
by the verb.
Factitive (F). The case of the object or being resulting from the action or state
identified by the verb.
Locative (L). The case which identifies the location or spatial orientation of the stateor action identified by the verb.
Objective (O). The semantically most neutral case, the case of anything representable
by a noun whose role in the action or state identified by the verb is identified by thesemantic interpretation of the verb itself (conceivably, the concept should be limited
to things which are affected by the action or state identified by the verb).

This diagram can also be represented as a series of triples:
make
make
make
make
make
NP1
NP2
NP3
NP4
NP5

AGENT
OBJECT
INSTRUMENT
TIME
LOCATION
TOKEN
TOKEN
TOKEN
TOKEN
TOKEN

NP1
NP2
NP3
NP4
NP5
John
chairstools
Oct 20AustinA-2
LOCKHEED PALO ALTO RESEARCH LABORATORYLMSC-D246461

Part nSince this s e t of t r i p l e s r e p r e s e n t s the meaning of the sentence, Simmons and Bruce
suggest that to answer a question one simply finds the s e t of triples in the data basecorresponding to the triples in the question.

They do not deal with the problem ofp a r a p h r a s e , a problem that keeps such matching from being a straightforward
procedure.
Schank et a l . , Refs. 12, 13, use case concepts in their p a r s e r , Spinoza ELThe

p a r s e r makes use of relations between conceptual actions and the implications ofthese actions.

This enables the conceptual analyzer to discover not only the conceptual

content of an utterance, but also the intention of that utterance in context.

The

conceptual dependency analysis which they use is directly related to the case concept,
but they use an interesting generalization to handle the problem of p a r a p h r a s e .
example, the sentence,?

For'the man took the book, " would have the network

p
man <

• take

o
—

R
«

book

•to

man

1

X

1
from

where p = actor or p e r f o r m e r , o = object, R is the recipient, and the X denotes thefact that the entity from which the book was taken was not specified in the sentence.
However, this network is typical of an entire c l a s s of actions involving transfer of an
object from one entity to another, e . g . , "given, n n send, " "steal, " which have thegeneral network

Z * = * transoR— object «—The v e r b dictionary used in Spinoza n would indicate that for "give" the " t r a n s " network
above is relevant, and that Z = X, i . e . , the person performing the action is the s a m e
a s the p e r s o n from whom the object c o m e s .

A-3
LOCKHEED PALO ALTO RESEARCH LABORATORYL O C K H E E D

M I S S I L E S&

S P A C E

C O M P A N Y

LMSC-D246461

Partn

The cases used by.Schank are shown in Table A-2, and those of Tesler, a coworker,
Ref. 8, in Table A-3.
Table A-2
CASES USED BY SCHANK[

Conceptual cases

Prepositions Used

RelationshipObjective(none)

Object of the action

Recipient
Instrumental

to, from

Receiver or transmitter

with, byInstrument used

Directive

to, from, toward

Direction of action

Attributive Cases

Prepositions Used

Relationship

Possessionof, with

Location-near

near, at, by, in before

Has
LocatedContainmentin, ofContained

Table A-3
CASES USED BY TESLER
Conceptuali zation
Class
Attributive
Classificational

Relationship

Behavioral

Behavior without change of state, victim, goal, or
direction,e.g. rotate, dance

MotiveSomething in motion from one station to another

Operative
MutualSomething active does something to something passive

Existence, has attribute, in a state
Something which is the same as something else

Two or more actors interacting to perform an action,
which would not be observed to be performed just byobserving one of them, e.g. fightHypothetical

An attitude toward an issue, e . g . , say, claim, know,
fear, hear

CausalSomething makes something happen, either intensionallyor unintentionally

A-4

LMSC-D246461
Part II

Appendix B

DESCRIPTIVE REPRESENTATIONS OF REMOTELY SENSED IMAGE DATA*
O. Firschein
M. A. Fischler
Lockheed Polo Alto Research Laboratory
Palo Alto, California 04304

There is a growing interest in high-altitude imago
data obtained from various s e n s o r s for use in earthr e s o u r c e s applications. The lines, patterns, andcolors in such imagery reveal what both nature and man
have done to the face of the earth. In photos taken fromhigh altitudes, highways appear as networks of fine lines,
farm fields produce recognizable patterns, and the areasfrom which forests have been cleared stand out brightly.
Colors are indicative of the vegetation, soil conditions,depth of water, and many other matters of vital concern
to mankind.
The present paper deals with the nature of theimage descriptions used to express the content or
meaning of earth resources satellite data, rather thanthe mechanics of handling such data, such as is given
in Ref. 1.
Natural Language Descriptions

bination of separate descriptions of the s a m e image.
Query and retrieval s y s t e m s would then be designed
around such representations.
Canonical Forms and Description Merging
We have examined two types of canonical forms,
the descriptor approach in which a set of words orphrases is selected to represent the image, (see ourreview in 5) and the relational graph approach in whicha conceptual representation is obtained by indicating
relationships between entities in the picture. If two
descriptor representations have been prepared for thesame image, a combined descriptor set can be obtainedby forming a single combined list of descriptors. If
two network representations for the same image areavailable, the problem is more complex but, abstractly,
the networks are combined based on detection of common
nodes. -Considerable variation occurs in the scope of natural
language description of imagery from the various earthr e s o u r c e s d i s c i p l i n e s . 2 Geology, geography, hydrology,
and oceanography tend to more global descriptions,while agriculture, forestry, and cartography tend to focus
on particular attributes of the image, often relying on o v e r lays or line drawing representations. In addition, we notethat the grammatical structures for describing imaged data
tend to be much simpler than conventional samples ofliterary or spoken language - the verb forms and relationships are s i m p l e r , and ambiguity is e a s i e r to r e s o l v e .

In the experiments described in 5 , using a set of
subjects having no special technical background, convergence to a merged form was obtained quickly; therelational net remained relatively unchanged when
additional descriptions of the same image were c o m bined with it. However, in the c a s e of experts dealing
with earth r e s o u r c e s data, an individual with a specialty
can introduce a whole new set of links and nodes into thenet, and therefore convergence to a particular r e p r e sentation does not guarantee completeness; it merely
indicates completeness for a particular set of specialties.

Several questions arise in dealing with such d e s criptions. (L) Mow can the descriptions be put into somenormal or canonical form to regularize variations In
description from observer to observer, and also for use
in computer based s y s t e m s ? (2) How can a number ofdifferent descriptions of a given image be combined toobtain a more comprehensive description? (3) How canthe entities and relations between entities in a description
be converted to a small set of basic primitives to simplify
the canonical f o r m s ?

The Concept of an Encyclopedic Entry

A basic philosophical question also a r i s e s in d e s i g n ing retrieval s y s t e m s : Should one determine the bestquery structure for a particular user population and thendesign a suitable data base structure, or should one
use a data base structure having certain desirable
properties and design the most appropriate query
structure?
We are taking the latter c o u r s e . By examining
various representations of natural language description,
we hope to determine structures that have certain d e sirable properties such as allowing automated structure
determination from text, and allowing automatic c o m -This study is concerned with the problem of whether
a representation of a picture could be used to answer
general questions about the picture. There would s e e m
to be too many diverse questions that could be asked
concerning the picture, and too many different requirements in terms of the level of response required to allow
such a procedure to be successful. Consider, however,an analogous situation which a r i s e s in the attempt tosummarize all human knowledge in such a way that questions can be answered using this summary. Although thistask would also s e e m to be overwhelming, collections ofknowledge, called "encyclopedias," have been used formany y e a r s , and turn out to be a useful means of a n s w e r ing a diversity of questions.
Similarly, it should be possible to develop aquestion-answering system based on the concept of anencyclopedia for the body of knowledge residing in eithera single photograph or a set of photographs. The sameproblems of level of detail and use of technical terminology a r i s e in the preparation of both an encyclopedia
based on knowledge expressed in natural language and•The work described here was supported partially by the Office of Naval ResearchContract N00014-70-C-0239, and partially by the Lockheed Independent Research
Program.

Paper presented at Joint National Conference on Major Systems, sponsored by theIEEE and ORSA, Anaheim, California, October 1971.
T*_I

LMSC-D246461

Partn
one tn which knowledge IH expressed In pictorial form.
In both cases, one hafl to consider how the material IH
to be organized, the nature of the potential user, ami theamount of material to be gathered.
Relation of Queries to Descriptions
We can gain some insight into the requirements foran encyclopedic entry by examining the types of querythat aro likely to occur, it should be kept in mind thatan encyclopedia contains material suitable for answering
many questions, but not ail questions on a topic. In
addition, while several diverse items of information fromthe encyclopedia may be pulled together to answer aquestion, there is no guarantee that this can be done.
Similarly, for an encyclopedic entry for an image, weshould expect to be able to answer many questions, to, combine elements of the entry to answer other queries,
and to fail completely for the remaining queries. In our
examination of queries concerning earth resources
imagery, we have found that the following elements ofdescription are often required.

concerning the source of the Imagery, the equipmentused, time, date, geographical location, image quidlty,
etc. The image citation data usually constitute one of
the main content-bearing elements used for indexing
Imago collections today.
Descriptor data. Words or phrases concerning the
subject matter in the" photograph.
Natural language descriptions. Descriptions which
capture the theme, general layout and arrangement of
the photograph, objects pictured, and their relationships
with other objects. This type of description has been themain concern of the present paper.
Analysis of imagery. Measurements and delineationof aspects of the Image, such as identifying contours ofbodies of water, determining distance between objects,
etc.

Discontinuity and contrast. An indication of discontinuity and contrast by identifying regions homogenous
in such attributes as texture, color, pattern, andtemperature (on BR) imagery.
Terraine structure identification. Queries willoften ask for the existence of flow patterns, surface film
patterns, silting and sediment, circulation, thermal
structure, man-made features, etc.
Change detection. Queries often ask for changes incultural features, stream flow, reflectivity, crustal
structure, snow areas, sand bars, etc. Such questions
can be answered if previous and present descriptions ofan image are available and contain line drawing extracts
or equivalent. Such line drawing extracts can be readily
compared and the differences noted.
Boundary area, and volume analysis. The boundaries of snow and ice, lakes, tectonic features, cropareas, etc., are often required for area and volume
computations.
Counting of items. Queries will often ask for acount of the number of items, or some Indication ofwhat percent of the image is taken up with some items.
Content in a broad sense. These are queries thatconcern the global aspects of the image, such as: is thearea predominately rural? ; what are the general characteristics of the mountain ranges?; what are the transportation networks?; describe the shoal formations.

Conceptual maps. Indication of the content of thepicture by means of linked structures. Such relational
nets are used in present day question-answering systems
having limited data base sizes.
Line drawing extracts. Line drawings extracted
from an image for expressing specialized relationships
such as transportation networks, geologic boundaries,
and water courses.
These disparate elements must be tied together insuch a way that queries are suitably responded to, withdirect answers, extracts of imagery, measurements, or
discussion provided to the user.
A Dynamic Data Bank for Earth Resources Imagery
Wo havo examined ^ the automated processing ofdescriptions to obtain encyclopedic entries for an image.
Such automation can lead to what we have termed"dynamic data banks" for earth resources imagery.
Experts using the imagery would return to the archival
center their analysis and description of the imagery
developed as part of their investigation. Such descriptions would be processed and merged, so that withtime each image would have associated with it a combined or encyclopedic description.
Of course, such descriptions would have to be tagged
so that data on the background of the person making thedescription (his name, set, or field of specialty, analysis used, comments regarding the analysis, and reports
produced using this image data) could be retrieved if
desired. This "metadescriptive" material gives thecontext of the individual descriptions, the confidence thata user can have in the description, and links to related
work.
References

The natural language descriptions are mostappropriate for the last type of query.
The Nature of an Encyclopedic Entry
An enc>clopedic entry for an Image capable of dealing with the above queries would consist of one or moreof the following content-indicating elements.

1. H. M. Gurk, C. R. Smith, and P. Wood, "Data
Handling for Earth Resources Satellite Data," Proceedings of the 6th International Symposium on Remote
Sensing of Environment. Center for Remote Sensing
Information and Analysis, University of Michigan,
Ann Arbor, Oct 1969
2

Image citation data. Identification information

B~2-

Ecological Surveys From Space, NASA SP-230,

LMSC-D246461
Part II
National Aeronautics and Space Administration,
Washington, p. c . 1970
3. O. Firschein and M. A. Fischler, "Describing and
Abstracting Pictorial Structures," Pattern Recognition
Journal (in press, 197 1)
4. M. A. Fischler, "Machine Perception and DOHcription of Pictorial Data," Proceedings of the International Joint Conference on Artificial Intelligence,
Washington, D.C., May 1969
5. O. Firschein, and M. A. Fischler, A Study In
Descriptive Representation of Pictorial Data, 6-83-71-3,
Lockheed Palo Alto Research Laboratory, Palo Alto,
California, Feb 1971 (also, Second International Joint
Conference on Artificial Intelligence, Sept 1971, London,
England)
6. O. Firschein and M. A. Fischler, Descriptive
Representations of Remotely Sensed Image Data,
6-83-71-5, Lockheed Palo Alto Research Laboratory,
Palo Alto, California, May 1971 (complete versionof the present short paper)

B-3