Mi
I
UNITED STATES DEPARTMENT OF COMMERCE • John T. Connor, Secretary NATIONAL BUREAU OF STANDARDS • A. V. Astin, Director

Statistical Association Methods

For Mechanized Documentation

Symposium Proceedings

Washington 1964

Edited by Mary Elizabeth Stevens, Vincent E. Giuliano,

and Laurence B. Heilprin

National Bureau of Standards Miscellaneous Publication 269

Issued December IS, 1965

For sale by the Superintendent of Documents, U.S. Government Printing Office, Washington, D.C., 20402

Price £2.75
Abstract

A Symposium on Statistical Association Methods for Mechanized Documentation was held in Washington, D.C., in March 1964. The Symposium was jointly sponsored by the Research Information Center and Advisory Service on Information Processing, Institute for Applied Technology, National Bureau of Standards, and by the American Documentation Institute. Topics covered include the

historical foundations, background and principles of statistical association techniques as applied to problems of documentation, models and methods of applying such techniques, applications to citation indexing, and tests, evaluation methodology and criticism. This volume contains 22 of the papers

included in the program, the abstracts of 4 additional papers that were presented, and the text of the

talk given by R. M. Hayes at the banquet.

Library of Congress Catalog Card No. 65-60077
Foreword

The Research Information Center and Advisory Service on Information

Processing was established at the National Bureau of Standards in 1959 under

the joint sponsorship of the National Science Foundation and the Bureau, with

the assistance of the Council on Library Resources. The Center is engaged in a continuing program to collect information and maintain current awareness of

research and development activities in the field of information processing and

retrieval and to encourage cooperation among workers in the field. On March 17, 18, and 19, 1964, the Center, in cooperation with the American

Documentation Institute, sponsored a Symposium on Statistical Association

Methods for Mechanized Documentation. The Symposium was held in Washington, D.C., and was attended by approximately 250 subject-matter specialists.

This volume contains the texts or abstracts of the papers presented. Primary

responsibility for their technical content must rest, of course, with the individual

authors. A. V. ASTIN, Director.

iii
Introduction

The Symposium on Statistical Association

Methods for Mechanized Documentation was con- vened March 17, 1964 at the Smithsonian Institu- tion Auditorium, Washington, D.C. An introduction by Dr. Donald A. Schon, Director, Institute

for Applied Technology, National Bureau of Standards, emphasized the different but interdependent

interests of the user of scientific and technical

information, the machine technologist, and the

information specialist. The keynote address was

given by the late Hans Peter Luhn, pioneer in the

practical application of statistical techniques to mechanized documentation operations, on the

subject, "Physical prototypes of meaning and their

manipulation." During the three-day sessions,

26 technical papers were presented and provocative

panel discussions were given by Pauline C. Atherton, Cyril W. Cleverdon, Calvin N. Mooers, and

Alan M. Rees on problems of evaluation and by

Phyllis B. Baxendale, Edward C. Bryant, John

O'Connor, Herbert C. Ohlman, H. Edward Stiles,

John W. Tukey, and the members of the Program

Committee on problems, progress, and prospects.

In recent years there has been a growing interest

in the use of computers and machine aids for the

processing of documents. Systems for machineaided document classification, for automatic indexing and abstracting, and for both document and

"fact" retrieval have been the subject of research

investigations and/or pilot operations. The growing interest in the use of statistical association

methods for such applications appears to be justi- fied for two quite excellent reasons.

First, our present understanding of digital computers and computing techniques is such that these

machines are best suited for the high-speed repetitive execution of simple arithmetic and logical operations. The statistical association techniques are based on the counting of simple observable entities such as words in text, index terms, term co-occur- rences, document citations, etc. They also involve

the computation of simple arithmetic decision functions based upon such counts. Digital computers

are particularly suited to such tasks. In contrast,

the handling of complex logical, syntactic, or semantic structures by machine requires comparatively

arduous and intricate techniques, and the application of these methodologies for purposes of documentation remains the subject of long-range re- search. The application of statistical procedures

to mechanized documentation thus capitalized on and matches a significant attribute of existing data- processing machinery — its numerical capability.

Secondly, the techniques appear to be based

upon excellent theoretical foundations drawn from

the fields of statistics and mathematical psychology.

Analogous or identical techniques have previously

been applied to a number of closely related problems

in other fields besides documentation. As a con- sequence, considerable experience has been gained

with the details of the methodology itself— and the

effectiveness of the techniques has been established

in analogous areas of application.

Because of this, the study of statistical association

techniques for mechanized documentation offers the real potential of creating powerful tools for solution of the problems at hand. The resulting

effect has been to enable concentration of most of the research effort on the real problems at hand

without the need to divert attention to study the

methods.

The major purposes of the Symposium were to bring together in one place a representative group

of individuals working in a common area to ex- plore the interrelationships among the different techniques being researched, and to explore further

the foundations and methods common to all of them.

To further this objective, the papers in this volume

have been grouped, for convenience, into sections

treating Background and Principles, Models and

Methods, Applications to Citation Indexing, and

finally Tests, Evaluation Methodology, and Criti- cisms. The area is still young and is now passing

into a more vigorous stage of research. Much re- mains to be done, for many important topics can be treated only in a preliminary and tentative fash- ion at the present state of knowledge and understanding. It can be hoped that the communication

provided by the Symposium will contribute towards

the identification of areas requiring intensive in- vestigation. More significantly, it can be expected

that more purposeful research on and testing of

iv
the basic premises will emerge from the discussions

and deliberation that were held. We, the members of the Symposium Committee,

wish to express our appreciation to those who contributed to this conference, the authors and the

discussants.

Mary Elizabeth Stevens, Chairman,

National Bureau of Standards

Vincent E. Giuliano

Arthur D. Little, Inc.

Laurence B. Heilprin

Council on Library Resources

v
Contents

Foreword iii Introduction iv 1. Background and principles

Historical foundations of research on statistical association techniques

for mechanized documentation 3

Paul E. Jones

Arthur D. Little, Inc. Cambridge, Mass. 02140

Mechanized documentation: The logic behind a probabilistic interpretation 9

M. E. Maron

The RAND Corporation

Santa Monica, Calif. Some compromises between word grouping and document grouping 15

Lauren B. Doyle

System development Corporation

Santa Monica, Calif. ^0406

The interpretation of word associations 25

Vincent E. Giuliano

Arthur D. Little, Inc. Cambridge, Mass. 02140

The continuum of coefficients of association 33

J. L. Kuhns

The Bunker-Ramo Corporation

Canoga Park, Calif. 91304

A correlation coefficient for attributes or events 41 H.P. Edmundson

The Bunker-Ramo Corporation

Canoga Park, Calif. 91304

2. Models and methods

A modified statistical association procedure for automatic document

content analysis and retrieval 47

Joseph Spiegel and Edward M. Bennett

The Mitre Corporation

Bedford, Mass. 01730

The construction of a thesaurus automatically from a sample of text 61

Sally F. Dennis

International Business Machines Corporation

Chicago, III. 60620

Latent class analysis as an association model for information retrieval... 149

Frank B. Baker

University of Wisconsin

Madison, Wis. 53706

Problems of scale in automatic classification (abstract only) 157

Roger M. Needham

University of Cambridge

Cambridge, England

A nonlinear variety of iterative association coefficient (abstract only) 159

Robert F. Barnes, Jr. The measurement of information from a file 161

Robert M. Hayes

University of California

Los Angeles, Calif. 90014

Vector images in document retrieval 163

Paul Switzer

Harvard University

Cambridge, Mass. 02138

Threaded term association files 167

Mark Seidel

Datatrol Corporation

Silver Spring, Md. 20910

VI
Statistical vocabulary construction and vocabulary control with optical

coincidence 177

Basil Doudnikoff and Arthur N. Conner, Jr. Jonker Business Machines, Inc. Washington, D.C. 20760

A computer-processed information-recording and association system... 181

G. N. Arnovick

Planning Research Corporation

Los Angeles, Calif. 3. Applications to citation indexing

Statistical studies of networks of scientific papers (abstract only) 187

Derek J. deSolla Price

Yale University

New Haven, Conn.

Can citation indexing be automated? 189

Eugene Garfield

Institute for Scientific Information

Philadelphia, Pa. 19106

Some statistical properties of citations in the literature of physics 193

M. M. KESSLER

Massachusetts Institute of Technology

Cambridge, Mass.

4. Tests, evaluation methodology, and criticisms

An evaluation program for associative indexing 201

Gerard Salton

Harvard University

Cambridge, Mass. 02138

The unevaluation of automatic indexing and classification (abstract

only) 211

T. R. Savage

Documentation, Inc. Bethesda, Md. 20014

Evaluation of automatic indexing using cited titles 213

Mary Elizabeth Stevens and Genevie H. Urban

National Bureau of Standards

Washington, D.C. 20234

Results of classifying documents with multiple discriminant functions... 217

J. H. Williams

International Business Machines Corporation

Bethesda, Md. 20014

Rank order patterns of common words as discriminators of subject content in scientific and technical prose 225

Everett M. Wallace

System Development Corporation

Santa Monica, Calif. 90406

Clumping techniques and associative retrieval 230

A. G. Dale and N. Dale

University of Texas

Austin, Tex.

Statistical association methods for simultaneous searching of multiple

document collections 237

William Hammond

Datatrol Corporation

Silver Spring, Md. 20910

Studies on the reliability and validity of factory-analytically derived

classification categories 245

Harold Borko

System Development Corporation

Santa Monica, Calif. 90406

Postscript: A personal reaction to reading the conference manuscripts 259

Vincent E. Giuliano

Arthur D. Little, Inc. Cambridge, Mass. 02140

vii

Background and Principl

Historical Foundations of Research on Statistical Association Techniques

for Mechanized Documentation*

Paul E. Jones

Arthur D. Little, Inc. Cambridge, Mass. 02140

Ultimately, in statistical association research of the type which is discussed in this Symposium,

the data under analysis are taken from a symbol system generated by man. The symbol system may

comprise, for example, a text prepared for communication in natural language, or it may be a pattern

of terms assigned to a document collection where the purpose of the indexing relates to the retrieval of the documents. Ordinarily the purpose of the system can be well defined, but the mechanism for producing the symbols (uttering words, indexing documents) is poorly understood. As attempts are made to unravel the statistical properties of these symbol systems, the unknown processes which

underlie formation of the data are in fact under scrutiny. Thus in examining the effects of the unknown

symbol-producing mechanism, problems continue to be studied which have caught the attention of the greatest intellects of Western culture.

1. Introduction

"Historical Foundations" may seem a surprising

title to people who consider our subject to be brand

new. After all, "information retrieval" is terminology no more than 20 years old, "mechanized

documentation" is perhaps younger, and computers

are so new that "historical" seems a curious term

to apply to so short a period. The work is ob- viously derived from the pioneering work of Luhn

[l], 1 Maron and Kuhns [2], and Stiles [3], all of

whom are clearly identified with the use of computers for mechanized documentation. Where

does one derive a historical view when developments

have been so recent?

Actually, the statistical association approach

draws its point of view, its objectives, and its ideas

from at least five major areas of study. Enumerating them is almost a commonplace: psychology,

philosophy, technology, linguistics, mathematics.

Many of the problems now under investigation have

been looked at before with a different perspective,

and all five disciplines are involved in the current work. Psychology enters because the data subjected to statistical study were generated, and

ultimately are interpreted, by man for his own

purposes and objectives. Technology, especially

digital computer technology, has had enormous

influence: The approach would be an empty theoretical conjecture were it not for the vast data-processing capabilities now at our disposal. Linguistics

has its influence, since the data being analyzed

fall into its province. The work is obviously

mathematical, not only because of the prominent

role of statistics but also because of the structure

of the approach. Finally, philosophy contributes

"Support for the preparation of this review was provided, in part, by the Decision Sciences Laboratory. ESD. U.S. Air Force under Contract AF 19(628»-331 1. ESTTDR-64-528.

1 r inures in brackets indicate the literature references on p. 8.

the epistemological basis for our work in ways to be touched upon in later paragraphs. Investigations along related lines, and important developments, are to be found in each of these areas, much

of it independent of computers and the advent of

serious thought about mechanized documentation.

1.1. A Linguistic Perspective

Most workers in the area of statistical association

techniques have applied their techniques to data

consisting of the term assignments in a mechanized

retrieval system. In general, the problem of document retrieval has served as a useful medium within

which to formulate the purpose of the approach;

also it has served as a source of guidelines to identify potentially fruitful fines of research.

Similarly, the environment of a retrieval system has

served in practice as the practical situation within

which the "improvement" that might be provided

by a statistical association technique can be

observed.

In studying an information retrieval system, or, more generally, a system for mechanized documentation, we may consider that we are studying

a language made up of the marks and symbols used

in indexing. These marks and symbols, which

were assigned to documents by indexers when the

document was entered into the documentation

system, are used in the system for such tasks as finding documents that are relevant to a user's request. The tags assigned to a document serve

as a representation — admittedly incomplete — of what the indexer tried to say the document was

about. In a mechanized system, where some degree of

orderliness and regularity is to be expected, these

marks and symbols are observable representations

of what the indexer was trying to convey. As such,

the symbol system functions as a language in the

intuitive sense: It serves as a vehicle for conveying

3
information about some universe, where the uni- verse is of course the content of the set of documents

being described. There ordinarily is an effort on

the part of the indexer to choose index tags which

describe the document's content, just as in a natural language there ordinarily exists a willful relationship between the words an author writes

down and that aspect of reality he is trying to convey. In each case, the symbols of the language

are all that is observable, whereas it is the "content"

of the message that has the major interest and

potential utility. In each case the symbols are purposefully related

to the universe under discussion. Thus either a term system or a natural language text may, at least in concept, serve as the data operated upon

by the statistical techniques which are devised

2. A Dualistic

2.1. Explaining Statistical W6rd Associations

Ultimately when a set of events is subjected to

statistical study, one is inevitably making assertions

about, and thus dealing with, the process which

brought the given data into being. Yet what is the underlying process which is being dealt with

when we perform statistical analysis of term co- occurrences in an information retrieval system, or analyze word co-occurrences in text? Are the

associations to be explained in terms of a phenomenon involving the representation, with symbols,

of entities that "really do" co-occur in the "real

world"? This hypothesis regards the data as strongly constrained by the external world of physical nature. Or, on the other hand, are the associations a manifestation of the "association of ideas" on the part of the author or the indexer? This

hypothesis regards the data as strongly constrained

by the internal world of mental phenomena. No

complete explanation has been given for the ap- parent success of the statistical association techniques in discovering the provocative regularities

in the data which have been reported. And since our work is interdisciplinary, it is probable that both the above explanatory mechanisms have been

employed simultaneously as working hypotheses.

(This fact alone is worth underscoring, for the view

that allows both explanatory mechanisms to be

seen as equivalent is relatively recent. 2

) Yet

historically speaking, they may be considered poles

apart, with roots in two distinct schools of thought.

There are two conflicting frameworks within

which the studies being discussed at this Symposium

may be embedded. On the one hand, since we

cannot ignore the user's mental processes, we are quite content to consider ideas, concepts, meanings

as perfectly respectable entities which are observable by introspection. We are capable of

2 See for example the discussion of language in [9].

3 For a detailed discussion of the development of the model, see [12],

for mechanized documentation purposes. Although some of the statistical association techniques have

been applied directly to words occurring in text, others cannot be applied directly since they explicitly assume that the data will exhibit certain

properties peculiar to a mechanized documentation

system. Nevertheless, if only because automatic

indexing of texts could be performed to obtain data

to which the statistical association techniques apply

[3], both types of data — text and index tags — appear

at the present time to be analyzable by the same

approach. A large body of work relevant to the

topic of this Symposium is thus to be found among

analogous aspects of the study of natural language,

especially those studies in which extra-linguistic

inferences are drawn from a given body of textual data [4-8].

Historical Base

talking quite rationally about relationships among

them, their degree of similarity, and the like, with- out quibbling about their reality. As scientists, on

the other hand, we are under strong influences to exclude man's mental processes from any system

under objective study. As Bridgman put it, in the indroduction to a philosophical discussion of

modern physics [10],

It is of course ifre merest truism that all our experi- mental knowledge and our understanding of nature is impossible and non-existent apart from our own mental

processes, so that strictly speaking no aspect of psy- chology or epistemology is without pertinence. For- tunately we shall be able to get along with a more or

less naive attitude toward many of these matters. We

shall accept as significant on common sense judgment

that there is a world external to us, and shall limit as far as possible our inquiry to the behavior and

interpretation of this "external" world.

Bluntly, the physicist says, "You can't observe an

idea." Yet because of the nature of our work we

also cannot define ideas out of the universe of discourse.

To circumvent this extreme dualism, introduced

by Descartes, in which mind and physical nature

are completely separate, we may employ the

epistemological framework developed by the British

empiricists between 1750 and 1900. Beginning

with Locke and Hobbes, the mind at birth was

treated as a tabula rasa upon which experience

about the external world was recorded, henceforth,

in a form and pattern that led ultimately to knowledge. Berkeley and Hume, among others, completed the epistemological framework and

hypothesized the associational mechanism to ac- count for and explain the higher mental processes.3 Some scholars claim that Aristotle had a crude

formulation of the association of ideas by "similarity" and by "contiguity." But Hume [11]

wrote of the associational mechanism:

And even in our wildest and most wandering reveries, nay in our very dreams, we shall find, if we reflect, that
the imagination ran not altogether at adventures, but

that there was still a connection upheld among the dif- ferent ideas, which succeeded each other. Were the

loosest and freest conversation to be transcribed,

there would immediately be transcribed, there would

immediately be observed something which connected it in all its transitions. Or where this is wanting, the person who broke the thread of discourse might still inform you, that there had secretly revolved in his mind

a succession of thought which had gradually led him

from the subject of conversation. Though it be too obvious to escape observation, that different ideas are connected together; I do not find that any philosopher

has attempted to enumerate or class all the principles

of association; a subject, however, that seems worthy

of curiosity. To me, there appear to be only three

principles of connection among ideas, namely, resemblance, contiguity in time or place, and cause or

effect. As an epistemological framework, the work of the

British empiricists has served as the principal route

for transfer between the external world and the

reality known by introspection.

2.2. The Psycholinguistic Route

But the associationists' model was also inter- pretable as a psychological doctrine [13], and as such it was severely attacked in the early twentieth

century. The model failed, for example, to provide

for quantifiable observations; the inadequacy of

introspection as a workable observational tool pre- vented the use of the associational model as the

basis for a scientific theory. Though the associationists' ideas were generally encompassed by

the newer psychological theories, the mainstream

of activity diverted from the epistemological interest

explored by the British empiricists. It goes with- out saying that psychologists retained their interest

in studying the laws that govern the mind, yet a sharp trend away from a dualistic philosophy

accompanied the rise of objective psychology and

behavioristics. Clearly this trend involved a move- ment away from the intuitive reality of ideas

and towards the study of external, observable

manifestations.

Many of the developments in psychology most

closely related to our present interests are derived

from the resulting experimental activity, especially

the efforts to analyze and quantify psychological

data. Naturally, modern psychologists have always

been interested in issues of scaling [14], computation, and statistical analysis of observed behavior,

but their objectives have involved interest in studying individual psychological parameters.

Workers in statistical association techniques for mechanized documentation have not shared this objective. But though our motivation is somewhat

different, there is much to be learned from the tools and approaches the psychologists developed in the early decades of the twentieth century. For

example, it was this school, with its interest in drawing inferences about psychological variables

4 These techniques figure importantly in the work of Borko and his followers.

from the outcome of behavioral experiments, which

developed and applied the techniques of factor analysis [15, 16] with its accompanying methodology. 4 In addition, psychologists became increasingly

interested in the analysis of linguistic behavior.

An important body of experimental work on human

word associations was performed [17]. This atten- tion led slowly to the notion that language data

could be analyzed for content by studying word

frequencies and interpreting the pattern that emerged [18, 19]. For example, one vigorous fine of development

in the 1940's was directed at the analysis of mass

communications to ascertain the objectives behind

the propaganda being transmitted or published.

. . . Content analysis was initially developed some

years before World War II, as a tool for the scientific study of political communication. Those who pioneered

with Harold D. Lasswell in its development were

interested in acquiring scientific knowledge about

political communication. Accordingly, content analysis was originally defined and developed in order to list and measure the frequency of occurrence of certain characteristics of the political communication under

study and to classify them under general terms, or content categories, which were suggested by a tentative theory of political communication. The objective of the research in this original content-analysis ap- proach was to make general inferences, or scientific generalizations, in the form of one-to-one regularities or correlations between some content indicator (or class or indicators) and some state or characteristic of the communicator or his environment [20].

This activity employed various techniques which

are now familiar to us, but the methodology suffered

from being excessively laborious. And although

simple frequencies of occurrence were taken as

clues, frequencies of co-occurrence were not. Work on this faded at the end of World War II, but then in 1955 a remarkable conference was held

at Atherton House at the University of Illinois. The proceedings [21] were not published until much

later (1959), but the deliberation reflects a great

deal of thought about problems very similar to those we are discussing this week.

The conferees were psycholinguists, interested

in drawing inferences from analysis of language

data. They counted co-occurrences. They dis- cussed a number of association formulas. They

used factor analysis. They talked about wordassociation profiles, meaning measures, and employed a vector space representation. They

performed cluster analysis.

For instance, in the introduction, Pool writes

It was . . . somewhat of a discovery for a group of scholars assembled in the mid-1950's, when content

analysis seemed to be in a decline, to find that other

scholars also had seen unexplored potentials in content

analysis if certain new tacks were taken to meet the unsolved problems of the previous decade. The conferees, each starting from different directions and

generally unaware of each other's work, did not of course see eye to eye on all issues. The discussions were vigorous .... But the striking fact was the degree of convergence.

5
It is not for this introduction to attempt to state what

the convergences of viewpoint were .... Suffice it here

to note that they centered above all on two points:

1. a sophisticated concern with the problems of infer- ence from verbal material to its antecedent conditions, and

2. a focus on counting internal contingencies between

symbols instead of the simple frequencies of symbols. Both these points arose out of the concern of the analysts to make their elaborate quantitative method produce something beyond what could be produced without its paraphernalia — to produce

something that would go beyond the reaffirmation of the obvious."

In the same volume (pp. 54-55) Osgood points out An inference about the "association structure" of a source — what leads to what in his thinking — may be made from the contingencies (or co-occurrences of symbols) in the content of a message. One of the earliest published examples of this type of content analysis

is to be found in a paper by Baldwin [22] in which the contingencies among content categories in the letters of a woman were analyzed and interpreted. For some

reason this lead does not seem to have been followed

up, at least in the published reports of people working on content analysis problems. On the other hand, it soon became evident in this conference that all of the

participants had been thinking about the contingency

method in one form or other as being potentially useful

in their work.

If there is any content analysis technique which has a defensible psychological rationale it is the contingency method. It is anchored to the principles of association which were noted by Aristotle, elaborated by the British Empiricists, and made an integral part of most modern

learning theories. On such grounds it seems reasonable to assume that greafer-than-chance contingencies

of items in messages would be indicative of associations

in the thinking of the source. If, in the past experience

of the source, events A and B (e.g., references to FOOD

SUPPLY and to OCCUPIED COUNTRIES in the experience of Joseph Goebbels) have often occurred to- gether, the subsequent occurrence of one of them

should be a condition facilitating the occurrence of the other: the writing or speaking of one should tend to

call forth thinking about and hence producing the other. In other words, out of a discipline with close in- volvement in understanding certain mental pa- rameters (like anxiety) these gentlemen did some

early work on statistical measures of association

with emphasis upon the psychological conse- quences. Their work differed from ours in that they were prepared to introduce a priori encoding

of the data under study. Thus they were prepared

to exercise human judgment in coalescing "ref- erences to factories, industry, machines, production,

and the like" into the single content category

FACTORIES. Less defensible from our view,

they were prepared to encode, by means of human

judgment, the attitude expressed toward such a content category in a given context. This posi- tion reflects, of course, a principal difference in motivation and objectives. (See also [23].) But although their motivation was different, their

procedure was very closely related to that we are now discussing in the context of mechanized documentation. It is of interest that their work has

had no significant influence upon the foundations

upon which the present work rests.

2.3. Natural Science

The mainstream of the statistical association ap- proach discussed at this conference comes rather

from the natural sciences and developments pro- vided by workers quite remote from psychology.

The trend of this work has been in the opposite

direction — away from exclusive attention to the

external world and towards increased incorporation of selected human intellectual activities within

the province of a totally objective science.

The advent of the twentieth century was accompanied by an enormous increase in the use of

statistical methods in all of science. Indeed the use of statistical methods was sufficiently broad

that workers in a multiplicity of areas invented

data-interpretation formulas appropriate to the task

at hand. Goodman and Kruskal [24], in a survey

of measures of association, critically examine a

large number of closely related formulations.

There were, for instance, developments in drawing

inferences from medical data which were contributed by experimenters in that field. A method of

analysis was developed by an ecologist who was

interested in the association between species and

the character (e.g., marshland) of the environment

in which they were discovered. A technique for evaluating the efficacy of a forecast of a tornado was developed by a meteorologist. In each case, the original report serves as a source for the philosophy and logic of the measure that was used

and the important rationale for its interpretation.

These efforts were, of course, subjected to crit- icism and debate. Yule, K. Pearson, Fisher, Kendall, and others continued to probe the rationale

underlying statistical analysis of observations.

While applied work increased in scope, they fo- cused attention on fundamental issues, delimiting

the range of applicability of the approaches, clar- ifying the inherent assumptions, and creating new

concepts of data analysis. But a more brutal

objectivity was needed by Dirac, Einstein, and other

physicists working early in this century [10, 25]. To make quantum mechanical and relativistic con- cepts comprehensible and consistent — in the face

of experimental evidence that defied intuitive explanations — they found it necessary to develop

and use a strict epistemological formalism [25]

which state'd explicitly what could be observed

and the limitations on the inferences one might

draw. Generally speaking, they limited the uni- verse of discourse to the observable physical

reality of the "external world," defining man entirely out of the picture. But in a highly mathematical formalism, they gave impetus to what has now become an increasingly symbolic point of view

toward making, and drawing inferences from, observations of the real world. The important objectivity

employed in the statistical association approach

derives, to a considerable extent, from a corre- sponding insistence that the data from a system (e.g, an information retrieval system) are to be processed

according to procedures which are spelled out in

6
advance. No human interpretation of the data is allowed until all processing is completed.

One would hardly expect that such a cold, scientific methodology could reveal semantic regularities

when applied to uninterpreted language data. After all, language is meant to be interpreted. Yet

it was approximately contemporary with the Atherton

House conference that Luhn began evangelizing

the use of word frequencies in text as a key to content [1]. There was no a priori encoding of words

into content catagories, and he and his followers

had to overcome significant skepticism. Yet his experiments and demonstrations were persuasive.

Luhn drew attention to information retrieval and

indexing as potentially tractable tasks, and combined

the objectivity of frequency analysis with the prag- matic objectives of doing something useful. More

significant, he gave great impetus to a movement

away from the use of manually assigned classifications in indexing and retrieval. The next big step was made by Maron and

Kuhns [2], who provided an overview of the act and

method of information retrieval. In synthesizing

a new model of the process, they broke far from

previous restraints, especially in introducing "arith- metic (as opposed to logic alone) into the problem

of indexing." They also argued for the statistical analysis of the co-occurrences of index tags, a sig- nificant departure which has had great influence.

Their emphasis was on the retrieval of relevant documents, rather than on interpretation of the association measures obtained among the terms. Thus

they were quite careful in their discussion of index

space to point out that The distinction between semantical and statistical relationships may be clarified as follows: Whereas the semantical relationships are based solely on the meanings of the terms and hence independent of the "facts" described by those words, the statistical relationships between terms are based solely on the relative frequency

with which they appear and hence are based on the

nature of the facts described by the documents. Thus,

although there is nothing about the meaning of the term

"logic" which implies "switching theory," the nature

of the facts (viz., that truth-functional logic is widely used

for the analysis and synthesis of switching circuits) "causes" a statistical relationship. (Another example

might concern the terms "information theory" and "Shannon" — assuming, of course, that proper names

are used as index terms.) 5

This comment, indeed their whole discussion, is quite free of a hypothesis regarding the "association of ideas" — rather they point to the external

world as the explanatory mechanism for the statistical relationships discovered.

It remained for Stiles [3] to synthesize his uncompromisingly operational view of the problem. First,

he made the entire process automatic in his proposal

to begin directly with the text of documents, index

them automatically, perform co-occurrence analysis

of the words so selected from the text, and thus

obtain association measures defined from text. Though in practice he employed data from a co- ordinate index, he specifically included the possibility of text analysis by the same approach. Sec- ond, he dispensed with heuristics, and with this step Stiles went beyond his predecessors. He

introduced the important idea of using term profiles to obtain second-generation terms.6 Finally

he observed of this step that "It projects us beyond

the purely statistical relationships and into the

realm of meaningful associations. . . . Among these

second-generation terms we find words closely

related in meaning to the request terms."

Stiles thus formulated a process which has enor- mous implications. Starting with text, a completely formal process leads to relationships which

admit plausible interpretations in the domain of meaning. The computer, one need hardly state, does not interpret the data — they are uninterpreted

symbols. At one blow it puts "The Measurement

of Meaning" in an entirely new light.

3. Conclusion

Despite differences in motivation, emphasis, and

perspective, the two main avenues that have been

sketched very briefly in this paper have led, quite

independently, to very similar constructs for the

determination of meaningful measures of word as- sociation. Despite their similarity of technique,

different explanatory mechanisms are suggested

in each of the two traditions. On the one hand,

the association of ideas is regarded as a defensible rationale for the method, while on the other, it is the "nature of the facts" in the external world

which provides the "cause" of the statistical re- lationships observed.

The roots of each tradition are found in the

epistemological framework erected by the British

S P. 225.

• Cf. Harris [6].

empiricists. A historian would thus be expected

to regard the significance of the present effort not

in terms of its mechanized documentation objectives but in terms of the larger movements of which

it is a part. For while the two traditions from which

statistical association techniques have emerged

have tended to split over the value of using "ideas"

as explanatory constructs, steps have been taken

in both to replace the introspective method by a more quantifiable and objective technique. The

discovery that the indexing language of a retrieval system is perculiarly susceptible to scientific analysis is an important step. But perhaps more sig- nificant is the degree to which the two traditions,

in treating substantially the same data with substantially the same techniques, are finding a common

experimental ground after a long historical

separation.

7
4. References

[1] Luhn, H. P., A statistical approach to mechanized encoding and searching of literary information, IBM J. Res. and

Devel. 1, 309-317 (1957).

[2] Maron, M. E., and J. L. Kuhns, On relevance, probabilistic indexing and information retrieval, J. Assoc. Comp. Mach.

7, 216-244 (1960).

[3] Stiles, H. E., The association factor in information retrieval,

J. Assoc. Comp. Mach. 8, 271-279 (1961).

[4] Morris, C. W., Signs, Language and Behavior (PrenticeHall, New York, N.Y., 1946).

[5] Harris, Z. S., Discourse Analysis, Language 28,1-30(1952).

[6] Harris, Z. S., Distributional structure, Word 10, 146-162

(1954).

[7] Saporta, S., Psycholinguistics (Holt, Rinehart and Winston,

New York, N.Y., 1961).

[8] Osgood, C. E., The nature and measurement of meaning,

Psych. Bull. 49, 197-237 (1952).

[9] Langer, S., Philosophy in a New Key (Harvard Univ. Press, Cambridge, Mass., 1951).

[10] Bridgman, P., Logic of Modern Physics (Macmillan, New

York, N.Y., 1946).

[11] Hume, D., An enquiry concerning human understanding,

pp. 596-597, in E. A. Burtt, ed., The English Philosophers From Bacon to Mill (Random House, Inc., New York,

_N.Y., 1939).

[12] Warren, H. C, A History of the Association Psychology

(C. Scribner's Sons, New York, N.Y., 1921).

[131 Hartley, D.. Observations on Man (1749).

[14] Mosier, C. L., A psychometric study of meaning, J. Soc. Phsych. 13, 123-140 (1941).

[15] Thurstone, L. L., Multiple Factor Analysis (Univ. of Chicago

Press, Chicago, 111., 1947).

[16] Harman, H. H., Modern Factor Analysis (Univ. of Chicago

Press, Chicago, 111., 1960).

[17] Kent, G. H., and A. J. Rosanoff, A study of association in insanity, Am. J. Insanity 67, 37-96 (1910).

[18] Lasswell, N., N. Leites, et al., Language of Politics; Studies

in Quantitative Semantics (G. W. Stewart, New York,

N. Y., 1949).

[19] Berelson, B., Content Analysis in Communications Research (Free Press, Glencoe, 111., 1952).

[20] George, A. L., Propaganda Analysis (Row, Peterson & Co., White Plains, N.Y., 1959), pp. 29-30.

[21] Pool, I. S., ed., Trends in Content Analysis (Univ. of Illinois Press, Urbana, 111., 1959).

[22] Baldwin, A. L., Personal structure analysis: a statistical method for investigating the single personality, J. Abnor- mal and Social Psych. 37, 163-183 (1942).

[23] Osgood, C. E., G. Suci, and P. H. Tannenbaum, The Meas- urement of Meaning (Univ. of Illinois Press, Urbana, 111., 1957).

[24] Goodman, L., and W. Kruskal, Measures of association for cross-classification, J. Am. Stat. Assn. 49, 732-764;

. . . Further discussion and references, Mar. 1959, 124-163.

[25] Dirac, P. A. M., The Principles of Quantum Mechanics, 3d

ed. (Clarendon Press, Oxford, 1947).

8
Mechanized Documentation: The Logic Behind a Probabilistic

Interpretation

M. E. Maron *

The RAND Corporation

Santa Monica, Calif. The purpose of this paper is to look at the problem of document identification and retrieval from

a logical point of view and to show why the problem must be interpreted by means of probability concepts. We show why one must interpret the transition between a user's request for information and

the library's response as an inverse statistical inference. Furthermore, we show how a mechanized

library system can elaborate automatically upon and improve a given request, and why this requires

association techniques based on statistical as well as semantical relationships. The paper concludes

with some remarks indicating how these notions may be extended to put the problem of mechanized

documentation on an even firmer base.

1. Introductory Remarks

Mechanized documentation a few years ago occupied a relatively small sector of the computing field; however, it may well overshadow and perhaps even

dominate conventional numerical uses of computers.

This prediction may appear extravagant in view of

the fact that we have had larger, faster, more re- liable, and more flexible computing machines each

year since the publication of Vannevar Bush's

classic discussion in 1945 [l], 1 and yet the problems of mechanized documentation are still largely

unresolved. This suggests, of course, that the

problems of mechanized documentation do not

relate primarily to hardware — if they did, they

would doubtless be more tractable. They are

intellectual problems, and they have remained

unsolved because the proper framework within

which to view them has not been firmly constructed.

Perhaps one reason for this has to do with the fact that the technology was ready — and as a result we

had an information storage and searching machine

(the Rapid Selector) — before we were clear about

the logic and the strategy to be used in mechanized

searching. But a more basic reason that solutions

to our problems have eluded us thus far has to do

with the fact that our subject is very difficult because some of its key aspects are basically

epistemological, having to do with the activity of

knowing.

2. Communication, Information, and Language

2.1. Knowing and the Notion of an Internal

Model

In order to get at fundamentals, we must be clear about the function of a library; we have to be clear about the circumstances under which someone

would want to use a library. The simple answer,

of course, is that someone comes to the library

because he doesn't know something and wants to find out about it by reading the appropriate books.

So first of all we have to ask: What does it mean

to say that someone knows something?

For present purposes, we will equate one aspect

of knowing with having an internal model (sometimes called a "cognitive map") of the world, which,

in a sense, is consulted and which determines the

•Any views expressed in this paper are those of the author. They should not be

interpreted as reflecting the views of The RAND Corporation or the official opinion or

policy of any of its governmental or private research sponsors.

' Figures in brackets indicate the literature references on p. 13.

appropriate behavior in terms of knowing what to do and what to expect under various circumstances [10]. We receive information when our internal model

of the world is updated or changed. In fact, we

might say that information is that which changes

what we know; i.e., it modifies our internal model

[3, 4]. The amount of semantic information in a message could, in principle, be measured in terms

of the amount by which it changes the internal

model of the receiver [6]. It is important to recognize from these remarks

that information is not a stuff contained in books

as marbles might be contained in a bag -even

though we sometimes speak of it in that way.

It is, rather, a relationship. The impact of a given

message on an individual is relative to what he

already knows, and, of course, the same message

could convey different amounts of information to different receivers, depending on each one's

internal model or map.

772-957 O-66—

9
2.2. The Notion of a Question

When an individual, A, wants some part of his internal map updated, he may ask a question of

another individual, B. Notice, that there are dif- ferent aspects of the map that may stand in need

of updating — scope, depth, detail, etc. But, the

point is that A characterizes the gap in his map in the form of a question. B receives the question

and responds after consulting his own map. Hopefully, he responds by describing those facts re- quested by A. An important feature of this type of information

exchange is that unless A and B are already familiar

with the background, education, and experiences

of each other, the process of communication between them may require several cycles of iteration before B is quite sure of what A "really" wants,

relative to depth and detail, and therefore how the answer must be framed. This requires that B

incorporate within his model of the world some

representation of A's model of the world [5].

2.3. Interrogating a Library Computer

Suppose the individual consults a library computer

instead of another person to obtain information.

Since current computers cannot comprehend [2], they must be instructed (programmed) as to how to

manipulate incoming requests on the basis of a description of the form of the input request and

stored data. That is, in order to compensate for the fact that computers only manipulate the symbols on the basis of stored instructions, appropriate

procedures must be initiated in order to have a computer automate certain library tasks. In conventional library systems the procedures are as

follows: A human indexer reads the library documents and assigns the appropriate tags (this could

be mechanized and executed by the computer [8]). Conventionally, an indexer reads the documents

and assigns index tags according to his notion of

where each document would fit, relative to the maps

of the library users who will interrogate the system.

To what extent, however, can he anticipate the

needs of future users who might find the document

relevant? The second step in the operation of

conventional systems is that information needs of

the users are described in the form of a library

request — usually framed in the vocabulary of the

library indexing language and the grammar of truth- functional logical connectives. Given a request,

the machinery begins to grind, the computer

searches its store trying to match the description

of the need with descriptions of documents. A

document is considered relevant to a user's infor- mation need if there is an exact logical match or

if the document description implies the request

formulation.

3. The Fallacy of Conventional Indexing

We have argued elsewhere [9] that the conventional search strategy described above is based on

an invalid inference scheme, and that once the

logical fallacy behind such systems is unmasked, we will recognize why retrieval effectiveness is poor. The fallacy can be pointed out as follows: An

indexer in the process of deciding whether or not

to assign index tag Ij to document D considers the

following sentence S:

If document D satisfies the information

need of a library user, then he will describe

that need in terms of index tag Ij.

S is a conditional sentence of the form: "If X,

then F", where X= document D satisfies the infor- mation need, "and Y— index tag Ij describes the

user's information need. So we can schematize

the transition from a user's request to the library

response as follows:

If X, then Y

Y

Therefore, X

(The inference consists of two premises, one of

which is sentence S, the truth of which is not now in question.)

To say that an inference is invalid is to say that

it is possible for its premises to be true and conclusions be false. The above inference is clearly

fallacious. We cannot even assert that the premises confer a degree of partial truth on the conclusion. It is not surprising that retrieval effectiveness suffers when based on an invalid

search strategy.

4. The Need for a Probabilistic Interpretation

What is the probability that a document indexed

by a given description will satisfy the information

need of a user who has described his need in an

identical way? The probability may be high or

rather low depending, among other things, on the

richness and flexibility of the library indexing lan- guage. However, in a communication situation of the type described above, where information needs

10
\ are to be related to documents in terms of the impact of their "contents" on the cognitive map of the

receiver, one must use the language of probability

to represent properly the relationship between need

and description and also to schematize properly the

logic of the transition from input request to output

documents.

A document can be understood properly, for index purposes, only in terms of its impact on a per- son with an information need. That is, documents

and their users stand in a relationship to each other,

this relational aspect of the situation must be rec- ognized and made explicit when designing a search

strategy. Therefore, it can be argued that index

descriptions should not be viewed as properties of

documents: They function to relate documents and

users. The corollary to this is that the relationship be- tween a document and a user admits of degrees and

must be interpreted probabilistically.

Given an understanding of the logic of this sit- uation—namely, that an index tag for a given document can "characterize" to some degree — one is in a position to recognize the rationale behind weighted

index tags [7]. The weight of an index tag, /,-, relative to a given document, can be interpreted as an

estimate of the probability that if a user were to read the document in question and find it to satisfy

his information need, then he would have described

his need in terms of Ij. This is what an intelligent individual does in- tuitively in deciding how to index a document for

the purpose of information retrieval. (And in conventional systems he converts his intuitive estimate

of this probability to either 1 or 0, depending on

which extreme is closer to his intuitive estimate.)

If we want to construct a valid inference of the

type required by the transition from a given infor- mation request, R (consisting of some function of

index tags), to the library response, which is, we

suggest, an inverse probability inference, then the

inference must be schematized in terms of the

theorem of Bayes.

We would argue as follows: That the logic behind

valid mechanized documentation implies the rela- tional aspect of index tags, that the weights associated with index tags can be interpreted in terms of

probabilities,2 and that the transition between a user's request and a library response must be viewed

as an inverse probability inference. Given this understanding of the logic of the situation, one can

explicate a comparative concept of relevance as a

relationship between probabilities of the following

kind:

The probability that if a user describes

his need in terms of a request R, then

he will find that document Di satisfies that need.

From an operational point of view, if, for a given

request, one document would more probably satisfy a user's need than another document, then the

former document is more relevant to his need, relative to that request.

The interpretation of weighted index tags and

this explication of relevance provide the logical

and mathematical tools needed to compute what

have been called relevance numbers [7] in order to rank the output documents resulting from a re- quest. And this ranking (ordering) provides an

optimal strategy in going through the class of re- trieval documents.

5. Statistical Association Techniques

The fallacious logic on which conventional search

strategies have been based gives rise to two typical

symptoms of the logical illness: too many documents

are retrieved, many of which are of very low rele- vance; some of the really relevant documents are completely missed in the search.

The first problem is handled once we cast the

search in its logically correct form; i.e., probabilistically, as described above. When we do that,

low-relevance documents are ranked accordingly

and hence can be trimmed automatically from the

output list. The second and more serious problem grows out

of the fact that the document descriptions or the

requests are inadequate because they contain in- sufficient redundancy. But we know that redundancy can be added automatically by the use of

statistical association techniques.

How can one increase the probability of retrieving

'For mathematical details, see [7],

a class of documents that includes relevant material

not otherwise selected? One obvious method suggests itself: namely, to enlarge upon the initial request by using additional index terms which have

a similar or related meaning to those of the given

request.

An intelligent librarian can always help an in- dividual enlarge upon his request, but a central concern of this Conference relates to the process

of mechanizing this procedure. To do this one would need to program a computing machine to make a statistical analysis of index terms so that the machine will "know" which terms are most

closely associated with one another and can in- dicate the most probable direction in which a given

request should be enlarged.

In 1960 [7], three techniques were analyzed for elaborating in so-called "request space" and a technique for elaborating in so-called "document

space." The rationale behind these techniques was to avoid the problem of missing relevant docu1 I
merits in the search process by enlarging upon a request in the most probable direction; i.e., by

adding the proper kind of redundancy. This can

be done using statistical association techniques

The library computer not only collects the relevant

statistics, but is also programmed to reformulate

the input requests to increase the probability of

selecting relevant documents, as described above.

Even though a redundant request implies a larger

class of retrieval documents and threatens further

to aggravate the problem of retrieving too many

documents of low relevance, probabilistic indexing

techniques provide relevance numbers so that the

enlarged class may be ranked and trimmed.

To enlarge upon a request in the most probable

direction presupposes that we can justify our elab- oration techniques in the sense that we can show

how the use of statistical association techniques

does in fact increase the probability of selecting

relevant documents. Thus, it would be useful to strengthen the theories (which presently are not always clear) behind some of the current techniques

in order to provide logical justification for their

preference (over alternatives); i.e., to have some

measures of the goodness of alternative association

techniques.

6. Toward a More General Theory of Association Procedures

The relational nature of indexing suggests that

statistical association techniques might be extended

and refined so as to deal more adequately with a

library whose users have heterogeneous backgrounds. For such a library, the relationship of

being statistically associated with, which ordi- narily holds between pairs of index terms, could

be enlarged and be interpreted as a three-place

relationship.

If a library user, Ui (who might have a background in psychology), uses the same request index

tag, say Ij, as another user, Ut (who might have a background in physics), then this background in- formation should not be missed. Given a request

using tag Ij by a user of type 1, we find the h-(Ui)

which has the highest coefficient of association

(by some measure) relative to a user of type 1. And,

for the physicist (as opposed to a psychologist)

who also uses Ij, we find the hiU^) which is most

highly correlated with Ij, relative to user class of

type 2. The suggestion that statistical associations between index tags become three-place instead of

two-place relationships implies that we look upon

a request as composed of two parts:

(1) Request data proper; i.e., the description of

the user's information need — of the gap

in his map.

(2) Background data; i.e., the description of the

background of the user — the "texture" and terrain of his map.

Given these data, a computer could keep records

and learn that a user who describes himself in one

particular way most probably belongs to user class

1, whereas another individual who describes his background differently would probably belong to user class 2, etc. Just as a computer can be programmed to index

a document and decide the subject category to which it most probably belongs, so also a machine

could decide automatically to which class a user most probably belongs. Then there would be

separate and distinct correlation relationships

for each distinct class of users. This is not merely to suggest that by keeping a

"profile" of library users one could program a computer to disseminate automatically; but rather that

in order to respond more effectively — either for direct on-fine requests or for automatic dissemination—we need to recognize that at least some of the statistical association relationships that we

are trying to evaluate by various techniques are not two-place but are three-place relationships and,

therefore, that they require different methods for

| their estimation.

7. Concluding Remarks

Although in principle there is no reason that argues against the possibility of building an intel- ligent artifact which can truly comprehend language,

a solution to the library problem does not hinge on

such systems. If we make full use of human intel- ligence we can design an effective library computer.

A clear comprehension of the logic of the problem

can go a long way toward preventing false starts,

trivial experiments, and naive discussion. The

concepts of probability are required to properly

frame the logic of the problem because, basically,

the transition from a user's request to the resulting | retrieved documents must be schematized as an

inverse probability inference. Statistical asso- I ciation techniques are required because, like a good |. detective, the library computer must be designed

to use all the clues and inference techniques that j are available.

If we can think clearly about the logical problems

of mechanized documentation, the opportunities

offered by a fabulous computer technology can be

exploited to our great advantage.

12
8. References

[1] Bush, V., As we may think. The Atlantic Monthly 176,

101-108 (1945).

[2] Kochen, M., D. M. MacKay, M. Maron, M. Scriven, and L. Uhr, Computers and Comprehension (The RAND Corporation, RM-4065-PR, Apr. 1964).

[3] MacKay, D. M., Operational aspects of some fundamental

concepts of human communication, Synthese 9, 182-198

(1954).

[4] MacKay, D. M., The place of "meaning" in the theory of information, pp. 215-255 in E. C. Cherry, ed.. Information Theory (Butterworths, London, 1956).

[5] MacKay, D. M., The informational analysis of questions and commands, pp. 469-476 in E. C. Cherry, ed.. Information Theory (Butterworth, London, 1961).

[6] MacKay, D. M.. Communication and meaning— A functional approach, in Helen Livingston, ed., Cross-Cultural Understanding: Epistemology in Anthropology (Harper and Row, New York, N.Y., 1964).

[7] Maron, M. E., and J. L. Kuhns. On relevance, probabilistic indexing and information retrieval, J. Assoc. Comp. Mach.

7,216-244(1960).

[8] Maron, M. E.. Automatic indexing: an experimental inquiry,

J. Assoc. Comp. Mach. 8, 404-417 (1961).

[9] Maron. M. E., Probability and the library problem. Behavioral Sci. 8, 250-256 (1963).

[10] Maron, M. E., On Cybernetics, Information Processing, and Thinking (The RAND Corporation, P-2879, Mar.

1964).

L3

Some Compromises Between Word Grouping and Document Grouping

Lauren B. Doyle

System Development Corporation, Santa Monica, Calif. 90406

Statistical analysis of the text of document collections has yielded for information retrieval purposes two broad classes of output: word grouping and document grouping. Associative indexing comes under

the general heading of word grouping; automatic classification is a kind of document grouping. Doc- ument grouping and word grouping, however, can be combined to give a scheme of classification with more attractive features than could be achieved with either document grouping or word grouping alone. A hierarchical grouping program written by Joe H. Ward of Lackland Air Force Base for use in classifying personnel by skill and aptitude turns out to be nearly ideal as a basis for a mixed documentand-word grouping approach. The program will derive four- or five-level hierarchies from key-word

lists drawn from 100 documents, will position document numbers or other numbers in the smallest subcategories, and is capable with additional routines of extracting appropriate labels from the key- word lists to describe the categories at all levels of the hierarchy. Additionally, homograph separation occurs as a natural outcome of the program's operation.

1. Introduction

Information retrieval technology in the 1950's was based largely on principles of logic, 1 an emphasis which was perhaps a "logical" result of the

emphasis on use of computers in information re- trieval. Computers are (above all) logical. Then

a well-known logician [1]

2 said that logic was at least being grossly misapplied or at worst nearly

useless in the information retrieval field. Judging by the trend of interest in statistical approaches in general and associative indexing

in particular, the 1960's will see information re- rieval based more and more on principles of redundancy. This is more appropriate because, as we

are often so painfully aware, the literature is quite

redundant and not very logical.

Redundancy has the adverse connotations of

undue length and repetition. It is these very characteristics that make a statistical approach to text analysis and retrieval both feasible and desirable.

Undue length favors a statistical approach because

it increases the sample size, and needless to say

the world's technical literature is unduly sizable as a sample. Repetition, of course, gives us something to count, without which we would have no

statistics; but more important than that, selective repetition by authors can be a highly reliable clue

to topic, as recognized by H. P. Luhn [2].

2. Document Grouping

There seem to be two broad uses of redundancy

among those who try to employ it as a means of

automatically generating an organized structure by which we may have access to the literature; these are document grouping and word grouping.

Document grouping was the basis of library classification long before computers, and it is expectable that those of a statistical orientation would

try to duplicate by automatic means what the

librarian can do intellectually, because similarity of word content in a group of documents implies

similarity of topic. Of course, documents or ref- erences thereto (titles, etc.) can be grouped 3 in ways other than by word content similarity; as examples, permuted title indexing groups them

alphabetically, and citation indexing groups ac1 Mainly the principles of Boolean algebra.

- Figures in brackets indicate the literature references on p. 24.

3 "Grouped" in the loosest sense, which might mean '"ordered" or even "inter- connected."

cording to author-implanted cues. These ap- proaches to document grouping currently outrun

the statistical approach in popularity because,

among other things, they are cheaper; neither

method requires the entire text of an article to be

processed, or for additional intellectual work to be done other than that done by the author himself.

But we value the statistical approach in spite

of its current expense, not only because costs are rapidly declining and will result inevitably in feasible digital storage for entire documents, but also because it is a whole technology, whose applications to text analysis go beyond what we talk about

herein. As one example of that, statistics can be

shown to be a strong right arm for syntactic analysis

[3], and perhaps — eventually — for machine translation. This is so because the redundancy in text can manifest itself through the grouping of words,

as well as through the grouping of documents.

15
3. Word Grouping

In my own work I have been preoccupied with

word grouping [4]. Others, such as H. E. Stiles [5], have in effect used index-term grouping, which is equivalent to word grouping, as a basis for improving

the performance of literature-searching systems.

Words or terms can be grouped statistically as a re- sult of their high co-occurrence in the same documents as tags or key words; when co-occurrence is high, as measured by some statistic, we speak of

the co-occurring words as being strongly "associated." Both word grouping and document grouping can be seen to spring from the tendencies of

many words to co-occur strongly.

Developments in statistical word association are proceeding along two paths. The majority ap- proach is that of Stiles, which is a modified coordination indexing in which users formulate search

requests and in which the machine acts on those

requests in such a way that the retrieved documents

contain not only the words specified by the request,

but also words which are associated statistically to those in the request.

The second approach, which is still a rather small

minority, is that in which the computer is used to generate an "association map" as a printout or cathode ray tube display. The best way to visualize

the difference between these two approaches is in analogy to the difference between straight machine

searching of text and automatic indexing. In machine searching one makes a request, which is fed

into the machine as a criterion that the machine can

use in searching for relevant references. In auto- matic indexing, the machine is used not as a searching instrument but as an arranger of references

which can be scanned in printout form by the human

eye. In associative indexing, by analogy, the first approach involves user specification of what the

machine should look for and the second approach

MODERNIZE —

T-34-L

T-54<

1954—1

MOLOTOV

BLOC

SECURITY

ALLIANCE —

j—AUSTRIA INVITATION —J

GERMANY

I TROOP J

FORCES

SPRING WITHDRAWAL 1955 REDUCE UNITS

ANNUAL

FALL

i ROTATION MANEUVERS

WINTER

REGIME

TRUCK

TOLL NEGOTIATION

KHRUSHCHEV

ISOLATE

RETALIATION

HARASSMENT

SPY

ACCESS

BORDER- -QUADRIPARTITE

EAST BERLIN

U. S. A.

MILITARY

j

-POLAND

REFUGEE

UNIVERSITY- -STUDENT- -INTELLECTUALS

REPRESSIVE

RESISTANCE

TRAIN ING--RECRU IT

TENSION

ODER-NEISSE GOMULKA

HUNGARY- -RUMANIA

Figure 1. Association map.

If,
RECOGNITION KHRUSHEHEV NEGOTIATION BERLIN^

HARASSMENT ISOLATE

SPY

ACCESS

STUDENT,

UNIVERSITY, INTELLECTUAL

RESISTANCE

BLOC ODER- GOMULKA

NEISSE

FIGURE 2. Hierarchical association map.

generates a printout or display by which the user himself can search.

The analogy here might even extend to developmental history. Recent years have seen a shift away from machine searching toward automatic

indexing, especially permuted title indexing. We

might well be on the point of seeing a shift from machine associational searching to machine associative indexing. I am assuming so, and for this reason have habitually placed my eggs in that basket.

Association maps can take on a bewildering variety of forms. The forms with which I have be- come most familiar are shown in figures 1 and 2. Figure 1 is a map hand-drawn from computergenerated statistical co-occurrence data, and figure 2 is a "hierarchical map" generated from the same

text. Both of these forms were first discussed in 1961 [6] and both are capable of completely auto- matic generation from text. The map of figure 1 could be called a "raw association map," in that it faithfully reflects the most strongly co-occurring

word pairs in the corpus; the hierarchical map of

figure 2 sacrifices strong co-occurrences between

words of roughly equal frequency for the sake of

better organization. The hierarchical effect is achieved by discriminating against the relating

(i.e., linking) of words of more or less equal frequency and by relating words of high frequency 4 to words of lower frequencies in a cascade of cate- gories and subcategories, as shown; since the words

of high frequency apply to a larger number of documents, it follows that these would be used to label the larger categories. One can construct ad hoc

statistical functions by which one can bring about

the desired discrimination against co-occurrences between equally frequent words. The most ef- fective one I have found so far is:

(6/a-0.35)2 + 0.03'

where a = the value of the higher frequency, 6 = the

value of the lower frequency, and c = the frequency

of co-occurrence of words a and b. The numerator's purpose is to maximize F as documents with

tokens of word b as tags or key words approach 100

percent inclusion in the larger set of documents

having word a. The denominator maximizes F as the ratio of the two frequencies, a and b, approaches

0.35; such a function would thereupon favor hier- archies having on the average three subcategories

per category. The presence of the constant 0.03

in the denominator is to prevent the function from

approaching infinity.

4. Disadvantages of Pure Word or Document Grouping

The reason I now search for compromises between word grouping and document grouping is that I have become aware of certain disadvantages

of either approach used in a pure way. Pure document grouping, for example, suffers from two

weaknesses:

4 By "frequency," here, we mean "number of documents having this word or tag"

rather than "number of words." The author, in a previous article [4], has defined

this kind of frequency as "prevalence."

(1) There is no obvious clear-cut way to represent

the groups of documents for perusal by

literature searchers. Grouping of titles in correspondence to the document groups

is not entirely adequate because the similarities leading to group formation may not be evident, and because a flock of titles may

contain too much information to characterize
whole groups, leading to cognitive strain

for searchers who would like to inspect numerous groups.

(2) The organization of the groups themselves,

though potentially achievable automatically,

may not be representable in a scheme which

can be followed by a searcher.

These faults would not seem important to those

who take the viewpoint of Maron [7] and others,

which pictures "heuristics in document space"

as a means of machine retrieval of closely related

documents. These workers would not be inclined

to emphasize representation for search by the

human eye. Word grouping (association maps, hierarchical

maps) has three weaknesses as a pure approach:

(1) Since the basic idea of an index based on word groups is to find word clusters of

interest or pertinence, and to proceed

from such a cluster to references containing more information about the documents

whose co-occurring words caused the

cluster, it is important that word maps have

document numbers (or other indicators)

positioned properly on them. This proves

difficult to do reliably by automatic means.

(2) Homographs are a problem in word-grouping

techniques. Though statistical separation

of homographs has been shown feasible by

Stiles [8], it ordinarily would require an

additional statistical technique to be used

along with whatever is used for the word

grouping. We would like to find a statistical technique from which both word

grouping and homograph separation come

in natural consequence.

(3) Though word grouping (particularly the

"hierarchical map") suggests organization

of something, the literature searcher is given no sense of what it is that has been

organized. A map, in order for one to accept it as a meaningful entity, ought to be a map "of something." An organized

set of document clusters, if it can be repre- sented in a maplike way, would have much

more reality to a searcher because it would

be perceived as a map of the document

collection.

5. A Procedure Permitting Both Document and Word Grouping

I could not have expected that these grim doubts

about either document grouping or word grouping

could be cleared up by a single computer program

which was used in a field quite remote from document retrieval. However, early in 1963 an article by Ward and Hook [9] came to my attention which

described a hierarchical grouping procedure used

by the U.S. Air Force in grouping aptitude profiles for personnel assignment. I was fortunate enough

to obtain the corresponding Fortran II computer

program, which was implemented and run on our

Philco 2000. I used this program, in effect, as a document grouping program.

As a natural outgrowth, perhaps, of my preferred

orientation toward word grouping, I found that one

can superimpose a highly organized word pattern on the document grouping pattern which the pro- gram generates, and that this superimposed word

pattern not only describes the document groups, but also overcomes the three weaknesses of a "pure

word-grouping" approach.

I do not wish to discuss herein the mathematical

principles of the grouping program, which are de- scribed well enough in the Ward paper [9]. Adherents of the statistical approach spend much

time arguing among themselves as to whether this or that statistical technique is more appropriate,

but those who have a chance to compare them

[10] often find that the difference in output between

one technique and another is not appreciable. In- deed, even if one technique led to substantially

different output from that of another, it would be

hard to say that one result was right and the other

wrong. / have usually found that selection of

technique on purely mathematical grounds is ap- propriate only when there is full and complete

understanding of what the technique is supposed

to do; otherwise the only sensible thing to do is to base selection of technique on an after-the-fact ap- praisal of the utility and quality of output. When

there is no underlying theory of what it means that a word occurs in text once, twice, thrice, or n times,

it is only the naive who would apply "sophisticated"

statistical formulae. Insight, on the other hand,

might well lead to the choice of a completely ad hoc

statistic with no foundation in mathematical theory,

as in the case of the hierarchical map shown in

figure 2. Several runs of the Ward program were made,

each having 100 12-word lists as input. Each 12- word list can be regarded as a list of index tags or most-frequent content words of one document. The

output, then, can be viewed as the organization by

similarity of a 100-document library. Three runs

will be described herein, one on 100 lists corresponding to reports on German affairs, one on 100 lists corresponding to information retrieval papers, and

100 which include 50 fists each from German affairs and physics collections.

L8
6. Principle of Operation of Ward's Grouping Procedure

Before presenting the results of these computer

runs, it is desirable to give a nonmathematical description of how the program operates. Its objective is to form groups whose members have maximal

similarity to each other. In the runs described

above, it begins with 100 ungrouped lists, or, it would be better to say, 100 groups having one member each. Each program "pass" forms one group

of two members or more according to any of the

following three rules:

(1) Combine one list and another list to form a group of two lists.

(2) Add one list to a group of two or more lists. (3) Merge two groups of two or more lists. Note that never more than two entities (lists or groups of lists) are combined on a given pass; therefore any one pass diminishes the total number of

groups (remembering that we've designated un- grouped lists as "groups with one member only")

by one; and also, therefore, the total number of

passes must be n— 1 for a collection of n lists. In other words, the program accepts n lists as input,

forms a new group (in accordance with the rules

just given) in each of n — 1 passes, and on the (n—l)th pass forms one large group consisting of all n lists. There are of course a larger number of paths

which the program could follow to reach the all- inclusive group at the (n — l)th pass. For example,

for a collection of four lists two possible paths

exist if we think of the lists as indistinguishable:

(1) form two groups of two each, and merge these

i to form a group of four; or (2) form a group of two,

add a third, and add a fourth. When we introduce

y combinations, however, i.e., regard the lists as distinguishable and count all possible ways of combining them, we find that the program has 18 possible paths by which to achieve the final group of

four. On the first pass it can form any of six possible groups of two. On the second pass it can —

for each of the six possible pairs — do three things:

(1) group the two ungrouped lists, (2) add one of the

ungrouped lists to form a group of three, or (3) add

the other of the ungrouped lists to form a group of

three. On the third pass all roads lead to Rome, i.e., the final group of four. As the number of items to be grouped increases,

the number of possible paths the program is allowed

to take increases enormously. According to an

earlier report of Ward's [11], for a group of five there are 180 possible paths; for six, 2700; for seven, 56,700; and for eight, 1,587,600.

The essence of Ward's grouping procedure is

i [• ( nl)2 that out of the , *_,, possible paths for n items,

n(z" l)

it selects some one pathway which brings together

the items of greatest similarity the soonest. This

selection is not as difficult as it may sound, at first hearing. Each of the (n—1) iterations is involved

in selecting the total pathway, for on each program

pass a group is formed such that the following function is maximized:

F=A0(no— 1)— Ai(ni — 1)—Az{n2 — 1) — C.

In this function, no stands for the size of the group

which is a candidate for formation on a given pass. On the first pass n0 must equal 2. On later passes

the upper limit of n0 is the number of the pass plus one; the lower limit, however, is always 2 except on

the final pass, where no must equal n. The n\

and n2 are the sizes of the groups to be merged

on a given pass, and their values are restricted by

the relation n0 = «i + "2, with a lower limit of

+ 1 for either or both.

Ao, Ai, and A2 are the corresponding average

similarities for the groups, which we define as A = — tttt:, n in this case being the group

n{n — 1)1

size and x being some measure of the similarity of

two of the items (in the case of the word fists used

in this study, x was simply the number of words

which two lists have in common). The summation

of x is over all combinations of the n items taken two

at a time. C is an arbitrary constant usually set

at the maximum possible A value. The above function F acts in effect as a threshold,

being set at its highest achievable value at the beginning of the first pass, and "highest achievable

value" means here that only items which are identi- cal in all respects could be formed into groups.

If all n items of a collection were identical to each

other, the threshold F need never be lowered.

But in that case, of course, there would be no point

in forming groups.

In a typical collection of complex items no two

of which are identical, the program lowers the value

of threshold F until two items are found similar

enough to each other to constitute the "most

similar pair in the collection." After the first pair is formed, the role of F becomes more complicated—and correspondingly more difficult to describe. For a comprehensive mathematical explanation, one should consult the Ward article [9].

I have described the function to the extent I have

only for the benefit of those who might want to construct their own grouping algorithm without having

to decipher what in some cases might prove to be

unfamiliar mathematical notation.

It will suffice for the purposes of this paper to state that F's role is to select at any given pass that group which has the most satisfying blend of similarity and homogeneity. The Ward program contains an alternative mode in which groups are formed based solely on maximum average similarity; however, my experience with this mode has

convinced me that better classification is achieved

(for my material, at least) in the mode which

maximizes F, rather than average similarity of the

next-to-be-formed group (i.e., Ao). Close scrutiny

of F will show the reader that a candidate for group

formation is penalized to the extent that the average

19
similarity of the new group differs from the average

similarities of the component groups. This has the

result that on many passes groups are formed whose

Ao values are substantially less than the maximum

possible on those passes. To put the action of this mode of the program in sociological terms, it tends

to "group the nonconformists" rather than to parcel

them as individuals into the tightly knit groups

of high average similarity.

The practical significance of the grouping

procedure described can be better understood if we think about the problems involved in grouping

common objects in terms of their attributes. Suppose, for example, that we apply the three rules given at the beginning of this section to forming

groups from four objects: a plum, a walnut, a flower

pot, and a jar of mustard. Without splitting too many hairs on the question of specifying their attributes, it might seem reasonable to group the

walnut and the plum first because they are both

small, edible, tree-grown objects; furthermore even

without a knowledge of biology, we suspect that they have many more things in common that we

could perceive with the eye. The next question is what to do on the second

pass. There are three things which can be done.

One (grouping the flower pot with the plum and the

walnut) appears unreasonable, since the flower pot

has practically nothing in common with either of

the other two. The jar of mustard, however, can

either be grouped with the flower pot (because it is a non-metallic container which just happens to contain mustard), or it can be grouped with the walnut

and the plum (because it has the common quality

with them of being partly edible — the edible part

being likewise derived from vegetable sources

primarily).

Which of the above two choices we would

want to make would depend on which attributes are of greatest interest to us. For example, if we

were running a store we would unquestionably

want to group the edibles, whereas it we were

in the transportation business we would tend to group jars of mustard with flower pots because they

present fewer problems in handling than the perishable walnuts and plums.

Coming now to the world of document retrieval,

how would we want to group books about walnuts,

plums, flower pots, and jars of mustard? Of

course, a lot depends here on the aspects of these

four subjects which are being discussed — for ex- ample, plums can be discussed as crops or as plants

(under biology or botany). It is to be noted, however, that since jars of mustard and flower pots

are finished products, it is somewhat more difficult to think of any book which might treat them in a

scientific (i.e., natural science) light, whereas any

book "all about walnuts" or "all about plums"

would of necessity have to begin with a biological

discussion. From a librarian's viewpoint, then, it might be logical to group a book "all about jars

5 With some assistance from the Encyclopedia Britannica.

of mustard" with similar books under the topic

"manufacturing." A book all about flower pots

would probably also be found under the "manufacturing" heading, though not specifically in the area of food processing.

Fortunately, in the area of statistical methods of

classification, we do not (yet) have to worry about

such hard intellectual choices as the above librarian might have to make; at this point we have nothing

better than the simple and somewhat comfortable

hypothesis that documents containing similar quantities of roughly the same words must be on roughly

the same topic. This makes it quite easy for us to decide how we want things to be grouped.

In particular, it was easy for me to decide by

what criteria I wish to group the 12-word lists (described above) — group lists according to the - number of words held in common. Let us assume

that, based on a word count of books about walnuts, j plums, flower pots, and jars of mustard, I have

derived 5 the following 12-word fists:

(1) Walnut (2) Plum (3) Clay (4) Mustard

Tree Fruit Plant Seed Nut Species Pot Bottling Hull Tree Mold Blend

Species Plant Fire Spice

Wood Color Pottery Vinegar

Shell Grow Dry Process Lumber Blossom Color Flour Kernel Soil Heat Spread

Black Prune Horticulture Flavor Crops Pit Home Sandwich

Soil Hybrid Flower Sharpness

One now notes that lists (1) and (2) have three

words in common ("tree," "soil," and "species"),

and that lists (2) and (3) have two words in common | ("plant" and "color"). List (4) has no words in common with any of the others. The outcome of our grouping procedure would be

that the first program pass would group fists (1) and (2). The second pass has no choice but to put | list (3) in with (1) and (2), since each of the other

two grouping possibilities would involve list (4), which has nothing in common with any other list. Note that grouping on the basis of "words in common" gives us a grouping which we have already

decided (above) was unreasonable on intuitive grounds, namely, to group flower pot with plum

and walnut. These sample word lists were fabri- cated deliberately not just to illustrate the basic

principle by which the lists are grouped, but also

to illustrate the apparent weaknesses of the method.

We enumerate and discuss these apparent weaknesses in terms of the above sample lists: A. Word choices can accidentally relate documents on dissimilar topics. Let us suppose that word list (2) had the word "flower" rather than

"blossom," and that (with somewhat greater emphasis on the production of prunes) the word "dry"

appeared on the list. We would now have the situa- tion in which lists (2) and (3) would have four words

in common, leading to the most unlikely initial grouping of all — plum and flower pot. Can we

20
permit such subtle shifts in vocabulary and emphasis to have such drastic effects on the outcome

of the classification? As we shall eventually see, such inappropriate groupings become less and less

likely as (1) the size of the document collection

increases, (2) the topical spectrum narrows, and (3) the amount of information (about each document)

which is used in grouping is enlarged — i.e., list length is increased.

B. Ties in number of words in common can lead

to instability. Let us assume that lists (2) and

3) were to have three words in common. Now

there is a tie between fists (1) and (3) in how similar

they are to fist (2). In such a case which group

would be formed first, (2) and (3), or (1) and (2)? Since a computer program, unless suitable provision were made, would have no way to decide this issue

except through comparison of similarity as we

have defined it, a typical program would simply

choose the first pair inspected. In other words, we can affect the program's classification simply

by physically rearranging the order in which the

fists are input. Such instabilities have actually

been observed in the computer runs to be described

in this paper, but it is not at all clear that this instability is related in any way to the quality or use- fulness of the output. We are perhaps uncomfortable with the thought that such instability could

lead to many alternative classifications, and that somehow there ought to be only one organization

inherent in the document collection. It remains to be seen whether such a viewpoint is really necessary. C. Raw lists of words omit semantic information

which ought to affect the classification. Two important kinds of information omitted would be ho- mography-resolving information and relationship

indicators (showing which words on a list are related

to each other and how). An example of both

imagined deficiencies is found in the word "plant.'*

On fist (2) the word in relation to plums actually

refers to a verb "to plant." On list (3) the word is a noun, describing what the flower pot is to contain,

although as far as the information given on the list is concerned, it could be referring to a "plant which

manufactures pottery." It could even have both

usages in the text of the parent document. The

answers to these arguments (tentative answers,

admittedly) are that statistical separation of homographs has been shown to occur [8, 12], and that relationship indicators — however useful they might

be to a user consulting a classification scheme — do

not contribute enough information to affect the

outcome of the classification significantly. From

an information theory viewpoint, the bulk of the

informational bits are contributed by the choices

of the words themselves.

7. Automatic Assignment of Labels to Groups

Four sample word fists have been used in showing

the most elementary of the principles of the Ward

grouping procedure, as well as the most apparent

of its possible deficiencies as applied to grouping

of word fists. Given that appropriate groups can

be formed by such a program, what more can be

done? One question is: if we can derive a classi- fication through such statistical procedures, can we

also derive labels for the various groups? The

answer is that we can, and the mechanism is shown

in figure 3. Six objects are pictured along with their

six corresponding attribute fists. The purpose

of the diagram is to illustrate that words can be

drawn automatically from the attribute lists to give

adequate descriptions of the groups, i.e., to describe

which common attributes have been most influenMETALLIC (5/6)

POINTED, CYLINDRICAL

NAIL

LONG

CYLINDRICAL POINTED HEADED SMOOTH

METALLIC

LONG, HINGED CIRCULAR HEADED

SAFETY BELT POKER CIRCULAR SCREW

LONG

CYLINDRICAL POINTED HEADED GROOVED

METALLIC

PIN BUCKLE CHIP

HINGED

POINTED

METALLIC CYLINDRICAL U-SHAPED

EYED

HINGED

FLAT METALLIC CYLINDRICAL POINTED RECTANGULAR BALANCED

FLAT CIRCULAR

PLASTIC GROOVED

SYMBOLIC

CAM

FLAT CIRCULAR

METALLIC SMOOTH

EYED UNBALANCED

CATEGORIES BASED ON COMMON ATTRIBUTES

FIGURE 3. Derivation of category labels.

21
tial in leading to the formation of each group. The groups of figure 3 were derived via the same

considerations of list similarity that we have al- ready used. The first program pass groups "nail"

and "screw," whose lists have five common attri- butes. On subsequent passes we must lower

threshold F to permit the formation of groups of more and more dissimilarity and heterogeneity.

On the second pass "safety pin" and "belt buckle,"

having four attributes in common, are combined.

On the third pass different things happen, de- pending on whether one uses the maximum-F or the maximumvlo mode of Ward's program. Since

I have chosen to use the maximum-F mode,

I shall discuss it in those terms. "Poker chip"

and "circular cam," having only 2 attributes in common, are paired, whereas the formation of the

group of four consisting of "nail," "screw," "safety

pin," and "belt buckle," with an average of 3.5 attributes in common, is delayed till the fourth

pass; the penalty for reduction of homogeneity

which formation of the group entails outweighs its lead in average similarity, as may be seen by calculating and comparing values of F for the possible

groupings on the third pass. The fifth pass has

only one choice, formation of the final group of

six.After the groups are formed, by what rules can we

assign labels? Ideally, for any group we would

like to select a label which described all and only

the members of that group. Our first-formed pair,

"nail" and "screw," have the attributes "long"

and "headed" which apply to them alone. Each

of the other groups of two have at least one such

attribute. (In deciding how to specify attributes, I arbitrarily distinguished between "cylindrical"

and "circular" so that the former could be used to pertain to cross section of structural members and

the latter to pertain to gross form.) The group of

four has two attributes "pointed" and "cylindrical"

present on all four lists, but not present elsewhere.

As we ascend upward in the hierarchy, we find some tendency for the attributes to be used up as

labels for the smallest categories. There is no

attribute, therefore, which perfectly describes the

group as a whole. The closest we can come to perfection is "metallic," which describes five out

of six of the objects. If the number of objects is increased to the point that five or six levels are generated in the hierarchy, we must either increase

the number of attributes per object or else accept

group descriptors which do not apply to every group

member, or which apply to objects which are not

part of the group. Figure 4 shows a closeup view

of the grouping pattern involving seven out of the

100 12-word lists of German affairs, and even though

each of the corresponding reports might be said to have "12 attributes," there are still not many satis- factory choices of labels. The only "perfect de- scriptor" in figure 4 is the word "toll," which

describes the three members of that group and no

outside member.

The notation alongside each label specifies to what extent if any the label is not a perfect descriptor

of the group. Thus, "allied" describes only 5 out

of 8 of the lists in that group (one member of which

is not shown), and also describes an additional list at some remote location in the hierarchy; the total number of "allied" tokens is outside of the parentheses, and the fraction of lists described by "allied"

is within the parentheses.

EAST GERMANY (33/37)

® GROUP LABEL ALLIED (5/8),

'15

BONN (4/6), .

1

W. BERLIN TOLL

(3/3), (3/3) RETALIATION 1

I TRUCK

(2/2) 4 1 (2/2

>3 1

31 #95 »23 #24 *87

p/2)

•75 ®W. BERLIN ©RETALIATION W. GERMANY

REARMAMENT W. EUROPE ©ALLIED

APRIL ®E. GERMANY

CHARGE

SPY THREATEN BLOCKADE

}E. GERMANY

REGIME

HARASS

TRAFFIC

& ALLIED ACCESS 9W. BERLIN RESISTANCE CONTROL

^RETALIATION W. GERMANY

5 BONN

BUNDESTAG

W. GERMANY

PROTEST

(ALLIED INTELLIGENCE INTERROGATION

REFUGEE

IW. BERLIN GERMANY

(BONN

OPPOSITION

U.S.A.

SOVIET COMMISSIONER

PUSHKIN ®E. GERMANY

STRUCK

®TOLL

W. GERMANY

DE FACTO RECOGNITION

POSITION

RETALIATION EMBARGO

W. GERMANY

GOVERNMENT

TRANSPORT i MINISTRY l NEGOTIATION ®TRUCK l ®TOLL

®E. GERMANY

DEMAND ®BONN

RECOGNITION

BERLIN

BERLIN BARGE

I TOLL

IE. GERMANY

RETALIATION BONN

SHIPMENT

STEEL NEGOTIATION W. BERLIN WATERWAY GERMANY

(BORDER CONTROL

E. BERLIN CURRENCY CUSTOMS

VIOLATE

I QUADRIPARTITE

BERLIN (ALLIED MOVEMENT

GERMANY

COMMUNIST

FIGURE 4. Extent to which lists contain group labels.

22
SOVIET

EAST GERMANY

WEST BRITAIN, GERMANY FRANCE

COMMUNIST L-rEUROPE WEST POLAND, FORCES, GERMANY EAST GERMANY

GERMANY

EAST ULBRICHT,

BERLIN OPPOSITION

MILITARY, TRAINING

REGIME

ALLIED

KHRUSHCHEV

AUSTRIA

,_L ALLIANCE

WEST, REARMAMENT

HUNGARY

RUMANIA

I T-54,

1954 TROOPS

ODER- FALL ' NE | SSE WINTER

1 WITHDRAW,

REDUCE CONTROL

I QUADRIPARTITE BORDER

SOVIET

J~~l U.S A.

POLICY

BERLIN

INVITATION, RAAB SATELLITE r—1

, SECURITY

'

,

REFUGEE

BONN

W. BERLIN, RETALIATION TRUCK

TOLL

CURRENCY

WEST

EUROPE

TROOPS LOAN GOVERNMENT

DEFENSE NEGOTIATION

I ECONOMIC

NUCLEAR

PROJECT, OBSTACLE

BUNDESTAG

FORCES

TREATY GOVERNMENT

TRADE

MIDDLE

EAST

AGRICULTURE REPRESSION CZECH OFFICIAL

WARSAW, NATO

YUGOSLAV,

DECLARATION

HARASS, WEST

BERLIN STUDENT, CHURCH,

INTELLECTUAL, EVANGELICAL

UNIVERSITY

SOVIET ADENAUER PARIS

I SOCIAL DEMOCRATIC

1 COALITION

ELECTION

TREATY, SAAR

RATIFICATION, ACCORD

FIGURE 5. Classification scheme for 100 articles based on application of Ward's grouping program.

As can be seen in figure 5, which shows the hier- archy 6 for all 100 reports, there are in general four

or five levels; this accounts for part of the difficulty.

But also, however, there is less similarity — on the

average — between these lists, even with 12 attri- butes, than there is between the lists in figure 3. From a pragmatic viewpoint, a reasonable degree

of imperfection of description may not be a serious

deficiency. As is well understood in the document

retrieval field, there are explicit index tags for a document and there are implicit tags — tags which

might well have been chosen to describe the document but which were not. Implicit descriptors,

unfortunately, are one reason why relevant documents are missed in a search, and this is why people

are so interested these days in associative indexing.

Thus, though the word "allied" pertains to only five out of eight documents in its group, one can sense

that for the documents not tagged by "allied,"

which — as is seen from figure 4 — are about the

various tensions involving East Germany and Berlin,

it is reasonable to regard "allied" as one of the implicit tags for those documents. That we should

retrieve documents which are relevant to the term

"allied," but which do not actually bear the term

as a tag, is the whole point of associative indexing.

We must take care, of course, not to stretch the

"implicit tag" viewpoint too far. The other kind of labeling imperfection — that a given tag describes members outside of the group

as well as in it — is even less serious, and in fact may be regarded as not an imperfection at all under

conditions of adequate system design. In figure 5

8 The smallest shown categories generally contain two or three — seldom more than

four — lists.

some words, such as "Soviet," describe several

categories and subcategories in different parts of

the hierarchy; an alphabetical index of the hier- archy's label can permit a thorough search of

groups described by "Soviet," if such is desired,

and could even reference individual documents.

It is in this multiple usage of the same word as a

label that we find the homograph-separation power

of the Ward grouping procedure. In the third of

the three computer runs enumerated earlier, 50

lists in the field of physics and 50 in the field of

German affairs were pooled as input to the program.

In each field there was substantial usage of the

words "satellite" and "force," which are homographs in the true sense of the word as we proceed

from the one field to the other. For "satellite"

all of the German affairs items used the word to mean

"vassal state of the U.S.S.R." All of the physics

items used it to mean "manmade earth-circling

object." The Ward program not only yielded a perfect separation of reports containing the variant

meanings of both "satellite" and "force," but also began the 99th pass with two groups of 50 each —

pure physics and pure nonphysics.

When one peruses the similarity matrix for all of

the lists, however, the clean-cut separation of the

two subjects hardly seems miraculous. That half of the matrix which describes similarities between

individual physics documents and individual German affairs documents contains mostly zeroes. There is a small percentage of document pairs

having a similarity of one. When these are looked

up, they turn out to be tagged by either "force" or

"satellite." So there is nothing mysterious about

statistical separation of homographs. The reports

containing the word "force" in the physical sense,

23
also just naturally have words in common like "nucleus," "electron," "magnetic," "field," and

"charge," and are therefore just as naturally grouped

together by the Ward procedure.

The results of the second run — on 100 lists corre- sponding to documents in information retrieval —

were not so satisfactory as the results for the

German reports or for the mixed library just

described, chiefly because no words adequately

described the largest categories (as in the case of

the four major categories of figure 5). This result

is expectable whenever the subject matter in a document collection is too diverse. Another reason

for dissatisfaction is vocabulary. A typical structure from the information retrieval hierarchy

is:

INDEX

Word Search

entry document system language

begin order retrieval abbreviate artificial symbol

I . f

1 generation , I

I hnghsh property Alongside of hierarchies containing such crisp

words as "Bundestag," "troops," "Khrushchev,"

"Hungary," and "rearmament," structures such

as the above would not seem to shed much fight on

the organization of the literature in the information

retrieval field. I have often contended that the

greatest difficulty in retrieving information will be

found in information retrieval's own documentation.

Nevertheless, even in an area as semantically fuzzy

as information retrieval, there is great reason for optimism if statistically processed material is touched up with an appropriate amount . of postediting [13]. Earlier in this paper we fisted five weaknesses

of pure word grouping and pure document grouping.

It may be evident after the subsequent discussion

that the Ward grouping procedure is one approach

which, with further development, offers great

promise of overcoming these weaknesses. It permits:

(1) Terse and reasonably accurate labeling of

groups of all sizes.

(2) Intricate and meaningful organization of

groups in relation to each other.

(3) Optimum positioning of references to indi- vidual documents in a network of descriptive words.

(4) Homograph separation and aspect coordination 7 as natural outcomes of the grouping and

labeling procedures.

(5) A scheme or map which is more easily comprehensible as a result of being analogous to something which is — or could be — a physical

arrangement of objects.

8. References

[1] Bar-Hillel, Y., Some theoretical aspects of the mechanization of literature searching, U.S. Office of Naval Research

Tech. Rept. 3 (Washington, D.C., Apr. 1960).

[2] Luhn, H. P., A statistical approach to mechanized encoding and searching of literary information, IBM J., 309-317

(1957).

[3] Doyle, L. B., The microstatics of text, Information Storage and Retrieval 1, 189-214 (Nov. 1963).

[4] Doyle, L. B., Indexing and abstracting by association. Am.

Documentation 13,378-390 (Oct. 1962).

[5] Stiles, H. E., The association factor in information retrieval,

J. Assoc. Comp. Mach. 8, 271-279 (Apr. 1961).

[6] Doyle, L. B., Semantic road maps for literature searchers,

J. Assoc. Comp. Mach. 8, 553-578 (Oct. 1961).

[7] Maron, M. E., and J. L. Kuhns, On relevance, probabilistic indexing and information retrieval, J. Assoc. Comp.

Mach. 7, 216-244 (July 1960).

[8] Stiles, H. E., Progress in the use of the association factor in

information retrieval, unpublished memorandum, Nov.

15, 1962.

[9] Ward, J. H., Jr., and M. E. Hook, Application of a hierarchi- cal grouping procedure to a problem of grouping profiles, Educ. and Psych. Measurement 23, 69-82 (Spring

1963).

[10] Borko, H., and M. D. Bernick, Automatic document classification: Part II. Additional experiments, J. Assoc. Comp. Mach. 11, (Apr. 1964).

[11] Ward, J. H., Jr., Hierarchical grouping to maximize payoff, WADD-TN-61-29 (Wright Air Development Division, Air Research and Development Command, USAF, Lackland Air Force Base, Texas, Mar. 1961).

[12] Doyle, L. B., Statistical semantics, Information processing

1962 (Proc. IFIP Congr., Munich, 1962), pp. 335-336

(North Holland Publ. Co., Amsterdam, The Netherlands,

1963).

[13] Doyle, L. B., Expanding the editing function in language

data processing (to be published).

7 An alphabetical list of words could easily be generated as part of any system in- volving the method discussed. Coordination of one term with another (i.e., the Boolean "and") can be incorporated into the system also, through the hierarchy itself or (more marginally) through document number postings in the alphabetical list.

24
The Interpretation of Word Associations*

Vincent E. Giuliano

Arthur D. Little, Inc. Cambridge, Mass. 02140

It is argued that it is possible to measure at least two kinds of word associations: "synonymy"

associations, which relate words according to likeness of meaning, and "contiguity" associations, which relate words according to probable relationships among their physical designata. Formulas

which measure both types of association are developed for content analysis and automatic abstracting. This paper is concerned with possible linguistic interpretations of such word association measures.

1. Introduction

Several of the papers presented at this Conference describe experiments involving the application of machine-computed association measures

to solutions of practical problems of documentation;

such experiments have also been discussed in the

previous literature [1, 2, 3, 4, 10, ll]. 1 This

paper is concerned with the interpretation of as- sociation measures which relate words to other

words. In previous publications it has been mentioned that it may be possible to measure at least two kinds of semantic associations among words,

"contiguity association" and "synonymy association" [3, 4, 5]. Procedures for measuring the two

kinds of associations are discussed more thoroughly

in the present paper. Some investigators have dealt with words auto- matically selected out of unedited running text, others with index terms manually assigned to documents, others yet with contexts which are abstracts,

extracts, or other documents. However, despite

the differences in the types of vocabulary or context, many of the techniques used for computing

associations are basically similar [4, 12]. Almost

all of the techniques deal with words and contexts

as fundamental units. However, depending on

the objectives and inclinations of individual re- searchers, a word may be of a particular kind, for example, a Uniterm, a descriptor, an index term,

a key term, etc. Likewise, depending on the application of interest, a context may be a document,

the index set of a document, an abstract, a para- graph, a sentence, a phrase, a pair of contiguous

words, etc. The discussion given here is meant to comprise all cases where the units being associated

are drawn from the vocabulary of natural language.

However, the discussion is specifically phrased in

terms of perhaps the most difficult situation — that which exists when the given raw material is running

text and when there are no well-defined criteria for either isolating a vocabulary subset or for selecting

units of context.

In dealing with natural language text using a computing machine within the context of a documentation application, semantics is often of para- mount importance — in short, it is desirable to have means for dealing by machine With the meanings

of words. Basically, one has two choices of strategy

available. On the one hand, one may proceed

initially to think about and write down certain

relationships among words which are felt to be

present within natural language and of importance

in relating meanings; on the other hand, one can

look for such relationships directly within a large

body of text at hand. Following the first kind of

strategy, the a priori route, many investigators have

attempted to model the manner in which words are related semantically by directly creating a the- saurus—simply by writing down relationships of

word meaning which seem to be relevant. These

association patterns can then be encoded for subsequent computer usage. Several of us at this meeting have taken the

second viewpoint — that perhaps the most relevant

relationships of meaning pertinent to the auto- matic processing of a text are inferable from the

way the words are set down in the text itself. This

second kind of approach must necessarily be based on certain observations and assumptions about the nature of word relationships which can be measured

statistically, and I would like to review a few of

these assumptions here.

2. Some Observations and Assumptions

First of all, it may be observed that natural

language is used to encode and transmit ideas with

*This work has been supported in part by the Decision Sciences Laboratory ESD,

U.S. Air Force Systems Command under contract No. AF19(628)-3311. ESD-TDR64-527.

1 Figures in brackets indicate the literature references on p. 32.

fairly high fidelity — that a sufficiently large and comprehensive sample of natural language text can

contain within it a useful representation of the most

germane conceptual relationships employed within

a given area of discourse. Naturally, the way in which conceptual relations are represented in text need not at all be in any simple correspondence to

772-957 O-66—

25
the way in which they are represented in human

minds, let alone in correspondence with the way

objects actually relate to each other in the real world. 2 I wish merely to assert that to a proper

decoding device (i.e.. an educated human being)

a body of text oi proper size and composition can

be decoded in such a manner as to reveal conceptual

relationships unknown previously to the decoder.

The text may in some cases offer a fairly complete

representation of the concepts and conceptual

relationships apphcable within some areas of

discourse.

A second observation of significance is that con- ceptual relationships are encoded at least in part

by means of the word order and proximity relation- ships present in text. That is, conveyance of

conceptual relationships depends not only on the

words used, but also crucially on the order in which

these words are set down in text. To justily the interpretation of statistically computed word association patterns as having semantic

significance, it is necessary to go somewhat further

and to assume that the word order and proximity

relationships in text are often the primary vehicle

by means of which conceptual relationships are encoded. The validity of this assumption is in part self-evident, but still it must be taken as a hypothesis whose range of validity is to be established by experiment.

There are in fact at least three ways to view a body

of natural language text and, correspondingly,

three ways to view association measures computed

with respect to that body. The text can be viewed

as a closed formal system which represents only

itself. In this case computed association measures

are descriptive rather than predictive statistics. The same formula applied twice yields the same

results, and therefore one can argue about the importance of an association statistic, but hardly

about its value. Secondly, one can view a body of

text as representing a much larger corpus of text,

in the sense of being a sample of that larger body

of text. Thus, for example, the text of a Sunday's

New York Times can be viewed as a sample of what

might be expected in a whole year's worth of the

Sunday issue of the same publication. Taking

this viewpoint, certain of the statistics descriptive

of the sample can be expected to have a predictive

value; they can be used to infer patterns likely to be present in the larger population. In this case

it becomes meaningful to ask questions relating to

3. Contiguil

The basic hypothesis to be considered first is that contiguity association can, under appropriate

circumstances (to be examined shortly), be meas2

It must be recognized that such relationships can be viewed two ways, corresponding to two distinct philosophical viewpoints. On the one hand, one can hold that the

relationships of interest appertain among actual physical objects. On the other hand, one can hold that the only meaningful relationships are among conceptual repre- sentations of objects. This point is treated further in the paper by Paul Jones pre- sented at this Symposium [131-

-1 Comments apropos to this topic may be found in the paper presented at this Conference by Maron [14].

sampling, i.e., how well does the corpus represent

the parent population?

Thirdly, a text can be regarded as representing an encoding of concepts and of conceptual relation- ships which are of importance to some area of dis- course. Computed association measures are then viewed as being correlates of actual relation- ships which exist among the concepts which are the designata of language expressions — this is the

viewpoint taken in this paper. Moreover, to the

extent that practical applications of documentation

require recognition of semantic relationships,

the utility of computed word associations depends

largely on this third kind of interpretation.3

I would like to advance the hypothesis that it is possible to obtain at least two types of measure- ments from text which are under certain conditions

interpretable as applying to relationships among the

designata of words. The first type of association measure reflects what has long been called contiguity association by psychologists [13]. Roughly

speaking, two words are considered to be contigunyassociated if the objects or properties denoted by

them are contiguous (have to do with one another)

in the real world (or, depending on one's philosophical viewpoint, in man's conceptualization of the

real world). Thus "hammer" and "tack" are re- lated in the contiguity sense; so are "hand" and

"glove."

The connection between "liquid oxygen" and

"rocket fuel" is a contiguity one. Strictly speaking,

liquid oxygen is not actually rocket fuel, but is commonly used along with the fuel to enable proper

combustion. "Subway" and "station" are also contiguity-related, as are "syndicate" and "crime."

Contiguity associations need therefore not be logical

in any well-defined sense; they include part-whole

relations, partial synonymy, cause-effect relations,

etc. They frequently are indicative of what documentationalists call facets of words.

The second type of association to be discussed

might be called synonymy association. Two words

may be regarded to be synonymy-associated

(i.e., synonymous) to the extent that they are com- monly used to denote the same thing (concept, i object, or property).

The position taken in this paper is that under certain conditions measurements which reflect these

two specific relations of meaning, contiguity and

synonymy, can be based upon counting procedures

applied to words and word pairs found within text. Association

ured in terms of the statistics of co-occurrences of

words within context of text. For example, if "air- craft" and "pilot" co-occur with a frequency more

than is plausibly explainable on the basis of chance

alone, it may therefore be inferred that these co1 occurrences are not due to chance, but due to the

fact that the words are contiguity-related, i.e., that concepts designated by "pilot" and "aircraft" in fact have to do with one another.

26
It should be recognized that there are in fact two

interrelated assumptions involved here: the first is that it must be possible to identify contexts in which

word co-occurrences reflect contiguity relationships,

and the second assumption is that an adequate

statistical procedure can be found for combining

observations made from many different contexts. Experimentally, these assumptions seem to be

valid. In fact, one of the problems facing any

researcher in the area is that there appear to be

many different (at least different on the surface)

ways for selecting contexts and measuring contiguity association — and all of them seem more or

less to work.

First of all, there is the question of what consti- tutes a proper context of co-occurrence. Ideally,

such a context would be a natural unit readily

isolable out of text which has the property that every

word within it is contiguity-related to every other

word within it. When running text is given, the

situation offers considerable choice.4 The context "ships have decks" can certainly be

said to contiguity-relate the two substantive words

within it, while the co-occurrence of two words

within the whole of the text of the Encyclopedia

Britannica should surprise no one. Proximity in running text therefore seems generally to be re- quired for a contiguity relationship to be asserted.

However, proximity does not guarantee the presence

of a direct and meaningful contiguity relationship.

Consider the sentence "The contract providing for the delivery of the concrete required to build the west sluice of the dam was signed in red ink

yesterday." The sluice of the dam was not signed

in red ink, but the contract was!

Despite sentences like that just illustrated, and

despite a large number of other readily constructable counter-examples,5

it is fair to assume that substantive words located together or in close proximity in text are in most cases contiguity-related by

the context. It is not absurd, as a matter of fact,

to hold that any sentence or other coherent passage

asserts some contiguity relationship or the other

(perhaps a complicated or indirect one) among

any pair of substantive words contained within it. That is, "red ink" in fact had something to do with

the "dam," and the sentence is a statement of what

that something was.

In some preliminary experiments performed by

the writer and his colleagues and described else- where [5], the precise nature of the contexts used to generate association measures for purposes

* When the contexts are given beforehand and there is no order relationship present among the words within a given context, for example as within a given set of uniterms

assigned to a document, the situation is relatively simple. A reasonable course of

action in this case is to assume that any Uniterm assigned to a given document is contiguity-related with each other Uniterm assigned to that document.

S A pointed but humorous treatment of how one's view of language can be colored by concocted counter-examples is given by Lauren Doyle in reference [6], as is an

excellent common-sense discussion of the role of statistics in dealing with natural language text.

of retrieval of sentences were not found to be

crucial. Two types of contexts were used in this work as a basis for determining machine-computed

associations: co-occurrence within sentences as basic units of contexts, and co-occurrence within

syntactic subtrees of sentences as units of contexts. A passage of text 7,000 words long was syntactically

analyzed, and word association matrices prepared

on the basis of the two definitions of context. The

association patterns obtained using the two defini- tions of context were somewhat different, and both

sets of associations served to enhance recall of

relevant sentences in retrieval experiments. Within the limitations of the discriminating power of our

experiments, however, we found no basis for assert- ing that one set of associations was superior to the other. My own current feeling is that, for running

sequential text at least, a good unit of context is a "window" of fixed length, say seven words long,

which is progressively moved from one position

to the next throughout the text. Thus, if the

window length is seven words, every word is regarded to be contextually related to six words

on either side of it. This procedure makes all contexts the same length, which enables one to use

a much simpler association formula than would be

necessary if variable-length contexts were used.6 Also, for certain kinds of running text, sentence

or punctuation boundaries can often best be ignored;

the benefits to be gained in relating antecedents to consequent probably far outweigh the penalties

of the false connections generated.

At first, the problem of picking an association

formula for measurement of contiguity association

appears to be even more vexing than that of select- ing a unit of context. Goodman and Kruskal have

identified over 50 different formulas for measuring

associations [7]. Each such formula has its own

advantages as well as its drawbacks, and, given our present incomplete understanding of the

problem of semantic association, it would be pre- mature to suggest any one as ideal.7 Yet, to be

specific, I would like to devote a few paragraphs to the development of a simple measure of contiguity

association, one which will turn out to be a version

of the formula my colleagues and I have been using

in our recent experimental work [5]. It is desirable

to develop the explanation from an elementary

point of view in order to detail the methodology

implicit in using an association measure. Suppose that one is dealing with a corpus of run- ning text and, for sake of simplicity, that the cons It is shown in an appendix of reference [5] that, for use of the linear transformation method described in this paper, equal lengths of context are required if the Markov

process corresponding to the word association transformation is to generate the same

word frequency statistics as present in the original text. A more complete formula which normalizes for context length is discussed in the paper presented by Spiegel and Bennett at this Conference [12].

7 In a previous paper, P. Jones and I pointed out that formulas of a certain class lend themselves to representation in such a way that word association and document

retrieval can be described by matrix operations [3|. Moreover, under certain assumptions, these formulas can be computed instantaneously using analog electrical networks

[3, 8],

27
texts to be considered are adjacent word pairs

determined by a moving window which is two

words in length. Thus considering the sequence of

words ABCDEFG etc., the first context is the word

pair AB, the second is the pair BC, then CD, etc. For an N word corpus there are N— 1 such contextual pairs, and for the moment we will consider

the pairs to be ordered — that is, the context Wt W2 is regarded to be different from W2W\-

Now suppose that a frequency count has been

made of all words in the corpus and of all adjacent

word pairs, and that it is known that word Wa occurs fa times, word Wb occurs fi, times, and that the

adjacent word pair W„Wb occurs fob times, etc. Before the counts can be interpreted, it is necessary to have a statistical testing procedure in mind.

The steps in such a procedure are standard. The

first step is to identify a phenomenon under study

and to decide on a procedure for making observations. The next step is to formulate a null hypothesis Hq this being merely an assumption of

chaos, an assumption that the results of observations

are due to chance alone. The third step consists

of a selection of a statistical measure S, a formula

which assigns a value to the results of observation.

For the selected statistic S one knows beforehand

(usually from tables) the probability of any value of

the statistic being observed if the null hypothesis

is true. The next step is to select a level of probability a which represents significance. Usually a

is small, say a = 0.0001. This completes the apparatus. To use it, observations are made and a value is computed for the statistic S based on these

observations. The probability p(S) of the statistic having this (or greater) value is computed, estimated,

or looked up in a table. If p(S) > a then the null hypothesis is accepted — i.e., it is decided that the

observed event could have happened due to chance

alone. If on the other hand p(S)< a, then the null hypothesis is rejected. That is if p{S)< 0.0001,

then there is less than one chance in 10,000 that the observed event could happen due to chance

alone, and the null hypothesis is therefore rejected. In most practical applications of statistical tests, an alternative hypothesis is accepted instead —for example, the hypothesis that a certain substance

causes cancer. As has been mentioned, the observations to be

used for the measurement of contiguity association

consist of word frequencies and of word pair fre- quencies. An appropriate null hypothesis Ho is that the position of a word in text is determined by

chance alone. That is, Ho states that a word Wa

is sprinkled through the text fa times, with probabilities of word occurrences in adjacent text positions being statistically independent. The

alternative hypothesis is the presence of contiguity

association.

B A primary difficulty is that the measure Cab possesses a large variance when one

of the numbers fa , ft,, or fob is very smatJ. A good rule of thumb is that the measure is reliable only when each of these numbers is 3 or greater.

9 These values are roughly correct for the sampling distribution of a text of 45,000 running words with which we are currently experimenting.

Having defined the measurements to be made and

having formulated a null hypothesis, the next step

is to find a statistical test to determine whether

the null hypothesis is sufficient to explain the

observed phenomena, these phenomena being the

observed word-pair frequencies fob. The measure

I suggest is a very simple contingency coefficient.

If Ho is valid, the probability of the pair W„Wb being

located in any adjacent pair of text positions, say

the first and second, is, by statistical independence,

PoPb which equals . There are N— 1 text positions, so that the expected number of pairs W„Wb,

f f on the basis of chance (H0 ) alone, is -~(N— 1). For long texts, this becomes for all practical

purposes:

expected number of pairs assuming H0

- .fgjb

N

(1)

However, one also knows fob the actual measured

number of pairs Wa Wb, and therefore one can form

a contingency coefficient,

observed number of pairs

expected number of pairs assuming Ho = Nfab fa

'ab' (2)

This coefficient is the proposed measure of contiguity association; it measures the degree of sur- prise connected with finding fab pairs WaWo when

statistical independence and chance alone would

dictate instead finding only fa-Jb

N

pairs. A very

similar measure can readily be defined for the case when the context-size window is more than two

words wide. This measure, incidentally, has its faults as well as advantages, and can be considered

to be reliable only for certain ranges of values of

fa, /&, and fab.

8 For fa ,fb, and fab within the range that makes the measure reliable, there is associated with every

value Cab a probability p{C'ab) that C'ab or a greater

value could be observed due to chance alone — i.e., that an observed value 2* C'ab occurs when the null hypothesis is valid. This probability is extremely

small, being in a typical case less than 10-4 when

Ca6 = 50. 9 Say that one has picked a significance

level a = 10-4

. Then if the value Cab 3* 50, the

probability of the observed event assuming the null hypothesis is less than 0.0001, and it is necessary

to reject H0 and accept an alternative hypothesis.

When Wa and Wb are both substantive words, I propose that an appropriate alternative hypothesis

is that one or two of the following events is present:

(a) a significant contiguity relationship exists among

the concepts denoted by the associated words and

this relationship is asserted by the text, or (b)

28
the associated words combine together to denote

a new concept not already implied by one of the

constituent words, as for example in the case of

"hot dog." The distinction between these two

kinds of events, incidentally, is often one of degree,

and is being studied further. 10

In practice, it is not necessary to bother with

computing probabilities, for they vary monotonically

with the value of the statistic, the larger the value

of C the smaller the probability of observing it assuming Ho. Instead, one regards the statistic

itself to be a measure of "association strength,"

and orte lists word pairs according to decreasing

value of this statistic. Different workers on statistical association

methods use different formulas and often give their

measures different interpretations. What is important in every case, however, is the existence of an underlying statistical procedure such as that described above. To every value V of an association statistic, be this statistic Cab or some other,

there exists a probability of that measure having

value V^V under the H0 assumption of randomness. Generally, the larger the measure V the

smaller this probability and the greater the confidence that the observed event could not be due to chance alone. In fact, if words associated with

respect to a given word W§ are ranked in order of

decreasing value of a well-behaved association measure within the framework of a well-defined

statistical procedure, these words will actually be

ranked in order of increasing probability of the ob- served co-occurrences being due to chance alone.

4. Synonymy Association

Although universally accepted, synonymy is unfortunately an ill-understood concept. It is nearly impossible to find two words which are

precisely identical in meaning. In general, a

given object may be named by a number of words

or phrases. Not only will some of these names be

specific and others more generic, but an object may

be named by a term which describes part of it, by another term which describes a whole of which

it is a part, or by another term which describes

the object in terms of one or more of its properties.

For example, in various contexts the same object

may be denoted by the following expressions: "the

aircraft," "the airplane," "the 707 astrojet,"

"the jet," "the equipment for this flight," "the

common carrier vehicle," "The Sylvia Jane II," "she," and the like. Questions of what constitutes synonymy and in- quiries into the meaning of meaning can very rapidly

lead to an endless philosophical quagmire. For

the achievement of practical objectives, however,

it is necessary to have an operational criterion for synonymy which allows measurements to be made.

Interchangeability of usage seems to provide as good a criterion of this type as any I know of. Clearly, two words are perfect synonyms if and only

if either one can always be used in place of the

other; likewise, partial synonyms can sometimes

be used interchangeably.

The basic hypothesis advanced here (and which

has been advanced previously by my colleagues and

others [3, 11]) is that, in a sufficiently large corpus, many synonymous words are used interchangeably,

and that in proper circumstances the extent to which two words are synonymous can therefore be

measured by noting the extent to which these two

words are used interchangeably in various contexts.

10 If one or both of the words ffa Wt, are function words, a third possibility exists: The observed association may be due to the presence of a syntactic unit or of a standard

syntactic construction.

Ideally, it would be useful to measure inter- changeability of usage considering a wide variety

of contexts, not only linguistic contexts but also

extralinguistic ones involving patterned situations

of human behavior. In practice, however, the

relationship between behavioral situations and

verbal responses is poorly understood and difficult to measure, although it is under continued study

by psycholinguists [9]. Most of us present at this Conference have confined ourselves to contexts of written text. But

even here the best way to proceed is as yet not understood. At one extreme, interchangeability

could be defined rigidly in terms of requiring identi- cal usage in relatively long contexts. For example,

suppose that the sentence is selected as the unit

of context, and that two words Wa and Wb are regarded as being interchangeable and therefore

synonymous when and only when two large sets of

sentences exist which are pairwise identical except

that the sentences in one set employ Wa where as the sentences in the other set employ Wb. This

definition of interchangeability would lead to uninteresting results, simply because long contexts such as sentences cannot be expected to be repeated

so systematically, even in a very large corpus. That

is, most sentences are not simple variants of other sentences. At the other extreme, by regarding two

words Wa and Wb to be interchangeable and there- fore synonymous, if there is some sentence containing Wa which contains a word in common with

another sentence containing Wb, this definition

would make almost any pair of words appear to be

synonyms.

As in the case of measuring contiguity association,

then, there are fundamental questions as to what

are appropriate contexts for comparison of inter- changeability and as to what is a correct procedure

and for measurement of interchangeability. A simple approach, but by no means a unique one, is

29
described in the following paragraphs — this ap- proach closely parallels that described previously

for contiguity association.

As in the previous discussion, suppose that one

considers contexts to be ordered sequential word

pairs as would be measured by a sliding window

two words in length." Then, to the first order at least, it is possible to hold that interchangeability

in these pairwise contexts provides an approximate measure of interchangeability with longer contexts. This thought is developed in the following para- graphs and a measure of synonymy is derived. This measure will then be shown to be closely related to the contiguity measure described earlier. Let the null hypothesis Ho be the same as before,

that words are sprinkled in text according to their

frequencies of occurrence but without regard to position, so that word occurrence probabilities in adjacent text positions are statistically independent.

The alternative hypothesis is the presence of syn- onymy association, and the statistic proposed is different than that discussed previously. Suppose

that Wa and Wb are specific words, and let Wi

denote an arbitrary word-type found in the text. As before, there are /V contexts (pairs) in the text. The statistics to be developed will assign a measure

to any two words Wa and Wb depending on the

number of contexts in which Wa and Wb are inter- changeable. It would be possible to design a statistic which measures interchangeability in terms

of the number of interchangeable contexts shared

by Wa and Wb, or in terms of the number of types

of such contexts, or in terms of both. The pro- posed statistic for measuring interchangeability

in fact depends on both of these quantities. f f

To develop the statistic, note that p

ab = ° ' is

Ji the ratio of the observed number of ways Wa and

Wb can be interchanged in contexts with Wi to the

total number of contexts containing Wi. This

quantity is therefore an observed interchangeability measure for Wa and Wb, with respect to Wi\ it reflects frequency of usage of W%. To obtain an overall observed interchangeability measure,

the sum can be formed:

Observed interchangeability:

R

ab =^pf=^-£f

i- (3)

The value of the same interchangeability measure

expected under the null hypothesis is obtained by

substituting expected co-occurrence frequencies

fafi fbfi N N

11 As in the case of contiguity association, the extension of the discussion given here

to longer contexts or to symmetric contexts is straightforward.

for the observed ones /<,;, /&,. One then obtains

instead of Rab the sum: Expected interchangeability GIVEN HQ =

n _ y-> (fafi ) {fafi

_fafb r fafb ... yy Nf -JF+fi-lT (4)

Analogous to what was done previously for contiguity association, one can now obtain a contingency measure for synonymy association:

„ _ Observed interchangeability Rab

ab Expected interchangeability given H0 Rab

Sab = N-^-F7 (5) Jajb

The process of interpreting this measure is similar to that described previously for interpreting

the contiguity measure. A high value of this measure corresponds to a low probability of the observed

interchanges occurring given the null hypothesis,

and leads to rejection of H0 and acceptance of the

alternative hypothesis — the presence of synonymy.

Example:

It is instructive to go through a highly simplified

example — one that is concocted to show how the

above measures work. Consider the corpus consisting of the sentence:

The U.S. Army launches rocket missiles while the U.S. Navy launches jet missiles; however, although

the Navy flies jet planes, strangely it is not the case

that the Army flies rocket planes.

In this corpus /V = 32. It can readily be verified by computing formulas (2) and (5) using the two- word sliding window procedure with asymmetric

contexts described above that the contiguity matrix

f Cab = N r

a £ (deleting portions of the matrix of

Ja \tb

minor interest) is:

launches rocket missiles jet flies planes

Army 8 0 0 0 8 0

launches 0 8 0 8 0 0

rocket 0 0 8 0 0 8

Navy 8 0 0 0 8 0

jet 0 0 8 0 0 8

flies 0 8 0 8 0 0

30
The corresponding synonymy matrix

^faifulfi

Sab 7 7 is '. Ja-Jb Army launches rocket Navy jet flies

5-

Army 8 0 0 8 0 0

launches 0 8 0 0 0 8

rocket 0 0 8 0 8 0

Navy 8 0 0 8 0 0

jet 0 0 8 0 8 0

flies 0 8 0 0 0 8

The pairs of words thus related by the synonymy

measure S are (Army, Navy), (launches, flies), (rocket, jet), together with the self-associations

(Army, Army), (Navy, Navy), (launches, launches),

etc.

5. Matrix Representation

I would like to comment briefly on the relation- ship between the two proposed statistics, Cy for contiguity association and Sy for synonymy association. The relationship can most readily be seen by writing the formulas in matrix notation. Let A be a diagonal matrix with A., = ^ and let F= {fij},

Ji C={dj}, and S = {Sij}. Then formula (2) can be

written

C = NAFA (6)

and formula (5) can be written 12

S = N(AF)2 A = NAFAFA. (7) AF is a stochastic matrix which can be thought

of as corresponding to a Markov process which

describes a conditional contiguity transformation

in (6). The synonymy measure (7) employs the

square of this matrix instead. In other words, the

synonymy measure (7) in essence matches the

profiles of contiguity strength of different words.

The argument pursued in the previous section is therefore equivalent to asserting that measuring the

interchangeability of words in pairwise contexts

12 This expression is valid only when the F matrix is symmetric, i.e., when each con--

text ab is thought of as generating two pairs: ab and ba. Otherwise, S=N(AF)(AF)TA. a Current experimental research on statistical association techniques at Arthur

D. Little, Inc., includes investigation of the association patterns within a corpus of about 45,000 running words of transcribed speech, within a 10,000 document subcollection of an operational mechanized retrieval system, and within a collection of 45,000 abstracts containing about a million and a half running words of text.

(using the measure S) is equivalent to matching

their conditional contiguity profiles; a necessary

and sufficient condition for a pair of words Wa and

Wb to have a Jiigh synonymy coefficient Sab is that words a and b have like profiles of contiguity association with the other words in the corpus. A final comment with respect to retrieval is that higher order association matrices (AF)"A can also be interpreted as contingency coefficients, and that these matrices can be combined together to obtain

association matrices which represent combined

contiguity and synonymy measures [3]. In ex- perimental work on retrieval [5], we have used the

matrices:

/ + AKA + (AK)2 A + ( AK)3A

as well as I+AKA + (AK)2 A.

Examples of association profiles computed using

the above Cab and Sab formulas (or using linear

combinations of them) applied to various data col- lections involving vocabulary sizes of up to 1,000

words have been exhibited and discussed elsewhere

[3, 4, 5]. 13 Although a large proportion of the as- sociation profiles which have been generated ap- pears to be remarkably good (in the sense of being

intuitively plausible), others are equally difficult to interpret. There is little point in exhibiting fur- ther examples until carefully controlled experiments

to determine the validity of the hypotheses mentioned in this paper are completed. Such experiments are now in progress, and will be reported

separately.

3]
6. References

[1] Maron, M. E., and J. L. Kuhns, On relevance, probabilistic tion for cross classifications II: Further discussion and

indexing and information retrieval, J. Assoc. Comp. Mach. references, Am. Statist. Assoc. J. 54, 123-163 (Mar. 1959). rm rw 7

? 2 16~244 ( I960). [8] Giuliano, V. E., Analog networks for word association, IEEE

12] Doyle, L. B., Indexing and abstracting by association, Am. Trans. Mil. Elec. MIL-7, 221-234 (Apr-July 1963) Documentation 13, 378-390 (1962). [9] Saporta, S., and J. R. Bastian, Psycholinguistics (Holt, Rine-

[3] Giuliano, V. E.. and P. E. Jones, Linear associative infor- hart and Winston, New York, N.Y., 1961). mation retrieval, in P. Howerton and D. Weeks, eds., [10] Salton, G., Associative document retrieval techniques using

Vistas in Information Handling 1, ch. 2 (Spartan Books, bibliographic information, J. Assoc. Comp. Mach. 10,

Washington, D.C., 1963). 440-457 (Oct. 1963).

[4] Giuliano, V. E., Automatic message retrieval by associa- [11] Stiles, H. E., The association factor in information retrieval,

tive techniques, Proc. 1st Congr. Information System J. Assoc. Comp. Mach. 8, 271-279 (1961). Sciences (The Mitre Corporation, 1962). [12] Spiegel, J., and E. M. Bennett, A modified statistical asso-

[5] Arthur D. Little, Inc., Automatic Message Retrieval, Studies ciation procedure for automatic document content analfor the Design of an English Command and Control Lan- ysis and retrieval, this volume, p. 47-60. guage System, Rept. CACL-3 (ESD-TDR-63-673) (Nov. [13] Jones, P. E., Historical foundations of research on statisti- 1963). cal association techniques for mechanized documentation

[6] Doyle, L. B., The Microsyntax of Text, Rept. SP-1083 this volume, p. 3-8. (System Development Corp., Feb. 1963). [14] Maron, M. E., Mechanized documentation: The logic behind

[7] Goodman, L. A., and W. H. Kruskal, Measures of associa- a probabilistic interpretation, this volume, p. 9-13.

32
The Continuum of Coefficients of Association

J. L. Kuhns*

The Bunker-Ramo Corporation

Canoga Park, Calif. 91304

This paper discusses the classification of various coefficients of association between properties

characterizing a collection of items. It is shown that it is useful to define a generalized coefficient of association as the product of a parameter and the deviation of the observed data from expectation

assuming the properties are independent. The values of this parameter are given for twelve coeffi- cients of association. The ordering of magnitudes of these coefficients is also given. Among the

coefficients discussed are "closeness" measures obtained from the Euclidian distance and rectangular

distance formulas, the cosine of the angle between the vector representations of the data, the coeffi- cient of linear correlation, Yule's coefficient of colligation and the index of independence.

1. Introduction

This paper describes a classification of a certain

broad class of coefficients of association among

properties which characterize a collection of items.

The results are useful for three purposes:

(1) The classification has an intrinsic interest in that it unifies the theory of coefficients of association and illustrates the several points of view from

which they arise;

(2) the classification admits of a generalization,

thus allowing the invention of new coefficients;

(3) in application, the classification simplifies the

problem of selecting a suitable coefficient for a particular purpose.

2. What Is a Coefficient of Association?

Let us consider the association of two properties.

What do we mean by this? We observe the phe- nomenon of association by noting how the properties

apply jointly and separately to a collection of in- dividuals. Before going further let us show the

pertinence of this to the field of documentation.

Example 1. Given a collection of documents (the

individuals), then the classification of a document

under a particular index term can be considered

to be a property of the document. Thus we may

want to study the association between the properties "classification under the subject term

'Aerodynamics' " and "classification under the subject term 'Biology'." Such an association can then

be used to induce an association between the index terms themselves and consequently be used as a

tool for associative retrieval. A part of this process is, of course, the answering of such questions

as: Is "Biology" more strongly associated with

"Aerodynamics" than "Computers" with "Aerodynamics"? Such applications are discussed in

detail in references [1]

1 and [2]. Example 2. Given a collection of index terms

(the individuals), then the classification of a particular document under an index term can be considered to be a property of the index term. Thus we may want to study the association between the

properties "apphcability to document 1" and "applicability to document 2." Such an association

* Present address: The RAND Corp., Santa Monica, Calif., 90406.

1 Figures in brackets indicate the literature references on p. 39.

2 This is not recommended as an evaluation procedure except under highly special

conditions. The reason is, of course, that the procedure does not take into account

the value of the information to the user. See [4].

can then be used to induce an association between

the documents themselves and, as in example 1, be used for associative retrieval. Other areas of application such as storage of

documents, redesign of index systems, and orga- nization of index files stem from these two examples.

Example 3. The sentences of a document can

be considered to be a collection of individuals. An

automatic abstracting (extracting) procedure can

then be interpreted as defining a property of sen- tences by the fact of its selection or nonselection of

a sentence. Reference [3] describes how the association of two such properties (selection procedures) can be used as an evaluation of automatic abstract- ing techniques.

Example 4. Given a collection of documents (the

individuals), then the association between the

properties of being retrieved in response to a given

request and of being relevant to the information

need that produced the request can be used to give a comparative evaluation of the effectiveness of

two retrieval systems under certain normative conditions. An example of an evaluation of this kind

is given in reference [l]. 2 We now introduce some terminology to discuss

the common features of these examples. Let the

collection of individuals be TV in number and designated by 'a'i, 'a 2, . . ., 'aV Let 'A' and '/?' denote

the two properties. The four combinations of properties A and B, A and not-B, B and noi-A, nol-A and

not-Z?, having numbers of individuals x, u, v, y, respectively, uniquely categorize the individuals.

We use ni to indicate the number of A's and ni to indicate the number of J5's.

33
There are four well-known methods to represent

such data. Method 1. Tabular Form

B not-fi

A

not I

X u = n\— x n,

v=n-i — x y = N— n\ — n-i + x N-n,

n2 N-m N

Figure 1.

This shows the number in each classification to- gether with the adjoined row and column sums. The "cell" numbers in terms of x, ni, n2 , N are

also shown.

Method 2. ^-dimensional vectors or points in ^-dimensional space.

a x a-z A 1 0

B 1 1

aN

0

(»

Each property is represented by a vector of /V components: the ith component is unity if a, has the

property and is zero otherwise.

Method 3. Venn Diagram.

Figure 2.

Each individual is represented by a point in the

rectangle. The properties are represented by

(possibly overlapping) regions and therefore display

the four categories.

Method 4. Mass Distribution in the Plane.

(0,1)

(0.0)

ii.i)

The four categories are represented by the vertices

of the unit square: (0, 0) is not-A and not-B, (1, 0)

is A and not-Z?, (1, 1) is ^4 andfi, (0, l)isnot-^ andfi.

The points are assigned masses y, u, x, v, respectively.

The problem is now to create from these data a measure of association between A and B. The

rules of the game are to use only the numbers

x, y, u, v, and not the meanings of the predicates

'.4 'and 'B'. Now, before saying what the coefficient of asso- ciation between A and B is, it is necessary to define

what we mean by saying A and B are unassociated,

i.e., independent. This is the logically prior consideration. The meaning of independence can be

expressed in terms of the (logically) more primitive

notion of probability. Suppose that we wish to bet

that an individual of the collection has the property

A given that it has the property B and that we have

knowledge of the numbers x, y, u, v (or the equivalent x, rii, ri2, TV). The betting quotient we offer

(ratio of amount offered to the total stake) we will designate by P(A\B). If we omit the condition

that the individual has the property B, the quotient

is designated by P(A). Now, if the information

that the individual has the property B is quite ir- relevant for our choice of betting quotient, i.e.,

P(A\B) = P(A), (1)

then we say B is independent of A. It can be shown

that for the betting quotient to be fair 3 we must

have

P(A) = mlN (2)

and

P(A\B) = x/n2 . (3)

The relation (1) is thus the case if and only if

x = mn2IN. (4)

This is called the independence value of x. The

excess of x over its independence value is what

will interest us, namely,

8(A, B) = x-mn2 IN. (5)

(1,0)

It can be seen from this that 8 may have positive

and negative values. If N, re,, n2 are fixed, then the

largest and smallest values of 8 are attained at the

largest and smallest values of x. The following in- equality gives these values: 4

Figure 3.

in-r.Vutr'"/ T

babili

fy U!,ed here i

s ,hal »f a theory of degree of confirmat.on, and

sec 94

3 inductive inference as described in reference [5), <We use min (a, b) lo indicate the smaller of the numbers a and A, max (a b) to indicate the larger. '

min (m, rc2 ) = * = max (0, re, + n2 — N) . (6) We note that in the four examples discussed and,

indeed, in most applications in documentation, the

situation ni + n2 ^N will be the case; thus the

smallest possible value of x will be zero. 34
Yule [6] has pointed out the importance of 8(A, B)

for the theory of coefficients of association. He has

shown that this quantity measures the excess over independence in all four categories in the sense

that if we did the similar calculations for the negations of the properties we would get

8(A, B) = 8(not-A, not-B) = -8(A, not-fl) = -8(not-A, B). (7)

Also, 8 is symmetric, i.e.,

8(A,B) = 8(B,A). (8)

Following Yule, we say that A and B are associated more or less according to the size of 8(A, B), and

consequently the measure of association should

vary as 8(A, B).

This paper will show, through an examination of

various coefficients of association, that the coefficients are comprised in the general form

Ca(A,B) = ^± (9) a

and hence specified by the value of a parameter a. The values of a will be given for each coefficient

and ordered according to magnitude. The result

is a "spectrum" of coefficients of association. Apparently intermediate values could be used as well,

hence the title "continuum" of coefficients. For

example, we will show that possible values of a

are min {ni, n-z), max (rei, n2), and intermediate

values given by the arithmetic and geometric means

of m, n2 . We will also show that if n\ + n2 ^= TV/2

then the range

NI2 ^ a ^ rumlN (10)

absorbs all the coefficients examined.

3. The Coefficients

In this section we will make an inventory of some

coefficients of association that all have the property

of vanishing when 8(A, B) is zero. These coeffi- cients will also have the property of symmetry with

respect to A and B.

3.1. Separation

In the Venn diagram (fig. 2) it can be seen that the area of the region given by A and not-Z? plus B

and no\.-A measures in some way the separation

between A and B. This area relative to N is given

by

r*i +10n-i — Zx N

Indeed, it is easy to show that it is permissible to define the distance between A and B to be given

by this expression.5 We now define the coeffi- cient of association to be this expression subtracted

from its independence value (nin2 /N substituted

for x). The result is

S(A,B) =

8{A, B)

N/2 (11)

("S" for "separation").

3.2. Rectangular Distance

In the representation by points in ^-dimensional

space, we can measure the distance between A and

B by simply summing the differences between the

components. However, before doing this, let us

s The three properties required of a distance function are satisfied: (1) the distance

is non-negative and is zero if and only if A = B; (2) the expression is symmetric with

respect to A and B; (3) the distance from A to B plus the distance from B to C is not

less than the distance from A to C.

"weight" the components in such a way that the

distance between any property and its negation

(the complimentary set of components) is unity.

The general expression for the distance with any

pair of weights / and g is N

where is the ith component of the A vector, tj, is the ith component of the B vector. Since only

four different values occur in the summation

(namely, [f— g\, f, g, 0, with the number of occur- rences x, ni ~x, Ti2~x, N— ni — n2 + x, respectively)

the distance expression becomes

Tlif+Tl2g-x{f+g- \f~g\).

But,

f+g-\f-g\ = 2mm(f,g).

Thus the distance is given by

nif+ ri2g—2x min (f, g). (12)

If we wish the distance between A and not-A to be unity then / and g must satisfy the equation

n xf+(N-ni )g=l.

Among the solutions of this equation are the simple ones f=g=UN (13)

and

t

1 f=7TT, g = : 2n, ,e 2(N-nt ) (14)

35
The solution (13) leads to the separation function

of 3.1. The solution (14) when generalized gives

, J_ = J_ 7 2n,'* 2n2

which upon substitution in (12) leads to the rec- tangular distance expression

1 — x min (1/ni, l/n2 ) = 1 max (ni, n2 )

If we subtract this from its independence value we get our second proposed coefficient of association:

R(A, B) =

8(A, B)

(15) max (ni, n2 )

("/?" for "rectangular distance".)

3.3. Proportion of Overlap

In figure 2 we consider the ratio of the area of

A and B to the total area covered by A and B. This is, in fact, the probability of an individual

having both properties A and B conditional on it having at least one of the properties. The ratio is

n\ + n2 — x

Since this is a measure of "closeness" of A to B, we subtract from it its independence value to get

P(A,B) =

8(A, B)

1-

n\ + n-i

(16)

(«i + n2 — ri\n>IN)

Unlike the previous parameter values, a(P) depends

on x. Its range of values is therefore determined

by the range of values of x. We have max (n u n2 ) n\ 4- n2

(/7, + n2 -n,«2 /A0^a(P) (17)

and

a(P)^ n, + n2 - n,n 2 //V if rc, + n2 ^ N. (18)

3.4. Conditional Probabilities

The probabilities P(A/B) and P(B/A) indicate

the association between A and B. These are:

P(AIB) = xln2

P(BIA) = xlnx

.

(19)

(20)

Since these are not symmetric with respect to A

and B we consider instead

and

min (n u n2 )

max (ni, n2 )

But the second would lead to the coefficient R(A, B)

(see (15)). Thus we take the first and subtract

from it its independence value to get:

W(A, B) =

8(A, B)

min [n\, n2) (21)

3.5. Probability Differences

Yule [6] suggests consideration of the two probability differences

and

P(A/B)-P(Alnot-B)

P(B/A)-P(Blnot-A)

as measures of the strength of association between

A and B. As in the case of the probability quantities of 3.4, these lead to nonsymmetric measures: The first gives

8(A, B) n2{\-mlN)

and the second gives

8(4, B)

rc,(l-n,/A0'

As in 3.4, we can create symmetric coefficients

by using the maximum and minimum values of the

denominators. Thus we define

U(A, B) =

V{A, B) =

8(A, B) max [n,(l - nJN), n2{\ - n2 IN)]

8(A, B)

min [n,(l - n,//V), n2 (l - n2/N)]

3.6. Angle Between Vectors

(22)

(23)

The cosine of the angle between the two vectors representing A and B measures the "closeness"

between them. This is

Vn\n2

so that by subtracting the independence value we

get

8(A, B)

G(A, B) (24)

36
Thus a{G) is the geometric mean of n t and n2 . The independence value is zero. This gives the

well-known coefficient of linear correlation. An

alternate derivation is obtained by applying the

formula for the linear correlation to the mass distri- bution in the plane (fig. 3).

7

3.7. Coefficient by the Arithmetic Mean

Since we have had the values max (ni, n2), min (rii, n2), and the geometric mean of the two

Vnin2, we ask: Is there a quantity that leads to the parameter value given by the arithmetic mean

of ni and rc 2 ? Such a quantity is 1-P = n _ x l+p ( ni + n2 )l2 (25)

where p is the proportion of overlap defined in 3.3.

This behaves like the complement of p in that it vanishes when p=l and is unity when p — 0; otherwise it is less than the complement of p. Subtracting (25) from its independence value gives

E(A, B) =

8(A, B)

(n, + n2 )/2 (26) f

3.8. Coefficient of Linear Correlation

The scalar product of two vectors, i.e., the sums

of the products of the corresponding components,

gives the product of the lengths of the vectors and

the cosine of the angle between them. If we

first subtract from the vector for A the vector whose

components are all equal to rii/N and from B the vector whose components are all equal to n2 /N,

then it turns out the scalar product for these modified vectors is exactly 8(A, B). Dividing by the

product of the lengths of the modified vectors gives

the cosine of the angle between them. This is

L{A,B) =

8(A, B)

Vn,rc2(l-n,/A0 (l-n2 /A0

(27)

3.9. Yule Measures

Yule [6] gives a detailed discussion of the follow- ing two quantities:

Q(A, B) _ xy— uv

xy-\- uv

Y(A, B) =

xy uv

\f~xy-\- \fu~v

(28)

(29)

The range for each is from — 1 to + 1 and the independence value of both is zero. The second is Yule's coefficient of colligation. In terms of 8(A,

B) we have

Q(A,B) =

Y(A, B)

8(A, B)

(xy+uv)/N

(A,B)

(Vxy+Vuv)2 IN

(30)

(31)

An application of Q to associative retrieval is to be found in [1]. 3.10. Index of Independence

We will show in the appendix that the denominator of Q(A, B) is the smallest of all the parameters

considered so far. It is of interest therefore to study what its minimum value is for fixed nt , re2 , N,

It turns out that if «i + ra2 = N/2 then the minimum

value is nin2 /N. Let us consider then, as a coefficient of association,

/= 8(4, B)

n\n2 /N (32)

This is the negative of the complement of the index

of independence

nin2 /N

4. Orders of Magnitude Among the Coefficients

Let us summarize our results. Each coefficient consists of 8 divided by the quantity a. These

are shown in the table below with a descriptive

phrase indicating the origin.

* We call this "£" because there is an alternate derivation using the square of

the Euclidean distance function. 'See ref. (7). p. 120. It is also of interest to note the relation between L and The x

2 formula (ref. [7], p. 164) when applied to the cells of the tabular representation

of figure 1 . gives x

1 = N ' L1

-

Section Coefficient

3.1 S: area of separation

3.2 R: rectangular distance

3.3 P: proportion of overlap

3.4 W: conditional probability on weak evidence

Parameter a

N/2 max (m, «2>

1 — n<2 + n (l—-5-1

\ n, + n2 / («! + n2 — W1/12/AO min (ni, n2 )

37
3.5 U: first probability difference max [n,(l — n,/N), n2(l-n2IN)]

3.5

3.6

3.7

V: second probability difference min [n t (\ — ri]IN), n-Al-n.JN)]

G: angle between vectors

E: modified proportion of overlap

V nin-2

(n, + m)l2

3.8 L: linear correlation

3.9 Y: Yule coefficient of colligation (Vxy + Vuv)2 IN

3.9 Q: Yule auxiliary quantity (xy+uv)/N

3.10 /: index of independence nin2 /N

The following results 8 hold. The proofs are given

in the appendix.

Result: Chain of Magnitude 1. If 8 0, then for

all x, ni, n2, N

(1) I^Q if m + n2 ^N/2

(2) Q^Y^V^L^U^P

(3) P^S if max (ni, n2)^NI2.

If 8 = 0, then the inequalities hold in the opposite

sense.

Result: Chain of Magnitude 2. If 8 ^ 0, then for

all x, ni, n2 , N

(1) / ^ Q if m + n2 ^ N/2

(2) Q^Y^V^L,W^G^E^R

(3) fl^Sif max(n1} n2)^N/2.

If 8 ^ 0, then the inequalities hold in the opposite

sense. We conclude that, from its position in the "spectrum" and its computational simplicity, the coefficient W characterized by a = min (rai, n2) appears

as a good choice for applications in documentation.

5. Appendix. Proofs of Inequalities

The proofs are given in terms of the a's.

1. Um+ ntSN/2, then <*(/) ^ a(Q).

We must show that nin2 =i xy + uv. Now xy+ uv

is the quadratic 2x2 + (N— 2n\ — 2n2 )x + nin2 . Thus, if ni + «2 = /V/2, the minimum occurs at the

minimum permissible value of x, namely x = 0.

2. a(Q) ^ a (Y). We must show that

xy + uv = ( y/xy + Vuv) 2. But = xy + uv + 2vxyuv .

3. a(Y) ^a(V)

Consider the vectors ||Vx, Vu||, ||Vy, V^||.

By the Cauchy-Schwarz inequahty,

"\/xy+ y/uv = y/x + u y/y + v .

But the righthand side is Vni (N—ni). Now apply

the Cauchy-Schwarz inequahty to the vectors \\Vx, Vv\\, \\Vy, Vu\\ to get

Vxy + Vuv ^ Vn2 (N-n2).

Combining these results, we get

( V^y+ Vw)2 ^ min [m(N— m), n2(N- n2)].

Dividing by N gives the desired result.

4. a(V) ^a(L) ^a(U).

a{L) is the geometric mean of a{V) and a(U)

and thus is an intermediate value.

5. a(U) ^a(P).

Let m— min (m, n2 ) and M— max (m, n2 ). Then

the minimum value of a(P) (given by (17)) can be

written as M

2M

8 We assume that ni, tVt are not zero.

m + M V" N J

"' V N )

" N(m + M)

But M(l — mlIV), is the largest of the four quantities

M(\ - m/N), M{\ - MIN), m(l - m/N), m(l - MIN)

and hence is at least as large as a(U) , the maximum

of two of them.

6. If max (m, n2) =i N/2, then a(P) ^ <x(S).

This follows from (18).

7. a{W) =i a(G) =i a(E)^a(R).

The geometric mean of two numbers is always

less than the arithmetic mean.

38
8. Note on the values of a(U) , a{V).

Using the notation of the proof of 5, we have

ni + n2 ^N if and only if a(U) =M(1-M/N).

For Tii + n2 ^ N gives M+m^N, thus

M2 -m2 ^N(M-m)

m(N-m) ^ M{N— M)

m(l-mlN) ^M(l-MIN),

6. References

[1] Maron. M. E.. and J. L. Kuhns, On relevance, probabilistic indexing and information retrieval, J. Assoc. Comp. Mach.

7, 216-244 (1960).

[2] Stiles, H. E.. The association factor in information retrieval.

J. Assoc. Comp. Mach. 8, 271-279 (1961).

[3] Study for automatic abstracting, Final report, AF 30(602)- 2223 C107-1U12 (Ramo-Wooldridge, a Division of Thompson Ramo Wooldridge. Inc.. Canoga Park. Calif., Sept. 19, 1961).

[4] Research on an advanced NASA information system, NASw538, (TRW Computer Division, Thompson Ramo Wooldridge, Inc., Canoga Park, Calif., Oct. 11, 1963).

[5] Carnap, R.. Logical Foundations of Probability (Univ. of Chicago Press, Chicago, 111., 1950).

[6] Yule, G. U., On measuring association between attributes,

J. Roy. Statist. Soc. 75 , 579-642 (1912).

[7] Hoel, P. G., Introduction to Mathematical Statistics, 2d ed. (John Wiley & Sons, Inc., New York and London, 1954).

39
I
A Correlation Coefficient for Attributes or Events

H. P. Edmundson*

The Bunker-Ramo Corporation

Canoga Park, Calif. 91304

This paper examines a correlation coefficient R(A, B), for attributes or events A and B, which

measures their probabilistic interrelation in a quantitative way. By means of indicator functions it is shown that the correlation coefficient R(A, B) is a special case of the classical correlation coefficient R(X, Y) for random variables X and Y, and hence, is a special case of Pearson's mean square contin- gency 4>

2 for a two-by-two contingency table.

1. Correlation of Attributes

The problem of measuring the degree of association or correlation between attributes is an old one and has been discussed by several investigators

(Yule [l], 1 Steffenson [2], Goodman and Kruskal

[3], [4]). Yule [1] lists several basic properties

that any "legitimate" coefficient of association

between attributes should be expected to have.

For example, he recommends that it should (1) vanish when attributes A and B are (statistically)

independent; (2) be a maximum when A implies,

is implied by, or is equivalent to B; (3) be a minimum

when A implies, is implied by, or is equivalent to non-5; and (4) have a simple range of values, say

from — 1 to 1. For reasons of conceptual and notational simplicity, the development of the results of this paper

will be in terms of events rather than of attributes or properties of things. This is theoretically

justifiable since attributes and events are in one-to- one correspondence. First, because in logic sets are defined intentionally as a collection of all things with a particular property; and second, be- cause in probability theory events are defined as subsets of a probability space. For example,

the event "x is green" corresponds to the set "all green things" which, in turn, corresponds to the

property "greenness."

As will be shown, the desiderata of Yule are generally met by the correlation coefficient for events discussed here. Hence, the event correlation coefficient can be regarded as "legitimate"

in the sense of Yule.

2. Classical Correlation Coefficient for Random Variables

Let X and Y be random variables with expectations E(X) and E(Y), standard deviations D(X) and

D(Y), covariance C(X, Y), and correlation R(X, Y).' Then, by the classical definition

R(X, Y) =

C(X, Y)

D(X)D(Y)

E{XY)-E(X)E(Y)

[E(X2

) - EHX^iEiY2

) - E2(Y) f'

2 (2.1)

The random variables X and Y are said to be uncorrelated provided R{X, Y) = 0, and to be independent

provided P(XeA and YeB ) = P(XeA)P(YeB) for all sets A and B. From correlation theory, the

following properties are well known (see Parzen [5]):

If X and Y are independent, then R(X, F) = 0

If Y=X, then R(X, Y)=l

If Y=-X, then R(X, F) = -l

\R(X, Y)\ 1.

(2.2)

(2.3)

(2.4)

(2.5)

3. Correlation C<

Let A and B be _sets (corresponding to events)

with complements A and B, union AUB, intersection A (IB, and probabilities P(A) and P(B).

*Present address: System Development Corp., Santa Monica, Calif., 90406.

1 Figures in brackets indicate the literature references on p. 44.

772-957 0-66—

fficient for Events

It is desired to define a correlation coefficient R(A, B) for events A and B that will be analogous

to the classical correlation coefficient R(X, Y) for random variables X and Y. Heuristically, this is suggested by formally mapping the algebra of ran-
dom variables onto the algebra of events by means

of the transformation:

Replace X by A

Replace ZF by A (IB

Replace E( • ) by P( • ). Then by strict formalism, since X2 maps into AHA—A, it would follow from the definitions of

variance, standard deviation, and covariance that

V(A) = P(A) - P%4) = P(A)[1 - P(A)] = P(A)P(A)

D(A) = [P(A)-P\A)]1 ' 2

C(A , B) = P(A HB)- P(A)P(B ).

With the appropriate substitutions (2.1) becomes

the symmetric function

R(A,B) =

C(A, B)

D(A)D{B)

P(AnB)-P(A)P(B)

[P(A ) - P*(A ) J

1 ' 2 [P(B ) - P

2(B ) f'

2 ' (3 - !)

which could be the heuristic definition of the cor- relation coefficient between events A and B. The appropriateness of the above formal mapping

is supported by the fact that the well-known CauchySchwartz inequality from probability theory

becomes

E2(XY) ^ £(X2 )£(P)

P2(Af)B)^P(A)P(B),

which is a valid theorem since AflBQA and A (IB QB imply P(AnB)^P(A) and P(A(1B)^P(B).

If R(A, B) is to be a measure of the correlation of

two events, then, like R(X, Y), it Should satisfy

Property 1: If A and B are independent, then

k(A, B) = 0

Property 2: If B=A, then R(A, B)= 1

Property 3: If B = A , then R(A , B ) = - 1

Property 4: \R{A, B)\ 1. The validity of these formally constructed propositions will now be examined.

4. Properties of the Event Correlation Coefficient

We shall now prove properties 1, 2, and 3 of the event correlation coefficient. It will be helpful to interpret R(A, B) in terms of the set theoretic ref- lations of A and B; for example BQA, B = A, BC.A,

B = (f) (null set), and B = S (event space). To do

this we shall express R(A, B) as a function of the

odds on A and the odds on B rather than as functions of the probabilities P(A) = a, P(B)=z b, and

P(AC\B) = c. Denote the odds on A by

0(A) =

P(A) P(A)

P(A~) l-P(A) 1-a

Note that 0(A~) = 0~\A). First, when B is a subset

of A we get

if BQA, then R(A, B) = [OXA)0(B)y'2 (4.1)

since

R(A,B)

b — ab

[a(l-a)b(l-b)Y'2

1/2 1/2 a -bj = [0(A)0(B)y'2

.

As a corollary, when B equals A we get property 2

if B = A, then R(A, B) = \. (4.2)

Second, when B is a subset of A (i.e., A and B are

disjoint) we get

if BQA, then R(A, B) = - [0(A)0(B)] ^ (4.3)

since

R(A, B) =

— ab

[a(l-a)b(l-b)yi2

1/2 1/2 a-a

= - [0(A)0(B>] i/2

.

As a corollary, when B equals A we get property 3

if B = A, then R(A, B) = -l. (4.4)

Next, what are the values of R(A, B) when B = 0

and 5 = 5? Direct substitution in (3.1) yields an

indeterminate form in each case. Instead, we shall use the facts that 0(0) = 0 and 0(S) = [0(0)]-' = oo. First, if B is the null set, then tyQA. So we get

if B = q, then R(A, B) = 0

since from (4.1)

(4.5)

42
R(A, 0) = [0(A)0(W2 = [0(A )

• OP = 0.

Second, if B is the universal set, then AGS. So

(4.6)

we get

if B=S, then R(A, fi) = 0

since again from (4.1)

R(A, S) =R(S, A)= [0(S)0(A)yi2 = WmO(A)]W = [o-0(A)y2 =o.

Finally, if A and B are independent, then c = P(A D B) = P(A)P(B) = ab. Hence, we get property 1

since from (3.1)

R(A,B) =

[a(l-a)6(l-&)] 1/2 = 0.

if A and B are independent, then R(A, B) = 0

(4.7)

It is interesting to observe that we can also get the

purely set-theoretic properties (4.5) and (4.6) as

corollaries to the non-set theoretic property (4.7); for A and Q are independent because P(Ar\(Jl) = P(A)P{^>) and A and S are independent because

P(AnS)=P(A)P(S).

The proof of property 4 can be given algebraically also, but it is indirect and lengthy. From the

fact that the proofs of properties 1, 2, and 3 are so easy, it should be suspected that something basic

is involved and that some fundamental relation

exists which will yield properties 1 through 4 di- rectly and immediately. In section 5, we shall show this to be the case.

5. Fundamental Relation Between the Two Correlation Coefficients

We will use indicator functions to expose the

fundamental relation between the classical correlation coefficient R{X, Y) for random variables and

the one R(A, B) for events. The indicator function

of a set A that is in the range of a random variable

Z is defined as the random variable

h(A)-

if ZeA

if ZeA

(5.1)

which can be seen to have the following properties

(see Parzen [5])

Iz(AnB) = Iz(A)h(B)

Iz(A)=l-Iz(A)

(5.2)

(5.3)

since from (5.2)

E(XY) = E[Iz(A)Iz(B)] = E[h(ADB)] = P(ZeAC)B) = P(ADB).

From (5.5) we get as corollaries

E(X2

) = P(A) and £(P) = P(B). (5.6)

Hence, we will define the correlation coefficient R(A, B) between events A and B to be

The justification of the heuristic mapping that led to the correlation coefficient between events A and B will now be given. Let X= IZ(A ) and Y=h(B). Then

E&)=P(A) and E{Y) = P(B) (5.4)

since from (5.1) E(X)=E[Iz{A)]

= P[Iz(A)=l]

=P(ZeA)=P(A).

Also

E(XY) = P(ADB) (5.5)

R(A,B) = R[Iz(A),Iz(B)) (5.7)

which is a special case of (2.1). Thus, substituting (5.4), (5.5), and (5.6) in (2.1), we get

P(ADB)-P(A )P(B)

R(A, B)

[P(A ) - P2(A ) ] 1I2[P(B) ~ P2{B)Y'2 (5.8)

which justifies the heuristic definition (3.1). From (5.2) it can be seen that h(A) and Iz(B) are independent if, and only if, A and B are independent;

so that independence and uncorrelatedness are equivalent for indicator functions. Hence we get

property 1

if A and B are independent, then R(A, B) — 0.

Similarly, property 2

if B=A, then R(A, B)=l

43
follows immediately from (2.3); and property 3

if B=~A, then R(A,B) = -\

follows immediately from (2.4). Also, it follows

immediately from (2.5) and (5.7) that property 4

holds

\R(A,B)\^1.

Therefore, properties 1 through 4 are satisfied by

the event correlation coefficient R(A, B).

Finally, it is fitting that the probabilistic inter- pretation F^iADB) =£ P(A)P(B) of the CauchySchwartz inequality E

2 (XY) =£ E(X)E(Y) follows di- rectly from the use of (5.4) and (5.5).

6. Pearson Mean Square Contingency

Of course, it is possible to show that R(A, B)

is a special case of R(X, Y) without making use of

the interesting properties of indicator functions.

By direct calculation when both X and Y assume

two discrete values corresponding to A and A for X and to B and B for Y, R(X, Y) reduces (see Cramer

[6], p. 279) to

R(X, Y) =P11P22 — P12P21

(6.1) (pipzp-ip-z)112

whose right side can be rewritten in our notation as

P(ADB)-P(A)P(B)

[P(A) - P*(A)] w[P(B) - PHB)] 1/2 = R(A, B).

Moreover, it follows that R(A, B) is equal to Pearson's mean square contingency

i=l k=l PiPk

where the p,* are given by the contingency table for m — n = 2

B B

A Pn P12 Pi

A P21 P22 P2

Pi P 2

since (see Cramer [6], p. 282)

cf>z = R(A,B).

and hence

(P11P22 — P12P21)

2 Pi P2 P ip 2 (6.2)

7. Estimation of Event Correlation Coefficient

The estimation of the event correlation coeffi- cient R(A, B) for two events A and B hinges on

estimating three probabilities P(A), P(B), and

P(ADB). One approach to the estimation of these

probabilities is through their corresponding relative frequencies fi(A), fj(B), and fk(A, B) where i,j, and k

are the respective sample sizes. It is to be noted

that i, j, and k are not necessarily equal since, in general, there will be differences in the sample

procedures for the three events A, B, and ADB.

The sample event correlation coefficient will be

defined by

, A m fk(A n B) —fj(A)fj(B)

[fiiA)-fJ(A)yi2 [fj(B)-f](B)]1/2

which can be computed readily, once the estimates

fi(A), fj(B), and fic(AC\B) are obtained from physical

observation. The accuracy of the sample value

r[A, B) as an estimation of the unknown parameter

R(A, B) can be determined by the application of standard statistical techniques from the theory of estimation of parameters. Finally, it should be

noted that if j{A), f(B), and f{AUB) are known,

then the unobserved f(ADB) can be computed

from

f(A n B) =f(A) +f(B) -f(A U B).

8. References

fl] Yule, G. H., On measuring association between attributes,

J. Roy. Statist. Soc. 75, 579-642 (1912).

[2] Steffenson, J. F., Deux problemes du calcul des probabilites, Ann. Inst. Henri Poincare 3, 319-344 (1932-3).

[3] Goodman, L., and W. Kruskal, Measures of association for cross classifications, J. Am. Statist. Assoc. 49, 732-764

(1954).

[4] Goodman, L., and W. Kruskal, Measures of association for cross classifications. II: Further discussion and references,

J. Am. Statist. Assoc. 54, 123-163 (1959).

[5] Parzen, E., Modern Probability Theory and Its Applications (John Wiley & Sons, New York, N.Y., 1960).

[6] Cramer, H., Mathematical Methods of Statistics (Princeton

Univ. Press, Princeton, N.J., 1946).

44
Models and Methods

A Modified Statistical Association Procedure

for Automatic Document Content Analysis and Retrieval

Joseph Spiegel and Edward Bennett

The Mitre Corporation

Bedford, Mass. 01730

The very large number of documents, reports, and the like that are being sponsored and produced

tend to overwhelm our indexing resources. This results in relatively poor retrieval results since re- trievals from a library of poorly indexed items are, at best, haphazard.

Bearing this problem in mind, we have been designing our system to operate without the necessity

for indexed documents although capable of operating with them if such are available. The system

is to be fully automatic, i.e., able to accept the full textual form of the document (in machine-readable

form) and to retrieve from its store those items statistically associated with the query. Let us make

it clear that this has not been achieved. However, we have completed some promising steps, enough

to indicate those paths that might lead to a successful system. The path we have started investigating uses a statistical association technique whereby word/ word matrix call weights are modified by means of a redundancy measure derived from statistical information theory. The result of this modification is to change cell weights of all terms in accord- ance with their corpus-bounded redundancy. Thus, some terms are elevated in association strength while some are downgraded.

In addition to reporting on the influence of redundancy on word associations, the retrieval program

will be described. The precise flow of operations within the computer system will be given together with the rationale for such flow. In addition, we will describe some of the validating work on machine

versus manual retrieval capability currently in progress.

1. Introduction

Much has been said about developing an auto- mated library where, if one is to believe the

visionaries, a simple verbal statement of a query,

introduced into some machine (usually specified

as a computer), will result at best in a direct and

correct answer or at least in a small list of references

all highly relevant to the query. Although we are unboundedly enthusiastic about the need for such

a system, we believe there are some theoretical

and engineering problems to be overcome before

its realization. In view of both the need and the problems, we

have tried to design an automatic retrieval system

'hat involves only a minimal number of constraints,

these constraints largely introduced by the engineering limitations of the machinery involved rather

than by any preset theoretical position concerning

the nature of language or documentation. In es- sence, we sought a system that could accept as an

input any type of material as long as it was in a form compatible with machine requirements. To

be more specific, the method or system should be

able to accept and analyze large amounts of natural

message content relating to a wide range of topics. In responding to retrieval search demands, the tech- nique should be able to draw upon its total resource

of stored information, not only to select an appropriate response, but more important, to improve

its program for interpreting such demands and re- sponding to them. The technique should be able

to improve with experience. The system should

be able to code the content from messages in a

fully mechanical manner. It also should be able

to relate new content to other relevant content

already in memory. From its reservoir of infor- mation, it should be able to elicit the necessary

clues as to which documents are relevant to each

other, especially in response to a message that is also a query. For such a system to be reasonably

adaptable, it also should be able to perform these

functions without an index, grammar book, dic- tionary, thesaurus, or other formal constraint.

What this suggested was a system for automatically content-coding various statistical properties

of documents and then using these codes for auto- matic retrieval or, for that matter, document routing. The statistical approach applies the most

elementary and primitive relation among message

units, that of co-occurrence probability patterns.

The basic strategy is to proceed as far as possible

using these patterns, with a minimum of assumptions about the linguistic or semantic organization

of the information within the message structure. This strategy implies a rather mechanistic ap- proach to language processing, and that is indeed

the case. We assume that the information contained in a message is carried by the words that make it up and by the manner in which they are strung together. Further, we assume a person

generating a message or document chooses words

in a nonrandom fashion and combines them ac- cording to semantic and syntactic rules that are regular and, at least in our culture, to some extent

predictable. That is, both the selection of elements

and their co-occurrence with other elements are subject to restrictions by the contexts in which

they occur. We intend to exploit the regularities

of these associations among words, ignoring the

specific nature of the rules which produce such

regularity and thereby restricting ourselves to the

resulting statistical features alone.

47
If one examines this approach carefully, it can be seen that we are defining an approach similar in many ways to the way humans appear to retrieve

information from their own memories. Typically,

humans seem to start with the query words and

then to associate these with other words until the

information they seek is brought to their conscious

attention. This process of association of elements

is so basic and obvious that Aristotle reasoned that

to learn was to associate. However, although

association theory has been known for many years,

little use has been made of it as a methodology for information processing. In fact, literature on the use of statistical associations for information processing is quite limited, although at least three

significant contributions of a methodological nature appear to be of direct relevance. All are concerned

with the use of index terms, from a specified library of index terms, to retrieve documents from a specified library of documents. All involve obtaining

descriptive statistics to indicate the extent to which specific index terms occur together in tagging

the various documents of the library. Such descriptive statistics then are used to expand from

one or more index terms used in a query to a set of associated terms, based upon evidence of the co-occurrence tendencies of the various terms.

2. Historical Background

Probably the most important early work in statistical association techniques comes from H. P. Luhn who in 1958 [1]

1 suggested that the clerical

ability of the computing machine be harnessed to develop statistical frequency counts of text. These

counts would then be used to determine "significant" terms. Almost as an addendum he sug- gested that one could take these "significant" terms

and determine their mutual co-occurrences, thus

yielding a series of connected terms. This suggestion was not followed through, as far as we can

determine, until 1960, when Maron and Kuhns

[2] published their investigations on statistical as- sociations as part of a more general methodological

attack on the problems of document retrieval. Starting with a catalog of index terms and a

library of documents, they develop a statistical matrix of association frequencies.

7*

x = N(Tj, Tk ) u=N(Th 7k) N(Tj)

Tj v=N(Tj, 7*) y = N(Tj,?k ) N{Tj)

ATO N(Tk) n

where

Tj is a tag in the original request.

Tk is a tag not in the original request.

N(Tj, Tk) = the number of documents in the

library tagged jointly with both

Tj and Tk. N(Tj, 7V) = the number of documents tagged

with Tj and not with Tk.

N(Tj) = the total number of documents

tagged with Tj.

iV(,7j) = the total number of documents not tagged with Tj. n = the total number of documents.

From these descriptive statistics, Maron and

Kuhns develop three different measures of close- ness of association for index terms. One is the

conditional probability that if a term in the original

request Tj is assigned to a document, then the additional term Tk also will be assigned:

N(Tj, Tk)

P(Tk\Tj) = - N(Tj)

(1)

The second measure is the inverse conditional

probability; that is, the probability that if the additional term Tk is assigned to a document, then the

original request term Tj also would be:

(2)

Finally, they use the contingency estimate, or estimate of the frequency of co-occurrence, independent of the individual and separate influences

of the two terms which form the co-occurrence in question. They remove the magnitude to be ex- pected on the basis of chance from the actual cell magnitude, taking into account the number of times

the individual tags are used.

8(7), Tk)=N(Tj, Tk ) N(Tj)N(Tk) (3)

Maron and Kuhns then introduce an arbitrary

coefficient of association, based upon 8(7}, Tk),

which ranges conveniently from — 1 to + 1 with a magnitude of zero for the condition:

1 Figures in brackets indicate the literature references on p. 60. 8(7}, 7V) = 0. (4)

48
This coefficient is of the form:

Q(Tj, ny

n8

(xy+ uv)

(5)

This work was followed by Doyle [3], who developed a measure drawn from a contingency table to indicate strength of association:

N(Th Tk)n N(Tj)N(Tk) (6)

Doyle [4] has subsequently repudiated this formula,

and has instead substituted

N(Th Tk) N( Tj) + N(Tk) — N( Tj , Tk) (7)

Following close on, Stiles [5] also started with a contingency table of the form given above. However, he introduced a different coefficient of as- sociation:

logi

nb\--

N(Tj)N(Tk)N(Tj)N(Tk) (8)

In each of the three approaches cited, the in- vestigators tend to adopt the same basic data

structure from which to develop their analyses.

They pass over the question of how many terms are used to index any particular document and start

with the total population of indexed documents

as a base. They divide this population of documents into those that exhibit the common property

of having been indexed by Tj, with and without

Tk , and those not indexed by Tj, with and without

Tk . Using various normalizing procedures, they

adjust the sizes of these various groups, especially

the group (Tj, Tk), to remove any effect that might

result from the tendencies of Tj and Tk , separately,

to occur frequently in general. Some kind of nor- malization is required, because the more fre- quently an index word occurs, the more likely it will co-occur with some other term, simply on the

basis of chance. The techniques used by Maron

and Kuhns, Stiles, and Doyle, however, do not treat the fact that the more lengthy the string of index

words used to index a document, the more likely that co-occurrences involving the terms in the string are due to chance.

For a library retrieval problem this might be

little more than a minor omission, if, for example,

the number of terms used to index all documents

is a constant. However, if data on statistical co- occurrence are drawn from the actual strings of words in natural language that comprise the body

of a document or message, then such factors as

string length, word position in the string, and

vocabulary size might significantly influence the

tendency of words to co-occur. Accordingly, we

would like to argue that a statistical association

technique should take into account such factors and, further, that it should not be dependent upon

the particular level of message aggregation being

considered.

3. Theoretical Development

Before discussing a method for accounting for these effects, it would be useful to define our terms

and examine their implications. As previously

stated, a message is a carrier of information or

content. The smallest message carrier of content

is probably the alphabetical letter, number, or

arbitrary punctuation mark. This is a message

of minimum size. A continuous string of such

marks, commonly a word, may be thought of as a somewhat larger message. At a still larger level

of aggregation, a string of words, perhaps a sen- tence or a paragraph, is also a message. Similarly, documents, books, clusters of books, and so

forth, are messages of increasing levels of

aggregation.

Analytical techniques for determining message

or document content do not necessarily have to change radically because of the magnitude of message aggregation being considered. The procedures

one uses to examine the subject matter index of a

library card file may be similar to the procedures

for understanding and searching the individual

book cards, which in turn may parallel the pro- cedures used with a book's table of chapter contents, its page index, or the paragraphs and

sentences of an individual page itself.

Therefore, to maintain stress upon the common

denominator, we will consider all of the strings that constitute messages as a class, becoming specific, when necessary, by indicating the size or

level of aggregation for any string. Alphabetical,

numerical, or punctuation mark messages are one

level of aggregation smaller than those considered

in detail at this point. The units of immediate

concern are words, strings consisting of a few

words, and strings of such strings, including those

larger strings that range from sentences or titles, to paragraphs or abstracts, to articles, and so forth. We establish the following working definition: a word type is the smallest unit of analysis and al- ways has the identical configuration of alphabetical,

numerical, and conventional marks. Thus, the

word type man is different from men or man's.

Similarly is, are, and am are different types.

Types may vary in size from one symbol to many.

The only requirement is that the symbol arrange- ment remains the same for the same type.

The ability of a person to react differently to the string of letters man in contrast to the string

men, man, or manx reflects the influence of differ- ing structural arrangements of identifiable elements.

The string man is a unique system that might be

49
represented by the simple flowgraph below, in which

the numbers give the distance between the ele- ments of the string

or, by the somewhat more redundant association

list m

1

2 -• n

The arrangement or association of words can be

represented in the same way to identify a sentence,

or the association of sentences can identify a para- graph. This also applies to messages of larger

aggregation. For example, the string Mary would

like John has an identity characterized by the co- occurrence of the four words, the specific sequence

of the words, and the distance among them:

would

Mary • like

John

In association list form the string would have the

representation:

Mary

Mary

Mary

would

would

like

1 would

like

John

like

John

John

2 Taken from the Defense Documentation Center's Technical Abstract Bulletin, dated 30 August 1961, No. AD-262 148.

In this way a message at any level of aggregation can be represented structurally by its co-occurring

units at the next lower level by merely specifying

the directions and distances among them.

As further illustration consider the following

title, descriptors, and abstract 2 as one message:

{title) Psychophysical relations in the visual perception

of length, area, and volume.

{descriptors) Visual perception, Perception, Stimulation, Tests, Measurement.

{abstract) Subjective length, area and volume as functions

of the corresponding stimulus variables were

studied in three experiments. The exponents of the psychophysical power functions scattered around 1 for perception of real space. For

perspective drawings of cubes and spheres, however, the exponents were about 0.75. It was

tentatively concluded that perspective is an in- sufficient cue to visual volume. The results are discussed with special reference to certain car- tographic symbols representing population

magnitude. Just for this example, we will establish the follow- ing convention. A word type consists of any unique

sequence of exclusively alphabetical symbols with one or more blank spaces preceding and following it, but without blank spaces in the sequence itself. Capital and lower case letters are to be considered

identical, and all numbers and punctuation are ignored in identifying types. A primary string is specified as terminating with the presence of a punctuation mark directly followed by two or more

spaces. This specification results in choosing as primary strings those sequences of words that cor- respond to what we ordinarily identify as sentences. Accepting these conventions we can represent the

message as a secondary string composed of sen- tence length primary strings:

Psychophysical relations in the visual perception of length area and volume. Visual perception, perception stimulation, tests measurement. Subjective length area and volume as functions

of the corresponding stimulus variables were studied in three experiments. The exponents of the psychophysical power

functions scattered around for perception of real space. For

perspective drawings of cubes and spheres however the ex- ponents were about. It was tentatively concluded that per- spective is an insufficient cue to visual volume. The results are discussed with special reference to certain cartographic

symbols representing population magnitude.

This message, or any part of it, also can be repre- sented by an association matrix, where the columns

represent the first word in a pair, the rows represent

the second word, and the cell entries indicate the

frequency for each of the co-occurrences. This

matrix is, in effect, a simple coded representation

of part of the structural content of this one message.

With the addition of other messages from the same

corpus, the matrix could gradually grow to reflect the co-occurrences of types in all the messages of the corpus in question. This matrix would reflect the statistical structure of the corpus, showing

which types were associated and to what extent. It

is this matrix that we use to develop our association

factor.

50
4. Statistical

The actual frequency of occurrence of any pair

of word types is partially a function of the relevant

tendency for the two word types to co-occur because

they are associated in some meaningful manner. However, it is also a function of the separate tendencies, irrelevant for this purpose, of either of the

word types to occur with all other word types in general. For example, a specific word type will be the first type in as many pairs as there are other

types following it in a string. Similarly it will be

the second type in as many pairs as there are other

types preceding it in a string. A word type will also form pairs as a function of how frequently it

Development

occurs as a type in the set of strings under consideration.

It is desirable to normalize to eliminate these extraneous influences: frequency of word occurrence, relative word position, and string length.

This can be accomplished by subtracting from the

actual frequency of pair occurrence an estimate of the frequency expected on the basis of chance due

to frequency and position of occurrences as well

as sentence length for each of the two words that comprise the pair in question, as follows. We

start with a matrix of frequencies of co-occurrences.

sE

C

o

N

D

FIRST POSITION

*k (ij, fa)

p n

o

s

i (fa fa)

T

IO

N

N(xh yj) N(xk , yj) N«£, ik), yj) N(yj)

N(xh yk) tyxk , n) NU, jfc), yk) N(yk)

N(xj, (fa fa)) N(xk , (fa fk)) N((fa fa), (fa fk )) N(fj, fk )

N(Xj) N(xk) N(fa fa) N0

where

N(xh yj)

N(xj, (fa fa))

the frequency of co-occurrences with

word type j preceding word type /. the frequency of co-occurrences with

word type j preceding tokens which

are not of word type j and not of

word type k. N(xj) = the sum of the frequencies of all co- occurrences with word type j in the first position.

N(yj) — the sum of the frequencies of all co- occurrences with word type j in the

second position.

Ao = the grand total frequency of co- occurrences. The total frequency of pairs that includes the word

type j in the first position, N(xj), is equal to the portion of the length of the string that follows the type

/, summed over the total number of occurrences of

the type. Similarly the total frequency of pairs

3 Note that this initial correction is identical to the contingency table correction made

by Maron and Kuhns, and Stiles on their matrix tabular data, although these investigators use row and column totals based upon frequency of type occurrence, ignoring the

variable of how many types are used to identify a document (our notion of string length).

that includes the type k in the second position,

N(yk), is equal to the length of the string that pre- cedes the type k, summed over the total number of occurrences of the type.

The row and column totals N(xj), N(xk), N(yj),

N(yk), and so forth, supply a statistical estimate of

the cell magnitude that could be expected because

of the extraneous factors of frequency, position,

and string length. Subtracting the customary

contingency table correction 3 from the actual cell magnitudes, this estimate of cell magnitude can

serve as a first-level normalization.

Even with this correction, the cell frequencies are

still a function of the actual magnitude of the total corpus of pairs and the total number of word types

included in the entire matrix. Thus the greater

the total number of pairs, the greater the number to be expected in any cell. Similarly, the fewer the

number of word types, the fewer the number of

matrix cells, and, therefore, the greater the number

of pairs to be expected in any one cell. Consequently, correction of cell frequencies proprotional to the total frequency of pairs and inversely

proportional to the number of matrix cells results

in a set of weights which is normalized for extra- neous factors. The resultant cell weights, Zs,

51
serve as one estimate of the influence of association

forces independent of individual frequencies,

sentence lengths, number of different types, and

total number of pairs within the corpus under consideration:

Z(xj, yk) = n

2 N(xh yk)

. No

N(xj)N(yky N2 iy

o (9)

where

N(xh yk)-

N(xj)-

N(yk)-

No'-

n =

the frequency of co-occurrences with

word type j preceding word type k. the total frequency of co-occurrences with

type j as first type.

the total frequency of co-occurrences

with type k as second type,

the total frequency of co-occurrence of

all types.

the number of different types.

When the direction of co-occurrence is not considered, the matrix can be collapsed into triangular

form which reflects joint occurrence, where pairs

with the words reversed in direction are combined.

Each matrix cell of such a triangular matrix, ex- cept the cell where j equals k, is, in effect, the sum

of two cells

N(xh yk) + N(xk , yj).

In this case, the correction for extraneous factors would be:

Z'(xj, yk)

: n(n+l) N(xj,yk) + N(xk , yj ) No

N(xj + yj)N(xk + yk)

' 2N2 (10)

where N(xj + yj) = the total frequency of pairs containing type j in either position. Therefore, N(xj,

yj) is counted twice.

If the matter of distance of displacement of the

words in the pairs is ignored for the moment, a matrix of co-occurrences based upon the statistic Z'(xj, yk ) would appear to reflect one statistical tendency of pairs of types to associate. The matrix

is adaptive in that it starts with no cell weights if there has been no input of strings. Then as the

inputs begin and continue, the matrix continues to grow and change as it digests ever-increasing

quantities of pairs. Each normalized cell weight,

Z', rises and falls with time as each specific association increases or decreases in relative frequency.

In this way, the matrix memory changes with

time, maintaining a cumulative pattern of associations reflecting one statistical characteristic of

messages fed into it in the past. In addition to this adaptive characteristic of

changing memory with time and with changes in inputs, the matrix is also readily subject to what

might be called "formal education." Any specific

cell weight can be strengthened by repeatedly

reading into the matrix memory the specific strings that contain the desired association. For example,

by introducing the strings is am, is are, am is, am

are, are is, and are am, we can increase the statistical tendency of the tokens is, am, and are to be associated.

More complex learning can be accomplished by

the introduction of strings such as man men, men

man, singular plural, plural singular, man singular, men plural. In a similar way, we can build chains,

fists, trees, and circles of associations. A chain

would be formed through the repetitive input of the strings of types such as a b, b c, c d, and so

forth. A list would involve input strings of the form

a b, a c, a d, a e, a f, where the word a is the list heading, and the other words are subordinate entries

in the list. A tree would involve introducing the

strings a b, b c, b d, c e, c f, d g, d h. Circular

associations of the form a b, b c, c d, d a could also be formed. In fact, any particular configuration

of links is possible through the development of an

appropriate set of input strings.

The retrieval algorithm that seems almost to arise as a result of such matrices is one that takes a set of given terms (the query) and expands the

set by finding other, highly associated, terms. Doing this, however, allows one to chain or pro- ceed down paths that have little or no relevance to the original query. For example, one could start with a capacitance

psychotherapy

resistance

test

1 ^ 2 psychological

neurosis

term such as "neurosis" and trace a path as shown

above until one reaches the term "resistance."

Here there are two equal bonds, one leading off into the electronics field through the term "capacitance" and the other continuing in the psychological area through the term "psychotherapy." Clearly,

it is this latter link we wish to use. This can be accomplished by providing a feed- back loop to the original query terms, by requiring

each candidate term for expansion to have co- 1 occurred at least once with the full set of query * terms. To state our retrieval algorithm more precisely:

Given a set of query types, the matrix is searched I to locate all types which have been associated with

each and every one of the query types in the set. From this group of words, those (equal in number

to the number of query types) that have the highest

52
;.e,

sum of normalized matrix weights (when summed

over all of the query types) are selected to form a

iset of first-order types.

Having obtained this set of first-order associates, we form a new set combining these first-order types

with the original query types. With this larger

set of joint first-order and query types, the matrix

again is searched to locate all types that have been

associated with each and every one of the types in

this expanded set. From this newly located group

of types, those (equal in number to the number of

joint first-order and query types) that have the highest sum of normalized matrix weights (when summed

over all of the first-order plus query types) now are selected to form a set of second-order types.

The procedure for determining first-order associates can be presented in a symbolic form as

follows: Let ajAr = the Z'.k for tj with respect to qu

where, qeQ

Q= {query terms}

Tj is any term in the normalized matrix but

y=any row of the normalized matrix

A: = any column of the normalized matrix;

then TjtA = (k)otjk & Sj is among the nq highest sum

where, A = {first-order associates}

n,

Sj = Lot-jk k=\ ng =the number of terms in the class Q.

The second-order associates are derived in a

sirtiilar fashion, as follows: Let /3jfc = Z'jk for Tj with respect to a*

where, aeA

Tj = any term in the normalized matrix but

iQiAthen TjeB = (k) ajk pjk & s! is among the 2nq highest sums

where, B = {second-order associates}

Sj = 2 ctjk + £ [ijk k=l k=l

ra a = the number of terms in the class A.

From the above it follows that Q, Z, B are mutually

exclusive.

Having derived the first- and second-order association terms, we can then note for each document

the occurrence of each query term, each first-order term, and each second-order term. The documents

then are ordered according to the following rules and definitions:

Let 7i6 = the number of terms in the class B

(second-order associates) nq — na — n.b/2 j — nq + na + rib k=l00nq +10na + nb

Z)j, k — a message or document with j and k

indices as defined above.

Dx r > D<i means that D\ is more relevant than D2.

The ordering of messages or documents on the

basis of relevance is then:

Djr>Dj . l

and within the j set of messages

Dj, kr> Dj, k -\-

In such an ordering each cut "/' is further subdivided by "k." This procedure, of course, pre- sumes that messages containing the query types

are more relevant than those that do not, those that contain first-order associates are more relevant

than those that do not, and so forth.

5. Natural Text Retrieval

Once the system was programmed and checked

out,4 a search was undertaken to locate suitable

natural language corpora already in a computercompatible form. Certain criteria of adequacy

were (1) representative of a heterogeneous message

or document file; (2) pre-indexed so that criteria of

retrieval success could be simply developed; (3) relatively recent; and (4) in a form convenient for input.

We found that the Defense Documentation Center's Technical Abstract Bulletin met these criteria,

since the TAB's provide many different types of

system inputs: author names, titles, descriptors, as

4 See Appendix A for an informal discussion of the program details.

well as an abstract. In addition, the TAB's were

already being printed from punched paper tapes.

Arrangements were made to borrow the punched

paper tapes for two TAB issues, 15 March and

1 April 1962. With the use of a paper tape reader,

the TAB's were transferred directly onto magnetic

tape in a form compatible with the particular computer we had available.

Initial retrievals were carried out using the

descriptors as the input corpus. However, the

intent of the project was to develop procedures to retrieve unindexed materials. To this end, we

then tried the technique using the natural text found in the abstracts as the input corpus for association. Table 1 shows the query terms and

their expansions for some representative efforts

53
using such natural text for association. As can

be seen, the weighting technique we were using was

unable to downgrade association to the "function"

or "little" words, words that are extremely fre- quent and that seem to add little or nothing for

retrieval.

Table 1. Examples of original expansions

H(y)=log2 P(y) (ID

Query number

Query

terms

Associated terms

First- order Secondorder

Analog Not requested

1 digital for computer on Camera on Not requested

2 data and

record to

Atomic the a

3 bomb to in explosions was of

Convection of progress, made

4 radiation in report, a thermal liquid this, two

There are two brute force ways to downgrade

these words. One is to establish an a priori list of these "function" words and then delete them

from consideration. Another is to arbitrarily cut

off the most frequently occurring terms. Both

of these solutions we feel are unsatisfactory,

the first because such a list must be prepared anew

for each new corpus and the second because highfrequency terms may be deleted which quite reasonably should remain because they are central to the area of concern. For example, in the abstracts corpus, which approximates natural language, out

of 5,803 unique words, the terms, temperature,

data, results, design, effects, and others, were among

the 30 most frequently occurring. Clearly, some

terms like these should not be purged.

Ideally the approach we were looking for was one

that would downgrade only those terms that did

not materially aid in the association technique.

The terms we wish to suppress are those whose

occurrence in the text is not significantly conditioned by their associations — that is, these terms

occur more or less independently of their associated context of other words. More precisely stated,

the occurrence of such a term can be predicted

equally well whether one knows or does not know

the terms with which it co-occurs. A desirable

term, on the other hand, is one whose occurrence can be predicted with greater certainty knowing its associates in comparison with not knowing them.

This fine of reasoning led us to an investigation of some of the ideas developed in information theory,

particularly those dealing with the prediction of

the occurrence of a term when one is given its paired

associate. Along this fine, three related measures were found to be of use. The first gives the extent

to which the occurrence of a term y

is generally uncertain without having any information concerning its associations.

The second

P(y) P(x) P{y) (12)

gives the average extent to which the uncertainty

of the term y is reduced when knowledge of any of

its associates is given.

The third

P{x) (13)

gives the average uncertainty that is left remaining even after knowledge of any of the term's associates

is given. In fight of these, we were able to argue that in an

association scheme the terms to be suppressed or downgraded are those whose uncertainty of occur- rence remains great in spite of knowledge of their

associations. Using these aforementioned measures, we identified such terms by taking the ratio

/(y, X)

H(y) ' (14)

or the amount of reduction in uncertainty knowing

the term's associates divided by the term's total uncertainty.

All of the former association weights were now

multiplied by this additional correction factor. The system was then tried using the new matrices.

Some representative queries and their new expansions are shown in table 2. Table 2. Examples of original and revised expansions

Associated terms

Query Query numbers terms First-order Secon d -order

Original Revised Original Revised

Analog computation Not re quested

1 digital for equations computer on system

Camera on present Not requested

2 data and contained

record to unit

Convection of liquid progress in 3 radiation report made between

thermal liquid progress report made

a of

this this two too

The modification of the previous association

technique by the use of this additional measure seems to have added to the value of the technique.

This can be noted by comparing the ranks in order

of association magnitude of associated terms from

the normalized matrix before and after modification. Table 3 shows some of these comparisons.

54
'aBLE 3. Rank orders of associated terms to selected terms

before and after matrix modification

TERM

ASSOCIATED TERMS

rii n Raw NT1T\T : L> A ML'

DIAMINES of amines

amines radicals

radicals monovalent by tbutoxy

ethylene tertiary examples substituted formation ethylene

given examples

monovalent formation

oxidation reaction reaction oxidation substituted by- tbutoxy given

tertiary of are are

with with

the the HORIZON an horizon

at airspeed

of knots

achieving fa airspeed photographic coverage optimized

fa achieving

feet coverage

horizon terrain knots feet operating area optimized operating photographic while

terrain above

above

area minimum

been been minimum an

while has has for for of

to to a a

the the DUCTS in bile to rat bile duct

addition obstruction

after liver approximately regeneration changes hepatectomy common seen comparable common

duct addition hepatectomy comparable

hours ours known partial

liver after obstruction cells partial known

rat result regeneration approximately

result changes

seen well well of

cells found

found in number number

of to that was was that the the

Table 3. Rank orders of associated terms to selected terms

before and after matrix modification — Continued

TERM

ASSOCIATED TERMS

OLD RANK NEW RANK

FLORYS for lattice of theories

a molecules

certain monomer

consisting deriving

deriving polymer

energy review formula formula

free consisting

lattice solution monomer certain polymer energy

review presented

solution for theories a presented is

is of and and

the the ENGINES to centrally

a trackless cargo train centrally cargo

controlled highway

coupled ofTroad highway selfpropelled

into controlled

ofTroad operate

operate coupled program units selfpropelled into trackless under

train conditions under control

units program

can can

conditions systems

control test presented presented

results study study results systems that

test to that a and are are and

of of the the

6. Summary

We have reported upon a statistical association

technique and program which can accept any

natural language input as long as it is in a computercompatible form and, from this input, derive a term- term association matrix whose cell values provide a measure of the tendency of the two defining terms

to co-occur through other than chance factors. This matrix appears to have a number of potential uses; among them are automatic message retrieval, content analysis studies, message routing, and so

forth.

55
7. Appendix A. System Program

System Overview

The overall system flow chart is shown in figure

1. This system was written for the IBM 7090 computer. The system can be divided into two parts:

data preparation and query.

A. Text Tape

B. Tape for Concordance C. Pairs Tape(s)

IBM 9SORT

D. Sorted Tape for Concordance

CONCORDANCE

PROGRAM

F. Sorted P lirs Tape(s)

FREQUENCY MA1K1X PROGRAM

E. Concordance Tape G. Frequency Matrix Tape H. Row Tape

K. Document Tapes

QUERY PROGRAM r

NORMALIZED PROGRAM

I. Normalized Matrix Tape

J. Query Deck 1

QUERY EXPANSION PHASE

I

L. Expanded Query-word list

I\ CONCORDANCE SEARCH AND RETRIEVAL PHASE

M. On-line or off-line print of retrieved documents in order of relevance, and of expanded query-word list. FIGURE 1. Overall system flow chart.

Data preparation starts with the text and builds

from it a concordance and a list of pairs. Both of

these are sorted. The list of pairs is used to build a frequency matrix of word-word co-occurrences where the j— k entry tells how many times word j

and word k occurred together within a sentence,

summed over all of the sentences of the corpus. The frequency matrix is "normalized" in accordance

with formula (10) given above. This normalized

matrix is used in the query part of the system to produce an expanded query-word list; i.e., the origi- nal query words plus those additional words highly

associated with them.

The query part of the system has two phases: the

query-expansion phase and the concordance search

and retrieval phase. In the query-expansion phase,

the program first finds those terms (called first- order associates) strongly associated with the original query words, using as input the original query

words. It then iterates this process by finding

those words (the second-order associates) strongly

associated with the first-order associates and the

query terms, and so on. The concordance search

and retrieval phase then takes the expanded query- word list and using the concordance finds all of the messages or documents which contain one or more of the words from the expanded query-word

list. Each document gets a score, based on the

number of words from the expanded query-word fist which refer to it. The documents are then retrieved

and printed in order of score (highest score first). Description of Subroutines

The following sections informally describe the

subroutines and the tape formats found at each

stage of the system. 5

In general, in the machine formation and computation stages, a word is represented by a string

of 18 characters. If the word does not take up

the whole string, it is padded on the right with

blanks; if it is longer than 18 letters, it is truncated

after the first 18 characters. This word size is an

arbitrary parameter. One can choose to truncate

at 12 or even 6 letters or, for that matter, at 24 or 30 letters. Whatever length one chooses, it must

be a multiple of 6 since one 7090 register can contain 6 characters. However, word length does have

a material effect upon the total number of words that can be handled at one time within core. The

shorter the word, the more words that can be manipulated. Table 4 shows the effects of varying word

lengths, holding the vocabulary size constant, on

the data preparation time and on the retrieval time.

Table 4. Timing and size relations

Word Data

length Word Word prepa- Retrieval* Matrix** Com- Pairs

trunca- types tokens ration time density pres- produced**

tion time (min) (percent) sion** (millions) point (min)

18 7,500 110,000 487 15 1.5 3.5 3

12 7,500 110,000 330. 13 1.5 3.5 3 6 7,500 110,000 165 7 1.5 3.5 3

'' More precise, technical descriptions of each subroutine and tape can be obtained from the authors.

*This is the time required to retrieve the first 100 documents, and includes the time necessary to search the matrix, the concordance, and the text. Rather than merely

printing out document numbers and allowing the user to find them, we retrieve the

actual documents, and print out the document number, the title, the list of descriptors attached to the document, and an abstract of the document. If the user wishes to retrieve more than 100 documents, these additional documents, which merely involve another pass at the Text Tape, can be retrieved at the rate of 1 minute per 100 docu- ments. **We assume that the matrix density (relation between actual entries and total possible entries) remains constant while compression (relation between pairs and nonzero entries) increases. The pairs figure is then implied by the vocabulary size. It seems reasonable to assume that as the corpus gets larger the same word patterns tend

to be repeated; i.e., the old patterns are repeated much more frequently than new ones

appear. The assumption about density, however, is simply made for convenience. We do not know what happens when new words are introduced because of an expanding

corpus. Do the new words appear in sentences mainly with the old words, or do they

tend to form a subgroup of their own? Much more experience with large samples of English text is needed before we can give an informed answer to this question.

56
For the present program, the relation of corpus

size to data preparation and running time is linear;

i.e., assuming that the mean sentence length stays

the same, doubling the corpus will double the preparation time. The overriding consideration in terms of the data preparation time is the number of

pairs produced. The number of pairs produced is critical because the single largest expenditure of time is incurred by the sorting program. The main

variable in relating size of text to number of pairs

produced is the mean string length. A string of

length n will produce n(n— 1) pairs. Thus, five 20-word sentences produce 1900 pairs, whereas

one 100-word sentence produces 9900 pairs. The

number of pairs which a corpus will produce can

be estimated by the relation: N0 = T0 (S-1) (15)

where:

No = total number of pairs

To— total number of tokens

S= mean string length (in tokens).

If S remains constant, a linear relation exists between pairs produced and corpus size. Since

the relation between sorting time and the number

of pairs to be sorted is more or less linear, a linear

relation exists between corpus size and preparation

time.

The main size limitation for the present program

is the necessity for having all the row names and

row sums in a core at once. There seems to be

no simple relation between the size of the corpus

and the size of the vocabulary, but after a certain

point vocabulary size increases very slowly.

Text Preparation (TAPE A)

Concordance Preparation (TAPE B)

Pairs Preparation (TAPE C)

These three subroutines and their resultant out- put tapes represent the first step in the data preparation phase. The text (tape A) must contain all of

the input data necessary to build the matrices.

The words on the text tape are processed in two

ways: associated with numbers to form the con- cordance and paired to form the basic information

for the association matrix. The only restrictions on the text tape are:

1. Input may not exceed one tape for any given

run.

2. The records on the tape need not be of uni- form length. However, no record may exceed 2000

registers (computer words) in length.

3. The end of information on the tape must be

indicated by an end-of-file record. The scan program will cease accepting input upon its first encounter with an end-of-file mark.

The program scans the input data by bytes,

each register (or word) of data contributing 6

bytes, or characters. In turn, these strings of

characters are extracted to form English words.

The words are then used to generate the two out- put tapes, tape B (tape for concordance) and tape

C (pairs tape). The input data is treated as having

a certain simple structure (groups of words form

sentences when a period followed by two spaces

is encountered). Groups of sentences form

messages when either a special code or 10 or more

blanks are encountered. A period, blank, and

comma are all treated as word separators.

The pairing procedure has a large range of options. These are shown in table 5.

TABLE 5. Parameters for scan program.

Parameter numberParameter name Value Meaning

1 Unit of pairing. 3

2

Words within the same message are

paired. Only words within the same sentence

are paired.

2 Common word list. 1

0

Words on the restricted list* go into the concordance. Words on the restricted List do not go

into the concordance.

3 Restricted word

List pairing.

1

0

Words on the restricted list are paired.

Words on the restricted list are not paired.

4 Repeated occur- rence pairing.

1

0

A word will be paired even if it has appeared previously in the same pair- ing unit. A word will not be paired if it has appeared

previously in the same pairing unit.

5 Word distance. D Suppose two words Wx and W-i within the same pairing unit are separated by n intervening words. If n + L < D, W\

and Wz will be paired, otherwise not.

6 Word direction.

10

Suppose Wi occurs before Wi in the

pairing unit: Both (Wu Wt) and (W-u Wx ) will be listed. Only {Wu W% ) will be listed.

7 Sentence

terminators. 0

1

2

All periods are considered sentence

terminators. Only periods followed by one or more

blanks are so considered. Only periods followed by two or more

blanks are so considered.

8 Message

terminators. 0

1

The character 528 is the message terminator. Either 528 or a tape record starting with

10 or more blanks will be treated as an end of message.

9 Use of restricted

list. 0

1

Do not use. Do use restricted list.

*The restricted list is an arbitrary list of words assembled into the program by the

user. It can be a common word list.

Sorted Tape for Concordance (TAPE D)

Sorted Pairs Tape (TAPE F)

These tapes contain the same information as the scan program output tapes B and C. However,

57

772-957 0-66—
tapes B and C must be sorted to get all the information relevant to a word (or a pair of words) to- gether. The sorting is straightforward in conception, and, for the tape for concordance

(tape B), in execution as well. However, the

number of pairs produced by even a relatively small

sample of text renders the job of sorting the pairs

tape (tape C) a major undertaking. Because of

this, sorting is the major bottleneck to quick and

efficient preparation of the input text as the program now stands. The IBM 9SORT program was

chosen because it was the only one available which

could handle the large quantities of pair data

produced.

Concordance Tape (TAPE E)

The concordance, which is essentially an index

of every word, is a series of lists; i.e., each word

is followed by a series of numbers defining where

that word appeared in the corpus. The concord- ance is produced as follows: The scan program

first fists each instance of the word with its asso- ciated information (at the present time this infor- mation is: document number, sentence number

within document, and word position within sentence). When the list is sorted to produce the

input to the concordance program (tape D), each

word is repeated for every change of information.

The concordance program then strips these re- dundant words and lists a word only once together

with all the relevant information. Since this program does not employ any buffering or input/

output overlap, it runs at about half tape speed.

However, this is not too serious a disadvantage

because the tapes tend to be short.

Frequency Matrix Tape (TAPE G)

The frequency matrix is a word co-occurrence

matrix; i.e., the j-k entry tells how many times word

j and word k co-occurred in the same string summed

through all of the strings of the corpus. The defi- nition of co-occurrence is a function of the particular

parameters selected by the user for the scan

program.

Because the frequency matrix is sparsely filled

(in our experience fewer than 5 percent of the

possible co-occurrences actually occur), only the non-zero entries are listed. This reduces the fre- quency matrix to a list, or rather a series of lists;

first a row name, then a fist of column names and

entries for that row; then the next row name, fol- lowed by its list of non-zero entries, etc. At the

end of the matrix, information regarding total rows,

total pairs, and total non-zero entries is appended.

The frequency matrix program is essentially a pair-counting program. The scan program pro- duces the pairs, the sort program sorts them, and

the j-k entry is obtained by counting the number

of i-k pairs. When the first different pair is en- countered, the program checks to see if the difference is in the last word (which indicates another

column entry in the same row) or whether the

difference is in the first word (which indicates the

beginning of another row).

The frequency matrix program will accept more

than one tape of input information. If it finds an

end-of-file, it will call for another input tape via the on-line printer. If there are several input tapes

to be mounted, they must, of course, be mounted

in the correct sequence. It will also call for a new

output tape if the old one fills before all the pairs are processed. Finally, the frequency matrix

program is interruptable. To resume operation,

the tapes must be positioned and the core filled, and the program will then continue where it left

off. However, it will take some time to position

the tapes, especially if the program has been in- terrupted with the tapes near the end of the reel. The frequency matrix program does not use any

buffering or input/output (I/O) overlap, so that

it runs at about half tape speed.

i Row Tape (TAPE H)

The row tape (tape H) summarizes some of the

information on the frequency matrix tape (tape G). ; Every entry on the row tape provides a row name,

the number of non-zero entries for the row (Sj), and the sum of the frequencies for the row N{yj). In addition, there is a second file on the row tape i that contains five items: the maximum row sum

(maximum /V(yj)); the maximum entry (maximum

N(xj, y>)h the total number of pairs (No); the total rows (TV); and the total number of non-zero entries

(Tnz ). The main use of the row tape is to provide

the values N(yj) and iV(y>) for the normalizing pro- gram. In addition, the row tape furnishes a fist of

all the row names, allowing a preliminary search

at the beginning of the query program to make sure

that all of the query words are actually present in the matrix. These processes involve a search of the row tape, or of an edited version of it. When

normalizing, for example, a table is required in core whose entries are row name and N(yj). Such ? a table can be obtained from the row tape by reading the entry tape into core but omitting Sj for each

entry. This requirement, that the whole table be

in core at once, sets an upper limit to the size of the vocabulary. Since each entry uses four regis- ters (three for the word and one for N(yj)), only about

7500 entries can fit in core, thus limiting the pro- gram to a corpus which does not exceed 7500 18- character word types. To extend this the words

must be truncated at 12 or even 6 characters to extend the matrix size.

i Normalized Matrix Tape (TAPE I)

The normalized matrix tape (tape /) looks just

like the frequency matrix tape, but with different

j-k values (thus only the non-zero entries — those on the frequency matrix tape — are normalized).

58
The present version of the normalization program

is limited in the size of the matrix it can handle.

By size, we mean the number of rows in the matrix,

which is equal to the number of word types in the corpus.6 To understand the reason for this limi- tation, it is necessary to consider the operation

of the program in a little more detail. The row

tape (tape H) is read in synchronization with the

frequency matrix tape (tape G) so that as each new

row name is encountered on tape G the corresponding N{yj) can be obtained from tape H. Since (No) is always available, N(yk) is the only ingredient of

Z'(Xj, yk

) that remains to be accounted for. The most

obvious solution entails having all the N(yic) data

in core all the time. Toward this end, the first thing the program does is to list alphabetically the

entire row tape in core. As the binary search subprogram is presently constituted, the list cannot

exceed about 7500 row names in length.

The sameness of size of these two principal

tapes suggests the efficacy of buffering so the

program is designed to carry on input, output,

and computation simultaneously. In point of

fact, however, the program is computation bound:

i.e., computation time is greater than input or out- put time. Consequently, the approximate time

required for the program is a linear function of the

number of N(xj, yk) entries to be processed:

Time in minutes = (number of N(xh yk) entries)/40,000. (16)

Query Deck (J) The query deck contains the query words,

punched one word per card. They are read into the computer via the on-line card reader.

Document Tape (TAPE K)

The document tape (tape K) contains the actual

documents or messages to be retrieved. In some

cases the document tape will be identical to the

text tape (tape A). However, when one wishes to develop the matrices on key terms, and then print out the full abstract or document, tape A would

contain only the key terms tagged with a document

identifier, and tape K would contain the material

to be printed out also tagged with the same document identifier.

Query Program — Query Expansion Phase

The job of the query expansion phase is to produce an expanded query-word list. The program

makes at most (n— 1) passes down the normalized

5 In the event of a one-word sentence no pairs would be formed. It is possible that

this word, not having co-occurred with any word in the corpus, would not be recorded

as a row entry (it would turn up in the concordance in any case). In this case, the number of unique words (typesl encountered would be greater than the number of rows. However, for practical purposes, one can state that the number of matrix rows

equals the number of unique words (types).

' Relevance is operationally defined by the number of words from the expanded

query list which references the document.

matrix tape, where n is the number supplied by the

parameter cards of the query deck. Let us consider a typical tape pass. We start with a query list "(?" containing k words. Consider

also a potential query word list "P." This list "P" has been initialized with the words and values

from the row of the alphabetically first original

query word. As the tape pass is made, list "P"

is continually shrunk as follows: each time a "Q"

word is encountered as a row name, its row is logically "anded" word-by-word into "P," and the

corresponding nonzero values are added into "P."

At the end of the pass, the top k (with respect to numerical value) surviving words are skimmed off

list "P" and added to "()" to form a new "Q"

list for the next pass.

It should be noted that the nature of this pro- cedure has important consequences from the pro- gramming standpoint. After the "P" list has

been initialized, the only rows in the matrix that can

be of any possible use in further computation are those that correspond to words in list "P." Thus,

it pays to edit the matrix tape (tape I) as we run down

it. Each successive row name that is not a "(?"

list word is checked against list "P." If the row name appears on list "P," the row is copied into the edited tape; otherwise not. The next tape pass

is run on the edited tape, producing still another

edited tape in the process. The shrinking "P"

list is thus reflected in a shrinking edited matrix

tape. Since input -output is buffered with computation, this procedure does not cost us any time. In fact, time saving can be considerable, especially

in a multipass expansion phase and/or with a large

matrix.

Expanded Query-Word List (L)

Printout of Retrieved Documen ts (M)

The expanded query-word fist (L) is made up of

the original query words plus the first-, second-,

and higher-order associates as chosen by the user. These associates have been generated on the basis

of the normalized matrix entries and the whole list is then used with the concordance tape (tape E)

to find those documents most heavily referenced by

the expanded query-word list. The documents

are then retrieved and printed either "on-" or

"off-line" in order of relevance.7

Query Program — Concordance Search and Retrieval

Phase

This phase of the query program takes the ex- panded query-word list (L) and uses it to reference

documents for retrieval. First, a list of all possible

document numbers is made. Each document is represented by two registers, one for the document

acquisition number and one, initially zero, which

accumulates the document score. We next construct a table of all possible increments. These

increments are partitioned into two scores; one,

59
always incremented by 1, for simply being refer- enced (this is called the ./-value), and the other,

the K-value, incremented by powers of 10. The

particular power chosen depends on the referencing

word; documents referenced by the original query

word will have the X-value incremented by 10",

where n is the number of orders of association se- lected by the user. As the order of associates

increases, the power decreases to 10". Thus, in a

retrieval where the user requests three orders of

association, the original query words will have

their /lvalue incremented by 10 3

, the first-order associates by 10 2

, the second-order associates by

10 1

, and the third-order associates by 10°. In other words, each document gets two scores, J

and K. The ./-score is the number of words from

the expanded query-word list which reference the

document. The X-score is: r0io»+riio'i - i +r2io*-2 + . . . +r„io° a 7) where:

Wq= number of original query words which refer- ence the document.

W\ — number of first-order query words which refer- ence the document.

Wz= number of second-order query words which

reference the document.

Wn = number of nth-order query words which refer- ence the document.

n = orders of association selected by the user. When documents are ordered for "relevance"

the ./-score supplies the primary order and the Kscore is the "tie-breaker."

The concordance tape (tape E) is now read into

core. Since the concordance indexes every word,

the document number associated with each word on

the expanded query-word fist can easily be found.

The score of each document is given the appropriate

increment. After all increments have been given, we have a table of two-register items in a core, each item representing one document. These tworegister items are then sorted on the second register, i.e., on the score. Since the ./-component of

the score is contained in the left-half of the register,

the ./-values supply the primary ordering, with the

^-values serving as "tie-breakers."

The next step is to retrieve the documents from

the document tape (tape K). First, the program

selects the top 100 documents from the table (since

the table is ordered at this time on the scores, the

J- and if-values, the top 100 documents are the 100

"most relevant" documents) and makes a new table containing three registers for each document.

These three-register entries are then resorted on

the first register; i.e., on the document acquisition number. We then pass down the document

tape (tape K), picking up the required documents.

As each document is picked up, it is read into core and third register is used to record the core location

and size of the document. These three-register

items are then sorted again on the score contained

in the second register. This sort puts the table

in order of relevance again, so we go down the table printing the messages in order of relevance by using

the third register of each item to give the core lo- cation and size of the document to the print routine.

When all 100 documents have been printed, the

process can be repeated to get the next "bite" of

100, etc., until either there are no more messages

with non-zero scores or until the limits set by the user in the query deck stop the process.

8. References

[1] Luhn, H. P., Auto-encoding of documents for information

retrieval systems, presented at the Symposium on Documentation or a Study of Information Retrieval Systems,

Univ. of Southern California School of Library Science

(Apr. 9-11, 1958). Published in M. Boaz [id.], Modern

Trends in Documentation, p. 45-58 (Pergamon Press, N.Y.,

1959).

[2] Maron, M. E., and J. L. Kuhns, On relevance, probabilistic

indexing and information retrieval, J. Assoc. Comp. Mach.

7, 216-244 (1960).

[3] Doyle, L. B., Semantic road maps for literature searchers, 1

J. Assoc. Comp. Mach. 8, 553-578 (1961).

[4] Doyle, L. B., Indexing and abstracting by association, Am.

Documentation 13, 378-390 (1962).

[5] Stiles, H. E., The association factor in information retrieval,

J. Assoc. Comp. Mach. 8, 271-279 (1961).

60
The Construction of a Thesaurus Automatically

From a Sample of Text 1

Sally F. Dennis

International Business Machines Corp.

Chicago, 111. 60620

This paper reports the results of processing the first two phases of the automatic indexing project, which is a part of the American Bar Foundation-IBM joint study. From a data base consisting of the raw text of 2649 appealed cases taken from the Northeastern Repdrter, a dozen statistical parameters have been calculated to describe the distribution of each unique word in the file. The statistical information then has been used to determine which words are discriminating for a file similar

to the sample, and hence candidates for inclusion in a thesaurus. The frequencies of co-occurrence

within paragraphs of pairs of discriminating or "informing" words have been used to calculate an

association factor, which can be converted to a between-word "distance" for each significant pair. The work described here is part of an investigation aimed at producing an automatic method for thesaurus construction, and then indexing of text with respect to that thesaurus. It is hoped that the complete system will have the ability endlessly to reclassify the documents contained in it in response

to questions posed.

1. Introduction

In my early conversations with Mr. Eldridge on

the subject of the legal literature, he emphasized

the twin points that a lawyer frequently must have

a high degree of assurance that he has seen all of

the documents relevant to a question and that he

would tolerate a rather large proportion of "false drops" in order to gain confidence in completeness.

As a matter of fact, he stated on one occasion that

"if the lawyer found a third of the references fur- nished to him relevant, he would be well satisfied." Therefore, my efforts should be understood to in- clude the assumption that the important goal is completeness, although eliminating useless diluent

naturally is a desirable secondary goal. If completeness were unimportant, I should think a straightforward system such as John Horty's [l] 2 word concordance, or perhaps some sort of KWIC

index might be perfectly adequate for the material.

Also a part of the early discussions, with both Mr.

Eldridge and the other members of the American

Bar Foundation staff, was the question whether legal

literature and scientific literature are fundamentally

different. I have become convinced that they

really are not different, as long as you are talking

about literature couched in words. Some of the

lawyers have argued that scientific words are more

"precise" than legal words, and that this feature

changes the literature problem. I think it is true

1 The work reported in this paper is part of a system design study aimed at producing

, an automatic indexing program for documents consisting mainly of words. The

design and experimental implementation of this system are IBM's principal contri- butions in the American Bar Foundation-IBM joint study of legal information retrieval. Other parts of the total investigation, which are being carried out by American Bar

Foundation personnel under the direction of William B. Eldridge, include an analysis of the West "keynumber" indexing system, experimental manual key word indexing,

the collection of a set of real-life questions from practising lawyers for use in testing various legal information systems, and an analysis of users of legal literature.

I have received help from many people in the course of carrying out the work

reported here. Mr. Eldridge has contributed much information about the philosophical background of the law, the nature of legal literature, the uses that may be made of it, and the meaning of the specialized vocabulary. S. E. Furth of IBM Data Processing Headquarters has supported the project from its inception. My IBM technical advisory committee, Manfred Kochen, C. T. Abraham, Hugh Fallon, John Garland, and John Williams, have participated in a number of discussions £bout methods. The

personnel at the Chicago Scientific Service Bureau Corporation have been most helpful

in carrying out machine operations. Thirteen members of the regular research staff of the American Bar Foundation have participated in an evaluation of intermediate

results. I also have had illuminating conversations at various times with Mr. A. R. Geiger, and Miss Phyllis Baxendale, of IBM.

- Figures in brackets indicate the literature references on p. 72.

that the scientist has access to a tighter logic than

has the lawyer, but when he is using tight logic, he

reduces his comments to such economical forms as

tables, graphs, structural formulas, mathematical

equations, or other theoretical models. "One

hundred dollars" or "30 days" is about as precise

as "two thousand BTU's" or "65 nanoseconds,"

if the error is viewed in proportion to the measurement. On the other hand, a word such as "chromosome" calls to mind a living aggregate whose character is sharp in some aspects and blurred in others; "county" might be a possible legal analog.

The word "catalysis" has existed for many years

in chemistry as a grand but vague idea and much

effort has been invested in prying apart what it really means. I suppose a legal counterpart to "catalysis" might be something like "natural law."

At a more philosophical level, it seems to me that law and science have some clear-cut differences.

The most striking example is in the observance of

the principle of stare decisis, which says to the

lawyer that if a thing has been decided before, then

that decision is correct. No conscientious scientist would deliberately follow the principle of stare

decisis, although it may happen at times that inertia causes him to fall into it [2]. (It probably is less than accurate to state this so flatly. It is my impression that the lawyer sometimes judiciously

abridges the principle. But he does regard it as a principle.) This brand of difference causes the

legal literature to be used for somewhat different

purposes and to become obsolete less rapidly than

the scientific literature, but I believe that it does no affect the basic problem of storing and retrieving

the information: In either case the customer wants

to know what is there. In addition to examining with Mr. Eldridge the

character of legal literature, I read a good chunk

of the published material on document retrieval

and emerged from that exercise particularly impressed with the papers of Stiles [3], Maron [4], and Doyle [5]. Before my assignment to this project

I had been familiar with the ideas of Luhn [6] and

61
with the Western Reserve University semantic

code [7]. It seemed to me that a modification of Luhn's autoindexer that worked through a suffi- ciently powerful thesaurus might be an appropriate

solution to the problem. Then I commenced to think about the possibility of building a thesaurus

mechanically by adapting some of the ideas of Stiles,

Maron, and Doyle. The analogy between Doyle's

"semantic road map" and some psychological

models of the brain [8] appealed to me, and I in- dulged in lengthy introspection about what goes on in a man's head when he is thinking.

On the practical side, it seemed to me that eventually you should arrive at a point where it would not be necessary to start from scratch to develop a custom-made information system for each new document file that is to be automated.

There ought to be some basic mechanical recipe

that could be used to "grow" the system from a sample of the material that would be contained in it. In the fall of 1962 1 laid out a reasonably detailed

plan for constructing an experimental system embodying these general ideas. Grossly, the plan consisted of building a thesaurus from a sample of

text by combining association with the generation

of a "map" of words, which would be the the- saurus to the machine system and in another sense

a crude model of a composite man's head. Indexing and searching would take place with reference

to the map, and the map would be improved continuously by incorporating new information added

to the system as indexing proceeded. (Or, the composite man would "learn" by "reading.") To

reduce this foggy notion to a working outline, I described a five-phase experiment:

Phase I. Selection of "informing words." Informing words are the words to be included in the

thesaurus. I conjectured that words that behave

pretty much alike across a file would be noninforming, because they were nonselective, while

those that were used inconsistently in the frequency

sense should be "informing." A way to analyze

the difference between the two types of behavior

in a computer would be to assume that noninforming

words would exhibit a symmetrical distribution,

while informing words would appear skewed, if number of documents were plotted versus norNO. OF

DOCS.

SYMMETRICAL NONINFORMING WORD

NO. OF

DOCS.

SKEWED:

INFORMING WORD

NORMALIZED WITHIN- DOCUMENT

FREQUENCY

NORMALIZED WITHIN - DOCUMENT

FREQUENCY Figure 1.

malized word frequency within documents (figs. 1 and 2). If this measure of informingness seemed

to have any practical sense, it would circumvent

the objections made to selecting key words via

Figure 2.

frequency on the ground that rare words may be

the most important index tags. A word that

qualified as "informing" for the file always would

be used as an indexing word for a document, regardless of whether it appeared once or a hundred times

within a given document.

Phase 11. Computation of "association factor" or "between-word distance" for informing words. This was to be done by measuring the departure from

the behavior that would be expected of any pair of

words, if they were presumed to occur independently in the statistical sense. In other words, if a pair appeared independent, that pair was of no

interest. Pairs whose behavior could not reasonably be explained by assuming independence would

be called "significant."

Phase III. Construction of a word map from the information learned about between-word distances

in Phase II. Each word in the thesaurus would

be assigned a position on the map compatible with

the association information, and the coordinates of

its position would serve as a numerical "definition"

of the word. The definition of a given word would

carry with it information about the other words

with which the word was associated (fig. 3). Homographs would have only one numerical definition,

but the patterns of associated words in different

orientations with respect to a homograph would

distinguish its multiple meanings.

Phase IV. Indexing of new documents with respect

to the word map. The computer would read the

document, discard noninforming words, and plot

the remaining words on a clean map (or "grid")

FIGURE 3. Between-word distances plotted onto word map.

62
by referring to their numerical definitions. Clumps

of words appearing on the map would be treated

as "concepts" and the document finally would be

indexed, not by its words, but by the coordinates

of evenly spaced grid points that fell within the

domains of the "concepts" (fig. 4). In this way a concept would be defined by the profile of a document seen against the superstructure of the vocabulary of the file. It would not necessarily be named:

It could be broad or specific, and reclassification

would take place continuously.

Phase V. Searching test questions. A search

question could be framed either as a narrative or

as a string of words. No "ands" or "ors" would

be used, since the question would be analyzed by

the indexing mechanism. Words falling within a concept would be treated disjunctively and separate

concepts would be treated conjunctively. Density

of words possibly could be used to estimate order

4 j of relevance.

The five-phase experiment was to be carried

out using a sample of 5000 appealed cases, selected

chronologically from the Northeastern Reporter [9] by starting with the latest available case and working back. For this experiment, the first three

phases (comprising thesaurus building) would be

executed using half the cases in the sample, and

the remaining half would be used for experimental

indexing in Phase IV. At some point in this paper I want to defend myself to the nonbelievers in this symposium, and I suppose that this is as good a place as any to do it. Therefore, I now will delve into some more of the

thoughts that He behind this proposal.

Luhn's contention, that you ought to be able

somehow to take advantage of the organization that an author has injected into his writing, strikes me

as eminently reasonable. The people who start

. from this assumption seem now to be divided into

.

i two camps: the grammar worshippers versus the

statistics worshippers. I belong to the second camp

and will try to explain why. It seems intuitively

|J | plausible to me that there is something fundamental

about communication between human beings, via

words, that is independent of grammar. I can

understand people who do not speak grammatically, as long as I can make out their words. Chili

I dren learn English well before they know anything

of grammar. One method that they use extensively

to accomplish this feat is inference from context,

and as a matter of fact adults continue to use that method at least a part of the time after they have

been initiated into the rites of dictionaries. I see

the thesaurus-building program in one sense as simulating learning from context. Such a proce- dure is not error-free; statistical procedures are

a handy aid for separating error from (probable)

truth. Although I am personally a grammar addict

(I resent "Winstons taste good like a cigarette

should"), I have come to the point of view that the

rules are a finicky ritual, the knowledge of which

admits one to certain in-groups, but not the meat of

meaning. Really, the superstructure of grammar

arises only after language already exists in fact. The rules are changing rapidly in our language, and

obviously they change even more rapidly as you

pass from one language to another. It seems to me that it is wasted effort to try to teach the computer to understand grammar — unless your assignment is to perform machine translation, or you must

wring the entire meaning out of one sentence. As

long as you have multiple sentences available to scan, I am convinced that you can do a better job

of extracting meaning by examining words in context, with the aid of statistics, than you can by de- voting an equal amount of attention to grammar.

And I think further that in the course of studying

language in this way, you may learn some fundamental things about it that to date have not been

realized. Attacking from another angle, it seems to me

that it is the defeatist role to contend that indexing

cannot be done intelligently by machines. After

all, indexing is a very dull job for humans, and

they do it inconsistently, as they perform all dull jobs. Energy expended on learning how to relegate

those boring decisions to the machines, who dote on dull jobs and perform them very consistently,

will be more valuable in the long run than energy

spent on continuing to make the decisions in the same old way.

My next brand of iconoclasm is to push the argu- ment that you should not have to redesign a system

every time you encounter a new file. Taking the

fundamental approach, which is (by my own definition!) to tackle directly the problem of meaning,

with the aid of statistics, you eventually will know

enough about language to make a general-purpose

indexer. Such a general-purpose indexer may be

"calibrated" as this one is by feeding it a sample

of the material to be digested or by some other

means, but otherwise it should adapt itself to the

quirks of any individual file, and then it should

continue to adapt itself to the changing of those

quirks with time.

My IBM colleague in this Symposium, Jack

Williams [10], is studying the same problem from

a better ordered point of view. Our basic assumptions agree partly; they differ principally in that he

supplies a priori information in the form of a human-made hierarchical classification. I like his mathematics better than mine because it has a solid theoretical base, while mine does not.

I like my model better because it does not require

a starting classification, but at the moment my hope

that you can operate without a fixed classification has yet to be supported. Within the next few

months we expect to make operating comparisons on the legal text base to obtain an estimate of

what is bought and sold as you go from a system

supplying a classification at the outset to one that

tries itself to reclassify to meet the demand of the

moment.

63
2. Experimental Procedure and Results

2.1. Sample Data Base

The data base consists of approximately 5800

cases taken chronologically from the Northeastern

Reporter [9]. The geographic area covered by the

section of the Reporter includes New York, Massachusetts, Michigan, Ohio, Indiana, and Illinois. The time period runs from 1959 to 1962. The

material was keypunched and transferred to magnetic tape by John Horty's group at the University

of Pittsburgh; this work was supported by a grant

from the Council on Library Resources, which was

obtained for this project by the American Bar

Foundation. No key numbers or headnotes are included in the keypunched material. The format was chosen for compatibility with Prof. Horty's

"key word-in-combination" system [1] so that comparative operating tests could be made to include his system easily. No verifying was done,

but the tapes were edited at the completion of

their preparation with the help of a word con- cordance. A partial set of uncorrected tapes was made available in July 1963, and I commenced the

work on Phase I in exploratory fashion at that time. In November 1963 the complete corrected

tapes were delivered, and I thereupon redid the

earlier work, taking advantage of the first experience

to improve procedures where possible. Except

where noted, the work reported in this paper will be

that carried out on the corrected tapes.

2.2. Computer Processing — General Remarks

Part of the exploratory processing was done on

the IBM 7090 at the Datacenter in Cleveland, Ohio.

All of the final work was done on the IBM 7090 at the Service Bureau Corporation in Chicago. Both

machines were equipped with 12 tapes and no other

form of bulk storage. All programming has been

done using FORTRAN IV and the 90 SORT under

the IBSYS monitor. (Since the FORTRAN IV

programs are experimental, they are not available

for distribution.) The programs cited in the fol- lowing are in general those by which final processing of the bulk data was done. As a matter of

expedience, however, I resorted to considerable

debugging of theory during the debugging of pro- grams. That is, I would incorporate a selective trace into each program while debugging, which

gave me an opportunity to examine partial results before bulk processing was executed; in several

instances the partial results caused me to decide

to change the intent of the main program. Some

results of such partial processing will be reported

in the sequel, where they seem to be of interest. To convey an idea of the bulk of material that was

handled in order to carry out this work, I have noted

in the program tables (tables III and XII) the number of reels for the large files. All large files were

blocked at approximately 1350 words, which was

the limiting input block size in the FORTRAN

system that I used. The original text file, prepared on the IBM 1410 at the University of Pittsburgh, was blocked at the equivalent of 332 7090 words.

Many of the bulk processing runs were of several

hours' duration. For example, to prepare the con- cordance of Phase I, Step 1 (table III), 20 seconds

per document was required, and for 2649 documents this turns out to be almost 15 hours. Service

Bureau Corporation personnel performed all of

the bulk operations.

2.3. Computer Programming — Phase I One matter that had to be settled at the beginning was what unit was to be regarded as a "word."

Should there be a program to strip off prefixes and

suffixes so as to unite stems, should each word

be retained in its entirety, or should it be truncated

at some arbitrary number of characters? What

should be done, for example, with hyphenated

words? I interviewed a number of people about

these questions before starting, in the hope that I would run across data that would support one decision or another. Unfortunately, there seems to exist no information other than opinions — and these

are diverse. In the absence of information, I made

the decision that would simplify programming and

reduce bulk, which was to define my "word" as an uninterrupted string of alphabetic characters

three to six characters in length. Therefore, all one- and two-character words have been dropped

and a hyphen behaves as a blank to begin or end

a word. The possible number of combinations of

three- to six-character alphabetic strings, if one presumes two vowels, is about 12 million, and so

it would seem that there is ample room to accom- modate the vocabulary in this format. The question that remains, of course, is whether or not the

real vocabulary makes even moderately efficient use of six characters. Truncating produces arti- ficial homographs, which may be a loss, but it collects words related through their roots, which

probably is a gain. Since I propose to deal with

homographs through their relationships with other

words anyway, I have shrugged off that problem

for the moment.

Data from other sources [11] would indicate that about 25 percent of the words in a sample of text

will be one- and two-character words. Therefore,

my document word counts should be adjusted by a

factor of 4/3, if one wishes to compare them with

other word counts.

In planning the program content for Phase I,

I took the position that since my theory was merely

that "informing" words could be selected on the

basis of their unusual distribution in the file, it would be in order to examine as many parameters

as I could think of, that might serve as a measure

of this characteristic. (As a practical matter it also is true that calculations come very cheap on

the 7090, once you have pushed the input in and

64
the output out — and so you might as well find out everything that you can all at once.) Therefore,

at the beginning I decided to calculate the follow- ing quantities for each word:

(1) NOCC, the total number of occurrences or

tokens. The standard procedure for eliminating

words is to make a list by hand; if any mechanical

assist is used, it ordinarily is the frequency of

the word in the file. It would be of interest to compare this method with the others.

(2) AVG, the average normalized frequency within

documents. This was calculated as the number of

tokens for the given word within a document divided

by the number of tokens for all words in the document, averaged over all documents in the sample.

Using the normalized frequency raised the question

of what spurious effect would be produced by including very short documents, and therefore, in the

first round I used only those documents whose

length equalled or exceeded 750 words.

(3) S2, the variance of the figure described in (2).

(4) S, the standard deviation or square root ofS2.

(5) G, gamma, the coefficient of skewness. G was

calculated as the third moment about the mean

divided by S cubed.

(6) B, beta, the coefficient of excess. B was

calculated as the fourth moment about the mean

divided by S2 squared.

(7) PZD, the percent of documents in which the word occurred. This calculation was suggested

by Fred Kochen. He pointed out that if the

basic idea were right, PZD might be a cheap way

of approximating it. (8) EK, the Erlang K number, which has been

used to characterize Poisson distributions in queueing problems. My attention was attracted

to this measure by an internal IBM research paper

by Yin-Min Wei. The number actually is the mean

squared divided by the variance, and so it can be

regarded alternatively as the square of the signalto-noise ratio, or as the reciprocal of the square of the coefficient of variation.

(9) The fraction of the expected number of documents in which the word occurred. I added this measure to the fist while debugging during the first round of testing, because I could see that while

the words that appeared in all or nearly all of the

documents obviously were noninforming (e.g.: the 100 percent, and 99 percent, that, for 98 percent,

not, which was, this, with, from, court above 90

percent), there were many noninforming words,

by subjective judgment at least, that appeared

in a low percent of documents. For example, be- come appeared in 25 percent, instead in 10 percent,

quite in 9 percent. The reason for this would seem

to be that these words just are not used as much in the total vocabulary. Perhaps to the extent that they are used, they would exhibit a flatter distri- bution across the documents than would informing

words. To calculate the expected number of documents I supposed that each document receives its words from a pool consisting of all of the tokens

in the total file. For example, when the quite

tokens, of which there were 307, come up for dis- tribution, assume that all documents are waiting

for words, but concentrate attention on document

number one. The probability that document one

will not receive the first quite token, assuming all documents have an equal chance, is {N—l)/N,

where N is the total number of documents participating in the pool. The probability that document

one will receive neither the first nor the second quite

token is ((TV— 1)/A02. And the probability that document one will not receive any quite tokens at all

is ({N— 1)/AO NOCC

. From this probability can be

calculated the expected number of documents

in which quite would appear at least once, if the

number of tokens in the pool were distributed by

chance. The fraction of expected then is the observed number of documents in which the word

actually appeared divided by the expected number.

This measure proved to be an almost unbelievably

poor test of the sought-for property in the first round of results, and so I dropped it from the refined

program. The fractions varied over a range from

about 0.25 to 1.008, but unquestionably were not measuring the right thing. For example, the word

and and the word ketchup both appeared in 1.008

times the expected number of documents.

The error in logic would seem to be in presuming

that the words in the pool were the words avail- able; certainly it can be argued that any word at

all is available to an author at the time he is generating a document. All of the other measures,

which imply that all words in the universe are avail- able to every document, perform much better than

this one. (Nevertheless, it still seems to me privately that for the individual writer producing a document about some subject, there are more

tokens of some words than of others available for him to use, and I do not fully believe this rationalization.) In addition to the quantities defined in the fore- going, I obtained as byproducts during the first round of processing the following extra information

for every fiftieth word and some selected words

{the, and, above, about, law):

(1) All of the mentioned parameters at intervals

of 100 documents.

(2) A discrete tabulation of the final distribution.

(3) The average normalized frequency (AVG),

segmented for documents containing (a) less than

375 words, (b) from 375 to 750 words, (c) from 750

to 1500 words, and (d) more than 1500 words.

Samples of some of these data are shown in tables

I and II and plots of some of the distributions are shown in figure 5. (All of this information comes

from the first set of unedited tapes, but there was

nothing in the final processing that would cause one

to doubt the approximate correctness of these data.)

The behavior of the parameters in the intermediate

document intervals would seem to suggest that about 600 or 700 documents are sufficient to characterize the information. Jumping the gun in

this account a bit to presume that one can select

65
AND FOR -, LAW

ABOUT NJURI

Ihh.

FIGURE 4. Concepts defined as grid-points contained within boundaries determined by document words. from these parameters a criterion for distinguishing informing from noninforming words, I would

suggest that if one were to apply these procedures

to a real-life file rather than in a research project,

it would be reasonable to follow the progress of

each word as documents are added to the file and to commence to drop out a word whenever its criterion has passed a conservative threshold.

The average normalized frequencies for documents of different lengths tell about the story that one might expect. The clearly trivial words (e.g., and) appear at about the same normalized frequency

regardless of document length, but the potentially

informing words decline in normalized frequency

as document length increases.

When the edited tapes arrived at the end of

November 1963, the above work had been completed

and it appeared to me that the best criterion for distinguishing informing from noninforming

words was going to be EK, followed by G and

PZD in that order. About that time I received an

informal communication from H. E. Stiles, in which

he proposed that the "information statistic" be

tried as a measure of "roughness" of the word in a file. The information statistic can be calculated

from the formula

f fi "fu

where fy is the frequency for word i in document

j and fi is the frequency for word i summed over the

j documents. This expression is equivalent to negative entropy.

It seemed to me that Stiles was thinking about

the same general idea as I, but with a fresh ap- proach, and so I decided to include his suggestion

in the final round of processing, along with the three

"best bets," EK, G, and PZD.

I also decided to give further consideration to the effect of length of document on normalized

frequency. Instead of eliminating all documents

shorter than 750 words from the distribution cal- culations, as I had done in the first round, I pro- grammed those calculations three ways:

(a) as before, but using all documents longer than

650 words;

(b) using all documents, but normalizing with the

log of the length of the document rather than with

the true length;

(c) dividing the file into sequences of documents

so that the boundaries came at the end of the first document in the sequence such that the total word

length of the sequence would be greater than 5,000. In the course of debugging the program, I found

that the 5,000-plus word boundary made every word

appear to be a trivial word, and so I deleted that calculation! This second example of establishing

what is a very bad idea bears a more clear-cut message than the first: the document boundary is highly important in characterizing this behavior

of words.

To summarize, final bulk processing of Phase I was executed as follows:

(1) The sample was 2649 documents, of which

2023 were longer than 650 words. All computations were done two ways: using only the documents longer than 650 words and normalizing with

true document lengths, and using all documents

but normalizing with the log of the document length.

(2) A "word" was a 3- to 6-character continuous

alphabetic string.

(3) The following distribution criteria were cal- culated for each word: NOCC, PZD, E (negative

entropy), EL (negative entropy using log normalizing

factor), EK, EKL (Erlang K using log normalizing

factor), G, GL (gamma using log normalizing factor).

The processing steps required to develop the in- formation are outlined in table III. Each step

represents a "batch." For each batch there is input, a processor program, and output. In most

cases the output from one batch becomes the input

for the next batch.

66
Some summary figures from the final Phase I processing are: DOCUMENT STATISTICS:

Docs longer than 650 All words documents

The results summarized by word are the following:

Number

Avg length Std dev of length

Gamma of length

WORD STATISTICS:

Total token count

Total type count

Total type count, eliminating words appearing in only one document

2023

1712.5

1149.7 2.495

3.8 million 30 thousand

16.2 thousand

2649

1384.9

1167.5 2.270

The range of the length of documents is from about

30 to about 10,000 words.

The first output from Phase I processing that be- comes interesting in the sense that the theory of

selecting informing words from file distribution can be tested is the eight sorted lists of 16,200

words, noted in table III. These lists are a bit too long to include in this paper! However, sorted

lists of a subset of 454 words, selected because

by any one criterion they appeared among the first 300 words are to be found in tables IV through XI. (The first 300 words or the "top of the list" are in general the words to be skimmed from the file as noninforming words, plus borderline words.)

By visually appraising the various sorted fists and

then checking my reactions against those of Mr.

Eldridge, I came to the conclusion that a practical

rule for skimming the noninforming words would be

to eliminate all having either an EK value greater

than 0.30 or a GL value less than 4.0, and this criterion was used actually to purge the concordance

file (see table III) for use in Phase II. However, on a subsequent date I had an opportunity to test the list against the subjective reactions

of a committee of 13 members of the research staff of the American Bar Foundation. I submitted to each member of the committee a list of the 454

words, together with an explanation of how they

had been obtained, and asked each individual to form an opinion as to whether or not he would want

to have access to any of the words on the fist in an index. (The list that the committee worked from was ordered alphabetically, not by any test parameter.) He was to check off the word if he wished

it retained and to construct an example of the

way in which it would be used.

The number of words checked off by any one indi- vidual ranged from 13 to 156, with an average of 64

words. The individual who voted to retain only 13 words was the only active librarian in the group. The rest are research attorneys and administrative

people who know the vocabulary, but would have

had no reason to give extended thought to the problems of indexing.

of Votes No. of Words Total No.

Retain of Votes

0 226 0

1 56 56

2 44 88

3 29 87

4 23 92

5 27 135 6 13 78

7 12 84

8 8

9 10 90

10 3 30

11 1 11 12 0 0

13 2 26

Totals 454 841 To study the relation of the committee's evaluation

to the proposed tests, I summed the votes for each

page (59 words per page — see tables IV through

XI) for each ordering. Because of the disparity

of opinion within the committee, I also summed

the votes per page, eliminating those for which fewer

than six individuals had agreed that a given word

should be eliminated.

The cumulative sums, page by page, including

all votes, are the following:

Test used as basis for ordering:

NOCC PZD E EL EK EKL G GL

1 113 81 44 59 38 44 36 39

2 248 227 182 184 141 143 130 105

3 435 339 302 297 251 267 202 155

3 587 508 467 425 371 360 283 234

5 714 718 618 564 425 478 394 305

6 778 772 728 718 663 640 517 457

7 823 822 815 816 798 773 659 646

8 841 841 841 841 841 841 841 841

The corresponding sums including only those votes

for which there was agreement by six or more

individuals are: NOCC PZD E EL EK EKL G GL

1 64 51 18 27 18 18 19 7 2 135 124 87 89 61 68 43 40

3 227 164 139 135 92 115 65 54

4 312 238 214 186 154 142 98 74

5 354 357 288 231 229 172 150 100

6 363 363 327 319 281 277 193 177

7 383 383 383 383 367 352 271 261

8 383 383 383 383 383 383 383 383 By both methods of counting, the tests improve in

ability to push words that should be retained to the

bottom of the fist as you move from left to right across the list of tests. This performance is shown

schematically in figure 6. In the tests where ordi- nary normalization is compared with log normalization, the log test consistently exhibits a small

improvement.
WORD RANK

Figure 5. Selected discrete distribution.

Horizontal axis is scaled to 11 equal segments. In this analysis NOCC (total number of occur- rences) walks away with honors as the very worst

test for locating insignificant words. Perhaps

the reason why it has appeared moderatefy satis- factory in the past is that when it has been used,

it has been post-processed editorially; and enough

of the time the performance measured by NOCC

happens to coincide with the performance that is really desired to make it appear a satisfactory

screening criterion in the absence of other

information.

It is not terribly taxing to construct an explanation for the relative efficacy of the test called GL:

Discrimination increases with the skewness of the

word distribution in the file, and the log of the

document length is a slightly better normalizing

factor than the raw length because writers tend to avoid repeating discriminating words.

If one studies the lists of words in tables VIII through XI, he can see that EK and EKL seem to be selecting different types of words from G and

GL. Arithmetically, EK is the squared signalto-noise ratio. But this ratio is highest for the

least informing words; the low values select the

informing words. Therefore, it appears that the

word that behaves like noise in the file is the one

that tells the story, and this is consistent conceptually with the skewness measure of selection.

Perhaps the indication that these two tests select words somehow different in type is a clue that there

could exist a test better than either for the purpose.

An interesting specific example is the difference

between the words shall and will. At first glance

both words are merely auxiliary verbs, which should

be trivial, but on second thought one notes that will

carries at least two special legal meanings: a major

meaning in the sense of "last testament" and a sec- ondary meaning in terms of "against his will." Will occurs 7140 times in the sample, while shall occurs only 6240 times. However, will is classified as informing by both tests, while shall is noninforming by the EK test and borderline by the GL

test.Another interesting point is that of the set of

454 words assumed to be nonimforming, only two were rated informing by the entire panel of 13 at- torneys. The two words so rated were notice and

jurisd. The test also retains both words. At the

other extreme, the panel and the EK-GL test agreed on 189 words as noninforming. I suspect

that the panel might agree that some of the words

rejected by the test should have been rejected by

the panel; for example states was not marked by

any of the panel, but it probably is an informing

word in the sense of "United States" or "states'

rights," if not others. States has, of course, also a trivial usage. One of the uses of the test is to distinguish between words that really are used

trivially most of the time and those that sometimes

have a specialized meaning.

2.4. Computer Programming — Phase II The objective of Phase II was to find all of the

significantly occurring word-pair combinations in the concordance from Phase I, once the noninforming words had been purged. The bulk processing

steps for Phase II are outlined in table XII. Purging by the combined EK-GL rule and eliminating

words occurring twice in a paragraph reduced the

number of words to be processed from 3,800,000 to 1,225,000. I had decided to analyze the pairs in terms of their concurrence within paragraphs, be- cause this seemed to me the "best bet," although one could make a case for doing this with a document, sentence, or phrase boundary — or perhaps

within a string or words of some arbitrary length.

There were about 64,000 paragraphs in the file and therefore an average of about 20 informing

words per paragraph. On a machine with no bulk

random access storage, all combinations would

have to be written out and then sorted and counted.

The number of combinations based on the 20-word

average would be 20 X 19 X 64000 X 0.5 or about 12

million, which, blocked at 1350 words, could be

accommodated on about 25, 2400-ft 556 BPI reels. However, the lengths of paragraphs varied greatly, some paragraphs containing as many as 80 informing words. The number of combinations per para- graph increases exponentially with the words per

paragraph, and so it would not be reasonable to write out all combinations, even if one regarded

25 reels as a feasible quantity of intermediate output. (The Service Bureau did not!) Therefore, in order to fit the job into the physical facilities, I decided to sample the words that occurred in large

numbers of paragraphs. (The number of para- graphs in which any one word occurred ranged from

2 to 7,000. For words that occurred in only a few

paragraphs you would want all the information

that could be extracted from the file, but for those

occurring a large number of times, hopefully, you

could estimate from a sample.)

In order to explain how the sampling was done,

it is necessary to describe the association calculation to follow (Step 5, table XII). For word pairs

in which the words behave independently of each

other, their expected number of co-occurrences

within paragraphs is (NPl)(NP2)/64,000, where

NP1 is the number of paragraphs in which the first word is known to occur, and NP2 is the correspond68
ing number for the second word. Any word pair

for which the co-occurrences summed to less than

this number would be of no interest as a potentially

significant pair. However, for words appearing

a small number of times in the file, e.g., 15 and 20,

the expected number of co-occurrences is a fraction,

and so by this formula one co-occurrence would

appear potentially significant. Obviously, every

word has to appear with some other words, whether

the co-occurrence is significant or not, and so one

co-occurrence cannot be taken seriously. Therefore, as a preliminary screen in the association

calculation, I planned to drop out all pairs whose

co-occurrence did not exceed one or (NP1)(NP2)/

64,000 — whichever was larger. I made the assumption that for the words that appeared most, the pairs

of primary interest would be the ones in which both

words occurred large numbers of times. (This

assumption at best is only partly true; it was forced more because something had to be done than from

conviction.) Then I sampled in such a way that for

a word occurring, e.g., in 600 paragraphs, I would

expect, on the average, to count only one co- occurrence with another word occurring 600 times,

if in fact in the case of independence, the real number of co-occurrences would be (600)(600)/64,000 or about six. As a practical matter, this boiled down

to a rule that said: If the word occurs in more

paragraphs in the total file than the square root of

64,000, reduce the number of times that you use

it in generating pairs to the square root of 64,000,

divided by the number of paragraphs in which the

word actually occurs. The rule was implemented

using a random number generator, and only 18

reels of intermediate output were generated. Of

course with random access storage, the pairs

could have been looked up, counted, and the association calculations made without the need for writing them out and sorting, and one then would

use somewhat different procedures.

In the course of debugging Step 5 (table XII),

where the actual calculations were performed, I tried several different association or "distance" measures. Calling A the number of paragraphs in which word A occurred, B the number of paragraphs

in which word B occurred, AB the number of para- graphs in which both occurred, /V the total number

of paragraphs, and letting A be the smaller of A

and B,

R2 =

(AB-(A)(B)IN)2 (A-A2 IN)(B-B2 IN)

(1)

The above formula for R2 corresponds to the ordi- nary statistical formula if occurrences are coded 1 for present and 0 for absent. R2 would be an at- tractive measure, if it seemed to make sense

empirically, since 1 — R2 is the square of a geometric

distance to which can be attached the idea of error

or noise. And so you would have a geometric

distance with an operational meaning that fits the context of the problem: the longer the distance,

the less closely associated the words. However,

because N is very large in relation to most A's, fi's, and AB's, the calculation is in most cases approximated by (AB)2 /(A)(B); that is, it is not telling the

statistical story that one tacitly expects. Stated

qualitatively, the situation is that you are assigning

as much value to the information about A and B in the paragraphs in which neither of them occurs, as you are to those in which one or both occurs. This

makes no sense if you consider the fact that you

could be looking in the wrong file! Therefore, I also

tried a modification of R2 based on the established

fact [12] that if you are sampling for a 2 X 2 contingency table, the most efficient sample size is 2A (A

less than B). That is, you sample A paragraphs containing word A and A paragraphs not containing

word A. The legitimate way to count the number of ZTs in the no\-A group would have been to simulate

sampling using the random generator, but since B

was known for the population, I calculated theoretical B for the sample of size 2A. This would

make the variance of the /? 2 's less than it should

be theoretically, but that seems hardly a drawback

for an empirically based investigation.

(2) (AB)/(A+B — AB). This measure has some

intuitive sense as the number of actual co-occur- rences divided by the total number of paragraphs

in which there possibly could exist a co-occurrence.

(3) AB/A. This one is the conditional probability of finding B in the set of paragraphs containing A.

(4) (AB - (A)(B)IN)/\/(A)(B)IN. This is an ap- proximation derived from the formula for standardizing a binomial di stribution. The conventional

formula is (S — np)lvnpq, where S is the observed

number of "successes," p is the probablility of success, n is the number in the sample, and q is \—p or the probability of failure. Since p is always small (for two words each occurring separately 1000 times, p would be about 0.00025), q is effectively 1, and therefore q has been omitted

from the approximation. The "meaning" of this measure is the number of standard deviation units

the observed co-occurrence falls to the right of the

value expected, if the words in the pair were oc- curring statistically independently (fig. 7). The

larger the number, the more reason to presume

dependency.

In the course of debugging I observed the behavior of the four measures for about a thousand

word pairs. Exercising entirely subjective judgment as to the sense of the results, I decided that

T

EXPECTED FREQUENCY

OF OBSERVATION

2 3 4 5 6 7 UNITS OF STANDARD DEVIATION

FIGURE 6. Schematic representation of test comparison.

69
the fourth measure was best. Therefore, the bulk

processing program calculated units of standard

deviation only, although the raw occurrence data were carried along so that if at a later date it seemed

desirable to examine other quantities, the input

information would be easily available.

The final report (from Step 6) is a list, for each

word in the thesaurus, of all the words co-occurring

significantly with it, listed in descending sequence

of number of standard deviation units. All words

occurring in fewer than 15 paragraphs in the total

file and all word pairs for which the number of

standard deviation units was less than 15 have

been deleted from the list — again, as the result of a subjective judgment as to where the "garbage

level" became obnoxious.

At this point there were approximately 7,000

words (types) remaining in the thesaurus. The

starting set had been 15,780 words — 16,200 minus

those deleted by the EK-GL test. The "lost"

words fall into two categories: Words that are trivial in addition to those skimmed off by the EK-GL test, and words that have not yet appeared sufficiently

often in the file to build a case for themselves

statistically. The extra trivial words are the ones

that do not occur in significant quantities in the main

file, but occur more or less randomly when they do

turn up. In a real, dynamic system, you would

continue to collect information about these words

and some of them eventually would arrive in the

thesaurus, as would some new words that as yet

have not appeared in the input.

Several sample pages from the list that comprises

the final report from Phase II are shown as table

XIII. In an effort to form some summary opinion

of the content of the fist or thesaurus, I have

played some games using the full fisting. The first was to start at the beginning of the alphabetical

Relation of associated to main word

Main

word

Root male Synonym or near-syn Antonym Otherwise

related Related to each other* Notobviously

related

aaron unsoun death mind poison deeds

elizab norm an

quickl proper

error

abando desert

discern use suppor

evicti

unfit provoc electi discon govern

city

moline

hurt reeder

abate abated abatem

caused nuisan

tax sanita slande sewage libel rubbis

fires

fox proper

additi

abate abated

abate

cliange pollut board

sanita truste assess

plea tax answer taxes

pleadi

proper

abilit inabit skill inabil financ impair suppor

perfor

list and analyze the "thesaurus set" for several

words in terms of grammatical relationships:

Those words are more or less fact-oriented, and

so I moved on in the list and chose a more "legal"

word, admiss:

Main

word

admiss

Root mate

admit! admit

Synonym or near-syn Antonym

reject exclud

Otherwise

related

wigmor

introd object statem

denial memory

view accuse

recove commit

hearsa

proof prove

arrest manner

addict confes except judge

lenien

parol sponta answer

declar

guilt infere

settle equivo

gestae

indict instru observ

convic

credib

disc ha

prejud

Not obviously

related

bryson

proper

teachi

writte

arts chicag conver

itemiz

*bul less obviously to the main word

Stiles, in his association work with descriptors

[3], suggested that, while it would be unlikely that one would find synonyms for the main word in its "first-generation profile" (corresponding to the

thesaurus set here), because an indexer would tend

to avoid indexing a document with synonyms, the

synonyms might be found in the second-generation

profile — that is to say, the terms with which the

words in the first-generation profile were highly

associated. His comment with respect to syno- nyms does not apply to the thesaurus profile gen- erated from within-paragraph co-occurrence; evi- dently writers of discursive text (or at least legal

writers) do use synonyms within paragraphs.

However, it would be of interest to examine the

"second-generation profiles" to see if more syno- nyms are unearthed in this manner than would be

found simply by inspecting the original set. To pursue this idea, I took as a starter the word

debt. The original thesaurus set for debt is debtor,

debts, mortga, indebt, proper, exting, ohio, discha,

exoner, credit, money, bankru, finds, pay, taxes, paymen — oi which the first four words are near- synonyms for the header word. (I am counting the

word a synonym or near-synonym if it deals with

the Same idea, even if it does not take the same

grammatical form.) Running through the associ

ated lists for all of the words in the original set, I collected the following synonyms or near-synonyms

for debt:

70
debtor debts mortga

indebt claim owing

debent lien liens obliga pledge loan

loans fines bonds

levies balanc defici encumb liabil shares

Choosing a second word of rather different type,

saturd, I found the names of all seven days of the

week included in the second-generation profile.

This says to me that at the least, the ' thesaurus

sets" constructed by this mechanism could be used

as an aid to refining questions posed to a file consisting simply of concordance, or they could be used

as the starting-point for the preparation of a handedited thesaurus, and further, that it is not unrea- sonable to think they can be used to make the "map"

proposed as Phase III of this study.

3. Summary of Results and Discussion

Since this really is a progress report on the entire

study and no system performance data are yet at hand, I will summarize the results only in terms of what seem to me to be the fair statements that can

be made now about the theories underlying the work.

(1) I think it is true that you can filter off the meaningless words from a document body by examining

their distribution in the file, and that if you are in a

position to do automatic indexing, this is a better

method than hand selection. The best measure

that I have uncovered to perform this job is the coefficient of skewness and the second best is the

ratio of the mean squared to the variance. Both

statistics are computed with respect to normalized

within-document frequency. The log of the length

of the document is a somewhat better normalizing

factor for this purpose than the true length.

(2) I suspect that if someone were to carry out a more sophisticated analysis of the word distributions, he would have a good chance of finding a more powerful measuring tool than the coefficient of skewness.

(3) Neither the raw frequency in the file nor the

number of documents in which a word occurs is a very good measure for distinguishing trivial words.

(4) There probably is enough information in word

pair association within paragraphs to form the basis

for the construction of a thesaurus suitable for reference in indexing and retrieving documents.

(5) There are uncountable bypaths that would

be interesting and possibly useful to investigate. In addition to further examination of the relation

of distribution to word significance, some questions

to study would be boundaries for pairing behavior

other than paragraph, how many times a word must

appear in a file before the pairing data become significant, whether words that have not yet appeared

enough times are important as index terms, and

what is the most efficient procedure for sampling

words. Since my plans are to move on to the next

phase, I do not expect to pursue any of these points.

If I were to turn around and apply the methods

of Phase I and Phase II to a "for-real" file, I would

use them dynamically rather than statically, as was done here. That is, for Phase I, I would choose some conservative threshold of the test criterion

for distinguishing noninforming words, and whenever a word in my input data passed over that threshold (after some minimum number of documents, say 400), I would commence to drop it from

the file. Concurrently, I would generate all pair

combinations, but when the total number of oc- currences of any given word exceeded some pre- determined limit, I would commence to sample its pairing performance in proportion to its total number of occurrences. I would, of course, use a machine system having available a large, random

access storage! When for an interval of, say 100

documents, I had found no new noninforming words

to drop, I would discontinue the Phase I test. The

pair sampling, however, would go on indefinitely,

although the basis for sampling might be changed

from time to time. In the specific case of the legal literature (and

analogous comments may apply in others), my

original file sample would come from as broad a base as possible. The sample used in this work was chosen as a broad base, and in terms of subject

matter it is, but it is limited geographically and

chronologically. This did not occur to me at the

time that the sample was being chosen, but I believe now that if instead of having picked 5000

cases from the Northeastern Reporter sequentially

in time, we had sampled 5000 cases from across the

country and over a period of perhaps 10 years, the

present thesaurus sets would be rid of many of

the individuals' names that are meaningless for

this particular file. Some names are not meaningless with respect to subject content. For example,

taft and hartle turn up as an associated pair, as do

wigmor and hearsa. But names of individuals

participating in suits such as aaron and elizab as well as names of judges are not meaningful. Possibly other provincial influences would appear in

files dealing with other subject matter; in selecting

samples it would be advisable to consider what these

might be so as to minimize their effect. The final object of this study is, of course, to build a working pilot information storage and re- trieval system — not simply to construct a thesaurus

with which to become fascinated. A still more

final object is to compare the pilot system with other

pilot systems based on different theories. At this writing the machinery for executing comparative

testing is getting slowly underway. Mr. Eldridge

is collecting a set of 200 questions; as of now he

has received commitments to participate in framing

questions and evaluating answers from 80 Fellows

of the American Bar Foundation, and thirty ques71
tions have been actually received. Formal plans

have been made for comparing only two systems,

the discriminant function classification method of

John Williams, and this one. It would be desirable

if more methods of other types, including grammatical analysis and citation indexing, were a part

of the test. I suggest to the members of this Symposium that the direct path to finding out which

methods or combinations of methods are really

going to do the job is to make such comparative

tests, and I hope you will consider these comments

an invitation, if not a challenge, to join in.

4. References

[1] Horty, J. F., The "keyword in combination" approach, M.U.L.L. 62M, 54 (1962). Also Kehl, Horty, Bacon, and

Mitchell, An information retrieval language for legal studies. Assoc. Computing Machinery Commun. 4, 380-89 (1961).

[2] Kenyon, R. L., Problems with conservatism, Chem. Eng.

News, p. 7 (Apr. 20, 1964).

[3] Stiles, H. E., The association factor in information retrieval,

J. Assoc. Computing Machinery 8, 271 (1961).

[4] Maron, J., Automatic indexing: an experimental enquiry,

J. Assoc. Computing Machinery 8, 404-17 (1961); also Maron and Kuhns, On relevance, probabilistic indexing, and information retrieval, J. Assoc. Computing Machinery 7, 216-44 (1960).

[5] Doyle, L. E., Semantic road maps for literature searchers,

J. Assoc. Computing Machinery 8, 553 (1961); also, Doyle,

Indexing and abstracting by association, Am. Documentation 13, 378 (1962).

[6] Luhn, H. P., The automatic creation of literature abstracts, IBM J. Res. 2, 159 (1958).

[7] Perry and Kent, Machine Literature Searching, Interscience Publ. (1959).

[8] Good, I. J., The mind-body problem; and Russell, C, and W. M. S. Russell, Raw materials for a definition of mind.

Theories of Mind, ed. J. Scher (Glencoe Press, 1962).

[9] West Publ. Co. Reporter Series.

[10] Williams, J. H., Jr., An application of discriminant analysis

to automatic document classification (this Symposium).

[11] Count of words in the book Digital Computer Primer, by

E. M. McCormich (McGraw-Hill).

[12] Anderson, Introduction to Multivariate Analysis (John Wiley & Sons).

Table I. Selected intermediate data from Phase I

THE LAW INJURI No.

of Docs. NOCC AVG G B EK NOCC AVG G B EK NOCC AVG G B EK

24010 11.74 0.419 3.44 43.7 448 0.23 1.84 6.66 0.74 62 0.036 2.54 8.96 0.20 100 46912 11.91 -.91 9.00 37.6 976 .26 1.59 5.60 .93 107 .030 2.79 10.76 .18 200

71181 12.00 -.58 7.30 38.2 1581 .27 1.46 5.28 1.10 180 .035 3.69 22.40 .16 300

94022 12.14 -.36 6.10 38.2 2196 .29 2.79 10.00 .94 218 .031 3.89 23.91 .14 400

116045 12.06 ' -.22 5.70 39.8 2690 .29 2.76 16.90 .96 299 .032 3.67 21.52 .15 500

138378 12.07 -.56 7.34 38.8 3149 .28 2.77 16.42 .93 387 .033 3.55 19.05 .15 600

158785 12.09 -.51 6.95 40.1 3545 .28 2.75 16.27 .93 436 .033 5.05 43.21 .13 700

181783 12.07 -.44 6.48 40.1 4042 .27 2.64 15.13 .91 522 .035 5.06 41.41 .13 800 225044 12.10 -.57 6.90 38.8 5059 .27 2.55 13.59 .88 664 .036 4.78 37.52 .13 1000 270081 12.07 -.46 6.24 38.8 6027 .27 2.57 13.94 .89 818 .036 5.18 43.38 .13 1200 314863 12.06 -.40 5.83 39.4 6937 .26 2.57 13.71 .87 951 .036 5.22 41.85 .12 1400

72
Table II. Selected data showing change of average normalized frequency with document length

Table III. Programming steps to accomplish Phase I.

Length of 1500 words 751-1500 376- 750 30-375

documents words woi ds \ No. of Avg. No. of Avg. No. of Ave. No. of Avg. Term \ docs. docs. docs. docs.

About 462 0.14 351 0.20 95 0.32 16 0.60 Above 431 .09 260 .14 92 .25 33 .83 Accoun 256 .16 161 .24 40 .50 8 .62 Affirm 569 .12 536 .19 224 .33 129 1.04 And 742 3.49 814 3.45 416 3.34 316 4.06 Case 718 0.43 738 0.46 347 0.59 151 0.88 Consti 470 .18 352 .23 108 .39 46 1.49 741 1.18 813 1.31 412 1.36 275 1.97

Injuri 187 0.15 148 0.20 53 0.36 7 0.46 Law 691 .28 648 .35 275 .47 110 .90 Writ 135 .17 130 .24 68 .50 69 1.81

Step Input

1 raw text of 2649 cases

(3.5 reels)

concordance

alpha-sorted concordance

word-statistics

list

word-statistics

list

Nature of Processor FORTRAN: locates "words" consisting

of 3 to 6 consecutive

alphabetic characters; tags each word with word

number, and docu- ment number; writes bibliography tape SORT: orders con- cordance by word,

document number,

and word number

FORTRAN: devel- ops document statistics, counts tokens

for each word, finds quantities NOCC,

PZD, AVG, S, S2, E, EL, EK, EKL, G, GL

(defined in text) and

writes list of words

appearing in only one document on to separate tape SORT 1 SORT 2 SORT 3 SORT 4 SORT 5 SORT 6 SORT 7 SORT 8 FORTRAN: selects

all words that appear

in top 300 by any cri- terion and summarizes statistics on ex- ception bases

Output

(1) concordance

3,800,000 words;

(7 reels)

(2) bibliography

alpha-sorted con- cordance (7 reels)

(1) document statistics

(2) list of 14,000 words (types) ap- pearing in one docu- ment only

(3) list of 16,200 re- maining word types with statistics noted

sorted by NOCC

sorted by PZD

sorted by E

sorted by EL

sorted by EK

sorted by EKL

sorted by G

sorted by GL

summary of words

possibly to be delet- ed with related statistics

772-957 0-66—

73

VOTES WORD

THE

ANO

THAT

WAS

FOR

NOT

10 COURT

THIS

5 DEFEND

WHICH

WITH

2 PLAINT

FROM

HIS

SUCH

HAD

3 CASE

3 APPELL

ANY

HAVE

ARE

THERE

WERE

9 EV I DEN

BEEN

UPON

ITAL

ITS

UNDER

SAID

9 JUDGME

HAS

2 SECTIO

7 TRIAL

WOULD

2 LAW

MAY

1 ONE

2 STATE

BUT

9 APPEAL

1 ALL

OTHER

2 OUESTI

4 ILL

2 OHIO

3 TIME

6 ACTION

7 CONTRA

MADE

5 PETITI

HER

5 STATUT

7 WILL

THEY

4 PERSON

WHEN

2 REASON

1 SEC

3 ORDER

442506

128355

89026

56044

45223

35835

33021

29490

25773

25522

21624

20986

19879

19529

18195

15451

15261

14543

13855

13825

13721

12925

12911

12726

12072

11816

11360

11061

10893

10747

10581

10530

10226

9898

9673

9658

95T0

9388

9231

9174

9096

9021

8966

8776

8605

8519

8254

8248

8033

7999

7623

7548

7283

7140

7042

6980

6875

6845

6808

6773

7.87

7.83

7.80

7.69

7.73

7.75

7.45

7.66

7.20

7.70

7.64

7.02

7.62

7.32

7.50

7.43

7.45

6.53

7.47

7.53

7.46

7.48

7.43

7.10

7.50

7.46

6.67

7.31

7.40

7.07

7.06

7.36

6.83

6.97

7.34

7.23

7.37

7.39

6.85

7.48

6. 80

7.36

7.43

7.25

6.49

6.49

7. 17

6.94

6.56

7.32

6. 19

6.30

6.89

6.84

7. 14

7.01

7.28

7.17

6.65

6.78

EL PZD

7.65 99.99

73

15

73

7.61 99

7. 6 J 98

7.55 95

7.61 98.07

7.60 96.97

7.41 93.58

7.59 96.67

7.12 71.19

7.56 94.41

7.51 92.03

6.94 57.71

7.51 92.18

7.22 78.63

7.35 85.80

7.30 82.44

7.36 84.74

6.44 50.16

7.37 83.12

7.44 85.99

7.39 84.37

7.40 84.25

15

19

31 79.91

,02 65.64

,41 83.76

40 82.93

,57 45.18

7.20 75.34

7.31 80.44

6.93 69,

7.17 73,

7.37 81.76

6.76 55.75

6.98 62.85

7.23 73.12

7.20 74.29

7.30 76.70

7.31 76.40

6.80 62.06

7.37 78.89

7.06 77.61

7.26 74.78

7.31 76.17

7.28 77.08

6.46 32.88

6.35 34.39

7.20 70.40

6.92 64.55

6.49 52.96

7.29 74.51

6.44 40.39

6.20 31

6.80 53,

6.74 62.55

7.03 64.47

6.94 60.81

7.24 69.87

7.25 72.48

6.62 49.60

6.77 58.32

89

15

AVG

12.1192

4562

4343

5630

2529

0.9798

0.9097

0.8106

0. 7468

0.6984

0. 5840

0.6097

0. 5456

0.5396

0.4817

0.4205

0.4182

0.3877

0.3703

0.3761

C.3766

0.3545

0.3486

0.3461

0.3306

0.3232

0..2755

0.2888

0.2937

0.2803

0.3119

0.2838

0.2858

0.2884

0.2580

0.2554

0.2605

0.2540

0.2417

0.2485

0.2637

0.2361

0.2397

0.2395

0.2551

0.2212

0.2237

0.2329

0.2158

0.2213

0.2198

0.2095

0. 1985

0. 1944

0. 1897

0.1897

0.1866

0.1850

0. 1929

0.1918

G

-0.19

0.53

0.70

0. 52

1.03

0.55

1.64

1.15

1.34

0.64

1.15

1.25

1.25

1.55

1.49

1.49

1.64

3.05

1.29

1.17

1.56

1.30

1.43

1.64

1.41

1.37

3.12

1.71

1.82

4.45

3.01

1.34

2.91

2.75

1.43

2.34

1.45

1.61

3.06

0.84

4.94

1.45

1.18

2.17

1.95

35

55

64

98

60

3.73

4.05

2.26

5.49

2.45

2.61

1.54

2.15

3.75

3.68

EK

41. 17

15.25

9.48

3.68

5.00

6.95

1.26

4.02

0.79

4.89

3.46

0.64

3.01

1.03

1.78

1.38

1.43

0.23

1.87

2.52

1.85

1.87

1.55

0.71

1.96

1.76

0.37

1.13

1.31

0.50

0. 54

1.51

0.38

0.45

1.34

0. 88

1.38

1.48

0.39

2.21

0.30

1.46

1.79

1.03

0.34

0.28

0.92

0.39

0.23

1.25

0.19

0.20

0.48

0.26

0.77

0.57

1.20

1.11

0.27

0.31

GL

1. 87

2.14

1.92

1.78

1. 87

1.90

3.97

2.45

2.43

1.79

2.15

2.24

1.83

2. 83

2.91

2.68

2. 38

5.26

2.37

2.53

2.55

2.17

2.67

3.09

2.07

83

32

49

98

6. 83

4.08

2.41

4.29

2.96

2.49

3.39

2. 50

2.40

4.64

2.06

5.35

3.34

2.45

4.30

3.00

5.51

2.17

4.77

7.29

1.97

5.82

4.75

4. 39

12.86

3.52

5.09

2.24

2. 86

4.50

11.48

EKL

1.93

1.57

1 . 54

1 .33

1 .59

1.56

0.76

1 .41

0.53

38

16

0.43

1.19

0.60

0.74

0.69

0.80

0. 16

0.83

0.97

0.86

0.91

0.70

0.43

0.95

0.95

0.19

0.54

0.69

0.27

0.49

0.J.3

0.27

0.41

0.64

0.54

0.72

0.75

0.25

0.89

0.33

0.64

0. 76

0.62

0.24

0.17

0.62

0.31

0.15

0.76

0.18

0.14

0.29

0.15

0.45

0.33

0.69

0.64

0.21

0.19

Table IV. Sorted by NOCC

75
VOTES

8

10

10

6

84

1

1

4

1

6

13

1

1

2

5

9

WORD

MOTION

THEIR

END

AFTER

COUNTY

SHALL

DID

ONLY

REQUIR

RECORD

FOLLOW

EMPLOY

CITY

PROPER

BEFORE

WHERE

PROVID

AGAINS

DIRECT

SHOULD

FOL

PRESEN

HIM

JURY

RIGHT

FILED

CONSID

GENERA

WHO

ALSO

MUST

WHETHE

ACT

TWO

COULD

DETERM

PROCEE

SAME

AUTHOR

APP

OPINIO

THESE

PART

NEW

SEE

MASS

COMPAN

FACT

PUBLIC

WITHOU

CHARGE

THEN

WITHIN

FURTHE

PROVIS

CAUSE

OUT

THAN

MATTER

DOES

NOCC

662 1 6514

6422

6340

6245

6240

6224

6218

6103

6093

6076

6062

5969

5913

5814

5794

5792

5725

5706

5689

5682

5653

5613

5530

5447

5362

5288

5262

5241

5230

5208

5173

5147

5130

5096

5030

5021

4992

4898

4769

4764

4753

4746

4744

4704

4687

4677

4658

4658

4652

4622

4583

4561

4546

4479

446 3

4389

4378

4313

4264

E

6.71

7.08

6.81

7.24

6.62

6.81

7.24

7.33

7.06

6.91

7.28

5.98

6.24

6.40

7. 19

7.19

7.03

7.04

6.95

7.20

6.67

7. 18

6.91

6.41

6.76

6.67

7.15

6.87

11

29

18

22

6.65

7.11

7. 16

7.02

6.79

7.05

6.78

6.74

7.02

7. 11

7. 12

6.68

6.93

5.77

6. 19

7.06

6.33

7.10

6.48

7. 12

6.85

7.11

6.80

6.77

7.00

7. 11

6.91

7.09

EL

6.84

7.02

6.71

7.2 1 6.52

6.73

7.17

7.31

7* 10

6.98

7.24

5.89

6.23

6.34

7.23

7.16

7.02

7.06

6.92

7.20

6.57

7.20

6.85

6.31

6.86

6.91

7.14

6.82

7.03

7.23

7.22

7.19

6.59

7.11

7.11

7.01

6.84

7.07

6.81

6.72

6.98

7.07

7.09

6.72

6.88

5.73

6.05

7.10

6.30

7. 17

6.47

7.07

6.97

7.13

6.77

6.90

6.99

7. 10

6.96

7.20

Table IV. Sorted by

PZD

53.90

61.75

51.86

68.47

52.43

49 . 18

66.70

72. 14

63.98

60.51

69.38

32.50

38.05

36.91

68.55

65.26

60.02

61.83

58.62

66.59

45.18

68 .25

54.24

34.27

54.24

55.26

63.72

52.92

59.64

67. 15

66.70

66. 13

45.56

60.51

61 .79

59.45

55.19

60.73

52.32

44.92

58.85

59.79

60.62

48.09

55.00

16.98

32.65

60.28

35.78

63.57

40.69

59. 19

55.56

61 .94

47. 18

54.28

57.04

59.38

55.19

63.30

NOCC

AVG

0. 1942

0. 1756

0. 1570

0. 1745

0.1787

0. 1705

0.1665

0. 1693

0. 1665

0. 1675

0. 1661

0.1653

0. 1706

0. 1591

0.1612

0. 1562

0. 1599

0. 1605

0.1575

0.1511

0. 1378

0. 1558

0.1531

0. 1470

0. 1464

0. 1589

0. 1379

0.1338

0.1416

0. 1410

0. 1412

0. 1408

0. 1370

0. 1408

0. 1383

0.1314

0.1373

0. 1299

0. 1319

0.1292

0.1218

0. 1275

0.1287

0. 1295

0. 1297

0.1483

0. 1180

0.1249

0.1226

0.1274

0. 1234

0.1242

0. 1294

0. 1230

0.1251

0.1255

0. 1 164

0. 1198

0. 1 166

0.1175

G

3.78

2.19

3.07

1.62

5.00

2.77

1.55

1.57

2.34

5.25

1.30

5.38

3.90

3.62

2.12

1.64

2.56

2.56

5.12

1.89

3.12

2.26

2.49

3.35

2.91

4.09

2.06

3. 11

1.89

1.08

1.83

1.69

3.30

1.59

1.59

3.04

3.56

2.47

4.35

2.51

2.05

1.97

2.57

3.77

2.95

3.41

4.27

2.10

4.86

2.02

3.96

2.04

2.63

1.92

2.55

2.98

3.00

2.23

3.11

1.80

EK

0.30

0.70

0.44

1.06

0.23

0.43

1.03

1.38

0.74

0.41

1.18

0.11

0.18

0.23

0.95

1.03

0.64

0.63

0.44

1.02

0.37

0.88

0. 52

0.24

0.47

0.33

0.93

0.47

0.79

1.33

08

04

32

85

0.95

0.64

0.40

0.76

0.37

0.41

0.71

0.83

0.78

0.31

0.47

0.12

0.17

0.80

0.20

0.91

0.24

0.82

0.50

0.91

0.45

0.43

0.65

0.81

0.53

0.96

GL

3.36

3.29

6.84

2.27

8.51

4.34

2.52

1.88

4. 53

4.95

2.44

7.48

5.82

5.71

2.63

2.43

3.62

3. 13

6.63

2.45

7.39

3.49

6.64

4.31

3.87

3.46

2.68

5.01

3.51

1.95

2.79

2.57

6.21

2.47

2.58

3.95

6.15

3.32

4.61

3.31

4.63

3.27

2.85

4.33

3.89

4.36

10.01

2.40

5.07

2.39

4.95

2.60

3.59

3.44

3.69

4.08

6.13

2.63

4.12

2.11

EKL

0.33

0.42

0.22

0.65

0. 14

0.27

0.59

0.82

0.47

0.35

0.69

0.08

0.13

0.15

0.66

0.58

0.42

0.46

0.29

0.63

0.19

0.58

0.29

0.17

0.32

0.36

0.56

0.28

0.44

0.71

0.64

0.61

0.20

0.55

0.54

0.40

0.26

0.48

0.28

0.29

0.37

0.48

0.52

0.26

0.33

0.10

0.09

0.54

0.15

0.62

0.18

0.51

0.41

0.53

0.30

0.34

0.37

0.54

0.38

0.67
VOTES

85

12

5 13

1

5 11

16

WORD

INVOLV

ENTERE

SPECIF

WHAT

RESPON

PERMIT

BOTH

REVERS

SUBJEC

NOTICE

CAN

RECEIV

CONDIT

GIVEN

SINCE

D ISM IS

WHILE

STATEM

ACCORD

OBJECT

ASSIGN

USED

AAAAAA

PARTY

THEREO

INCLUD

GROUND

OVER

YEARS

SUSTAI

HEREIN

RESPEC

SUPRA

CLAIM

CIRCUM

MAKE

RELATI

THOSE

SUBSTA

HEARIN

TAKEN

SUFFIC

CANNOT

THREE

SECOND

NOW

CONTIN

PARTIC

PRIOR

UNTIL

REVIEW

STATES

PAID

CONCUR

WELL

DURING

DAY

PRINCI

ENTITL

RIGHTS

NOCG

2933

2920

2900

2883

2872

2869

2868

2857

2855

2855

2822

280 1 2779

2766

2756

2755

2749

2732

2721

2703

2654

2650

2649

2643

2640

2632

2629

2622

2601

2600

2599

2579

2573

2565

2543

2535

2530

2527

2527

2525

2518

2484

2467

2437

2415

2384

2382

2381

2379

2347

2347

2343

2316

2290

2259

2216

2189

2158

2141

2108

E

6.56

6. 78

6.65

6. 76

5.94

6.35

6.85

6.66

6.70

6.04

6.93

6.52

6.46

6.80

6.89

5.96

6.82

6.32

6.87

6.27

6.00

6.45

7.07

6.26

6.69

6.71

6.68

6.72

6.53

6.65

6.23

6.80

6.29

6.24

6.75

6.76

6.54

6.73

6.62

6.28

6.67

6.72

6.74

6.70

6.53

6.60

6. 37

6.48

6.69

6.65

6.02

6.38

6.25

6.65

6.77

6.58

6.41

6.46

6.53

6.30

EL

6.90

6.87

6.68

6.79

6.00

6.49

,88

93

8 1 ,18

,94

57

6.47

6.82

6.93

6.48

6.85

6.36

6.96

6.31

6.12

6.58

7.87

6.33

6.75

6.76

6.77

6.71

6.56

6.89

6.70

6.82

6.25

6.24

6.75

6.84

6.53

6.77

6.71

6.31

6.76

6.81

6.92

6.73

6.61

6.80

6.40

6.76

6.74

6.7G

6.30

6.33

6.25

7.30

6.83

6.62

6.46

6.43

6.69

6.33

Table IV. Sorted by

PZD

47 . B6

4 H . 5 8

4 2.28

44 .30

29.21

39.63

46 . 54

46.96

45.48

30.76

49. 15

39. 10

35. 52

45 .07

48.6 5 35.90

46.31

34. 16

49.64

32.50

29.82

38.16

99.99

31.93

41 .60

43.41

44. 16

40.99

37.10

46.24

41.75

44.43

29.21

32.27

41.94

43.94

37. 10

42.43

41.60

31.59

43.07

42.92

46.54

41.18

38.50

43.29

34.35

42.12

40.88

39.22

32.72

33.37

28.16

63.91

43.14

36. 50

34. 16

34.61

38.42

30.38

NOCG

AVG

0.0789

0.0873

0.0 7 90

0.0725

0.0772

0.0820

0.0771

0.0842

0.0784

0.0853

0.0739

0.0764

0.0760

0.0744

0.0753

0.0790

0.0751

0.0720

0.0745

0.0742

0.0715

0.0734

0.0783

0.0726

0.0697

0.0716

0.0728

0.0701

0.0687

0.0753

0.0670

0.0678

0.0636

0.0735

0.0679

0.0681

0.0662

0.0642

0.0693

0.0716

0.0697

0.0708

0.0694

0.0677

0.0656

0.0629

0.0634

0.0625

0.0654

0.0 62 8 0.0 676

0.0582

0.0616

0.0643

0.0592

0.0609

0.0607

0.0564

0.0591

0.0581

G

2.29

3.29

3.75

2.52

6.24

6.17

1.87

2.65

2.72

5.70

1.61

6.76

3.52

2.27

1.76

5.16

5.29

4.77

2.12

8.66

6.48

5.62

0.42

4.28

2.61

3.86

3.25

2.40

3.24

3.40

3.17

1.99

3.34

5.91

2.08

2.35

3.61

3. 12

3.48

4.03

3.27

2.35

2.06

3.19

3.97

2.79

5.85

3. 17

2.87

2.31

5.34

6.26

3.21

2.45

2.87

2.73

3.92

6.01

2.60

5.59

EK

0. 56

0.42

0.34

0.51

0.12

0.17

0.59

0.48

0.46

0.14

0.67

0.27

0.26

0.50

0.62

0. 16

0.43

0.20

0.62

0.15

0.12

0.24

4.32

0.20

0.42

0.39

0.38

0.43

0.31

0.40

0.36

0.54

0.23

0.15

0.49

0.54

0.30

0.46

0.36

0.21

0.37

0.45

0.57

0.40

0.31

0.46

0.21

0.41

0.41

0.42

0.15

0.22

0.23

0.73

0.51

0.36

0.26

0.24

0.38

0.20

GL

2.99

4.02

5.03

3.76

11.25

6.36

2.81

3.60

3.64

6.77

2.68

5.74

3.88

3.10

2.78

5.01

4.31

5.32

2.92

5.60

7.19

4. 18

2. 55

5.91

3.06

3.68

5.73

3.50

4.19

2.63

5.86

3.71

4.77

7.77

2.94

3. 17

5.77

3.52

4.62

6.14

4.04

3.24

2.46

3.87

5.63

3.10

10.10

3.48

3.12

3.46

7. 80

8. 54

4.69

2.51

3.49

4.42

9.83

7.85

3.68

4.76

EKL

0.40

'0.34

0.25

0.32

0.08

0.17

0.39

0.43

0.33

0.12

0.44

0.21

0.21

0.35

0.43

0.20

0.35

0.16

0.45

0.15

11

23

31.32

0.16

0.33

0.31

0.29

0.29

0.23

0.41

0.25

0.34

0.15

0.12

0.33

0.37

0.20

0.33

0.27

0.15

0.31

0.36

0.45

0.30

0.23

0.34

0.14

0.32

0.32

0.30

0.13

0.13

0. 16

0.86

0.36

0.26

0.17

0.16

0.30

0.17

77
vOTE.o WORD NOCC

3 OPEKAT 4 20 7 3 APPLI

4168

3 FIRST 4165

4 CODE 4152

2 PURPOS 4138

6 CONST I 4132

1 FACTS 4095

6 RULE 4090

7 OFFICE 4060

3 COMMON 4042

5 JUDGE 4000

2 DECISI 3988

HELD 3978

3 COMPLA 397 1 4 AFFIRM 3897

CASES 3896

CONTEN 3888

THEREF 3871

2 BEING 3858

5 APPEAR 3855

8 SERVIC 3855

3 USE 3852

6 ERROR 384 1 4 CONST

3805

2 ALLEGE 3766

5 EFFECT 3759

2 STATED 3698

7 CONCLU 3665

7 TESTIM 3650

8 I NTERE 3637

4 FOUND 3608

6 EXCEPT 3589

I NTO 358 3 BECAUS 3553

THEM 3505

5 PARTI

3496

1 TESTIF 3484

9 NECESS 3477

HERE 3448

3 FINDIN 3437

5 ANSWER 3398

SOME 3394

HOWEVE 3333

1 EACH 3332

I RESULT 3328

BETWEE 323 1 ABOUT 3228

PAGE 3218

OUR 3179

3 SUPPOR 3151

5 EXAM I

3117

5 I SSUE 3113

4 AMOUNT 31 10

2 CERT AI 3069

13 JURISD 3056

MORE 3050

9 COUNSE 3030

SET 2964

1 ESTABL 2947

2 CONTRO 2941

Table IV.

E EL

6.52 6.4:>

6.58 6.60

7.01 7. OA

6.21 6.18

6. 76 6.76

6.41 6.49

7.00 7.01

6. 56 6.70

6.26 6.12

6.46 6.48

6. 52 6.64

6. 52 6.69

7.04 7.02

6.40 6.45

6.89 7.23

6. 86 6.90

7.02 7.09

7.01 7.18

7.04 7.08

6.95 7.00

6.04 6.05

6.29 6.27

6. 56 6.66

6. 58 6.55

6.72 6.8 1 6.91 6.92

6.99 6.99

6.95 7.02

6.42 6.41

6.36 6.32

6.91 6.98

6. 58 6.82

6.93 6.92

7.00 7. 1 1 6.92 6.89

6. 55 6.59

6.35 6.35

6.93 6.93

6.93 6.9 7 6. 56 6.5 9

6.42 6.41

6.97 6.93

7.09 7.11

6.68 6.69

6.85 6.86

6.84 6.87

6.65 6.65

6.47 6.45

6.80 6.83

6.65 6.67

6. 19 6.23

6.61 6.66

6.49 6.52

6.87 6.96

6.00 6.10

6.94 6.95

6.22 6.27

6.71 6.84

6.74 6.72

6.48 6.55

Sorted by

PZD AVG

30.56 0.1145

4 7.37 0.1134

57.15 0.1116

29.55 0.1146

49.30 0.1096

42.99 0.1058

55.79 0.1137

47.18 0.1055

33.93 0.1033

42.58 0.1171

46.84 0.1181

46.58 0.1070

55.34 0.1058

37.44 0.1136

63.53 0.1109

51.41 0.1062

57.11 0.1094

62.21 0.1050

57.41 0.1040

57.68 0.1045

29.63 0.1114

36.12 0.1059

44.80 0.1051

40.50 0.1054

47.86 0.1091

52.39 0.1018

54.77 0.0975

53.90 0.1010

34.65 0.1010

35.33 0.0944

53.68 0.1017

49.79 0.1046

51.00 0.0952

57.19 0.0999

49.37 0.0943

41.71 0.0960

31.74 0.0969

52.20 0.0937

52.69 0.0938

41.56 0.0995

39.33 0.0913

50.88 0.0897

55.90 0.0923

43.90 0.0859

48.50 0.0911

47.45 0.0879

41.10 0.0882

33.71 0.0815

47.98 0.0833

46.35 0.0855

35.56 0.0831

42.88 0.0831

37.56 0.0869

50.62 0.0830

29.67 0.0812

49.49 0.0822

32.54 0.0868

46.54 0.0798

44.46 0.0788

39.93 0.0849

NOCC

G EK

3.54 0.27

4.97 0.25

2.30 0.71

4.17 0.17

3.99 0.41

3.48 0.28

3.05 0.60

4.23 0.31

4.82 0.17

5.85 0.19

10.30 0.19

4.00 0.30

1.92 0.75

4.27 0.22

2.26 0.78

2.58 0.54

2.14 0.71

1.43 0.90

2.13 0.75

3.97 0.56

5.82 0.13

4.86 0.18

3.69 0.29

3.38 0.30

3.04 0.40

2.86 0.56

2.37 0.68

2.50 0.64

3.30 0.25

5.26 0.20

2.73 0.53

5.95 0.26

2.51 0.57

2.04 0.75

2.56 0.56

3.86 0.29

3.53 0.24

3.31 0.52

1.92 0.66

4.00 0.26

5.64 0.22

1.97 0.67

1.47 0.90

4.53 0.36

3.50 0.49

2.33 0.55

2.68 0.39

2.83 0.31

2.15 0.55

7.06 0.24

7.01 0.15

3.76 0.32

3.85 0.27

2.20 0.65

4.48 0.14

1.98 0.66

6.05 0.15

3.36 0.45

3.00 0.45

5.05 0.23

GL EKL

4.52 0.18

8.13 0.16

3.27 0.46

5.98 0.13

6.33 0.25

7.53 0.15

2.90 0.46

12.48 0.20

18.75 0.07

7.01 0.16

6.80 0.20

5.57 0.23

2.83 0.47

4.90 0.19

2.61 0.70

3.22 0.38

2.24 0.56

2.25 0.65

2.89 0.52

9.43 0.32

7.29 0.10

7.72 0.12

4.33 0.24

4.65 0.21

3.37 0.33

7.29 0.34

3.69 0.42

2.52 0.49

3.88 0.20

5.71 0.15

3.16 0.43

4.72 0.30

3.14 0.39

2.28 0.58

4.37 0.36

4.47 0.22

3.72 0.19

4.91 0.35

3.12 0.43

3.90 0.23

9.44 0.13

4.84 0.39

1.76 0.62

5.12 0.25

3.97 0.34

2.83 0.38

3.45 0.27

5.57 0.19

4.84 0.31

9.79 0.18

8.63 0.11

4.98 0.23

3.75 0.22

3.90 0.42

6.50 0.11

2.76 0.45

5.28 0.14

3.72 0.35

17.95 0.18

5.00 0.20

78
kjtes

8

5

12

5 13

1

5 11

16

WORD

INVOLV

ENTERE

SPECIF

WHAT

RESPON

PERMIT

BOTH

REVERS

SUBJEC

NOTICE

CAN

RECEIV

CONDIT

GIVEN

SINCE

D ISM IS

WHILE

STATEM

ACCORD

OBJECT

ASSIGN

USED

AAAAAA

PARTY

THEREO

INCLUD

GROUND

OVER

YEARS

SUSTAI

HEREIN

RESPEC

SUPRA

CLAIM

CIRCUM

MAKE

RELATI

THOSE

SUBSTA

HEARIN

TAKEN

SUFFIC

CANNOT

THREE

SECOND

NOW

CONTIN

PART I

PRIOR

UNTIL

REVIEW

STATES

PAID

CONCUR

WELL

DURING

DAY

PRINCI

ENTITL

RIGHTS

2933

2920

2900

2883

2872

2869

2868

2857

2855

2855

2822

280 1 2779

2766

2756

2755

2749

2732

2721

2703

2654

2650

2649

2643

2640

2632

2629

2622

2601

2600

2599

2579

2573

2565

2543

2535

2530

2527

2527

2525

2518

2484

2467

2437

2415

2384

2382

238 1 2379

2347

2347

2343

2316

2290

2259

2216

2189

2158

2141

2108

2

6. 56

6. 78

6.65

6. 76

5.94

6.35

6.85

6.66

6.70

6.04

6.93

6.52

6.46

6.80

6.89

5.96

6.82

6.32

6.87

6.27

6.00

6.45

,07

26

,69

71

6.68

6.72

6.53

6.65

6.23

6.80

6.29

6.24

6.75

6. 76

6.54

6.73

6.62

6.28

67

72

74

70

53

60

6. 37

6.48

6.69

6.65

6.02

6.38

6.25

6.65

6.77

6.58

6.41

6.46

6. 53

6.30

EL

6.90

6.8 7 6.68

6.79

6.00

6.49

6.88

6.93

6.8 L 6.18

6.94

6.57

6.47

6.82

6.93

6.48

6.85

36

,96

31

12

58

87

33

6.75

6.76

6.77

6.71

6.56

6.89

6.70

6.82

6.25

6.24

6.75

6.84

6.53

6.77

6.7 1 6.3 1 6.76

6.8 1 6.92

6.73

6.61

6.80

6.40

6.76

6.74

6.70

6.30

6.33

6.25

7.30

6.83

6.62

6.46

6.43

6.69

6.33

Table IV. Sorted by

?ZD

47.36

48.58

A 2 . 2 8 44 . 30

29.21

39.63

46 . 54

46.96

45.48

30.76

49. 15

39. 10

35. 52

45 .07

48 .65

35.90

46.31

34. 16

49.64

32.50

29.82

38.16

99.99

31.93

41 .60

43.41

44. 16

40.99

37.10

46.24

41.75

44.43

29.21

32.27

41.94

43.94

37. 10

42.43

41.60

31. 59

43.07

42.92

46 . 54

41.18

38.50

43.29

34.35

42.12

40.88

39.22

32.72

33.37

28.16

63.91

43.14

36. 50

34.16

34.61

38.42

30. 38

NOGC

AVG

0.0789

0.0873

0.0790

0.0725

0.0772

0.0820

0.0771

0.0842

0.07b4

0.0853

0.0739

0.0764

0.0760

0.0744

0.0753

0.0790

0.0751

0.0720

0.0745

0.0742

0.0715

0.0734

0.0783

0.0726

0.0697

0.0716

0.0728

0.0701

0.0687

0.0753

0.0670

0.0678

0.0636

0.0735

0.0679

0.0681

0.0662

0.0642

0.0693

0.0716

0.0697

0.0708

0.0694

0.0677

0.0656

0.0629

0.0634

3.0625

0.0654

0.0 62 8 0.0676

u.0582

0.0616

0.0643

0.0592

C.0609

0.0607

0.0564

0.0591

0.0581

G

2.29

3.29

3.75

2.52

6.24

6.17

1.87

2.65

2.72

5.70

1.61

6. 76

3.52

2.27

1.76

5.16

5.29

4.77

2.12

8.66

6.48

5.62

0. 42

4.28

2.61

3.86

3.25

2.40

3.24

3.40

3.17

1.99

3.34

5.91

2.08

2.35

3.61

3. 12

3.48

4.03

3.27

2.35

'..06

1. 19

3.97

2.79

5.85

3.17

2.87

2.31

5.34

6.26

3.21

2.45

2.87

2.73

3.92

6.01

2.60

5.59

EK

0.56

0.42

0.34

0.51

0.12

0.17

0.59

0.48

0.46

0.14

0.67

0.27

0.26

0.50

0.62

0.16

0.43

0.20

0.62

0.15

0.12

0.24

4.32

0.20

0.42

0.39

0.38

0.43

0.31

0.40

0.36

0.54

0.23

0.15

0.49

0.54

0.30

0.46

0.36

0.21

0.37

0.45

0.57

0.40

0.31

0.46

0.21

0.41

0.41

0.42

0.15

0.22

0.23

0.73

0.51

0.36

0.26

0.24

0.38

0.20

GL

2.99

4.02

5.03

3.76

11.25

6. 36

2.81

3.60

3. 64

6.77

2.68

5.74

3.88

3.10

2.78

5.01

4.31

5.32

2.92

5.60

7.19

4. 18

2.55

5.91

3.06

3.68

5.73

3.50

4. 19

2.63

5.86

3.71

4.77

7.77

2.94

3.17

5.77

3.52

4.62

6.14

4.04

3.24

2.46

3. 87

5.63

3.10

10.10

3.48

3. 12

3.46

7.80

8. 54

4.69

2.51

3.49

4.42

9. 83

7.85

3.68

4.76

EKL

0.40

0.34

0.25

0.32

0.08

0.17

0.39

0.43

0.33

0.12

0.44

0.21

0.21

0.35

0.43

0.20

0.35

0.16

0.45

0.15

0.11

0.23

31.32

0.16

0.33

0.31

0.29

0.29

0.23

0.41

0.25

0.34

0.15

0.12

0.33

0.37

0.20

0.33

0.27

0.15

0.31

0.36

0.45

30

23

34

14

32

0.32

0.30

0.13

0.13

0.16

0.86

0.36

0.26

0. 17

0.16

0.30

0.17

79
VOTES WORD NOCC E

4 V I EW 140 6 6.35

9 ATTEMP 1404 6.05

CITED 140 1 6.41

2 S I TUAT 1358 6.42

3 CORREC 1358 6. 14

> ENT I RE 1350 6. 30

THERE

1342 6.40

1 APPARE 1334 6.43

1 I NT END 1333 6. 29

3 REFERR 1309 6.24

THOUGH 130 1 6.43

2 REFUSE 1286 6.14

NQTH I

1275 6. 24

APPL I

1264 6.25

2 SUBSEQ 1263 6.25

MANNER 1259 6. 30

1 FAVOR 1249 6.22

4 OCCURR 1248 6.05

1 STAT 1245 5.90

2 S IMILA 1243 6.38

5 SEVERA 1243 6. 32

I NATURE 1185 6.16

OROERE 1180 6.14

BEL I EV 1176 6.22

BECOME 1158 6.07

CLEARL 1145 6.31

1 TRUE 1140 6.23

SOUGHT 1 132 6.11

MANY 1117 6.27

SHOWN 1 106 6.15

OTHERW 1095 6.14

SAY 1088 6.26

1 KNOWN 1083 6.12

TOOK 1080 6.15

DONE 1079 6.09

SHOWS 1078 6.16

THERE I 1068 6.13

MAKING 1060 6.19

MOST 105 1 6.25

RAISED 1050 6.00

LONG 1047 6.23

PREV I 0 1040 6. 16

4 PURSUA 1039 6.08

1 THINK 1035 6.18

DISCUS 1034 6.22

HOLD 1033 6.15

5 RECOGN 1033 6. 10

2 EXISTE 1029 6.06

THERET 1022 6.05

POSS I B 1018 6.18

4 EMPHAS 1012 5.96

1 HOLD I

1008 6.05

3 DISTIN 997 6.14

ITSELF 993 6.25

NEVER 976 6.01

5 PREVEN 956 6.00

2 FILE 943 5. 49

CONS I

94 1 6.19

MERELY 936 6.21

NE I THE 930 6. 16

Table IV. Sort

EL

6.4H

6.42

6.54

6.49

6.38

6.41

6.55

6.53

6*39

6.43

6.54

6.22

6.55

6.40

6.37

6.37

6.37

6.11

5.9 3 6.46

6.36

6.31

6.33

6.34

6. 30

6.45

6. 36

6.33

6.38

6.36

6.42

6.34

6.17

6.28

6.28

6.35

6.38

6.33

6.31

6.28

6.32

6.31

6.24

6.28

6.31

6.35

6.25

6.17

6.35

6.23

6.00

6.20

6.22

6.33

6.15

6.16

5.87

6.31

6.32

6.38

PZD

30.95

29. 18

30.95

29.40

2R.57

28.53

31.03

30.84

27.63

28.65

30.46

24.49

30.65

27.63

26.99

27.29

26.87

21 . 78

19.10

20.61

27.25

25.48

26.23

25.67

25.36

27.67

26.23

25.44

25 .82

25.74

27.18

25.44

22 .19

24.46

24.57

25.25

25.70

25.14

24.95

23.93

24.80

24.57

23.17

23.63

24.34

24.61

23.51

22.08

24.95

22.98

19.59

22.76

22.68

24.38

21.32

21.44

17.06

23.66

23.78

24.87

NOCC

AVG

0.0375

0.0376

0.0390

0.0368

0.0370

0.0369

0.0389

0.0364

0.0361

0.0341

0.0340

0.0351

0.0345

0.0351

0.0363

0.0329

0.0364

0.0347

0.0383

0.0339

0.0331

0.0313

0.0324

0.0322

0.0320

0.0304

0.0309

0.0316

0.0286

0.0303

0.0307

0.0294

0.0285

0.0302

0.0282

0.0297

0.0279

0.0282

0.0273

0.0290

0.0280

0.0277

0.0271

0.0298

0.0267

0.0270

0.0261

0.0286

0.0278

0.0272

0.0246

0.0265

0.0265

0.0260

0.0254

0.0265

0.0265

0.0260

0.0248

0.0252

G

4.33

4.42

2.52

2.40

4.3 5 5.20

2.78

3.26

3.14

8.37

2.57

4.26

2.76

2.95

3.67

3.46

3.45

3.73

3.51

2.91

3.47

3.80

3.50

3.33

3.89

2.81

3.33

3.80

2.52

3.38

4.16

2.94

3.59

3.21

3.94

3.55

2.72

4.11

2.65

3.56

3.39

3.93

2.92

3.00

2.85

2.49

3.33

5.05

3.03

3.04

3.16

3.62

2.77

2.40

4.0 3

3.86

5.51

2.47

2.46

2.65

EK

0.29

0.25

0.33

0.33

0.21

0.25

0.32

0.30

0.25

0.24

0.34

0. 19

0.33

0.27

0.24

0.27

0.23

0. 18

0.15

0.30

0.26

0.22

0.23

0.24

0.23

0.30

0.26

0.21

0.29

0.24

0.25

0.26

0.21

0.24

0.21

0.23

0.27

0.22

0.28

0.21

0.23

0.22

0.22

0.23

0.25

0.26

0.23

0.19

0.25

0.23

0.19

0.21

0.24

0.27

0.19

0.19

0.10

0.26

0.26

0.27

GL

7.01

7.93

3.08

3.07

4.34

6.76

2.92

3.32

4.27

5.55

2.82

4.13

2.84

3.46

3.97

6.32

4.09

4.81

6.23

3.18

7.53

4.10

6.13

3.34

3.96

3.28

4.42

4.23

2.73

3.23

3.79

3.71

4.34

4.38

4. 53

3.06

3.38

3.75

6.00

3.95

3.84

3.68

3.93

3.20

3.19

3.24

3.94

4.18

3.31

3.70

5. 19

4.43

4.15

3.32

4. 18

3.57

4.17

3.02

2.82

2.44

EKL

0.20

0.19

0.27

0.25

0.20

0.20

0.28

0.26

0.21

0.21

0.28

0.17

0.29

0.22

0.21

0.19

0.21

0.15

0.11

0.24

0.18

0.19

0.18

0.21

0.19

0.24

0.20

0.20

0.23

0.22

0.23

0.21

0.16

0.19

0.18

0.22

0.23

0.21

0.18

0.19

0.20

0.20

0. 18

0.20

0.21

0.22

0.18

0. 16

0.22

0.18

0.13

0.17

0.18

0.22

0.16

0.17

0.12

0.21

0.22

0.25

80
TTYYPTTQ W UAL/ 17£i

•t \/V I1 Fu Wn X140ft ^ 'J O ft 1

g A T T E M P 1 40 4 ft OS

140 1 ftO & 1 . *t X c <s I TH AT 1 3 S ft ft 4 ?

X1 J3Sft J o ftO • 14X ^

IX ENT IRE 13L J J50\J ft 30

i ( 1 1_ r\ [_ m i 34 ? ft A.n

11 ADD AD C M r r M fS C1 ll/ 1 Jjt ft A

1 I NT P n n 1 111 1 3 J J ABICTO Q

REFERR 1X 309 J W 7 ft 24

Turn 1 1 1 U Uion O 1

1 30 1 X JU X ft 4 ~k RppilC p 1L ?ft C O ftO ftO . 14X H

1X A?7Sf J ft 9A. APPL I

X1 264 4- U I ftO . A2 JS 2 O \J u -j u w 1X £2ftO J3 ft 2 S M A NI M F R 1 ? SQ ftO jUo

1X 1x a?49^ # ft ? ? A" nrr 1 1 rr 1X £?4ft ^ o ft OS

1 ^TAT 1 <_ J S 90

5 C T M r 1 A 1X 243T J ftD« 3ft JO

CCWCD

124 3 ftO . 3J A?

1& M ATI J D C llflS J. x o ft 1ft u i\ u l r\ l_ 1X 1X ftO O \J ftO . 14X ^ RFI I F\I X1 1J. 7r AO ftO . 22A A

U L L> U 1 1 L. 1X 1X 5ft ^> O ft 07

CLEARL 1145 ft 3

1 TRLIF X X " V/ fto . 2£ 3J

Jsought U U U 1 1 1 1X 1X J^?£. ftL> • 1X 1X MANY 1117 XXX/ ft 27

j i i un ii 1106 X X V u 6 - 1 S OTHF RW 1 09 5 ft - 1 4

S AY 1X 0ft w O ftO

1 KNOWN 1 OH 3 O • X c t nnK 1X OftO \J o u AO • 1X 5J ODNF X1 079 \J 1 * A 0 9

jnun o X U 1 o O • X o TuPR F

1 OA ft AO • 1X ^D MAKINT, l l m r\ x j >4 o 1X OftO w o w AO • 19X y

1X O S 1 \J J X A 9 5

R A T C F n 1 0 SO A 0 0

i dno 1047 APRFV t n 1X 040 w *T W AO • 1X Ao 4 PI* UIRr\ OS 1U1 MA 1 0 3Q A Oft

1 TH I NK 1 0 3 5 A 1ft 1034 A ??

n ul u 10 3 3 AO * 15X -J j1 10 3 3 A 10

2 1X 02Q W £ 7 A OA

TU CD C T i ncivc i A 0 5

X101ft w X O Aw *• 1X ftO 4 FMPH A<\ 1012 X w X £ 5 9 A

li uni nULUn T NJ 1 IH 1UUO 1 OO ft A 0 5 997 SSI Ao -• 1X 4"

1T T<i 1 O FltZ L. Fi 99 3 A ? 5

I N C V C r\ Q7 A A 0 1 j r r\ C V C ll Q Sft A 00

F T 1 F 94 3 5 . 49

ton*; t <; 94 1 6-19 w . X J ncl\tL ucppi vT 7 JO ft 9 1 NE I THE 930 A 1 A Table IV. Sort

EL

6. 48

6.42

6.54

6.49

6.38

6.41

6.55

6.53

6.39

6.43

6.54

6.22

6.55

6.40

6.37

6.37

6.37

6.11

5.9 3 6.46

6.36

6.31

6.33

6.34

6. 30

6.45

6.36

6.33

6.38

6.36

6.42

6.34

6.17

6.28

6.28

6.35

6.33

6.33

6.31

6.28

6.32

6.31

6.24

6.28

6.31

6.35

6.25

6.17

6.35

6.23

6.00

6.20

6.22

6.33

6.15

6.16

5.87

6.31

6.32

6.38

PZD

30.95

29.18

30.95

29.40

23.57

28.53

31.03

30.84

27.63

28.65

30.46

24.49

30.65

27.63

26.99

27.29

26.87

21 .78

19.10

28.61

27.25

25.48

26.23

25.67

25.36

27.67

26.23

25.44

25.82

25.74

27.18

25.44

22 . 19

24.46

24.57

25.25

25.70

25.14

24.95

23.93

24.80

24.57

23.17

23.63

24.34

24.61

23.51

22.08

24.95

22.98

19.59

22.76

22.68

24.38

21. 32

21.44

17.06

23.66

23.78

24.87

N0GC

AVG

0.0375

0.0376

0.0390

0.0368

0.0370

0.0369

0.0389

0.0364

0.0361

0.0341

0.0340

0.0351

0.0345

0.0351

0.0363

0.0329

0.0364

0.0347

0.0383

0.0339

0.0331

0.0313

0.0324

0.0322

0.0320

0.0304

0.0309

0.0316

0.0286

0.0303

0.0307

0.0294

0.0235

0.0302

0.0282

0.0297

0.0279

0.0282

0.0273

0.0290

0.0280

0.0277

0.0271

0.0298

0.0267

0.0270

0.0261

0.0286

0.0278

0.0272

0.0246

0.0265

0.0265

0.0260

0.0254

0.0265

0.0265

0.0260

0.0248

0.0252

G

<».33

4.42

2.52

2.40

4.3 5 5.20

2.78

3.26

3. 14

8.37

2.57

4.26

2.76

2.95

3.67

3.46

3.45

3.73

3.51

2.91

3.47

3.80

3.50

3.33

3.89

2.81

3.33

3.80

2.52

3.38

4.16

2.94

3.59

3.21

3.94

3.55

2.72

4.11

2.65

3.56

3.39

3.93

2.92

3.00

2.85

2.49

3.33

5.05

3.03

3.04

3.16

3.62

2.77

2.40

4.03

3. 86

5.51

2.47

2.46

2.65

EK

0.29

0.25

0.33

0. 33

0.21

0.25

0.32

0.30

0.25

0.24

0.34

0. 19

0.33

0.27

0.24

0.27

0.23

0. 18

0.15

0.30

0.26

0.22

0.23

0.24

0.23

0.30

0.26

0.21

0.29

0.24

0.25

0.26

0.21

0.24

0.21

0.23

0.27

0.22

0.28

0.21

0.23

0.22

0.22

0.23

0.25

0.26

0.23

0.19

0.25

0.23

0. 19

0.21

0.24

0.27

0.19

0.19

0. 10

0.26

0.26

0.27

GL

7.01

7.93

3.08

3.07

4.34

6.76

2.92

3.32

4.27

5.55

2.82

4.13

2.84

3.46

3.97

6.32

4.09

4.81

6.23

3.18

7.53

4. 10

6.13

3.34

3.96

3.28

4.42

4.23

2.73

3.23

3.79

3.71

4.34

4.38

4. 53

3.06

3.38

3.75

6.00

3.95

3.84

3.68

3.93

3.2C

3.19

3.24

3.94

4.18

3.31

3.70

5. 19

4.43

4. 15

3.32

4. 18

3.57

4.17

3.02

2.82

2.44

EKL

0.20

0.19

0.27

0.25

0.20

0.20

0.28

0.26

0.21

0.21

0.28

0.17

0.29

0.22

0.21

0.19

0.21

0.15

0.11

0.24

0.18

0.19

0.18

0.21

0.19

0.24

0.20

0.20

0.23

0.22

0.23

0.21

0.16

0.19

0.18

0.22

0.23

0.21

0.18

0.19

0.20

0.20

0.18

0.20

0.21

0.22

0.18

0.16

0.22

0.18

13

17

18

22

16

17

12

21

22

25

81
VOTES WORD

FAR

LESS

EVERY

CLAIME

RATHER

HEARD

VERY

JUSTIF

H I MS EL

TOGETH

RELATE

LATTER

WHOM

SHOW IN

VARIOU

APPLY

SUGGES

PLACED

READS

VALID

LEAST

AGAIN

BEYOND

TIMES

DISSEN

OCCASI

HOW

LIKE

BECAME

PUT

THEREB

NOTED

AGREE

APPROX

MENTIO

MUCH

COME

STILL

WHOSE

MERE

ESSENT

WHOLE

SEEMS

08VIOU

FOREGO

DOING

FULLY

QUOTED

ADDED

AMONG

DIFF IC

MAKES

WHERE I OPPORT

ALREAD

REACHE

ALONE

DESIRE

NONE

HERETO

Table

NOCC

923

923

922

921

9L7

903

888

885

86*

861

839

833

832

829

815

806

782

781

769

768

766

766

754

751

75 1 742

739

738

734

719

712

710

707

704

694

693

663

660

655

654

651

651

647

645

626

625

591

591

587

579

578

565

560

545

542

539

536

507

506

498

IV.

E

6.11

6.08

6. 1

5.97

6. 15

5.97

6.15

5.90

5.95

6.04

5.92

6.04

6.00

5.78

5.99

6.00

5.94

5.88

5.89

5.83

6.00

6.00

5.87

5.95

5.48

5.95

5.93

5.93

5.81

5.88

5.99

5.88

5.91

5.79

5.91

5.99

5.90

5.86

5.89

5.82

5.83

5.74

5.88

5.87

5.73

5.71

5.74

5.60

5.62

5.83

5.72

5.73

5.60

5. 53

5.68

5.63

5.73

5. 38

5.58

5.41

Sorted

EL PZD

6.2 4 2 2.61

17 21

22 22

1 / 21

66

6

6.21 22.00

6.0 7 19.93

63

31

44

6.22 21

6.C7 19

93

85

6.L0 19.85

6.16 20.91

,12 20.04

,14 20.23

,13 20.08

,16 20.53

,12 19.96

,08 19.63

.06 18.68

6.05 18.91

6.03 18.30

5.92 17.06

6.11 19.40

6.11 19.32

5.99 17.74

6.09 19.21

5.73 13.43

6.03 18

6.01 17

38

89

6.08 18.87

6.08 18.61

5.96 17.40

6.11 19.02

6.02 18.04

6.10 18.98

5.87 15.77

6.02 17.89

6.11 19.13

6.00 17.40

6.07 18.08

6.0 4 17.70

5.99 17.02

5.98 16.76

5.7d 14.87

5.98 16.87

6.09 18.23

5.96 16.64

5.89 16.04

5.93 16.00

5.85 15.13

5.77 13.96

5.93 15.81

5.87 15.06

5.98 16.27

5.92 15.66

5.7 5 13.70

5.80 14.08

5.86 14.91

5.87 14.79

5.78 13.74

5.82 14.23

5.64 12.60

by NOCC

AVG

0.0247

0.0250

0.0244

0.0261

0.0246

0.0241

0.0230

0.0235

0. 0241

0.0222

0.0233

0.0235

0.0228

0.0227

0.0214

0.0212

0.0208

0.0208

0.0220

0.0207

0.0206

0.0209

0.0209

0.0201

0.0191

0.0206

0.0191

0.0198

0.0196

0.0197

0.0192

0.0182

0.0187

0.0179

0.0191

0.0187

0.0173

0.0176

0.0179

0.0170

0.0173

0.0169

0.0179

0.0187

0.0163

0.0167

0.0 159

0.0149

0.0144

0.0152

0.0155

0.0151

0.0155

0.0146

0.0141

0.0139

0.0152

0.0143

0.0136

0.0121

G

4.89

3.44

3.05

4.84

3.00

3.35

2.80

3.52

5.07

3.31

3.10

3.47

3.43

3.37

4.01

3. 14

3.46

4.15

3.56

3.58

2.98

4.64

3.35

3.18

3.84

3.38

3.23

4.09

3.61

3.40

3.22

3.47

3.50

3. 77

4.96

3.85

3.24

3.47

3.34

3.36

3.67

3.54

4.19

3.36

3.55

3.56

4.28

3.88

4.33

3.05

3.98

3.28

4.62

5. 13

3.49

4.07

4.20

4.09

3.70

3.70

EK

0.20

0.21

0.22

0. 17

0.24

0.18

0.24

0.18

0.17

0.21

0.20

0. 19

0.19

0. 19

0.20

0. 19

0.18

0. 16

0. 16

0.16

0.20

0.18

0.17

0. 19

0. 12

0.18

0. 19

0. 17

0.18

0.17

0. 19

0.17

0. 19

0. 15

0. 16

0. 19

0.18

0.18

0. 18

0. 17

0. 16

0.14

0. 16

0. 18

0.16

0. 15

0. 14

G. 14

0. 13

0.17

0.14

0. 17

0. 13

0.11

14

14

14

12

0. 14

0.12

GL

4.79

3.99

3. 79

3.94

3.67

5.06

3.45

4.41

3.60

3.86

4.01

3.63

3.68

3. 12

3.74

4.78

3.55

4.20

3. 85

4.77

3.43

3.29

3.90

3. 80

3.90

5.02

3.80

3.62

3.09

5.70

3.25

4.48

3.35

4.01

4.13

3.99

3. 88

2.94

3.38

3.95

3.52

5. 73

3.41

2.92

3.70

5. 74

3.71

4.09

3.95

3.70

3.51

3.07

3. 89

4. 15

4.07

4. 15

3.50

3.97

4. 14

6.07

EKL

0.18

0.17

0.18

0.17

0.18

0.14

19

15

0.16

0. 17

0.16

0.17

0.17

0.18

16

15

16

15

0.15

0.12

0.17

0. 17

0.14

0. 16

0.11

0.14

0.15

0.16

0.17

0.13

0.17

0.14

0.17

0.12

0. 15

0.17

0.15

0.17

0. 16

0. 14

0.15

0.10

0.15

0.18

0.14

0.12

0.14

0.12

0.12

0. 14

0.13

0.15

0.14

0.11

0.12

0.13

0.13

0.12

0. 12

0.05
WORD NOCC E EL PZD G EK GL EKL

MOVED 492 5. 6 1 5.75 1 i . 40 0.0149 3.94 0.13 4.21 0.11

RELIED 487 5. 62 5.80 13.89 0.0134 4.43 0. 12 4.02 0.12

CONCED 48 5 5. 58 5.8 3 14 .00 0.0140 3.43 0. 14 3.42 0.13

EVER 481 5. 47 5.65 12.23 0.0127 4.47 0. 11 4.27 0.10

C ARE FU 453 5. 42 5.79 13.51 0.0118 3.79 J. 13 3.84 0.12

HENCE 447 5. 43 5.68 12.26 0.0118 4.38 0. 11 3.85 0.11

ARGUES 44 3 5. 52 5.67 12.23 0.0136 4.75 0. 11 3.96 0.11

SOLELY 441 5. 50 5.74 12 .87 0.0118 4.03 0. 12 4. 06 0.12

FAILS 426 5. 21 5.68 12.15 0.0125 4*84 0. 10 3.68 0.11

COMPAR 418 5. 42 5.57 1 1.09 0.0 121 4.96 0.09 4.20 0.09

ABLE 416 5. 37 5.64 11.77 0.0107 4.69 0. 11 4.20 0.10

LIKEWI 404 5. 52 5.64 1 1 .70 0.0106 3.26 0. 12 4.45 0.10

ARGUED 396 5. 47 5.71 12.15 0.0117 3.88 0. 12 3.34 0. 12

STATI

385 5. 43 5.67 11 . 77 0.0112 4.44 0.11 3. 86 0.11

EXISTS 376 5. 38 5.59 10 .94 0.0104 4.09 0. 11 3. 84 0.10

ONCE 375 5. 32 5.60 1 1 .02 0.0094 3.70 0. 11 3.77 0.10

SEEKS 374 5. 15 5.62 11 .32 0.0117 4.95 0.10 3.75 0.10

NEVERT 370 5. 50 5.71 1 1 .92 0.0096 3.19 0.13 3.20 0.12

INSIST 368 5. 36 5.51 10.41 0.0096 3.68 0. 10 4.72 0.09

INSTEA 328 5. 29 5.52 10.07 0.0088 4.25 0. 10 3.97 0.09

VIRTUE 322 5. 21 5.46 9.55 0.0091 4. 56 0.09 3.99 0.09

ALLEGI 320 5. 18 5.47 9.66 0.0088 4. 31 0.09 4.05 0.09

NAMELY 316 5. 27 5.44 9.36 0.0080 4.71 0.09 4.09 0.09

QUITE 307 5. 32 5.46 9.39 0.0083 4.11 0.09 3.74 0.09

RELIES 301 5. 28 5.48 9.62 0.0090 4. 16 0.09 3.91 0.09

WEYGAN 251 4. 57 5.40 8. 79 0.0050 6.09 0.05 3.57 0.09

MATTHI 249 4. 57 5.37 8.64 0.0049 6.34 0.05 4.17 0.08

SOiMET I 237 5. 05 5.22 7.39 0.0068 5.15 0.07 4. 18 0.07

SOMEWH 236 5. 13 5.27 7.73 0.0070 4.87 0.07 4. 12 0.07

DESMON 230 4. 86 5.24 7.47 0.0065 4.60 0.07 4.06 0.07

PECK 216 4. 34 5.22 7.43 0.0043 7.17 0.04 4.22 0.07

VOORHI 209 4. 80 5.23 7.32 0.0059 4.32 0.07 3. 98 0.07

FROESS 209 4. 78 5.18 6.98 0.0062 4.96 0.06 3.98 0.07

FULD 208 4. 73 5.20 7.09 0.0057 4.57 0.06 4.05 0.07

Table IV. Sorted by NOCC

83
VOTES WORD NOCC E

THE 442506 7.87

AAAAAA 2649 7.07

AND 128355 7. 83

THAT 89026 7.80

FOR 45223 7.73

NOT 35835 7.75

THIS 29490 7.66

WAS 56044 7.69

WHICH 25522 7.70

10 COURT 33021 7.45

FROM 19879 7.62

WITH 21624 7.64

HAVE 13825 7.53

SUCH 18195 7.50

3 CASE 15261 7.45

ARE 13721 7.46

THERE 12925 7.48

BEEN 12072 7.50

ANY 13855 7.47

UPON 11816 7.46

HAD 15451 7.43

HAS 10530 7. 36

UNDER 10893 7.40

WERE 12911 7.43

8UT 9174 7.48

HIS 19529 7.32

9 APPEAL 9096 6.80

2 QUESTI 8776 7.25

MAY 9510 7.37

1 ONE 9388 7.39

OTHER 8966 7.43

ITS 11061 7.31

1 ALL 9021 7.36

MADE 7999 7.32

2 LAW 9658 7.23

9 JUDGME 10581 7.06

WOULD 9678 7.34

2 REASON 6845 7.17

1 ONLY 6218 7.33

5 DEFEND 25773 7.20

3 TIME 8254 7. 17

WHEN 6875 7.28

1 FOLLOW 6076 7.28

SAID 10747 7.07

BEFORE 5814 7. 19

AFTER 6340 7.24

3 PRESEN 5653 7.18

ALSO 5230 7.29

DID 6224 7.24

MUST 5208 7. 18

SHOULD 5689 7.20

WHETHE 5173 7.22

9 EV I DEN 12726 7. 10

WHERE 5794 7. 19

6 ACTION 8248 6.94

THEY 7042 7. 14

2 REOUIR 6103 7.06

4 CONCUR 2290 6.65

8 CONS ID 5288 7.15

WITHOU 4652 7. 10

Table y. Sorted

EL

7.65

7.87

7.61

7.60

7.61

7.60

7.59

7.55

7.56

7.41

7.51

7.51

7.44

7.35

7.36

7.39

7.40

7.41

7.37

7.40

7.30

7.37

7.31

7.31

7.37

7.22

7.06

7.28

7.30

7.31

7.31

7.20

7.26

7.29

7.20

7.17

7.23

7.25

7.31

7. 12

7.20

7.24

7.24

6.93

7.23

7.21

7.20

7.23

7.17

7.22

7.20

7.19

7.02

7.16

6.92

7.08

7. 10

7.30

7.14

7. 17

by PZD

PZD

99.99

99.99

99. 73

98. 15

98.07

96.97

96.67

95.73

94.41

93.58

92. 18

92. 03

85.99

85.80

84.74

84.37

84.25

83.76

83.12

82.9 3 82.44

81.76

80.44

79.91

78.89

78.63

77.61

77.08

76.70

76.40

76.17

75.34

74.78

74.51

74.29

73.19

73.12

72.48

72. 14

71.19

70.40

69.87

69.38

69.15

68.55

68.47

68.25

67.15

66.70

66.70

66.59

66.13

65 .64

65.26

64.55

64.47

63.98

63.91

63.72

63.57

AVG

12. 1192

0.0783

3.4562

2.4343

1.2529

0.9798

0.8106

1.5630

0.6984

0.9097

0.5456

0.5840

0.3761

0.4817

0.4182

0.3766

0. 3545

0.3306

0.3703

0.3232

0.4205

0.2838

0.2937

0.3486

0.2485

0. 5396

0.2637

0.2395

0.2605

0.2540

0.2397

0.2888

0.2361

0.2213

0.2554

0.3119

0.2580

0.1850

0. 1693

0.7468

0.2237

0.1866

0. 1661

0.2803

0.1612

0. 1745

0.1558

0.1410

0. 1665

0.1412

0.1511

0. 1408

0.3461

0. 1562

0.2329

0. 1897

0. 1665

0.0643

0.1379

0.1274

G

-0.19

0.42

0.53

0.70

1.03

0.55

1.15

0.52

0.64

1.64

1.25

1.15

1.17

1.49

1.64

1.56

1.30

1.41

29

37

49

34

82

1.43

0.84

1.55

4.94

2.17

1.45

1.61

1.18

1.71

1.45

1.60

2.34

3.01

1.43

2.15

1.57

1.34

2.55

1.54

1.30

4.45

2.12

1.62

2.26

1.08

1.55

1.83

1.89

1.69

1.64

1.64

3.64

2.45

2.34

2.45

2.06

2.02

EK

41.17

4.32

15.25

9.48

5.00

6.95

4.02

3.68

4. 89

1.26

3.01

3.46

2.52

1.78

1.43

1.85

1.87

1.96

1.87

1.76

1.38

1.51

1.31

1.55

2.21

1.03

0.30

1.03

1.38

1.48

1.79

1.13

1.46

1.25

0.88

0.54

1.34

1.11

1.38

0.79

0.92

1.20

1.18

0.50

0.95

1.06

0.88

1.33

1.03

1.08

1.02

1.04

0.71

1.03

0.39

0.77

0.74

0.73

0.93

0.91

GL

1.87

2.55

2. 14

1.92

1.87

1.90

2.45

1.78

L.79

3.97

1. 83

2.15

2.53

2.91

2.38

2.55

2.17

2.07

2.37

1.83

2.68

2.41

2.98

2.67

2.06

2.83

5. 35

4.30

2.50

2.40

2.45

3.49

3.34

1.97

3. 39

4.08

2.49

2.86

1.88

2.43

2. 17

2.24

2.44

6.83

2.63

2.27

3.49

1.95

2.52

2.79

2.45

2.57

3.09

2.43

4.77

3.52

4.53

2.51

2.68

2.39

EKL

1.93

31.32

1.57

1.54

1.59

1 .^6

1.41

1.33

1.38

0. 76

1.19

1. 16

0.97

74

80

86

0.91

0.95

83

95

69

33

0.69

0.70

0.89

0.60

0.33

0.62

0.72

0.75

0.76

0.54

0.64

0.76

0.54

0.49

0.64

0.64

0.82

0.53

0.62

0.69

0.69

0.27

0.66

0.65

0.58

0.71

0.59

0.64

0.63

0.61

0.43

0.58

0.31

0.45

0.47

0.86

0.56

0.62

84
VOTES WORD

4

25

6

7

84

5

74

10

5

6

9

AFFIRM

DOES

TRIAL

WILL

THEREF

STATE

FURTHE

AG A I NS

COULD

THEIR

PERSON

SAME

PART

TWO

RECORD

FACT

PROVID

THESE

WHO

DETERM

THAN

THEN

OPINIO

DIRECT

ORDER

PLAINT

APPEAR

BEING

BECAUS

FIRST

CONTEN

OUT

HOWEVE

FACTS

SECTIO

WITHIN

HELD

FILED

MATTER

PROCEE

SEE

STATED

CAUSE

RIGHT

HIM

CONCLU

MOTION

FOUND

STATUT

CONTRA

GENERA

HERE

COUNTY

EFFECT

AUTHOR

NECESS

END

CASES

INTO

SOME

NOCC

3897

4264

9898

7140

3871

9231

4546

5725

5096

6514

6980

4992

4746

5130

6093

4658

5792

4753

5241

5030

4378

4583

4764

5706

6773

20986

3855

3858

3553

4165

3888

4389

3333

4095

10226

4561

3978

5362

4313

5021

4704

3698

4463

5447

5613

3665

6621

3608

7283

8033

5262

3448

6245

3759

4898

3477

6422

3896

3583

3394

6.89

7.09

6.97

6. 84

7.01

6.85

7. 11

7.04

7, 16

7.08

7.01

7.05

7.12

7.11

6.91

7.06

7.03

7.11

7.11

EL

7.23

7.2 0 6.98

6.74

18

80

13

06

11

Table V.

02

11

12

02

95

78

7.02

6.95

7.04

7.00

7.01

7.02

7.00

7.09

7.00

6.83

6.85

7.04

6.67

6.91

6.79

6.93

6.99

6.77

6.76

6.91

6.95

6.71

6.91

6.89

6. 56

6. 87

6.93

6.62

6.91

6.78

6.93

6.81

6. 86

6.93

6.97

Sorted

7.02

6.94

7.07

7.09

7.11

6.98

7.10

7.02

7.07

7.03

7.01

7.10

7.07

6.98

6.92

6.77

6.94

7.00

7.08

7.11

7.04

7.09

6.99

7.11

7.01

6.76

6.97

7.02

6.91

6.96

6.84

6.88

6.99

6.90

6.86

6.85

7.02

6.84

6.98

6.80

6.49

6.82

6.97

6.52

6.92

6.81

6.93

6.71

6.90

6.92

6.93

by PZD

PZD AVG G

6 3 .53 n.U i i n q . L1U7 9 3A

6 3 . 30 U • JL JL I j 1 • 0 u

6 2 .85 A TOO/. 9 7 <z 62 .55 n i q /. /. u . ivtt 3«t7AO

6 2 .21 1 A 3 6 2 .06 ^_> * OAU O

6 1 . 94 a 1 lift U • i. C. D\J 1 Q 9 61 .83 A 1 A A C 9 c i 6

< 79 a i "a fi i 1 • P V

6 1 .75 fiU •

11 7SA 1 DO C< 17

60 .81 OU 1 >)Q7 . 107I 9 A

60 .73 AU .

11 A

9 Q7 Q7 9 A7

60 .62 A 1 907 9 <=> 7

60 .51 A 1 A AO 11 .

•iJ Q7 60 .51 3a ^3

60 .28 A 1 9AQ 9 1 n 60 .02 a 1 r.qq 59 . 79 U . 1 A 1 J 11.71Q 7 59 . 64 n TATA 11 .

ftO Q7 59 .45 A 1 "J 1 A ^ OA

59 .38 V m l Ut) 9 ? "\ 5 9 . 19 Ti 1 ?i.7 C . 1 A H A

~> OA

58 .35 v.f 1 7 1 ft . 1 A L O ? n si C m \J 0* 5 8 • oZ -> . i c 0 8 • 32 O 1 O 1 ft 'Kj . AO RO

D I . /

n a n q 7 1.43

D J .68 A 1 nit U . 1 UtJ )•^ 7Q71 C ~J .41 9 113^

C 1 0 (1 o • IV D\J .

fiU Q7 Q7 Q7 ? r»A 5 7

1 c nu .

111111 a9 5

' * 1 1 t9 •

11 A*t K 7 5 I A / A 1 1 A A 3.UU

o j

•TV A .90 U . U 7 A .9 1 A7

r> j

~y o . f V n 11 77 n •i 55 .75 n 0 ft ft 9 Q

'•71

5 5 .56 A 1 9 A A U . 1 2V4 9 A 1 55 .34 P.O 1 (Tift • 1U3D 11.7^Q 9 55 .26 n i ^ ft o, a ncj 5 5 .19 PO .

111LOOA A Jdl

5 5 .19 U. i j 1 j

55 .00 OUrn 11 A9 Q7 7f ? Q ^ 54 .77 U . U 7 / J

9 ^7

54 .28 0, 1 9 s K 9 Q ft 54 .24 A 1 A A A 9'•71d 1 54 .24 0 1 111 9'•H7AQ

53 .90 iiJ .

11U1U m

5 3 .90 1 1 Q A 9 7ft 5 3 .68 n 1 m 7 9 7 7

5 3 . 1 5 a 1 001; U . 1 7UJ 9 . A

9 AO

5 2 .96 A 11 CO U • ^ 1 3o O7 Q ft . 7 0

52 . 92 A 1 9 9 Q .1.11 7 11 52 . 69 aU.U730 n q 9 ft 1 Q 9 1 * rf C 52 .43 U

A .11 707 / 0 / ^ on

52 . 39 0 1 ^ 1 ft A9 .

ft0 OA 52 . 32 A 1 i 1 Q H» 33

52 . 20 AU.U7J A Q 9 1

3a31 7 7 1 5 1 . 86 A 1 C70 U . 13/U 73>Un 71 5 1 . 4

DU •1 HA 9 1 U O c. 9 ft 51 .00 A A Q C 9 i.9 c>311 50 .88 0.0897 1.97

EK

0. 78

0.96

0.45

0.26

0.90

0.39

0.91

0.63

0.95

0.70

0.57

0.76

0.78

0.85

0.41

0.80

0.64

0. 83

0.79

0.64

0.81

0.82

0.71

0.44

0.31

0.64

0.56

0.75

0.75

0.71

0.71

0.65

0.90

0.60

0.38

0.50

0. 75

0.33

0.53

0.40

0.47

0.68

0.43

0.47

0.52

0.64

0.30

0.53

0.48

0.23

0.47

0.66

0.23

0. 56

0.37

0.52

0.44

0.54

0.57

0.67

GL

2.61

2.11

2.96

12.86

2.25

4.64

3.44

3. 13

2.58

3.29

5.09

3.32

2.85

2.47

4.95

2.40

3.62

3.27

3.51

3.95

2.63

2.60

4.63

6.63

11.48

2.24

9.43

2.89

2.28

3.27

2.24

6. 13

1.76

2. 90

4.29

3.59

2.83

46

12

15

89

6 9

C8

87

6.64

2. 52

3.36

3. 16

4. 39

7.29

5.01

3.12

8.51

7.29

4.61

4.91

6.84

3.22

3.14

4. 84

EKL

0.70

0.67

0.41

0.15

0.65

0.25

0.53

0.46

0. 54

0.42

0.33

0.48

0.52

0.55

0.35

0.54

0.42

0.48

0.44

0.40

0.54

0.51

0.37

29

19

43

32

52

58

0.46

0.56

0.37

0.62

0.46

0.27

0.41

0.47

0.36

0.38

26

33

42

34

0.32

0.29

0.49

0.33

0.43

0.29

0.15

0.28

0.43

0.14

0.34

0.28

0.35

0.22

0.38

0.39

0.39

85
VOTES

23

6

9

1

WORD

CEKTAI

APPELL

EXCEPT

ACCORD

SEC

MORE

THEM

PURPOS

SHALL

CAN

S INCE

ENTERE

RESULT

NEW

OUR

INVOLV

ALLEGE

BETWEE

APPLIC

PROVIS

RULE

REVERS

JUDGE

DECIS I CANNOT

BOTH

SET

SUPPOR

WHILE

SUSTAI

ACT

SUBJEC

ITAL

FOL

GIVEN

APP

WHAT

ERROR

EST ABL

RESPEC

GROUND

MAKE

EACH

INCLUD

NOW

NOR

WELL

TAKEN

CONSTI

SUFFIC

ISSUE

COMMON

THOSE

SPECIF

PARTIC

HAVING

CIRCUM

HEREIN

PART IE THEREO

NOCC

3069

14543

3589

2721

6808

3050

3505

4138

6240

2822

2756

2920

3328

4744

3179

2933

3766

3231

4168

4479

4090

2857

4000

3988

2467

2868

2964

3151

2749

2600

5147

2855

11360

5682

2766

4769

2883

3841

2947

2579

2629

2535

3332

2632

2384

2099

2259

2518

4132

2484

3113

4042

2527

2900

2381

2006

2543

2599

3496

2640

Table V.

E

6.87

6. 53

6. 58

6.3 7 6.65

6.94

6.92

6.76

6.81

6.93

6.89

6.78

6.85

6. 68

6. 80

6. 56

6.72

6. 84

6.58

6.80

6.56

6.66

6*52

6.52

6.74

6.85

6.71

6.65

6.82

6.65

6.65

6.70

6.67

6.67

6.80

6.74

6.76

6. 56

6.74

6.80

6.68

6.76

6.68

6.71

6.60

6.70

6.77

6.67

6.41

6.72

6.61

6.46

6.73

6.65

6.48

6.67

6.75

6.23

6.55

6.69

Sorted

EL ?ZD

6.9t> 50.62

6.44 50.16

6.82 49.79

6.96 49„64

6.62 49.60

95 49.49

89 49.37

76 49

7 i 49

94 49

6.93 48

,30

. 18

,15

,65

,87 48.58

,86 48.50

,72 48.09

,83 47.98

,90 47

,81 47

86

86

66

66

6.6.

6.8 7 47.4 5 6.60 47.37

6.77 47.18

6.70 47.18

6.93 46.96

6.64 46.84

6.69 46.58

6.92 46.54

6.88 46.54

6.84 46.54

6.67 46

6.85 46

35

31

6.89 46.24

56

48

18

18

6.59 45

6.81 45

6.57 45

6.57 45

6.82 45.07

6.72 44.92

6.79 44.80

6.66 44.80

6.72 44.46

6.82 44.43

6.77 44.16

6.84 43.94

6.69 43. 9C, 6.76 43.41

6.80 43. 2S

6.86 43.14

6.8 3 43.14

6.76 43.07

6.49 42.99

6.81 42.92

6.66 42.88

6.48 42.58

6.77 42.43

6.68 42.28

6.76 42.12

6.86 42.09

6.75 41.94

6.70 41.75

6.59 41.71

6.75 41.60

by PZD

AVG

0.0 830

C. 3877

C. 1046

0. 0745

0. 1929

0.0822

0.0943

0. 1096

0. 1705

0.0739

0.0753

0.0873

0.0911

0. 1295

0.0833

0.0789

0.1091

C.0379

0.1134

0. 1251

0.1055

0.0842

0.1181

0.1070

0.0694

0.0771

0.0798

0.0855

0.0751

0.0753

0.1370

0.0784

0.2755

0.1378

0.0744

0. 1292

0.0725

0.1051

0.0788

0.0678

0.0728

0.0681

0.0859

0.0716

0.0629

0.0581

0.0592

0.0697

0.1058

0.0708

0.0831

0.1171

0.0642

0.0790

0.0625

0.0548

0.0679

0.0670

0.0960

0.0697

G

2.20

3.05

5.95

2.12

3.75

1.98

2.56

3.99

2.77

1.61

1.76

3.29

3.50

3.77

2.15

2.29

3.04

2.33

4.97

2.55

4.23

2.65

10.30

4.00

2.06

1.87

3.36

7.06

5.29

3.40

3.30

2.72

3. 12

3.12

2.27

2.51

2.52

3.69

3.00

1.99

3.25

2.35

4.53

3.86

2.79

1.94

2.87

3.27

3.48

2.35

3.76

5.85

3.12

3.75

3.17

2.18

2.08

3.17

3.86

2.61

EK

0.65

0.23

0.26

0.62

0.27

0. 66

0.56

0.41

0.43

0.67

0.62

0.42

0.49

0.31

0.55

0. 56

0.40

0.55

0.25

0.45

0.31

0.48

0.19

0. 30

0.57

0.59

0.45

0.24

0.43

0.40

0.32

0.46

0.37

0.37

0.50

0.41

0. 51

0.29

0.45

0.54

0.38

0.54

0.36

0.39

0.46

0.53

0.51

0.37

0.28

0.45

0.32

0. 19

0.46

0.34

0.41

0.51

0.49

0.36

0.29

0.42

GL

3.90

5.26

4. 72

2. 92

4. 50

2. 76

4. 37

6. 33

4. 34

68

78

02

97

33

84

99

3. 37

2. 83

8.13

3.69

12.48

3.60

6.80

5.57

2.46

2.81

3.72

9.79

4. 31

2.63

6.21

3.64

7.32

7.39

3.10

3.31

3.76

4.33

17.95

3.71

5.73

3. 17

5. 12

3.68

3.10

2.78

3.49

4.04

7.53

3.24

4.98

7.01

3.52

5.03

3.48

2.07

2.94

5. 86

4.47

3.06

EKL

C . 42

0.16

0.30

0.45

0.21

0.45

0.36

0.25

0.27

0 .44

0.43

0. 34

0.34

0.26

0.31

0.40

0 .33

0.38

0.16

0.30

0.20

0.43

0.20

0.23

0.45

0.39

0.35

0.18

0.35

0.41

0.20

0.33

0.19

0.19

0.35

0.29

0.32

0.24

0.18

0.34

0.29

0.37

0.25

0.31

0.34

0.40

0.36

0.31

0.15

0.36

0.23

0.16

0.33

0.25

0.32

0.43

0.33

0.25

0.22

0.33

86
VOTES

3

3

1

3

7

3

9

57

8

12

17

11

2

10

157

WORD

SUBSTA

FINDIN

THREE

ABOUT

OVER

PRIOR

CHARGE

CONSTR

DENIED

PETITI

EITHER

CONTRO

PERMIT

OPERAT

ANSWER

UNTIL

RECEIV

EVEN

ALTHOU

SECOND

ENTITL

USED

CONTAI

CITY

FIND

INDICA

AMOUNT

COMPLA

YEARS

RELATI

PROPER

DURING

ANOTHE

USE

EXPRES

DISK IS

PUBLIC

EXAMIN

CONDIT

I NTERE

ABOVE

OWN

I NSTAN

THUS

C ONCER

TEST I M THROUG

PRINCI

OHIO

CONTIN

MIGHT

JURY

STATEM

DAY

OFFICE

SHOW

UNLESS

BROUGH

PAGE

CLEAR

Table

NOCC

252 7 3437

2437

3228

2622

2379

4622

3805

2053

7623

2033

2941

2869

4207

3398

2347

2801

1964

1762

2415

2141

2650

2096

5969

1954

1901

3110

3971

2601

2530

5913

2216

1881

3852

2022

2755

4658

3117

2779

3637

1812

1857

1867

1622

1797

3650

1954

2158

8519

2382

1734

5530

2732

2189

4060

1649

1520

1534

3218

1537

V.

E

6.62

6.56

6.7C

6.65

6.72

6.69

6.48

6.58

6. 30

6. 19

6.71

6.48

6.35

6. 52

6.42

6.65

6.52

6.64

6.67

6.53

6.53

6.45

6.55

6.24

6.51

6.64

6.49

6.40

6.53

6. 54

6.40

6.58

6.57

6.29

6.51

5.96

6.33

6. 19

6.46

6.36

6.40

6.53

6.54

6.58

6.57

6.42

6.52

6.46

6.49

6.37

6.57

6.41

6.32

6.41

6.26

6. 36

6.54

6.50

6.47

6.52

Sorted

EL ?ZD

6.7 1 4 1.60

6.59 4 1.56

6.73 41.18

6.65 41.10

6.71 40.99

74 40.88

47 40.69

55 40.50

77 40.39

6.44 40.39

6.78 40.20

6.55 39.93

6.49 39.63

6.45 39.56

6.41 39.33

6.70 39.22

6.57 39.10

6.75 38.80

6.77 38.65

6.61 38.50

6.69 38.42

6.58 38.16

6.65 38.12

6.23 38.05

6.66 37.75

6.70 37.67

6.52 37.56

6.45 37.44

6.56 37.10

6.53 37.10

6.34 36.91

6.62 36.50

6.65 36.35

6.27 36

6.61 36

6.48 35

6.30 35

6.23 35.56

6.47 35.52

6.32 35.33

6.63 35.18

6.60 34.99

6.60 34. £18 6.65 34.80

6.59 34.76

6.41 34.65

6.56 34.61

6.43 34.61

6.3 5 34.39

6.40 34.35

6.63 34.27

6.31 34.27

6.36 34.16

6.46 34.16

6.12 33.93

6.59 33.89

6.63 33.82

6.59 33.74

6.45 33.71

6.57 33.48

by PZD

12

,01

,90

,78

AVC

0.0693

0.0995

0.0677

0.0 8 82

0.0701

0.0654

0. 1234

0. 1054

0.0580

0.2198

0.0532

0.0 849

0.0820

0. 1145

0.0913

0.0628

0.0764

0.0509

0.0487

0.0656

0.0591

Q.0734

0.0578

0.1706

0.0519

0.0499

0.0869

0. 1136

0.0687

0.0662

0. 1591

0.0609

0.0500

0.1059

0.0546

0.0790

0.1226

0.0831

0.0760

0.0944

0.0483

0.0502

0.0494

0.0427

0.0468

0.1010

0.0531

0.0564

0.2212

0.0634

0.0465

0.1470

0.0720

0.0607

0.1032

0.0470

0.0418

0.0460

0.0815

0.0425

G

3.48

4.00

3.19

2.68

2.40

2.87

3.96

3.38

2.91

3.73

1.96

5.05

6.17

3.54

5.64

2.31

6.76

2.09

1.78

3.97

2.60

5.62

3.35

3.90

3.11

2.45

3.85

4.27

3.24

3.61

3.62

2.73

2.97

4.86

3.21

5.16

4.86

7.01

3.52

5.26

2.94

2.91

2.58

2.08

4.40

3.30

3.87

6.01

2.35

5.85

2.4-0

3.35

4.77

3.92

4.82

3.26

2.32

4.00

2.83

3.35

EK

0.36

0.26

0.40

0.39

0.43

0.41

0.24

0.30

0.37

0.19

0.50

0.23

0.17

0.27

0.22

0.42

0.27

0.49

0.50

0.31

0.38

0.24

0.35

0.18

0.35

0.42

0.27

0.22

0.31

0.30

0.23

0.36

0.37

0.18

0.34

0.16

0.20

0.15

0.26

0.20

0.35

0.35

0.36

0.42

0.34

0.25

0.30

0.24

0.28

0.21

0.39

0.24

0.20

0.26

0. 17

0.32

0.39

0.29

0.31

0.33

GL

4.62

3.90

3.87

3.45

3. 50

3. 12

4.95

4.65

2.72

5.82

3.10

EKL

0.27

,00

36

52

,44

,46

5.74

3.06

2.66

5.63

3.68

4. 18

5.43

5. 82

3.70

3. 59

3.75

4.90

4. 19

5. 77

5.71

4.42

3. 17

7.72

4. 18

5.01

5.07

8.63

3.88

5.71

3.03

3.93

3.01

2.88

3.67

3. 88

4.00

7.85

5. 51

10.10

2.78

4.31

5.32

9. 83

18.75

3.21

2.95

3.64

5.57

5.39

23

30

27

29

32

18

21

35

18

0.35

0.20

17

18

13

30

0.21

0.35

0.37

0.23

0.30

0.23

0.25

0.13

0.28

0.31

0.22

0.19

0.23

0.20

0.15

0.26

0.29

0.12

0.26

0.20

0.15

0.11

0.21

0.15

0.29

0.27

0.28

0.31

0.26

0.20

0.24

0.16

0.17

0.14

0.30

0.17

0.16

0. 17

0.07

0.28

0.30

0.27

0.19

0.24

87
VOTES WORD NOCC EL PZD

4

2

759

5

53

9

2

359

1

182

2

4

1

5 13

8

2

13

84

152

89

5

3

3

23

STATES

DIFFER

WAY

ILL

8ASED

CALLED

REVIEW

COMPAN

couNse

OBJECT

EMPLOY

PLACE

CLAIM

ADD I T I DUE

ORIGIN

PARTY

HER

TESTIF

RENDER

HEAR I

RETURN

COMPLE

DATE

COURTS

THEREA

CITED

VIEW

APPARE

REGARD

BASIS

NOTICE

NOTHIN

COURSE

THOUGH

OVERRU

REMAIN

RIGHTS

TAKE

FAILED

FAILUR

DECIDE

ASSIGN

GIVE

JURISD

SERVIC

CODE

LATER

POINT

REQUES

SITUAT

SUPRA

RESPON

ATTEMP

ADMITT

FORTH

ARGUME

REFERR

SIMILA

CORREC

Table

2343

1714

1771

8605

1605

1618

2347

4677

3030

2703

6062

1881

2565

1708

1937

2053

2643

7548

3484

1657

2525

2074

1709

1983

2033

1342

1401

1406

1334

1466

1500

2855

1275

1500

1301

1644

1592

2108

1484

1442

1630

1409

2654

1490

3056

3855

4152

1426

1487

1941

1358

2573

2872

1404

1667

1458

1528

1309

1243

1358

V.

6.38

6.46

6.21

6.49

6.38

6.40

6.02

6. 19

6.22

6.27

5.98

6.36

6.24

6.39

6.40

6.23

6.26

6.30

6.35

6.30

6.28

6.24

6.30

6.31

6.28

6.40

6.41

6.35

6.43

6.39

6.41

6.04

6.24

6.22

6.43

6.23

6.35

6.30

6.38

6.29

6. 16

6.41

6.00

6.32

6.00

6.04

6.21

6.43

6.35

6.11

6.42

6.29

5.94

6.05

6.32

6.25

6.26

6.24

6.38

6. 14

Sorted

6.33 33

6.55 33

37

14

6.45 32.91

6.46 32.88

6.56 32.84

6.57 32.76

6.30 32.72

6.05 32.65

6.27 32.54

6.31 32.50

5.89 32.50

6.45 32.27

6.24 32.27

6.49 32.12

6.47 32.08

6.39 32.01

6.33 31.93

6.20 31.89

6.35 31.74

6.45 31.74

6.31 31.59

6.32 31.48

6.45 31.40

6.41 31.37

6.36 31.21

6.55 31.03

6.54 30.95

6.48 30.95

6.53 30.84

6.52 30.80

6.47 30.76

6.18 30.76

6.55 30..65

6.45 30.53

6.54 30.46

6.42 30.46

6.38 30.46

6.33 30.38

6.47 30.35

6.48 30.31

6.43 30.16

6.50 29.89

6.12 29.82

6.45 29.78

6.10 29.67

6.05 29.63

6.18 29.55

6.47 29.48

6.42 29.48

6.29 29.44

6.49 29.40

6.25 29.21

6.00 29.21

6.42 29.18

6.32 28.87

6.40 28.80

6.37 28.69

6.43 28.65

6.46 28.61

6.38 28.57

by PZD

AVG

0.0582

0.0466

0.0472

0.2551

0.0431

0.0444

0.0676

0. 1180

0.0868

0.0742

0. 1653

0.0528

0.0735

0.0453

0.0542

0.0558

0.0726

0.2095

0.0969

0.0464

0.0716

0.0589

0.0455

0.0555

0.0553

0.0389

0.0390

0.0375

0.0364

0.0380

0.0412

0.0853

0.0345

0.0421

0.0340

0.0456

0.0428

0.0581

0.0407

0.0414

0.0459

0.0 381

0.0715

0.0399

0.0812

0. 1114

0. 1146

0.0387

0.0407

0.0545

0.0368

0.0636

0.0772

0.0376

0.0436

0.0391

0.0429

0.0341

0.0339

0.0370

6.26

3.96

6.65

1.95

2.60

4.43

5.34

4.27

6.09

8.66

5.38

6.46

5.91

5.06

4. 13

4.38

4.28

4.05

3.53

3.94

4.03

8.81

4.76

3.97

9. 19

2.78

2.52

4.33

3.26

3.05

5.82

5.70

2.76

6.86

2.57

4.78

4.99

5.59

3.85

3.32

3.81

2.48

6.48

3.06

4.48

5.82

4.17

2.75

4.43

7.47

2.40

3.34

6.24

4.42

3.82

3.68

5.01

8.37

2.91

4.35

EK

0.22

0.29

0.22

0.34

0.35

0.31

0.15

0.17

0.15

0.15

0.11

0.21

0.15

25

25

21

20

0.20

0.24

0.23

0.21

0.15

0.24

0.23

0.16

0.32

0.33

0.29

0.30

0.32

0.26

0.14

0.33

0.21

0.34

0.19

0.23

0.20

0.27

0.29

0.24

0.31

0.12

0.29

0. 14

0.13

0.17

0.31

0.25

0.15

0.33

0.23

0.12

0.25

0.23

0.25

0.20

0.24

0.30

0.21

GL

8. 54

3.56

10.08

3.00

3.70

3.42

7.80

10.01

5.28

5.60

7.48

5.21

7.77

4.68

3.79

5.63

5.91

4.75

3.72

6.39

6.14

9.23

5.48

4.85

5.77

2.92

3.08

7.01

3.32

3.05

5.60

6.77

2.84

4.36

2.8.2

4.35

7.12

4.76

3.52

3.79

4.43

3.99

7.19

3.67

6.50

7.29

5.98

3.52

4.24

5.99

3.07

4.77

11.25

7.93

5.59

4.54

4.22

5.55

3.18

4.34

EKL

0.13

0.25

0.16

0.24

0.26

0.27

0.13

0.09

0.14

0.15

0.08

0.19

0.12

0.22

0.22

0.18

0.16

0.14

0.19

0.19

0.15

0.14

0.20

0.19

0.17

0.28

0.27

0.20

0.26

0.26

0.21

0.12

0.29

0.21

0.28

0.20

0.16

0.17

0.23

0.23

0.21

0.25

0.11

0.23

0.11

0.10

0.13

0.24

0.21

0.15

0.25

0.15

0.08

0.19

0.17

0.20

0.19

0.21

0.24

0.20

88
WORD NOCC E EL PZD AVG G EK GL EKL

LEGAL 1650 6. 25 6

30 28

57 0« 0423 7.41 0. 19 9.77 0. 14

ENTIRE 1350 6. 30 6

4 1 2853 00369 5.20 0.25 6.76 0. 20

TERMS 1583 6. 33 6, 39 28

46 0424 3.43 0.25 3.35 0. 21

DUTY 1873 6. 25 6

30 28 . 35 00506 3.82 0.21 5.09 0. 17

GRANTE 1574 6. 25 6

34 28. 35 0. 0425 4.97 0.20 5.70 0. 17

PAID 2316 6. 25 6. 2 5 28

16 0. 0616 3.21 0.23 4.69 0. 16

CLEARL 1145 6. 31 A\j « 4 5 ?7 67 n 0304 2.81 0.30 3.28 0 • 24

APPLI

1264 6. 25 AU 4 ?7 A 3 n V J J L 2.95 0.27 3.46 022

INTEND 1333 6. 29 AW .1 2 7 - h 3 0.w « 0 361 3.14 0.25 4.27 0« 21

SUPREM 1904 6. 16 6

24 2744 0. 0474 3.73 0.21 6.65 0. 14

OBTAIN 1498 6. 18 6 • 30 2 7

40 0. 0397 3.28 0.23 5.62 0. 17

MANNER 1259 6. 30 6

3 7 27 29 0 - 0329 3.46 0.27 6.32 0. 19

SEVERA 1243 6. 32 h\j .• 36 27

2 5 00331 3.47 0.26 7.53 0. 18

OTHERW 1095 6. 14 6

42 27. 18 0. 0307 4.16 0.25 3.79 0. 23

SUBSEQ 1263 6. 25 6 37 26

99 0. 0363 3.67 0.24 3.97 0. 21

FAVOR 1249 6. 22 6

37 26

87 0

0 364 3.45 0.23 4.09 0. 21

TRUE 1140 6. 23 6 • 36 26

23 0. 0309 3.33 0.26 4.42 0. 20

ORDERE 1180 6. 14 6 33 26

23 0. 0324 3.50 0.23 6.13 0. 18

MANY 1117 6. 27 6 . 38 25 . 82 0. 0286 2.52 0.29 2.73 0. 23

LANGUA 1492 6. 22 6. 2 3 25 . 78 0. 0411 3.66 0.21 5. 17 0. 16

SHOWN 1106 6. 15 6 * 36 25. 74 0. 0303 3.38 0.24 3.23 0. 22

THERE I 1068 6. 13 6. 38 25. 70 0. 0279 2.72 0.27 3.38 0. 23

BEL I EV 1176 6. 22 6. 34 25 . 67 0. 0322 3.33 0.24 3.34 0. 21

NATURE 1185 6. 16 6 . 325 . 48 0. 0313 3.80 0.22 4.10 0. 19

SAY 1088 6. 26 6 . 34 25. 44 0. 0294 2.94 0.26 3.71 0. 21

SOUGHT 1132 6. 11 6. 33 25. 44 0. 0316 3.80 0.21 4.23 0. 20

BECOME 1158 6. 07 6 . 30 25 . 36 0. 0 320 3.89 0.23 3.96 0. 19

SHOWS 1078 6. 16 635 25 . 25 0. 0297 3.55 0.23 3.06 0. 22

MAKING 1060 6. 19 6 . 33 25. 14 0. 0282 4.11 0.22 3.75 0. 21

DAYS 1500 6. 05 6 . 22 24. 99 0. 0447 6.03 0.14 3.91 0. 17

THERET 1022 6. 05 6. 35 24. 95 0. 0278 3.03 0.25 3.31 0. 22

MOST 1051 6. 25 6. 31 24. 95 0. 0273 2.65 0.28 6.00 0. 18

NE I THE 930 6. 16 6 • 3b 24. 87 0. 0252 2.65 0.27 2.44 0. 25

LONG 1047 6. 23 6. 32 24. 80 0. 0280 3.39 0.23 3.84 0. 20

HOLD 1033 6. 15 6. 35 24. 61 0. 0270 2.49 0.26 3.24 0. 22

PREVIO 1040 6. 16 6. 3 1 24. 57 0. 0277 3.93 0.22 3.68 0. 20

DONE 1079 6. 09 6 . 28 24. 57 0 . 0282 3.94 0.21 4.53 0. 18

REFUSE 1286 6. 14 6 . 22 24. 49 0. 0351 4.26 0. 19 4.13 0. 17

TOOK 1080 6. 15 6 . 2 8 24. 46 0. 0302 3.21 0.24 4.38 0 . 19

ITSELF 993 6. 25 6. 33 24. 38 0. 0260 2.40 0.27 3.32 0. 22

DISCUS 1034 6. 22 6. 3 1 24. 34 0. 0267 2.85 0.25 3.19 0. 21

RAISED 1050 6. 00 6 . 28 23. 93 0. 0290 3.56 0.21 3.95 0. 19

MERELY 936 6. 21 6. 32 23 . 78 0. 0248 2.46 0.26 2.82 0. 22

CONSIS 941 6. 19 6. 3 1 23. 66 0. 0260 2.47 0.26 3.02 0. 21

THINK 1035 6. 18 6. 2b 23. 63 0. 0298 3.00 0.23 3.20 0. 20

RECOGN 1033 6. 10 6. 2 5 23 . 51 0. 0261 3.33 0.23 3.94 0. 18

PURSUA 1039 6. 08 6. 24 23 . 17 0. 0271 2.92 0.22 3.93 0. 1

POSSIB 1018 6. 18 6. 23 22. 98 0. 0272 3.04 0.23 3.70 0. 18

HOLDIN 1008 6. 05 6 . 20 22. 76 0. 0265 3.62 0.21 4.43 0. 17

REV 1484 6. 07 6. 08 22 . 72 0. 0446 3.55 0.18 9.27 0. 12

DISTIN 997 6. 14 6 • 22 2268 0. 0265 2.77 0.24 4. 15 V • 1 ft FAR 923 6. 11 6 . 24 22

6 1 0. 0247 4.89 0.20 4.79 1 8

EVERY 922 6. 11 6 . 22 22 . 31 0. 0244 3.05 0.22 3.79 0. 18

KNOWN 1083 6. 12 6. 1 7 22 . 19 0. 0285 3.59 0.21 4.34 0- 16

EXISTE 1029 6. 06 6. 1 7 22 . 08 0. 0286 5.05 0.19 4.18 0. 16

RATHER 917 6. 15 6. 2 1 22 . 00 0. 0246 3.00 0.24 3.67 0. 18

VERY 888 6. 15 6 . 22 21. 93 0. 0230 2.80 0.24 3.45 0. 19

OCCURR 1248 6. 05 6. 1 1 21

78 0. 0347 3.73 0.18 4. 81 15

LESS 923 6. 08 6. 17 21. 63 0. 0250 3.44 0.21 3.99 0. 17

PREVEN 956 6. 00 6. 16 21. 44 0. 0265 3.86 0.19 3.57 0. 17

Table V. Sorted by PZD

89

772-957 0-66—
• WOnU EL PZD AVG G EK GL EKL

CLAIME 921 5 .97 6 .17 21 .44 0 .0261 4..84 0..17 3 .94 0 . 17

NEVER 976 6 .01 6 .15 21 .32 0 .0254 4..03 0.,19 4 .18 0 . 16

TOGETH 861 6 .04 6 .16 20 .91 0 .0222 3..31 0..21 3 . 86 0 .17

SHOWIN 829 5 .78 6 .16 20 .53 0 .0227 3..37 0..19 3 . 12 0 .18

LATTER 833 6 .04 6 . 14 20 .23 0 .0235 3. 47 0..19 3..63 0 .17

WHOM 832 6 .00 6 .13 20 .08 0 .0228 3..43 0, 19 3 .68 0 .17

RELATE 839 5 .92 6 .12 20 .04 0 .0233 3..10 0..20 4,.01 0 .16

VARIOU 815 5,.99 6 .12 19 .96 0 .0214 4.,01 0. 20 3..74 0 .16

HEARD 903 5..97 6 .07 19 .93 0 .0241 3, 35 0. 18 5,.06 0 . 14

HIMSEL 864 5 .95 6 .10 19 .85 0 .0241 5. 07 0, 17 3..60 0 .16

JUSTI

885 5,.90 6 .07 19 .85 0 .0235 3. 52 0. 18 4..41 0 .15

APPLY 806 6,.00 6 .08 19 .63 0 .0212 3.14 0, 19 4..78 0 .15

EMPHAS 1012 5<.96 6 .00 19 .59 0 .0246 3. 16 0. 19 5. 1? 0 .13

LEAST 766 6..00 6 .11 19 .40 0 .0206 2. 98 0. 20 3. 43 0,.17

AGAIN 766 6..00 6 .11 19 .32 0 .0209 4. 64 0. 18 3. 29 0 .17

TIMES 751 5c 95 6 .09 19 .21 0 .0201 3. 18 0. 19 3, 80 0..16

MUCH 693 5< 99 6 .11 19 .13 0 .0187 3. 85 0. 19 3.99 0..17

STAT 1245 5, 90 5 .93 19 .10 0 .0383 3. 51 0. 15 6.23 0..11

THEREB 712 5. 99 6 .11 19 .02 0 .0192 3. 22 0. 19 3.25 0..17

AGREE 707 5 C 91 6 .10 18 .98 0 .0187 3. 50 0. 19 3. 35 0..17

PLACED 781 5 c 88 6 .05 18 .91 0 .0208 He 15 0. 16 4. 20 0.. 15

LIKE 738 5 c 93 6 .08 18 .87 0 .0198 4. 09 0. 17 3. 62 0. 16

SUGGES 782 5. 94 6 .06 18 .68 0 .0208 3. 46 0. 18 3. 55 0,,16

BECAME 734 5 c 81 6 .08 18 .61 0 .0196 3. 61 0. 18 3. 09 0. 17

OCCASI 742 5, 95 6 .03 18 .38 0 .0206 3. 38 0. 18 5. 02 0. 14

READS 769 5. 89 6 .03 18 .30 0 .0220 3. 56 0. 16 3. 85 0. 15

OBVIOU 645 5o 87 6 .09 18 .23 0 .0187 3. 36 0. 18 2. 92 0, 18

STILL 660 5. 86 6 .07 18 .08 0 .0176 3. 47 0. 18 2. 94 0, 17

NOTED 710 5. 88 6. .02 18 .04 0..0182 3. 47 0. 17 4. 48 0. 14

HOW 739 5c 93 6<.01 17 .89 0..0191 3. 23 0. 19 3. 80 0. 15

MENTIO 694 5. 91 6 .02 17 .89 o..0191 4. 96 0. 16 4. 13 0, 15

BEYOND 754 5o 87 5 .99 17 .74 0..0209 3. 35 0. 17 3. 90 0. 14

WHOSE 655 5o 89 6..04 17 .70 0, .0179 3. 34 0. 18 3. 38 0. 16

COME 663 5. 90 6,.00 17 .40 0..0173 3. 24 0. 18 3. 88 0. 15

PUT 7 1 9 5 c 88 5<,96 17 .40 0..0197 3. 40 0. 17 5. 70 0. 13

FILE 943 5. 49 5<.87 17 .06 0<.0265 5. 51 0. 10 4. 17 0. 12

VALID 768 5o 83 5,.92 17..06 0 0207 3. 58 0. 16 4. 77 0. 12

MERE 654 5. 82 5..99 17..02 Oc 0170 3. 36 0. 17 3. 95 0. 14

MASS 468 7 5 o 77 5, .73 16..98 0. 1483 3. 41 0. 12 4. 36 0. 10

SEEMS 647 5. 88 5, 98 16..87 Oc 0179 4. 19 0. 16 41 0. 15

ESSENT 651 5. 83 5 c 98 16.,76 0.0173 3. 67 0. 16 3. 52 0. 15

FOREGO 626 5. 73 5. 96 16,.64 Oc 0163 3. 55 0. 16 3. 70 0. 14

MAKES 565 5o 73 5c 98 16..27 Oc 0151 3. 28 0. 17 3. 07 0. 15

DOING 625 5o 71 5< 89 16. 04 Oc 0167 3. 56 0. 15 5. 74 0. 12

FULLY 59 ! 5.c 74 5 c 93 16, 00 Oc 0159 4. 28 0. 14 3. 71 0. 14

AMONG 579 5. 83 5c 93 15. 81 Oc 0152 3. 05 0. 17 3. 70 0. 14

APPROX 704 5. 79 5c 87 15, 77 0179 a 77 0. 15 4. 01 0. 12

WHERE I 560 5. 60 5c 92 15, 66 0. 0155 4. 62 0. 13 3. 89 0. 14

QUOTED 591 5. 60 5 c 85 15. 13 0. 0149 3. 88 0. 14 4. 09 0. 12

DIFFIC 578 5. 72 5» 87 15. 06 0. 0155 3. 98 0. 14 3. 51 0. 13

REACHE 539 5. 63 5 = 86 14. 91 0. 0139 4. 07 0 = 14 4. 15 0. 13

WHOLE 651 5. 74 5o 78 14. 87 0. 0169 3. 54 0. 14 5. 73 0. 10

ALONE 536 5, 73 5. 87 14. 79 0. 0152 4. 20 Q. 14 3. 50 0. 13

NONE 50 6 5. 58 5. 82 14. 23 0. 0136 3. 70 0. 14 4. 14 0. 12

ALREAD 542 5. 68 5. 80 14. 08 Oc 0141 3. 49 0. 14 4. 07 0. 12

CONCED 485 5. 58 5c 83 14. 00 0. 0140 3. 43 0. 14 3. 42 0. 13

ADDED 587 5. 62 5- 77 13. 96 0. 0144 4. 33 0. 13 3. 95 0. 12

RELIED 487 5. 62 5o 80 13. 89 0. 0134 4. 43 0. 12 4. 02 0. 12

DESIRE 507 5. 38 5. 78 13. 74 0. 0143 4. 09 0. 12 3. 97 0. 12

OPPORT 545 5. 53 5. 75 13. 70 0. 0146 5. 13 0. 11 4. 15 0. 11

Table V. Sorted by PZD

90
VOTES WORD

CAREFU

DISSEN

MOVED

SOLELY

HERETO

HENCE

ARGUES

EVER

ARGUED

FAILS

NEVERT

STAT I

ABLE

LIKEWI

SEEKS

COMPAR

ONCE

EXISTS

INSIST

INSTEA

ALLEGI

RELIES

VIRTUE

QUITE

NAMELY

WE YG AN

MATTHI

SOMEWH

DESMON

PECK

SOMETI

VOORHI

FULD

FROESS

NOGC

453

751

492

441

498

447

443

481

396

426

370

385

416

40 4

374

418

375

376

368

328

320

301

322

307

316

251

249

236

230

216

237

209

208

209

5.42

5.48

5.61

5.50

5.41

5.43

5. 52

5.47

5.47

5.21

5.50

5.43

5.37

5.52

5.15

5.42

5.32

5.38

5.36

5.29

5.18

5.28

5.21

5. 32

5.27

4.57

4. 57

5.13

A. 86

A. 34

5.05

4.80

4.73

4.78

EL

5.79

5.73

5.75

5.74

5.64

5.68

5.67

5.65

5*71

5.60

5.71

5.67

5.64

5.64

5.62

5.57

5.60

5.59

5.51

5.52

5.47

5.48

5.46

5.46

5.44

5.40

5.37

5.27

5.24

5.22

5.22

5.23

5.20

5.18

PZD

13.51

13.43

13.40

12.87

12.60

12.26

12.23

12.23

12.15

12.15

11 .92

11.77

11 .77

11.70

11.32

11.09

11.02

10 .94

10.41

10.07

9

9

9

9

9

.66

.62

.55

. 39

.36

.79

.64

.73

.4 7 .43

.39

7.32

7.09

6.98

AVG

0.0118

0.0191

0.0149

0.0118

0.0121

0.0118

0.0136

0.0127

0.0117

0.0125

0.0096

0.0112

0.0107

0.0106

0.0117

0.0121

0.0094

0.0104

0.0096

0.0088

0.0088

0.0090

0.0091

0.0083

0.0080

0.0050

0.0049

0.0070

0.0065

0.0043

0.0068

0.0059

0.0057

0.0062

3.79

3.84

3.94

4.03

3.70

4.38

4.75

4.47

3.88

4.84

3.19

4.44

4.69

3,26

4.95

4.96

3.70

4.09

3.68

4.25

4.31

4. 16

4.56

4.11

4.71

6.09

6.34

4.87

4.60

7.17

5. 15

4.32

4.57

4.96

EK

0.13

0. 12

0.13

0.12

0.12

0. 11

0.11

0.11

0.12

0.10

0.13

0.11

0.11

0.12

0.10

0.09

0.11

0. 11

0.10

0.10

0.09

0.09

0.09

0.09

0.09

0.05

0.05

0.07

0.07

0.04

0.07

0.07

0.06

0.06

GL

3.84

3.90

4.21

4.06

6.07

3. 85

3.96

4.27

3. 34

3.68

3.20

3. 86

4.20

4.45

3.75

4.20

3.77

3. 84

4.72

3.97

4.05

3.91

3.99

3.74

4.09

3.57

4. 17

4.12

4.06

4.22

4.18

3.98

4.05

3.98

EKL

0. 12

0.11

0.11

0,12

0109

O.U

0.11

0.10

0.12

0.11

12

11

10

10

10

0.09

0.10

0.10

0.09

0.09

0.09

0.09

0.09

0.09

0.09

0.09

0.08

0.07

0.07

0.07

0.07

0.07

0.07

0.07

Table V. Sorted by PZD

91
VOTES WORD NOGC E

THE 442506 7.8 7 AND 128355 7.83

THAT 89026 7. 80

NOT 35835 7.75

FOR 45223 7.73

WHICH 25522 7. 70

WAS 56044 7.69

THIS 29490 7.66

WITH 21624 7.64

FROM 19879 7.62

HAVE 13825 7.53

BEEN 12072 7.50

SUCH 18195 7.50

BUT 9174 7.48

THERE 12925 7.48

ANY 13855 7.47

UPON 11816 7.46

ARE 13721 7.46

10 COURT 33021 7.45

3 CASE 15261 7.45

OTHER 8966 7.43

WERE 12911 7.43

HAD 15451 7.43

UNDER 10893 7.40

1 ONE 9388 7.39

MAY 9510 7.37

HAS 10530 7.36

1 ALL 9021 7.36

WOULD 9678 7.34

1 ONLY 6218 7. 33

HIS 19529 7.32

MADE 7999 7.32

ITS 11061 7.31

ALSO 5230 7.29

1 FOLLOW 6076 7.28

WHEN 6875 7.28

2 QUESTI 8776 7.25

DID 6224 7.24

AFTER 6340 7.24,

2 LAW 9658 7.23

WHETHE 5173 7.22

5 DEFEND 25773 7.20

SHOULD 5689 7.20

WHEf^E 5794 7.19

BEFORE 5814 7. 19

MUST 5208 7. 18

3 PRESEN 5653 7. 18

2 REASON 6845 7. 17

3 TIME 8254 7.17

COULD 5096 7.16

8 CONSID 5288 7. 15

THEY 7042 7. 14

THEN 458 3 7. 12

1 PART 4746 7. 12

I TWO 5130 7. 11

WHO 5241 7.11

FURTHE 4546 7. 11

THESE 4753 7. 11

THAN 4378 7.11

9 EV I DEN 12726 7. 10

Table VI.

7.65

7.61

7.6u

7.60

7.6 1 7.56

7.55

7.59

7.51

7.51

7.44

7.41

7.35

7.37

7.40

7.37

7.40

7.39

7.41

7.36

7.31

7.31

7.30

7.31

7.31

7.30

7.37

7.26

7.23

7.31

7.22

7.29

7.20

7.23

7.24

7.24

7.28

7.17

7.21

7.20

7.19

7.12

7.20

7.16

7.23

7.22

7.20

7.25

7.20

7.11

7. 14

7.08

7.07

7.09

7.1

7.03

7.13

7.07

7. 10

7.02

Sorted by

EL PZD

99 .99

99.73

98. 15

96.9 7 98.0 7 94.41

95.73

96.67

92.03

92. 18

85.99

83.76

85.80

78.89

84.25

83.12

82.93

84.37

93.58

84.74

76.17

79.91

82.44

80.44

76.40

76.70

81.76

74.78

73.12

7 2.14

78.63

74.51

75.34

67. 15

69. 38

69.87

77.08

66.70

68.47

74.29

66.13

71. 19

66.59

65.26

68.55

66. 70

68.25

72.48

70.40

61.79

63.72

64.4 7 59.19

60.62

60.51

59.64

61 .94

59.79

59.38

65.64

E

AVG

12. 1 192

3.4562

2.4343

0.9798

1.2529

0.6984

1.5630

0.8106

0. 584Q

0. 5456

0.3761

0.3306

0.4817

0.2485

0.3545

0.3703

0.3232

0.3766

0.9097

0.4182

0.2397

0.3486

0.4205

0.2937

0.2540

0.2605

0.2838

0.2361

0.2580

0. 1693

0.5396

0.2213

0.2888

0.1410

0. 1661

0. 1866

0.2395

0.1665

0. 1745

0.2554

0. 1408

0.7468

0.1511

0. 1562

0.1612

0.1412

0.1558

0. 1850

0.2237

0. 1383

0. 1379

0. 1897

0. 1242

0.1287

0. 1408

0.1416

0. 1230

0.1275

0.1198

0.3461

G

-0.19

0.53

0.70

0.55

1.03

0.64

0. 52

1.15

1.15

1.25

1.17

1.41

1.49

0.84

1.30

1.29

1.37

1.56

1.64

1.64

1.18

1.43

1.49

1.82

1.61

1.45

1.34

1.45

1.43

1.57

1.55

1.60

1.71

1.08

1.30

1.54

2.17

1.55

1.62

2.34

1.69

1.34

1.89

1.64

2.12

1.83

2.26

2.15

2.55

1.59

2.06

2.45

2.04

2.57

1.59

1.89

1.92

1.97

2.23

1.64

EK

41.17

15.25

9.48

6.95

5.00

4. 89

3.68

4.02

3.46

3.01

2.52

1.96

1.78

2.21

1.87

1.87

1.76

1.85

1.26

1.43

1.79

1.55

1.38

1.31

48

38

51

46

34

38

1.03

1.25

1.13

1.33

1.18

1.20

1.03

1.03

1.06

0.88

1.04

0.79

1.02

1.03

0.95

1.08

0.88

1.11

0.92

0.95

0.93

0.77

0.82

0.78

0.85

0.79

0.91

0.83

0.81

0.71

GL

1.87

2.14

1.92

1.90

1.87

1.79

1.78

2.45

2.15

1.83

2.53

EKL

1.93

1.57

07

91

2.06

2.17

2.37

1.83

2.55

3.97

2.38

2.45

2.67

2.68

2. 98

2.40

2. 50

2.41

3.34

2.49

1.88

2.83

1.97

3.49

1.95

2.44

2.24

4.30

2. 52

2.27

3.39

2.57

2.43

2.45

2.43

2.63

2.79

3.49

2. 86

2.17

2.58

2.68

3.52

2.60

2.85

2.47

3.51

3.44

3.27

2.63

3.09

54

56

59

38

33

41

16

1.19

0.97

0.95

0.74

0.89

0.91

0.83

0.95

0.86

0.76

0. BJ

0.76

0.70

69

69

75

72

83

64

0.64

0.82

0.60

0.76

0.54

0.71

0.69

0.69

0.62

0.59

0.65

54

61

53

63

58

66

64

58

0.64

0.62

0.54

0.56

0.45

0.51

0.52

0.55

0.44

0.53

0.48

0.54

0.43

92
VOTES WORD

WITHOU

HOWEVE

DOES

THEIR

AAAAAA

SAID

FACT

REQUIR

JUDCME

SAME

HELD

BEING

AGAINS

PROVID

PLAINT

OPINIO

CONTEN

DETERM

THEREF

FIRST

PERSON

BECAUS

OUT

FACTS

STATED

SOME

TRIAL

CUNQ.U

APPEAR

DIRECT

MORE

ACTION

CAN

HERE

INTO

SEE

NECESS

THEM

HIM

FOUND

EFFECT

MATTER

RECORD

SINCE

AFFIRM

STATUT

ACCORD

CERT AI

GENERA

CASES

BOTH

WITHIN

STATE

RESULT

BETWEE

WILL

SECTIO

WHILE

SHALL

END

NOGC

4652

3333

4264

6514

2649

10747

4658

6103

10581

4992

3978

3858

5725

5792

20986

4764

3888

5030

3871

4165

6980

3553

4389

4095

3698

3394

9898

3665

3855

5706

3050

8248

2822

3448

3583

4704

3477

3505

5613

3608

3759

4313

6093

2756

3897

7283

2721

3069

5262

3896

2868

4561

9231

3328

3231

7140

10226

2749

6240

6422

Table VI.

E

7. 10

7.09

7.09

7.08

7.07

7.07

7.06

7.06

7.06

7.05

7.04

7.04

7.04

7.03

7.02

7.02

7.02

7.02

7.01

7.01

7.01

7.00

7.00

7.00

6.99

6.97

6.97

6.9!>

6.95

6.95

6.94

6.94

6.93

6.93

6.93

6.93

6.93

6.92

6.91

6.91

6.91

6.91

6.91

6.89

6.89

6.89

6.87

6.87

6.87

6.86

6.85

6.85

6.85

6.85

6.84

6.84

6.83

6.82

6.81

6.81

Sorted

EL

7.17

7.1 1 7.20

7.02

7.87

6.93

7.10

7. 10

7.17

7.07

7.02

7.08

7.06

7.02

6.94

6.98

7.09

7.01

7.18

7.04

6.94

7.11

6.99

7.01

6.99

6.93

6.98

7.02

7.00

6.92

6.95

6.92

6.94

6.97

6.92

6.88

6.93

6.89

6.85

6.98

6.92

6.96

6.98

6.93

7.23

6.80

6.96

6.96

6.82

6.90

6.88

6.97

6.80

6.86

6.87

6.74

6.76

6.85

6.73

6.71

by E

PZD

63.57

5 5.90

63. 30

61 .75

99.99

69. 15

60.28

63.98

73.19

60.73

55.34

57.41

61.83

60.02

57.71

58.85

57.11

59.45

62.21

57.15

60.81

57.19

57.04

55.79

54.77

50.88

62.85

!>).90

5 f .68

58.62

49.49

64.55

49.15

52.69

51.00

55.00

52.20

49.37

54.24

53.68

52.39

55. 19

60.51

48.65

63.53

53.15

49.64

50.62

52.92

51.41

46.54

55.56

62.06

48.50

47.45

62.55

55.75

46.31

49.18

51.86

AVG

0. 1274

0.0923

0. 1175

0. 1756

0. 0783

0.2803

0. 1249

0. 1665

0.311?

0. 1299

0. 1058

0. 1040

0. 1605

0. 1599

0.6097

0. 1218

0. 1094

0. 1314

0. 1050

0. 1116

0.1897

0.0999

0.1164

0. 1137

0.0975

0.0897

0.2884

0. 1010

0. 1045

0. 1575

0.0822

0.2329

0.0739

0.0938

0.0952

0.1297

0.0937

0.0943

0. 1531

0.1017

0.1018

0.1166

0. 1675

0.0753

0.1109

0.1985

0.0745

0.0830

0. 1338

0. 1062

0.0771

0.1294

0.2417

0.0911

0.0879

0. 1944

0.2858

0.0751

0.1705

0.1570

G

2.02

1.47

1.80

2.19

0.42

4.45

2.10

2.34

3.01

2.47

1.92

2.13

2.56

2.56

1.25

2.05

2.14

3.04

1.43

2.30

2.61

2.04

3.00

3.05

2.37

1.97

2.75

2.50

3.97

5.12

1.98

3.64

1.61

1.92

2.51

2.95

3.31

2.56

2.49

2.73

2.86

3.11

5.25

1.76

2.26

2.26

2.12

2.20

3.11

2.58

1.87

2.63

3.06

3.50

2.33

5.49

2.91

5.29

2.77

3.07

EK

0.91

0.90

0.96

0.70

4.32

0.50

0.80

0.74

0.54

0.76

0.75

0.75

0.63

0.64

0.64

0.71

0.71

0.64

0.90

0.71

0.57

0.75

0.65

0.60

0.68

0.67

0.45

0.64

0.56

0.44

0.66

0.39

0.67

0.66

0.57

0.47

0.52

0.56

0.52

0. 53

0.56

0.53

0.41

0.62

0.78

0.48

0.62

0.65

0.47

0.54

0.59

0.50

0.39

0.49

0.55

0.26

0.38

0.43

0.43

0.44

GL

2. 39

1.76

2.11

3.29

2.55

6.83

2.40

4.53

4.08

3.32

2.83

2.89

3.13

3.62

2.24

4.63

2.24

3.95

2.25

3.27

5.09

2.28

6.13

2.90

3.69

4.84

2.96

2.52

9.43

6.63

2.76

4.77

2.68

3. 12

3.14

3.89

4.91

4.37

6.64

3. 16

7.29

4.12

4.95

2.78

2.61

4.39

2.92

3.90

5.01

3.22

2.81

3.59

4.64

3.97

2.83

12.86

4.29

4.31

4.34

6.84

EKL

0.62

0.62

0.67

0.42

31.32

0.27

0.54

0.47

0.49

0.48

0.47

0.52

0.46

0.42

0.43

0.37

0.56

0.40

0.65

0.46

0.33

0.58

0.37

0.46

0.42

0.39

0.41

0.49

0.32

0.2S

0.45

0.31

0.44

0.43

0.39

0.33

0.35

0.36

0.29

0.43

0.34

0.38

0.35

0.43

0.70

0.29

0.45

0.42

0.28

0.38

0.39

0.41

0.25

0.34

0.38

0.15

0.27

0.35

0.27

0.22

93
VOTES WORD NOCC E

RESPEC 2 5 79 6.80

OUR 3179 6.80

GIVEN 2766 6.80

2 PROV I4479 6.80

9 APPEAL 9096 6.80

1 PROCEE 502 1 6.79

ENT ERE 2920 6.78

3 ORDER 6773 6 7ft 6 AUTHOR A - 7 ft WELL 2259 6.77

5 CAUSE 4463 6.77

MAKE 2535 6.76

WHAT 288 3 6.76

6 RIGHT 5447 6. 76

2 PURPOS 4138 6. 76

C I RCUM 2543 6.75

C ANNOT 2467 6.74

1 APP 4769 6. 74

ESTABL 2947 6.74

THOS E 2527 6.73

SUFF I

2484 6.72

OVER 2622 6. 72

2 ALLEGE 3766 6.72

E I THER 2033 6.71

SET 2964 6.71

a MOT I ON 6621 6.71

2 I NCLUD 2632 6.71

NOR 2099 6.70

5 SUB JEC 2855 6. 70

THREE 2437 6.70

THEREO 2640 6.69

4 PRIOR 2379 6.69

4 GROUND 2629 6.68

1 NEW 4744 6.68

1 EACH 3332 6.68

ALTHOU 1762 6.67

H AV I NG 2006 6.67

I TAL 1 1 360 6.67

FOL 5682 6.67

TAKEN 2518 6.67

FILED 5362 6.67

2 RE VERS 2857 6.66

UNT I L 2347 6.65

CONCUR 2290 6.65

ABOUT 3228 6.65

ACT 5147 6.65

3 SUSTAI 2600 6.65

1 SEC 6808 6.6 5 7 SPEC I

2900 6.65

3 SUPPOR 3151 6.65 even 1964 6.64

3 I NDI

a 190 1 6.64

3 SUBST

2527 6.62

1)0 rnuNTY 6245 6.62

5 ISSUE 3113 6.61

NOW 2384 6.60

THUS 1622 6.58

DURING 2216 6.58

CONST R 380 5 6.58

3 APPLIC 4168 6.58

EL PZD

6

66

6

7

666

6

6

66

6

6

6

6

G6

66

6

6

66

66

6

6

6

6

6

66

6

66

6

6

66

6

6

6

76

66

6

66

66

66

6

6

6

66

6 Table VI. Sorted

.82 44

.83 47

.82 4 5 .7/47

.06 77

.84 55

.87 48

. 77 58

.81 52

.83 43

.90 54

.84 43

.79 44

.86 54

.76 49

.75 41

.92 46

.72 44

.72 44

.77 42

.81 42

.71 40

.81 47

.78 40

.84 46

.84 53

.76 43

.86 43

.81 45

.7 3 41

.75 41

.74 40

.77 44

.72 48

.69 43

.77 38

.86 42

.57 45

.57 45,

.76 43.

.91 55,

.93 46.

.70 39,

.30 63,

.65 41.

.59 45.

.89 46.

.62 49,

.68 42.

.67 46.

.75 38.

.70 37.

.71 41.

.52 52.

.66 42.

.80 43.

.65 34.

.62 36.

.55 40.

.60 47.

by E

.43

.93

.07

. 18

.61

.19

.58

.32

.32

.14

.28

.94

. 80

.24

. 30

.94

.54

.92

.46

.43

.92

.99

.86

.20

. 54

.90

.41

. 14

.48

.18

.60

.88

.16

.09

.90

.65

.09

.18

.18

,07

,26

,96

,22

,91

, 10

, 56

,24

t>0 28

35

80

67

60

43

8 3 29

80

50

50

37

AVG

0.0678

0.0833

0.0744

0.1251

0.2637

0. 1373

0.0873

0.1918

0.1319

0.0592

0. 1255

0.0681

0.0725

0. 1464

0. 1096

0.0679

0.0694

0. 1292

0.0788

0.0642

0.0708

0.0701

0. 1091

0.0532

0.0798

0. 1942

0.0716

0.0581

0.0784

0.0677

0.0697

0.0654

0.0728

0. 1295

0.0859

0.0487

0.0548

0.2755

0. 1378

0.0697

0. 1589

0.0842

0.0628

0.0643

0.0882

0. 1370

0.0753

0. 1929

0.0790

0.0855

0.0509

0.0499

0.0693

0. 1787

0.0831

0.0629

0.0427

0.0609

0. 1054

0. 1134

G

1.99

2.15

2.27

2.55

4.94

3. 56

3.29

3.68

4.35

2.87

2.98

2.35

2. t>2 2.91

3. 99

2.08

2.06

2.51

3.00

3.12

2.35

2.40

3.04

1.96

3.36

3. 78

3.86

1.94

2.72

3.19

2.61

2.87

3.25

3. 77

4.53

1.78

2.18

3. 12

3.12

3.27

09

65

31

45

68

3.30

3.40

3.75

3.75

7.06

2.09

2.45

3.48

5. CO

3.76

2. 79

2.08

2.73

3.38

4.97

EK

0. 54

0.55

0.50

0.45

0.30

0.40

0.42

0.31

0.37

0.51

0.43

0.54

0.51

0.47

0.41

0.49

0.57

0.41

0.45

0.46

0.45

0.43

0.40

0.50

0.45

0.30

0. 39

0.53

0.46

0.40

0.42

0.41

0.38

0.31

0.36

0.50

0.51

0.37

0.37

0.37

0.33

0.48

0.42

0.73

0.39

0.32

0.40

0.27

0.34

0.24

0.49

0.42

0.36

0.23

0. 32

0.46

0.42

0.36

0.30

0.25

GL

3.71

4. 84

3.10

3.69

5. 35

6. 15

4.02

11.48

4.61

3.49

4.08

3.17

3. 76

3. 87

6.33

2.94

2.46

3.31

17.95

3.52

3.24

3. 50

3.37

3.10

3.72

3.36

3.68

2.78

3. 64

3. 87

3. 06

3.12

5. 73

4. 33

5. 12

2.66

2.07

7. 32

7.39

4. 04

3.46

3.60

3.46

2.51

3.45

6. 21

2.63

4.50

5.03

9. 79

3.06

3.59

4. 62

8. 51

4.98

3.10

2.88

4.42

4.65

8. 13

EKL

0.34

0.31

0.35

0.30

0.33

0.26

0.34

0.19

0.28

0.36

0.34

0,37

0.32

0.32

0.25

0.33

0.45

0.29

0.18

0.33

0.36

0.29

0.33

0.35

0.35

0.33

0.31

0.40

0.33

0.30

0.33

0.32

0.29

0.26

0.25

0.37

0.43

0.19

0.19

0.31

0.36

0.43

0.3C

0. 8fc 0.2 7 0.20

0.41

0.21

0.25

0.13

0.35

0.31

0.27

0.14

0.23

0.34

0.31

0.26

0.21

0.16

94
VOTES WORD NOCG

6 EXCEPT 3589

M I GHT 1734

ANOTHE 1881

1 CONCER 1797

I NVOLV 2933

6 ERROR 3841

7 CONTRA 80 3 3 3 FIND IN 3437

6 RULE 4090

1 CONTAI 2096

5 PARTI3496

UNLESS 1520

2 I NSTAN 1867

2 RELAT2530

1 ENTITL 2141

1 OWN 1857

3 APPELL 14543

2 YEARS 260 1 SECOND 2415

4 CLEAR 1537

3 OPERAT 4207

THROUG 1954

2 DECIS

3988

RECEIV 2801

5 JUDGE 4000

3 FIND 1954

7 EXPRES 2022

BROUGH 1534

4 ILL 8605

2 OHIO 8519

4 AMOUNT 3110

2 PARTIC 2381

8 CHARGE 4622

2 CONTRO 2941

PAGE 3218

7 CONDIT 2779

DIFFER 1714

3 COMMON 4042

11 PRINCI 2158

1 USED 2650

THOUGH 1301

LATER 1426

1 APPARE 1334

2 SITUAT 1358

7 TESTIM 3650

5 ANSWER 3398

DECIDE 1409

CITED 1401

10 JURY 5530

6 CONSTI 4132

5 DAY 2189

5 BAS IS 1500

THEREA 1342

ABOVE 1812

3 PROPER 5913

3 DUE 1937

3 COMPLA 3971

CALLED 1618

REGARD 1466

2 ADDITI 1708

Table VI.

E EL

6.53 6.82

6.57 6.63

6.57 6.65

6.57 6.59

6.56 6.90

6.56 6.66

6.56 6.49

6.56 6.59

6.56 6.70

6.55 6.65

6.55 6.59

6.54 6.63

6. 54 6.60

6. 54 6.53

6.53 6.69

6.53 6.60

6.53 6.44

6.53 6.56

6.53 6.61

6.52 6.57

6. 52 6.45

6.52 6.56

6.52 6.69

6.52 6.57

6.52 6.64

6.51 6.66

6.51 6.61

6.50 6.59

6.49 6.46

6.49 6.35

6.49 6.52

6.48 6.76

6. 48 6.47

6.48 6.55

6.47 6.45

6. 46 6.4 7 6. 46 6.55

6.46 6.48

6.46 6.43

6. 45 6.58

6. 43 6.54

6. 43 6.47

6.43 6. 53

6. 42 6.49

6. 42 6.41

6.42 6.41

6.41 6.50

6.41 6.54

6.41 6.31

6.41 6.49

6.41 6.46

6.41 6.47

6. 40 6.55

6.40 6.63

6.40 6.34

6. 40 6.47

6. 40 6.45

6. 40 6.57

6. 39 6.52

6. 39 6.49

Sorted by E

PZD AVG

49.79 0.1046

34.27 0.0465

36.35 0.0500

34.76 0.0468

47.86 0.0789

44.80 0.1051

52.96 0.2158

41.56 0.0995

47.18 0.105$

38.12 0.0578

41.71 0.0960

33.82 0.0418

34.88 0.0494

37.10 0.0662

38.42 0.0591

34.99 0.0502

50.16 0.3877

37.10 0.0687

38.50 0.0656

33.48 0.0425

39.56 0^1145

34.61 0.0531

46.58 0.J070

39.10 0.0764

46.84 0.1181

37.75 0.0519

36.01 0.0546

33.74 0.0460

32.88 0.2551

34.39 0.2212

37.56 0.0869

42.12 0.0625

40.69 0.1234

39.93 0.0849

33.71 0.0815

35.52 0.0760

33.14 0.0466

42.58 0.1171

34.61 0.0564

38.16 0.0734

30.46 0.0340

29.48 0.0387

30.84 0.0364

29.40 0.0368

34.65 0.1010

39.33 0.0913

29.89 0.0381

30.95 0.0390

34.27 0.1470

42.99 0.1058

34.16 0.0607

30.76 0.0412

31.03 0.0389

35.18 0.0483

36.91 0.1591

32.08 0.0542

37.44 0.1136

32.76 0.0444

30.80 0.0380

32.12 0.0453

G EK

5.95 0.26

2.40 0. 39

2.97 0. 37

4.40 0. 34

2.29 0.56

3.69 0.29

3.9 8 0.23

4.00 0.26

4.2 3 0.31

3.35 0.35

3. 86 0.29

2.32 0.39

2.58 0.36

3.61 0. 30

2.60 0. 38

2.91 0.35

3.05 0.23

3.24 0.31

3. 97 0.31

3.35 0.33

3.54 0.27

3.87 0. 30

4.00 0. 30

6.76 0.27

10. 30 0.19

3.11 0. 35

3.21 0. 34

4.00 0.29

1.95 0.34

2.35 0.28

3.85 0.27

3.17 0.41

3.96 0.24

5.05 0.23

2.83 0.31

3.52 0.26

3.96 0.29

5. 85 0. 19

6.01 0.24

5.62 0.24

2.57 0. 34

2.75 0.31

3.26 0.30

2.40 0.33

3. 30 0.25

5.64 0.22

2.48 0.31

2.52 0.33

3. 35 0.24

3.48 0.28

3. 92 0.26

5.82 0.26

2.78 0.32

2. 94 0. 35

3.62 0.23

4.13 0.25

4.27 0.22

4.43 0. 31

3.05 0. 32

5.06 0.25

GL EKL

4. 72 0.30

2. 78 0. 30

3. 17 0.29

3.67 0.26

2. 99 0.40

4. 33 0.24

7. 29 0.15

3.90 0.23

12*48 0.20

5.43 0.25

4.47 0.22

2. 95 0.30

3.01 0.28

5. 77 0.20

3. 68 0. 30

3. 93 0.27

5. 26 0.16

4. 19 0.23

5. 63 0.23

5.39 0.24

4. 52 0. 18

4.00 0.24

5.57 0.23

5.74 0.21

6. 80 0.20

3. 70 0.28

4.18 0.26

3. 64 0.27

3. 00 0.24

5.51 0.17

3.75 0.22

3. 48 0. 32

4. 95 0 . 18

5. 00 0.20

5. 57 0.19

3.88 0.21

3. 56 0.25

7.01 0.16

7. 85 0.16

4.18 0.23

2. 82 0.2-8

3. 52 0.24

3. 32 0.26

3.07 0.25

3.88 0.20

9.44 0. 13

3. 99 0.25

3.08 0.27

4. 31 0.17

7. 53 0.15

9. 83 0.17

5.60 0.21

2. 92 0.28

3.03 0.29

5.71 0.15

3.79 0.22

4.90 0. 19

3.42 0.27

3.05 0.26

4.68 0.22

95
1NUOO h, 2 BASED 1605 6.38

2 S IMILA 1243 6.38

TAKE 1484 6. 38

STATES 2343 6. 38

4 CONT I

2382 6.37

SHOW 1649 6.36

8 I NTERE 3637 6.36

3 PLACE 188 1 6. 36

1 TEST I

3484 6.35

V IEW 1406 6.35

1 POINT 1487 6. 35

6 REMAI

1592 6.35

5 PERMI

2869 6.35

2 TERMS 1583 6.33

9 PUBLI

4658 6.33

2 GIVE 1490 6. 32

5 SEVERA 1243 6. 32

ADMITT 1667 6.32

1 STATEM 2732 6. 32

CLEARL 1145 6.31

2 DATE 1983 6.31

1 DENIED 2053 6. 30

MANNER 1259 6. 30

1 RENDER 1657 6. 30

HER 7548 6. 30

4 COMPLE 1709 6. 30

1 ENTIRE 1350 6. 30

6 RIGHTS 2108 6.30

1 I NTEND 1333 6.29

FAILED 1442 6.29

SUPRA 2573 6. 29

3 USE 3852 6.29

8 HEARIN 2525 6.28

6 COURTS 2033 6.28

MANY 1117 6.27

5 OBJECT 2703 6.27

SAY 1088 6.26

9 PARTY 2643 6.26

7 OFFICE 4060 6.26

3 ARGUME 1528 6.26

I TSELF 993 6.25

MOST 1051 6.25

APPLIE 1264 6. 25

1 PAID 2316 6.25

2 SUBSEQ 1263 6.25

FORTH 1458 6.25

6 DUTY 1873 6.25

3 GRANTE 1574 6.25

1 LEGAL 1650 6.25

NOTHIN 1275 6.24

5 CITY 5969 6.24

9 CLAIM 2565 6.24

3 REFERR 1309 6.24

2 RETURN 2074 6.24

HEREIN 2599 6.23

I TRUE 1140 6.23

LONG 1047 6.23

5 ORIGI

2053 6.23

OVERRU 1644 6.23

DISCUS 1034 6.22

Table VI,

6

6

6

6

6

6

6

6

6

6

66

6

6

6

66

6

66

6

66

6

66

6

6

6

66

6

6

6

66

6

6,

6<

6.

6 , 6.

6o

6.

6.

6.

6.

6.

6.

6.

6.

6.

6 c 6.

6.

6.

6.

6.

6o

6.

Sorted

EL PZD

.56 32.84

.46 2 6.61

.4 7 30. 35" .33 33.37

.40 34.35

.59 33.89

.32 35.33

.45 32.27

.35 31.74

.48 30.95

.42 29.48

.38 30.46

.49 39.63

.39 28.46

.30 35.78

.45 29.78

.36 27.25

.32 28.87

.36 34.16

.45 27.67

.41 31.37

.77 40.39

.37 27.29

.45 31.74

.20 31.89

.45 31.40

.41 28.53

.33 30.38

.39 27.63

.48 30.31

.25 29.21

.27 36.12

.31 31.59

.36 31.21

.38 25.82

.31 32.50

.34 25.44

.33 31.93

.12 33.93

.3 7 28.69

.33 24.38

.31 24.95

.40 27.63

.25 28.16

.37 26.99

.40 28.80

.30 28.35

.34 28.35

.30 28.57

.55 30.65

.23 38.05

.24 32.27

.43 28.65

.32 31.48

.70 41.75

.36 26.23

.32 24.80

39 32.01

42 30.46

31 24.34

by E

AVG

0.0431

0.0339

0.0407

0.0582

0.0634

0.0470

0.0944

0.0528

0.0969

0.0375

0.0407

0.0428

0.0820

0.0424

0.1226

0.0399

0.0331

0.0436

0.0720

0.0304

0.0555

0.0580

0.0329

0.0464

0.2095

0.0455

0.0369

0.0581

0.0361

0.0414

0.0636

0.1059

0.0716

0.0553

0.0286

0.0742

0.0294

0.0726

0. 1032

0.0429

0.0260

0.0273

0.0351

0.0616

0.0363

0.0391

0.0506

0.0425

0.0423

0.0345

0. 1706

0.0735

0.0341

0.0589

0.0670

0.0309

0.0280

0.0558

0.0456

0.0267

G

2.60

2.91

3.85

6.26

5.8 5 3.26

5.26

6.46

3. 53

4.33

4.43

4.99

6.17

3.43

4.86

3.06

3.47

3.82

4.77

2.81

3.97

2.91

3.46

3.94

4.05

4.76

5.20

5.59

3. 14

3.32

3.34

4.86

4.03

9.19

2.52

8.66

2.94

4.28

4.82

5.01

2.40

2.65

2.95

3.21

3.67

3.68

3.82

4.97

7.41

2.76

3.90

5.91

8.37

8.81

3.17

3.33

3.39

4.38

4.78

2.85

EK

0.35

0. 30

0.27

0.22

0.21

0.32

0.20

0.21

0.24

0.29

0.25

0.23

0.17

0.25

0.20

0.29

0.26

0.23

0.20

0.30

0.23

0.37

0.27

0.23

0.20

0.24

0.25

0.20

0.25

C.29

0.23

0.18

0.21

0. 16

0.29

0.15

26

20

0.17

0.20

0.27

0.28

0.27

0.23

0.24

0.25

0.21

0.20

0. 19

0.33

0.18

0.15

0.24

0. 15

0.36

0.26

0.23

0.21

0. 19

0.25

GL

3. 70

3.18

3.52

8. 54

10. 10

3.21

5.71

5.21

3. 72

7.01

4.24

7. 12

6. 36

3.35

5.07

3.67

7. 53

5. 59

5.32

3.28

4. 85

2. 72

6.32

6. 39

4. 75

5.48

6.76

4. 76

4.27

3. 79

4.77

7. 72

6. 14

5.77

2.73

5.60

3.71

5.91

13.75

4.22

3. 32

6.00

3.46

4.69

3.97

4. 54

5.09

5.70

9.77

2. 84

5.82

7.77

5.55

9.23

5.86

4.42

3.84

5.63

4.35

3. 19

EKL

0.26

0.24

0.23

0.13

0.14

0.28

0.15

0.19

0.19

0.20

0.21

0.16

0.17

0.21

0.15

0.23

0.18

0.17

0.16

0.24

0.19

0.35

0.19

0.19

0.14

0.20

0.20

0.17

0.21

0.23

0.15

0.12

0.15

0. 17

0.23

0.15

0.21

0. 16

0.07

0.19

0.22

0.18

0.22

0.16

0.21

0.20

0.17

0.17

0.14

0.29

0.13

0.12

0.21

0. 14

0.25

0.20

0.20

0.18

0.20

0.21

96
VOTES WORD NOCC E

BEL I EV 1176 6.22

1 FAVCR 1249 6.22

2 LANGUA 1492 6.22

9 COUNSE 3030 6.22

2 COURSE 1500 6.22

MERELY 936 6.21

4 CODE 4152 6.21

WAY 1771 6.21

CONSIS 941 6. 19

5 PETITI 7623 6.19

MAKING 1060 6.19

5 COMPAN 4677 6. 19

5 EXAM IN 3117 6. 19

1 THINK 1035 6. 18

POSSIB 1018 6.18

OBTAIN 1498 6.18

NE ITHE 930 6. 16

SHOWS 1078 6. 16

1 SUPREM 1904 6.16

1 NATURE 1185 6. 16

5 FAILUR 1630 6. 16

PREVIO 1040 6. 16

HOLD 1033 6.15

VERY 888 6.15

RATHER 917 6.15

TOOK 1080 6. 15

SHOWN 1106 6.15

3 DISTIN 997 6. 14

ORDERE 1180 6.14

OTHERW 1095 6.14

2 REFUSE 1286 6.14

3 CORREC 1358 6.14

THERE I 1068 6. 13

1 KNOWN 1083 6. 12

1 EVERY 922 6.11

SOUGHT 1132 6.11

FAR 923 6.11

5 REOUES 1941 6.11

5 RECOGN 1033 6. 10

DONE 1079 6.09

4 PURSUA 1039 6.08

LESS 923 6.08

1 REV 1484 6.07

BECOME 1158 6.07

2 EXISTE 1029 6.06

THERET 1022 6.05

1 HOLDIN 1008 6.05

4 OCCURR 1248 6.05

9 ATTEMP 1404 6.05

1 DAYS 1500 6.05

TOGETH 861 6.04

LATTER 833 6.04

13 NOTICE 2855 6.04

8 SERVIC 3855 6.04

7 REVI EW 2347 6.02

NEVER 976 6.01

LEAST 766 6.00

APPLY 806 6.00

WHOM 832 6.00

RAISED 1050 6.00

Table VI.

EL

6.34

6.3 T 6.23

6.2 / 6.45

6.32

6.18

6.45

6.31

6.44

6.33

6.05

6.23

6.28

6.23

6.30

6.38

6.35

6.24

6.31

6.43

6.31

6.35

6.22

6.21

6.28

6.36

6.22

6.33

6.42

6.22

6.38

6.38

6.17

6.22

6.33

6.24

6.29

6.25

6.28

6.24

6.17

6.08

6.30

6.17

6.35

6.20

6.11

6.42

6.22

6.16

6.14

6.18

6.05

6.30

6.15

6.11

6.08

6.13

6.28

Sorted by

PZD

25.67

26 . 8 7 25.78

32. 54

30. 53

23.78

29.55

32.91

23.66

40.39

25.14

32.65

35.56

23.63

22.98

27.40

24.87

25.25

27.44

25.48

30. 16

24.57

24.61

21.93

22.00

24.46

25.74

22.68

26.23

27.18

24.49

28 .57

25.70

22.19

22.31

25.44

22.61

29.44

23.51

24.57

23. 17

21.63

22.72

25.36

22.08

24.95

22.76

21.78

29. 18

24.99

20.91

20.23

30.76

29.63

32.72

21.32

19.40

19.63

20.08

23.93

E

AVG

0.0322

0.0364

0.0411

0.0868

0.0421

0.0248

0. 1146

0.0472

0.0260

0.2198

0.0282

0.1180

0.0831

0.0298

0.0272

0.0397

0.0252

0.0297

0.0474

0.0313

0.0459

0.0277

0.0270

0.0230

0.0246

0.0302

0.0303

0.0265

0.0324

0.0307

0.0351

0.0370

0.0279

0.0285

0.0244

0.0316

0.0247

0.0545

0.0261

0.0282

0.0271

0.0250

0.0446

0.0320

0.0286

0.0278

0.0265

0.0347

0.0376

0.0447

0.0222

0.0235

0.0853

0.1114

0.0676

0.0254

0.0206

0.0212

0.0228

0.0290

G

3.33

3.45

3.66

6.05

6. 86

2.46

4.17

6.65

2.47

3.73

4.11

4.27

7.01

3.00

3.04

3.28

2.65

3.55

3.73

3.80

3.81

3.93

2.49

2.80

3.00

3.21

3.38

2.77

3.50

4.16

4.26

4.35

2.72

3.59

3.05

3.80

4.89

7.47

3.33

3.94

2.92

3.44

3.55

3.89

5.05

3.03

3.62

3.73

4.42

6.03

3.31

3.47

5.70

5.82

5.34

4.03

2.98

3.14

3.43

3.56

EK

0.24

0.23

0.21

0.15

0.21

0.26

0.17

0.22

0.26

0.19

0.22

0.17

0.15

0.23

0.23

0.23

0.27

0.23

0.21

0.22

0.24

0.22

0.26

0.24

0.24

0.24

0.24

0.24

0.23

0.25

0.19

0.21

0.27

0.21

0.22

0.21

0.20

0.15

0.23

0.21

0.22

0.21

0.18

0.23

0.19

0.25

0.21

0. 18

0.25

0.14

0.21

0.19

0.14

0.13

0.15

0.19

0.20

0.19

0.19

0.21

GL

3.34

4.09

5. 17

5.28

4.36

2.82

5.98

10.08

3.02

5.82

3.75

10.01

8.63

3.20

3.70

5.62

2.44

3.06

6.65

4. 10

4.43

3.68

3.24

3.45

3.67

4.38

3.23

4. 15

6.13

3.79

4.13

4.34

3.38

4.34

3.79

4.23

4.79

5.99

3.94

4.53

3.93

3.99

9.27

3.96

4.18

3.31

4.43

4.81

7.93

3.91

3.86

3.63

6.77

7.29

7.80

4. 18

3.43

4.78

3.68

3.95

EKL

0.21

0.21

0.16

0.14

0.21

0.22

0.13

0.16

0,21

0.18

0.21

0.09

0.11

0.20

0.18

0.17

0.25

0.22

0.14

0.19

0.21

0.20

0.22

0.19

0.18

0.19

0.22

0.18

0.18

0.23

0.17

0.20

0.23

0.16

0.18

0.20

0.18

0.15

0.18

0.18

0.18

0.17

0.12

0.19

0.16

0.22

0.17

0.15

0.19

0.17

0.17

0.17

0.12

0.10

0.13

0.16

0.17

0.15

0.17

0.19

97
5 PREVEN 956

1 3 JUR I S D 30 5 6 AGAIN 766

8 ASSIGN 2654

THEREB 7 L 2 MUCH 69 3

3 VARIOU 815

5 EMPLOY 6062

HEARD 90 3 CLAIME 921

EMPHAS 1012

3 D I SMI

275 5 2 T I MES 751

OCCAS I 742

HI MS EL 864

SUGGES 782

8 RESPON 2872

HOW 739

LIKE 738

RELATE 839

6 AGREE 707

MENT I 0 694

COME 663

1 STAT 1245

7 JUST I F 88 5 WHOS E 65 5 READS 769

PUT 719

NOTED 710

PLACED 78 1 SEEMS 647

BEYOND 754

OBV I OU 64 5

1 STILL 660

AMONG 579

7 VAL I D 768

2 ESSENT 651

MERE 654

BECAME 734

L APPROX 70 4 SHOWI

829

2 MASS 468 7

2 WHOLE 651

FULLY 591

MAKES 56 5 FOREGO 626

ALONE 5 36

DI FF I C 578

DOING 62 5 ALRE AD 542

REACHE 5 39

ADDED 587

RELIED 487

MOVED 492

2 QUOTED 59 1 WHERE I 560

1 CONCED 48 5 NONE 506

1 OPPORT 545

L I KEW I 404

Table VI.

Ill 6.00 6.16

z A A 6 . 0 J 6.10

6.00 6 . 1 i £ a A 6. OU 6.12

5 . 99 6.11

5.99 6.11

5.99 6.12

5.98 5.89

5.97 6*0 7 5.97 6.17

a a ( 5.96 6.00

5.96 6.48

5.95 6.09

c At l: 5.95 6.0 3

C A £~ 5.95 t. 1 A 6.10

5.94 6.06

5.94 6.00

5.93 6.01

5.9 3 6.08

5.92 6.12

5.91 6.10

c a 1 5.91 6.02

D. 90 £. A A 6.00

5. 90 5.93

5.90 6.07

5.89 6.04

5.89 6.03

5.8 8

C A / b . 96

5.88 6.02

5.88 6.0b

5.88 5.98

5.87 5.99

5.87 6.09

5.86 6.07

5.83 5.93

b . 8 3 b . 92

5.83 5.98

b . 82 5.99

5.81 6.08

5.79 5.87

5.78 6.16

5 • 7 ( b . 73

5 . 74 5.78

5 . 74 5.93

5.73 5.98

5.73 5.96

5.73 5.87

5.72 5.87

5.71 5.89

5.68 5.80

5.63 5.86

5.62 5.77

5.62 5.80

5.61 5.75

5.60 5.85

c t a 5.60 5.92

5.58 5.83

5.58 5.82

5.5 3 5.7b

5.52 5.64

uy n

flvu

2 1 . 44 0 .0265

2 9 . 6 7 .0812

1 9 . 32 .0209

29 . 32 0 .0715

1 9 .02 0 .0192

19 . L 3 0 .0187

19 .96 0 .0214

32 .5*0 0 .1653

19 .93 0 .0241

21 . 4-4 0 .0261

19 .59 0 . 0246

35 .90 0 .0790

19 . 2 1 0 .0201

1 8 .38 0 .0206

19 .85 0 .0241

18 .68 0 .0208

29 .21 0 .0/72

1 7 . 89 0 .0191

18 .87 0 .0198

20 .04 0 .0233

18 .98 0 .0187

17 . 89 0 .0191

17 . 40 0 .01/3

19 .10 0 .0383

19 .85 0 .02 35

17 . 70 0 .0179

1 8 . 30 0 .0220

17 . 40 0 .0197

18 .04 0 .0182

1 3 . 9 1 0 .0208

16 .87 0 .0179

17 .74 0 .0 209

18 .23 0 .0187

1.08 0 .0176

1 5 .81 0 .0152

17 .06 0 .0207

16 . 76 0 .0173

1 7 .02 0

A 1 "7 A .0170

18 . 6 1 0 .0196

1 5 . 7 7 0 .0179

20 .53 J .0227

16 .93 0 .1483

14 .87 0 .0169

16 .00 0 .0159

16 .27 0 .0151

16 .64 0 .0163

14 . 79 0 . 0 1 b2

15 .06 0 .0155

1 6 .04 c .0167

14 .08 0 .0141

14 .91 0 .0139

1 3 . 96 0 . 0 144

13 .89 0 .0134

1 3 .40 0 . 0 149

15 . 1 3 0 . 0 149

1 5 .66 0 .0155

1 4 .00 0 . 0 140

L4 .23 0 . .0136

13 . 10 00 146

11 .70 0

.0106

G EK

3.86 0.19

4.48 0.14

4.64 0.18

6.48 0.12

3.22 0.19

3.85 0.19

4.01 0.20

5.38 0.11

3.35 0.18

4.84 0.17

3.16 0.19

5.16 0.16

3.18 0.19

3.38 0.18

5.07 0.17

3.46 0.18

6.24 0.12

3.23 0.19

4.09 0.17

3.10 0.20

3.50 0.19

4.96 0.16

3.24 0.18

3.51 0.15

3.52 0.18

3.34 0.18

3.56 0.16

3.. 40 0.17

3.47 0.17

4.15 0.16

4.19 0.16

3.35 0.17

3.36 0.18

3.47 0.18

3.05 0.17

3.58 0.16

3.67 0.16

3.36 0.17

3.61 0.18

3.77 0.15

3.37 0.19

3.41 0.12

3.54 0.14

4.28 0.14

3.28 0.17

3.55 0.16

4.20 0.14

3.98 0.14

3.56 0.15

3.49 0.14

4.07 0.14

4.33 0.13

4.43 0.12

3.94 0.13

3.88 0.14

4.62 0.13

3.43 0.14

3.70 0.14

5.13 0.11

3.26 0.12

GL EKL

3.57 0.17

6.50 0.11

3.29 0.17

7.19 O.li

3.25 0.17

3.99 0.17

3.74 0.16

7.48 0.08

5.06 0.14

3.94 0.17

5.19 0.13

5.01 0.20

3.80 0.16

5.02 0.14

3.60 0.16

3.55 0.16

11.25 0.08

3.80 0.15

3.62 0.16

4.01 0.16

3.35 0.17

4.13 0.15

3.88 0.15

6.23 0.11

4.41 0.15

3.38 0.16

3.85 C.15

5.70 0.13

4.48 0.14

4.20 0.15

3.41 0.15

3.90 0.14

2.92 0.18

2.94 0.17

3.70 0.14

4.77 0.12

3.52 0.15

3.95 0.14

3.09 0.17

4.01 0.12

3.12 0.18

4.36 0.10

5.73 0.10

3.71 0.14

3.07 0.15

3.70 0.14

3.50 0.13

3.51 0.13

5.74 0.12

4.07 0.12

4.15 0.13

3.95 0.12

4.02 0.12

4.21 0.11

4.09 0.12

3.89 0.14

3.42 0.13

4.14 0.12

4.15 0.11

4.45 0.10

98
VOTES WORD NOCG E EL PZD

ARGUES 443 5.52 5.67 12.23

NEVERT 370 5.50 5.71 11.92

SOLELY 441 5.50 5.74 12.87

2 FILE 943 5.49 5.87 17.06

4 DISSEN 751 5.48 5.73 13.43

ARGUED 396 5.47 5.71 12.15

EVER 481 5.47 5.65 12.23

HENCE 447 5.43 5.68 12.26

STATIN 385 5.43 5.67 11.77

3 CAREFU 453 5.42 5.79 13.51

2 COMPAR 418 5.42 5.57 11.09

HERETO 498 5.41 5.64 12.60

1 DESIRE 507 5.38 5.78 13.74

EXISTS 376 5.38 5.59 10.94

1 ABLE 416 5.37 5.64 11.77

INSIST 368 5.36 5.51 10.41

ONCE 375 5.32 5.60 11.02

QUITE 307 5.32 5.46 9.39

INSTEA 328 5.29 5.52 10.07

1 RELIES 301 5.28 5.48 9.62

NAMELY 316 5.27 5.44 9. 2 VIRTUE 322 5.21 5.46 9,

FAILS 426 5.21 5.68 12.15

1 ALLEGI 320 5.18 5.47 9.66

SEEKS 374 5.15 5.62 11.32

SOMEWH 236 5.13 5.27 7.73

SOMETI 237 5.05 5.22 7.39

DESMON 230 4.86 5.24 7.47

1 VOORHI 209 4.80 5.23 7.32

FROESS 209 4.78 5.18 6.98

FULD 208 4.73 5.20 7.09

1 WEYGAN 251 4.57 5.40 8.79

2 MATTHI 249 4.57 5.37 8.64

1 PECK 216 4.34 5.22 7.43

36

55

AVG

0.0136

0.0096

0.0118

0.0265

0.0191

0.0117

0.0127

0.0118

0.0U2

0.0118

0.0121

0.0121

0.0143

0.0104

0.0107

0.0096

0.0094

0.0083

0.0088

0.0090

0.0080

O.0091

0.0125

0.0088

0.0117

0.0070

0.0068

0.0065

0.0059

0.0062

0.0057

0.0050

0.0049

0.0043

G

4.75

3.19

4.03

5.51

3.84

3.88

4.47

4.38

4.44

3.79

4.96

3.70

4.09

4.09

4.69

3.68

3.70

4.11

4.25

4.16

4.71

4.56

4.84

4.31

4.95

4.87

5.15

4.60

4.32

4.96

4.57

6.09

6.34

7.17

EK

0.11

0.13

0.12

0.10

0.12

0.12

0.11

0.11

0. 11

0.13

0.09

0.12

0.12

0.11

0.11

0. 10

0.11

0.09

0.10

0.09

0.09

0.09

0.10

0.09

0.10

0.07

0.07

0.07

0.07

0.06

D.06

0.05

0.05

0.04

GL

3.96

3*20

4.06

4. 17

3.90

3.34

4.27

3.85

3. 86

3.84

4.20

6.07

3.97

3.84

4.20

4. 72

3.77

3.74

3.97

3.91

4.09

3.99

3.68

4.05

3.75

4.12

4. 18

4.06

3.98

3.98

4.05

3.57

4.17

4.22

EKL

0.11

0.12

0 . 12

0.12

0.11

0.12

0. 10

0.11

0.11

0.12

0.09

0.09

0.12

0.10

0.10

0.09

0. 10

0.09

0.09

0.09

0.09

0.09

0.11

0.09

0.10

0.07

0.07

0.07

0.07

0.07

0.07

0.09

0.08

0.07

Table VI. Sorted by E

99
VOTES WORD

A A A AAA

THE

AND

FOR

THAT

NOT

THIS

WHICH

WAS

WITH

FROM

HAVE

BEEN

COURT

THERE

UPON

ARE

BUT

ANY

HAS

CASE

SUCH

OTHER

WERE

UNDER

ONE

ONLY

HAD

MAY

CONCUR

MADE

QUESTI

ALL

REASON

WHEN

FOLLOW

WOULD

ALSO

BEFORE

AFFIRM

HIS

MUST

AFTER

ITS

LAW

SHOULD

PRESEN

TIME

DOES

WHETHE

THEREF

DID

WITHOU

JUDGME

WHERE

CONS ID

FURTHE

DEFEND

COULD

TWO

NOCG

26 4 9

442506

128355

45223

89026

35835

29490

25522

56044

21624

19879

13825

12072

33021

12925

11816

13721

9174

13855

10530

15261

18195

8966

12911

10893

9388

6218

15451

9510

2290

7999

8776

9021

6845

6875

6076

9678

5230

5814

3897

19529

5208

6340

11061

9658

5689

5653

8254

4264

5173

3871

6224

4652

10581

5794

5288

4546

25773

5096

5130

E

7.07

7.87

7.83

7.73

7.80

7.75

7.66

7.70

7.69

7.64

7.62

7.53

7.50

7.45

7.48

7.46

7.46

7.48

7.47

7.36

7.45

7.50

7.43

7.43

7.40

7. 39

7.33

7.43

7.37

6.65

7. 32

7.25

7.36

7.17

7.28

7.28

7.34

7.29

7. 19

6.89

7.32

7. 18

7.24

7.31

7.23

7.20

7.18

7. 17

7.09

7.22

7.01

7.24

7. 10

7.06

7. 19

7. 15

7.11

7.20

7. 16

7.11

74

.80

17

Table VII. Sorted

EL PZD

.87 99.99

.65 99.99

.61 99.73

.61 98.07

.60 98.15

.60 96.97

.59 96.67

.56 94.41

.55 95.73

.51 92.03

.51 92.18

.44 85.99

.41 83.76

.41 93.58

.40 84.25

.40 82.93

.39 84.37

.37 78.89

.37 83.12

.37 81.76

.36 84

.35 85

.31 76

.31 79.91

.31 80.44

.31 76.40

.31 72.14

.30 82.44

.30 76.70

.30 63.91

.29 74.51

28 77.08

.26 74.78

.25 72.48

24 69.87

.24 69.38

23 73.12

23 67.15

.23 68.55

.23 63.53

.22 78.63

.22 66.70

.21 68.47

.20 75.34

.20 74.29

.20 66.59

.20 68.25

.20 70.40

.20 63.30

.19 66.13

.18 62.21

.17 66.70

,17 63.57

.17 73.19

.16 65.26

.14 63.72

.13 61.94

.12 71.19

,11 61.79

.11 60.51

by EL

AVG

0.0783

12. 1192

3.4562

1.2529

2.4343

0.9798

0.8106

0.6984

1.5630

0. 5840

0.5456

0.3761

0. 3306

0.9097

0.3 545

G.3232

0. 3766

0.2485

0.3703

0.2838

0.4182

0.4817

0.2397

0.3486

0.2937

0.2540

0.1693

0.4205

0.2605

0.0643

0.2213

0.2395

0.2361

0. 1850

0. 1866

0. 1661

0.2580

0.1410

0.1612

0. 1 109

0.5396

0.1412

0. 1745

0.2888

0.2554

0. 1511

0.1558

0.2237

0. 1 175

0. 1408

0.1050

0. 1665

0. 1274

0.3119

0. 1562

0. 1379

0. 1230

0.7468

0.1383

0. 1408

G

0.42

-0.19

0.53

1.03

0.70

0.55

1.15

0.64

0.52

1.15

1.25

1.17

1.41

1.64

1.30

1.37

1.56

0.84

1.29

1.34

1.64

1.49

1.18

1.43

1.82

1.61

1.57

1.49

1.45

2.45

1.60

2.17

1.45

2.15

1.54

1.30

1.43

1.08

2. 12

2.26

1.55

1.83

1.62

1.71

2.34

1.89

2.26

2.55

1.80

1.69

1.43

1.55

2.02

3.01

1.64

2.06

1.92

1.34

1.59

1.59

EK

4.32

41.17

15.25

5.00

9.48

6.95

4.02

4. 89

3.68

3.46

3.01

2.52

1.96

1.26

1.87

1.76

1.85

2.21

1.87

1.51

1.43

1.78

1.79

1.55

1.31

48

38

1.38

1.38

0.73

1.25

1.03

1.46

1.11

1.20

1.18

1.34

1.33

0.95

0.78

1.03

1.08

1.06

1.13

0. 88

1.02

0.88

0.92

0.96

1.04

0.90

03

91

54

03

93

0.91

0.79

0.95

0.85

GL EKL

2.55 31.32

1.87 1.93

2.14 1.57

1.87 1.59

1.92 1.54

1.90 1.56

2.45 1.41

1.79 1.38

1 . 78 I . 33

2.15 1 .16

1.83 1.19

2.53 0.97

2.07 0.95

3.97 0.76

2.17 0.91

1. 83 0.95

2.55 0.86

2.06 0.89

2.37 0.83

2.41 0 . 8 3 2.38 0.80

2.91 0.74

2.45 0.76

2.67 0.70

2.98 0.69

2.40 0.75

1.88 0.82

2.68 0.69

2.50 0.72

2.51 0.86

1.97 0.76

4.30 0.6;

3.34 0.6^

2.86 0 . 64

2.24 0.69

2.44 0.69

2.49 0.64

1.95 0.71

2.63 0.66

2.61 0.70

2. 83 0.60

2.79 0.64

2.27 0.65

3.49 0.54

3.39 0.54

2.45 0.63

3.49 0.58

2.17 0.62

2.11 0.67

2.57 0.61

2.25 0.65

2.52 0.59

2.39 0.62

4.08 0.49

2.43 0.58

2.68 0.56

3.44 0.53

2.43 0.53

2.58 0.54

2.47 0.55

100
VOTES WORD NOCC

HOWEVE 3333 7.09

BECAUS 3553 7.00

THAN 4378 7. 1

4 FACT 4658 7.06

2 REQUIR 6103 7.06

1 PART 4746 7. 12

CONTEN 3888 7.02

THEY 7042 7.14

2 BEING 3858 7.04

THEN 4583 7. 12

THESE 4753 7. 11

SAME 4992 7.05

AGAINS 5725 7.04

9 APPEAL 9096 6.80

3 FIRST 4165 7.01

WHO 5241 7.11

9 EVIDEN 12726 7. 10

THEIR 6514 7.08

HELD 3978 7.04

1 PROVIO 5792 7.03

7 CONCLU 3665 6.95

4 DETERM 5030 7.02

1 FACTS 4095 7.00

5 APPEAR 3855 6.95

OUT 4389 7.00

2 STATED 3698 6.99

3 OPINIO 4764 7.02

7 TRIAL 9898 6.97

4 FOUND 3608 6.91

6 RECORD 6093 6.91

HERE 3448 6.93

WITHIN 4561 6.8 5 MATTER 4313 6.91

9 ACCORD 2721 6.87

2 CERTAI 3069 6.87

MORE 30 50 6.94

2 PLAINT 20986 7.02

4 PERSON 6980 7.01

1 CAN 2822 6.93

SAID 10747 7.07

SOME 3394 6.97

9 NECESS 3477 6.93

SINCE 2756 6.89

2 REVERS 2857 6.66

5 DIRECT 5706 6.95

6 ACTION 8248 6.94

INTO 3583 6.93

5 EFFECT 3759 6.91

CANNOT 2467 6.74

FILED 5362 6.67

CASES 3896 6.86

CAUSE 4463 6.77

I NVOLV 2933 6.56

THEM 3505 6.92

3 SUSTAI 2600 6.65

SEE 4704 6.93

1 BOTH 2868 6.85

BETWEE 3231 6.84

ENTERE 2920 6.78

1 RESULT 3328 6.85

Table VII.

EL

7.11

7.11

7.10

7.10

7.10

7.09

7.09

7.08

7.08

7.07

7.07

7.07

7.06

7.06

7.04

7.03

7.02

7.02

7.02

7.02

7.02

7.01

7.01

7.00

6.99

6.99

6.98

6.98

6.98

6.98

6.97

6.97

6.96

6.96

6.96

6.95

6.94

6.94

6.94

6.93

6.93

6.93

6.93

6.93

6.92

6.92

6.92

6.92

6.92

6.91

6.90

6.90

6.90

6.89

6.89

6.88

6.88

6.87

6.87

6.86

Sorted by

PZD

55.93

57.19

59.38

60.28

63.98

60.62

57. 11

64.47

57.41

59.19

59.79

60.73

61.83

77.61

57. 15

59.64

65.64

61.75

55.34

60.02

53.90

59.45

55.79

57.68

57.04

54.77

58.85

62.85

53.68

60.51

52.69

55.56

55. 19

49.64

50.62

49.49

57.71

60.81

49. 15

69. 15

50.88

52.20

48.65

46.96

58.62

64.55

51.00

52.39

46.54

55.26

51.41

54.28

47.86

49.37

46.24

55.00

46.54

47.45

48.58

48.50

EL

AVG

0.0923

0.0999

0.1198

0. 1249

0. 1665

0.1287

0. 1094

0. 1897

0. 1040

0. 1242

0. 1275

0. 1299

0. 1605

0.2637

0.1116

0. 1416

0.3461

0.1756

0. 1058

0. 1599

0. 1010

0.1314

0. 1137

0. 1045

0.1164

0.0975

0.1218

0.2884

0.1017

0.1675

0.0938

0. 1294

0. 1166

0.0745

0.0830

0.0822

0.6097

0.1897

0.0739

0.2803

0.0897

0.0937

0.0753

0.0842

0. 1575

0.2329

0.0952

0.1018

0.0694

0.1589

0.1062

0. 1255

0.0789

0.0943

0.0753

0.1297

0.0771

0.0879

0.0873

0.0911

G

1.47

2.04

2.23

2.10

2.34

2.57

2.14

2.45

2.13

2.04

1.97

2.47

2.56

4.94

2.30

1.89

1.64

2.19

1.92

2.56

2.50

3.04

3.05

3.97

3.00

2.37

2.05

2.75

2.73

5.25

1.92

2.63

3.11

2.12

2.20

1.98

1.25

2.61

1.61

4.45

1.97

3.31

1.76

2.65

5.12

3.64

2.51

2.86

2.06

4.09

2.58

2.98

2.29

2.56

3.40

2.95

1.87

2.33

3.29

3.50

EK

0.90

0.75

0.81

0.80

0.74

0.78

0.71

0.77

0.75

0.82

0.83

0.76

0.63

0.30

0.71

0.79

0.71

0.70

0.75

0.64

0.64

0.64

0.60

0.56

0.65

0.68

0.71

0.45

0. 53

0.41

0.66

0.50

0.53

0.62

0.65

0.66

0.64

0.57

0.67

0.50

0.67

0.52

0.62

0.48

0.44

0.39

0.57

0.56

0.57

0.33

0.54

0.43

0.56

0.56

0.40

0.47

0.59

0.55

0.42

0.49

GL

1.76

2.28

2.63

2.40

4.53

2.85

2.24

3.52

2.89

2.60

3.27

3.32

3. 13

5.35

3.27

3.51

3.09

3.29

2.83

3.62

2.52

3.95

2.90

9.43

6.13

3.69

4.63

2.96

3.16

4.95

3. 12

3.59

4.12

2.92

3.90

2.76

2.24

5.09

2.68

6.83

4.84

4.91

2.78

3.60

6.63

4.77

3.14

7.29

2.46

3.46

3.22

4.0£

2.9?

4.37

2.63

3. 89

2.81

2.83

4.02

3.97

EKL

0.62

0.58

0.54

0.5,4

0.47

0.52

0.56

0.45

0.52

0.51

0.48

0.48

0.46

0.33

0.46

0.44

0.43

0.42

0.47

0.42

0.49

0.40

0.46

0.32

0.37

0.42

0.37

0.41

0.43

0.35

0.43

0.41

0.38

0.45

0.42

0.45

0.43

0.33

0.44

0.27

0.39

0.35

0.43

0.43

0.29

0.31

0.39

0.34

0.45

0.36

0.38

0.34

0.40

0.36

0.41

0.33

0.39

0.38

0.34

0.34

101
VOTES WORD NOGG E

6 RIGHT 5447 6. 76

NOR 2099 6.70

HAVING 2006 6.67

HIM 5613 6.91

WHILE 2749 6.82

1 PROCEE 5021 6.79

MAKE 2535 6. 76

8 MOT I ON 662 L 6.71

SET 2964 6.71

OUR 3179 6.80

WELL 2259 6.77

4 GENERA 5262 6.87

GIVEN 2766 6.80

RESPEC 2579 6.80

6 EXCEPT 3589 6.58

6 AUTHOR 4898 6.78

4 SUFFIC 2484 6.72

2 ALLEGE 3766 6.72

5 SUBJEC 2855 6.70

5 STATUT 7283 6.89

2 STATE 9231 6.85

NOW 2384 6.60

WHAT 2883 6.76

EITHER 2033 6.71

2 PROVIS 4479 6.80

3 ORDER 6773 6.78

THOSE 2527 6.73

4 GROUND 2629 6.68

ALTHOU 1762 6.67

1 DENIED 2053 6.30

2 SECTIO 10226 6.83

2 PURPOS 4138 6.76

2 I NCLUD 2632 6.71

TAKEN 2518 6.67

2 PART IC 2381 6.48

4 CIRCUM 2543 6.75

THEREO 2640 6.69

EVEN 1964 6.64

7 WILL 7140 6.84

4 PRIOR 2379 6.69

SHALL 6240 6.81

1 THREE 2437 6.70

1 APP 4769 6. 74

1 ESTABL 2947 6.74

1 NEW 4744 6.68

END 6422 6.81

OVER 2622 6.72

3 SUBSTA 2527 6.62

UNTIL 2347 6.65

3 I ND IC1901 6.64

6 RULE 4090 6. 56

HERE I N 2599 6.23

1 EACH 3332 6.68

1 ENT ITL 2141 6. 53

2 DEC IS I 3988 6.52

7 SPEC I2900 6.65

3 SUPPOR 3151 6.65

5 I SSUE 3113 6.61

6 ERROR 3841 6.56

3 FIND 1954 6.51

EL

6.R6

,86

86

85

85

Table VII.

6.84

6.84

6.84

6.84

6.83

6.83

6.82

6.82

6.82

6.82

6.8 1 6.81

6.81

6.8 1 6.80

6.80

6.80

6.79

6.78

6.77

6.77

6.7 7 6.77

6.77

6.77

6.76

6.76

6.76

6.76

6.76

6.75

6.75

6.75

6.74

6.74

6.73

6.73

6.72

6.72

6.72

6.71

6 . 7 1 6.71

6.70

6.70

6.70

6.70

6.69

6.69

6.69

6.68

6.67

6.66

6.66

6.66

Sorted by EL

PZD

54-24

43.14

42.09

5 4.24

46. 31

55.19

43.94

53.90

46.54

47.98

43.14

52.92

45.07

44.43

49.79

52.32

42.92

47.86

45.48

53. 15

62. C6

43.29

44.80

40.20

47. 18

58.32

42.43

44.16

38.65

40.39

55.75

49.30

43.41

43.07

42. 12

41 .94

41.60

38.80

62.55

40. 88

49. 18

41.18

44.92

44.46

48.09

51.86

40.99

41.60

39.22

37.67

47.18

41.75

43.90

38.42

46.58

42.28

46.35

42.88

44.80

37.75

AVG

0. 1464

0.0581

0.0548

0.1531

0.0751

0. 1373

0.0681

0.1942

0.0798

0.0833

0.0592

0. 1338

0.0744

0.0678

0. 1046

0.1319

0.0708

0.1091

0.0784

0. 1985

0.2417

0.0629

0.0725

0.0532

0.1251

0.1918

0.0642

0.0728

0.0487

0.0580

0.2858

0. 1096

0.0716

0.0697

0.0625

0.0679

0.0697

0.0509

0. 1944

0.0654

0. 1705

0.0677

0.1292

0.0788

0.1295

0. 1570

0.0701

0.0693

0.0628

0.0499

0.1055

0.0670

0.0859

0.0591

0. 1070

0.0790

0.0855

0.0831

0.1051

0.0519

G

2.91

1.94

2.1 8 2.49

5.29

3. 56

2.35

3.78

3.36

2.15

2.87

3.11

2.27

1.99

5.95

4.35

2.35

3.04

2.72

2.26

3.06

2.79

2.52

1.96

2.55

3.68

3.12

3.25

1.78

2.91

2.91

3.99

3.86

3.27

3. 17

2.08

2.61

2.09

5.49

2.87

2.77

3.19

2.51

3.00

3.77

3.07

2.40

3.48

2.31

2.45

4.23

3.17

4.53

2.60

4.00

3.75

7.06

3.76

3.69

3.11

EK

0.47

0.53

0.51

0. 52

0.43

0.40

0.54

0.30

0.45

0.55

0.51

0.47

0. 50

0.54

0.26

0.37

0.45

0.40

0.46

0.48

0.39

0.46

0.51

0.50

.45

.31

.46

.38

0.50

0.37

0.38

0.41

0.39

0.37

0.41

0.49

0.42

0.49

0.26

0.41

0.43

0.40

0.41

0.45

0.31

0.44

0.43

0.36

0.42

0.42

0.31

0.36

0.36

0.38

0.30

0.34

0.24

0.32

0.29

0.35

GL

3. 87

2.78

2.07

6.64

4.31

6.15

3.17

3.36

3.72

4. 84

3.49

5.01

3. 10

3. 71

4.72

4.61

3.24

37

64

39

64

3. 10

3.76

3. 10

3.69

11.48

3.52

5.73

2.66

2.72

4.29

6.33

3.68

4. 04

3.48

2.94

3.06

3.06

12. 86

3.12

4. 34

3. 87

3.31

17.95

4.33

6. 84

3.50

4.62

3.46

3.59

12.48

5.86

5. 12

3.68

5.57

5.03

9.79

4.98

4.33

3.70

EKL

0.32

0.40

0.43

0.29

0.35

0.26

0.37

0.33

0.35

0.31

0.36

0.23

0.35

0.34

0.30

0.28

0.36

0.33

0.33

0.29

0.25

0.34

0.32

0.35

0.30

0.19

0.33

0.29

0.37

0.35

0.27

0.25

0.31

0.31

0.32

0.33

0.33

0.35

0.15

0.32

0.27

0.30

0.29

0.18

0.26

0.22

0.29

0.27

0.30

0.31

0.20

0.25

0.25

0.30

0.23

0.25

0.18

0.23

0.24

0.28

102
2

1 10

4

3 WORD NOCC E EL PZD AVG G EK GL EKL

AbOUT 3228 t.O i A S AO i A S U" 1i

1 0 0\j « 0 ft R 7 a^ a ft R 0VJ « 39J 7 aJ 4 45T J1 0 .27

THUS 1622 AO i » 5-J Ao

AO 4 A 5 nV a 04 ? 7 a_? a 0 R VJ « 4? p 88 0 . 3

ANOTHE 188 1 AO * » 5-j 7/ AO a A <S n 4 0 500 C a 97 0. 37 aaJ <1 1X 7I 0

. 29

CONTAI 2096 AO 4 5 5 AO ft 5 3 a V-> a 0 5 7H -? 4 3 5 0. 35 5> a 43 nVJ 4 > 4? 5-a* JUDGE 4000 AVJ a 52 6

64 46 84 nw a 1X 1 ft 1 a. <J L 1iU40. 30 0.19 6 a 80 0VJ 4 .20 1 ta V MIGHT 1734 Ao a s<j 7i AVJ O J 34 ? 7 nw a 04ft5 al a 40 0. 39 PA a 7fl ft\J 4 4 »30 O VJ UNLESS 1520 AO a 5 a 9 4 AO a A ~X D J • 0 <i U a aC a a 0 0. 39 PA a 7P Au < 1 an3 VJ ABOVE 1812 A An4 J A A *a 0 j D D a in ,-1 U a ft A O 1 >J 4 0 3 £ a OAV4 0. 35 p3 a ft*a U3 AU 4 p 0 . A V SEC 6808 AO -

a A K AW a AW a£P a n nW < a. ^« y 5 a 7 »X

( P 0. 2 7 4 a c ft AU 4 p 1

at i DURING 2216 6i 58j vj 6VJ a 6 ? 50 nw a 0609 >

aC a 7 3 0 a O A 36 4." a 4? .26 1 <C *J SECOND 2415 AVJ a •5 3 AO a A 1 3R W a 0VJ AO 5 A0

aJ a 9 7 AOa 31 5J « ft3 ftVJ < ?i. 3J1 EXPRES 2022 Avj a 5J 1L AO a O L 3ft 0 1 n>J a 0 5 4 A "I

« A? 11 c. 34 A*T « X1 R0 nVJ 1 A?ftVJ APPLIC 4168 AO a 5 fl AO a O U 47 J a 1 1 *T a Q 7 0. 2 5

QO €

1X 3P

ftU a 1 ft t X VJ INSTAN 1867 AVJ a 54 AO a A 0 34 ft ft • 0 0 W a 0 4 C a 5P «0 0 a 1 A 36 -33 • 0

\J X

ftw a A? RO OWN 1857 AO a 5P 3j

AO a a n 3 4 q q O a 0 k n 0 A a 7 1 Urn 3 5

*3

J> 0 Q37 P

ft

VJ a A?7f ACT 5147 AO a A 5 AO a S Q 4 5 • 5JOA •"1 ^/ a

1 ^ 7H "33 a 3 0 u. a 0 3a AO a A? 1X

ftVJ • A? 0VJ CONCER 1797 Ao • 57 AO a S Q 34 7A J a 0U /» A ft *T 0 0

A

"T a 40 ftu • a3 a4 0 « ft7 0 1 ft

VJ a A?'ftO FIND IN 3437 6vj -• J VJ AVJ a Sj Qj 41 c; a V a 0 995 ^4 a 00 U . A

P AD

-3J • 907 \j ft

VJ a A? 3 PART I

3496 6 . 55 A»-> a 5 9 41 71 0 a 0U 9/ AOuu flft u . AP Q7 4 47 0 a atC 2?4- BROUGH 1534 6

50-* VJ 6w .a 5Q 3 3 74 a 04 AO A^ . 00 <j . A 7 A4 0 a 77

SHOW 1649 6 * 36 AVJ a 59 —» J ft 9 0 0470 • ?aC. AL> J3 vA 7£. 1X

ft

VJ 4 74- aVJ USED 2650 6 . 45 6

5 8 38 1 ft •g

0 734 5 . 62 ftu

PA 4 1aL 8U 0 a 23

ITAL 11360 6 . 67VJ I AU a 57 45 .18 0 . 2755 -a* * 1X ?^

ftu . 3371 7 3? 0 a 1X 97 FOL 5682 6 . 67 Av> • 57 45 1 ft C • 1 37R 1i. A? ftV . 17^ 1 7. 39 0VJ .c X1 97 CLEAR 1537 6 . 52 6

5 7 33 .48 0

042 5 3 . 3 5 ft 3-> 3p 539 0 a 24

RECEIV 2801 6. 52 6

57 39 . 10 0

0 764 6 . 76 ft A? 71 5. 74 0 a 2

CALLED 1618 6. 40i VJ Aw • 5 7 3? - 76 n 0444 4 43 ft

\J . 3 1 3 4?F at. 0VJ .• 27

YEARS 260 1 6 . 5 3 6vj -. 5 6 37 1 0 0

0687 3 . 24 ftv . 34, 1X 97 0

? 3

THROUG 1954 6 . 52 6 . 56 34 .61 0

0531 3. 87 ftv . -P30u 4. 00 0

24

BASED 1605 6 . 38 6 . 56 32 .84 0

0431 2. 60 ft 3a 5p 3. 70 0 a 26

CONSTR 3805 6vj .. 58J V Avj . 5 5 40 50 0 1i. 0Va/ —54' T -\

-J • 3fl ftU . 330u 4, 65VJ J* 0 2<£. X CONTRO 2941 6 • 48 AVJ • 5 5 39 93 0 0849 5 . 05 A<J . p "a A 3 5 00VJ VJ 0 . 20<C VJ DIFFER 1714 VJ • 46 6vj .. 5 5 33 • aL " n 0466 96 ft p 0 A 7

aj . -J5ftu 0VJ .« 75at— ^ THEREA 1342 Avj • 40T \J Au . 5 5 3 1 0 3 n 0 3 R9 7fl A a p 3 a

pA . 9? ftw . A?RO NOTHIN 1275 AL* * Au*T AVJ . 30 A5 0^45 A . 7 A AU. *a *a 33 PA . ft4 VJ . 79A 7

THOUGH 1301 Au .• 4^*T J Avj . 54 30 4 ft 0 0 ^40 ?a- . 57 A •a /. 34 PA . ft? O A

ft

VJ . A7 flO CITED 1401 AO • T L AO • 5 4 30 95 0\j • 0 "^90 4l . 5aV ?C A0. *a a 33 aj . Oft VJ 0

ftVJ . A7 7f RELATI 2530 Avj .• 54 AVJ .. 5 3 J f a 14. 0w n 0 A A? aj . Alw X A a ft 30 5 77 ft

VJ . A70VJ APPARE 1334 Ao • 4 ^ AO . 5 1 30 .84 nv • 0\J J3 A4\J "T j

j . ? A Au. a ft 3U •aj * 3Pv A

ft

VJ . A7ftO

COUNTY 6245 AO • A ? AO . 5 ? 5 ? 4 3 nU a 1 7R7 . no A 0 "a A 3

a0 . 5 1Xft

VJ .

1X 4*T AMOUNT 3110 AQ • 49 AO « 5 ? 5ft 0 0 ft A9 ft0 5j

ft p 7 A /•ap . 75 ft

VJ • A7 A7 REGARD 1466 AO • J397 AO . 5 ? 30 ftO <j . 0 ^ ftO •aj . 0VJ J5 A a p 3 a

ap . 0 5 ft

VJ . A7ftO DECIDE 1409 A . 4"T 1X 6 . 50 ?9 R9 0

0V 3fi J 0 1X 4ft ft 3 L a3 . 99 ftVJ . A7 5 CONTRA 8033 Avj _• 56J VJ 6 - 49 5? .96 0. aC? X1 -a/ 5RO a _ 98 ft p *a A 3 7 A?97

ft

VJ . 1X 5-J SITUAT 1358 Ao • 4? Ao . 49 ?9 40 n\j . 0w ^^ Aft 0 0 A . 40" VJ A 0 *a 3 3

aP . 07 ft

VJ . A7 5P CONSTI 4132 vjA • 41i X A\J . 49 4? 99 0 . 105ft l *J J u -J • 4ft A 28 7 53 ft

VJ . 1X 5J ADDITI 1708 vjA ^• 39 A _ 49 3? 0 . 0453 5 .• 06 u •

•a c A 5 4 ft fl ft

VJ . A77A PERMIT 2869 Ao • 35J J Au . 49 39 0. 0ft?0 A - 1 7 0. 1

OA • 3ft ftVJ . X

1 7« COMMON 4042 AO • 4A" vj AD . 4 A 4? 5 ft n 1a. 11 711 a. 5 R5 0. 1

7 01\J X

ft

VJ .

1X ft0

VIEW 1406 A AO . 4*t fto ^0 Q 5 nu • 0 3 75 A 0. 29 7 0U X

ft

VJ . A70VJ FAILED 1442 A AO . 4ft -> \j • Ok. nu * 0414 xO . J1 ^ A0. 29 •ap . 79f 7

ft

VJ . A9 P3 DISMIS 2755 c

J • Q7 AO

AO . 4 ft 90 n 0 790 Kj * 1x A0

A 1 6

KP . OlU Xft

VJ . A U CHARGE 4622 AO • 4ft AO . 47"T f 40 • AOO

~ ri\j .

1 ? 34 a^ . 9A A0. 24 A Q57 p

ft

VJ . 1 0

CONDIT 2779 AO • 4A AO . 47 -j j 0 7ft0 a 5? ft\j . A?ftO

ap . Q Q O OftU . 9 1 A 1 LATER 1426 Au • 4 ^ 6 . 47 29 .4ft 0. 03R7 ? . 751 a** 0. 31 a. ~*5?A 0VJ . A74*T BASIS 1500 6. 41 6. 47 30 .76 0. 0412 5. 82 0. 26 5. 60 0VJ . 7A 1X DUE 1937 VJ • 40 6 . 47 32 .08 0542 4 . 1

X <J 0. 25 -ap . 791 7

ftVJ • A7 A7 TAKE 1484 AO • 3fl AO . 47 30 35 0 0407 \J T w 1 a * ft 5 0. 27 ap .

ftu . A9 'a3

ILL 8605 6. 49 6. 46 32 .88 0. 2551 1. 95 0. 34 3. 00 0. 24

DAY 2189 6. 41 6. 46 34 .16 0. 0607 3. 92 0. 26 9. 83 0. 17

Table VII. Sorted by EL

103
2

3

3

3

2

4

12

35

11

3

5

1

s WUKJJ E ELi a t Try AVG G EK GL

S I MI LA 1243 6. 38 6 .46 28 .61 0 .0339 2. 91 0..30 3. 18

OPERAT 4207 6.52 6..45 39 .56 0 . 1 145 3. 54 0..27 4.52

PAGE 3218 6.47 6..45 33 . 7 I 0 .0815 2..83 0..31 5. 57

COMPLA 3971 6.40 6,.45 37 .44 0 .1136 4, 27 0..22 4. 90

PLACE 188 1 6. 36 6..45 32 .27 0 .0528 6. 46 0..21 5. 21

GIVE 1490 6. 32 6..45 29 .78 0 .0399 3. 06 0..29 3. 67

CLEARL 1145 6.31 6..45 27 .67 0 .0304 2. 81 0..30 3, 28

COMPLE 1709 6.30 6,,45 31 .40 0 .0455 4, 76 0.,24 5« 48

RENDER 1657 6. 30 6,.45 31 .74 0 ,0464 3. 94 0.,23 6. 39

COURSE 1500 6.22 6. 45 30 .53 0..0421 6. 86 0..21 4. 36

WAY 1771 6.21 6. 45 32 .91 0..0472 6. 65 0..22 10. 08

APPELL 14543 6.53 6. 44 50 .16 0..3877 3.05 0..23 5.26

PETIT I 7623 6. 19 6.,44 40 .39 0..2198 3. 73 0. 19 5. 82

PRINC

2158 6.46 6. 43 34 .61 0..0564 6. 01 0. 24 7. 85

REFERR 1309 6.24 6.43 28 .65 0..0341 8. 37 0. 24 5. 55

F AILUR 1630 6. 16 6 C 43 30 .16 0. 0459 3. 81 0. 24 4. 43

POINT 1487 6. 35 6. 42 29 .48 0. 0407 4. 43 0. 25 4. 24

OVERRU 1644 6.23 6. 42 30 .46 0. 0456 4. 78 0. 19 4. 35

OTHERW 1095 6. 14 6. 42 27 .18 0. 0307 4. 16 0. 25 3. 79

ATTEMP 1404 6.05 6. 42 29 .18 0. 0376 4. 42 0. 25 7. 93

TESTIM 3650 6. 42 6. 41 34 .65 0, 1010 3. 30 0. 25 3. 88

ANSWER 3398 6.42 6 = 41 39 .33 0. 0913 5. 64 0. 22 9. 44

DATE 1983 6.31 6. 41 31 .37 0. 0555 3. 97 0. 23 4. 85

ENTIRE 1350 6. 30 6. 41 28 .53 0. 0369 5. 20 0. 25 6. 76

CONTIN 2382 6. 37 6. 40 34 .35 0. 0634 5. 85 0. 21 10. 10

APPLIE 1264 6.25 6. 40 27 .63 0. 0351 2. 95 0. 27 "3. 46

FORTH 1458 6.25 6. 40 28 .80 0. 0391 3. 68 0. 25 4. 54

TERMS 1583 6.33 6. 39 28 .46 0. 0424 3. 43 0. 25 3. 35

INTEND 1333 6.29 6. 39 27 .63 0. 0361 3. 14 0. 25 4. 27

ORIGIN 2053 6.23 6. 39 32 .01 0

0558 4. 38 0. 21 5. 63

REMAIN 1592 6.35 6. 38 30 .46 0. 0428 4. 99 0. 23 7. 12

MANY 1117 6.27 6o 38 25 .82 0. 0286 2. 52 0. 29 2. 73

NEI THE 930 6. 16 6, 38 24 .87 0. 0252 2. 65 0. 27 2. 44

CORREC 1358 6. 14 6. 38 28 .57 0. 0370 4. 35 0. 21 4. 34

THERE I 1068 6.13 6. 38 25 .70 0. 0279 2. 72 0. 27 3. 38

MANNER 1259 6. 30 6. 37 27 .29 0. 0329 3. 46 0. 27 6. 32

ARGUME 1528 6. 26 6. 37 28 .69 0. 0429 5. 01 0. 20 4. 22

SUBSEQ 1263 6.25 6. 37 26 .99 0. 0363 3. 67 0. 24 3. 97

FAVOR 1249 6.22 6. 37 26 .87 0. 0364 3. 45 0. 23 4. 09

STATEM 2732 6.32 6. 36 34 . 16 Go 0720 4. 77 0. 20 5. 32

SEVERA 1243 6. 32 6. 36 27 .25 0 = 0331 3. 47 0. 26 7. 53

COURTS 2033 6.28 6. 36 31 .21 0. 0553 9. 19 0. 16 5. 77

TRUE 1140 6.23 6. 36 26 .23 0. 0309 3. 33 0. 26 4. 42

SHOWN 1 106 6. 15 6. 36 25 .74 0. 0303 3. 38 0. 24 3. 23

OHIO 8519 6.49 6. 35 34 .39 0. 2212 2. 35 0. 28 5. 51

TESTIF 3484 6.35 6. 35 31 .74 0. 0969 3. 53 0. 24 3. 72

SHOWS 1078 6. 16 6. 35 25 .25 0. 0297 3. 55 0. 23 3. 06

HOLD 1033 6.15 6. 35 24 .61 0. 0270 2. 49 0. 26 3. 24

THERET 1022 6.05 6. 35 24 .95 0. 0278 3. 03 0. 25 3. 31

PROPER 5913 6.40 6. 34 36 .91 0. 1591 3. 62 0. 23 5. 71

SAY 1088 6.26 6„ 34 25 .44 0. 0294 2. 94 0. 26 3. 71

GRANTE 1574 6.25 6. 34 28 .35 0. 0425 4. 97 0. 20 5. 70

BEL I EV 1176 6.22 6. 34 25 .67 0. 0322 3. 33 0. 24 3. 34

STATES 2343 6.38 6. 33 33..37 0. 0582 6. 26 0. 22 8. 54

RIGHTS 2108 6.30 6. 33 30..38 0. 0581 5. 59 0. 20 4. 76

PARTY 2643 6.26 6. 33 31 . .93 0. 0726 4. 28 0. 20 5. 91

ITSELF 993 6.25 6. 33 24..38 0. 0260 2. 40 0. 27 3. 32

MAKING 1060 6. 19 6. 33 25,. 14 0. 0282 4. 11 0. 22 3. 75

ORDERE 1180 6. 14 6. 33 26..23 0. 0324 3. 50 0. 23 6. 13

SOUGHT 1132 6.11 6. 33 25..44 0. 0316 3. 80 0. 21 4. 23

laoie Try T VII. Sorted by EL

EKL

0.24

0.18

0.19

0.19

0.^9

0.23

0.24

0.20

0.19

0.21

0.16

0.16

0.18

0.16

0.21

0.21

0.21

0.20

0.23

0.19

0.20

0.13

0.19

0.20

0.14

0.22

0.20

0.21

0.21

0.18

0.16

0.23

0.25

0.20

0.23

0.19

0.19

0.21

0.21

0.16

0.18

0.17

0.20

0.22

0.17

0.19

0.22

0.22

0.22

0.15

0.21

0.17

0.21

0.13

0.17

0.16

0.22

0.21

0.18

0.20

104
VOTES

8

52

10

8

5

14

13

1

WORD

INTEKE

ADM I TT

RETURN

LONG

MERELY

JURY

HEARIN

OBJECT

MOST

DISCUS

CONSIS

PREVIO

NATURE

PUBLIC

DUTY

LEGAL

OBTAIN

BECOME

REVIEW

REQUES

THINK

TOOK

DONE

RAISED

USE

COUNSE

SUPRA

PAID

RECOGN

CLAIM

SUPREM

FAR

PURSUA

CITY

LANGUA

EXAM I N POSSIB

VERY

REFUSE

DISTIN

EVERY

DAYS

RATHER

HER

HOLDIN

CODE

NOTICE

KNOWN

LESS

EXISTE

CLAIME

TOGETH

PREVEN

SHOWIN

NEVER

LATTER

WHOM

OFFICE

ASSIGN

VARIOU

Table

NOCC E

3637 6.36

1667 6.32

2074 6.24

1047 6.23

936 6.21

5530 6.41

2525 6.28

2703 6.27

1051 6.25

1034 6.22

941 6.19

1040 6.16

1185 6.16

4658 6.33

1873 6.25

1650 6.25

1498 6.18

1158 6.07

2347 6.02

1941 6.11

1035 6.18

1080 6.15

1079 6.09

1050 6.00

3852 6.29

3030 6.22

2573 6.29

2316 6.25

1033 6.10

2565 6.24

1904 6.16

923 6.11

1039 6.08

5969 6.24

1492 6.22

3117 6.19

1018 6.18

888 6.15

1286 6.14

997 6.14

922 6.11

1500 6.05

917 6.15

7548 6.30

1008 6.05

4152 6.21

2855 6.04

1083 6.12

923 6.08

1029 6.06

921 5.97

861 6.04

956 6.00

829 5.78

976 6.01

833 6.04

832 6.00

4060 6.26

2654 6.00

815 5.99

VII. Sorted by EL

87

48

80

78

12

54

EL PZD

6.32 35.33

6.32 28

6.32 3L

6.32 24

6.32 23

6.31 34.27

6.31 31.39

6.31 32.50

6.31 24.95

6.31 24.34

6.31 23.66

6.31 24.57

6.31 25.48

6.30 35.78

6.30 28.35

6.30 28.5?

6.30 27.40

6.30 25.36

6.30 32.72

6.29 29.44

6.28 23.63

6.28 24.46

6.28 24.57

6.28 23.93

6.27 36,

6.27 32,

6.25 29.21

6.25 28.16

6.25 23.51

6.24 32.27

6.24 27.44

6.24 22.61

6.24 23.17

6.23 38.05

6.23 25.78

6.23 35.56

6.23 22.98

6.22 21.93

6.22 24.49

6.22 22.68

6.22 22.31

6.22 24.99

6.21 22.00

6.20 31.89

6.20 22.76

6.18 29.55

6.18 30.76

6.17 22.19

6.17 21.63

6.17 22.08

6.17 21.44

6.16 20.91

6.16 21.44

6.16 20.53

6.15 21.32

6.14 20.23

6.13 20.08

6.12 33.93

6.12 29.82

6.12 19.96

AVG

0.0944

0.0436

0.0589

0.0280

0.0248

0. 1470

0.0716

0.0742

0.0273

0.0267

0.0260

0.0277

0.0313

0. 1226

0.0506

0.0423

0.0397

0.0320

0.0676

0.0545

0.0298

0.0302

0.0282

0.0290

0.1059

0.0868

0*0636

0.0616

0.0261

0.0735

0.0474

0.0247

0.0271

0. 1706

0.0411

0.0831

0.0272

0.0230

0.0351

0.0265

0.0244

0.0447

0.0246

0.2095

0.0265

0. 1146

0.0853

0.0285

0.0250

0.0286

0.0261

0.0222

0.0265

0.0227

0.0254

0.0235

0.0228

0.1032

0.0715

0.0214

G

5.26

3.82

8.81

3.39

2.46

3.35

4.03

8.66

2*65

2.85

2.47

3.93

3.80

4.86

3.82

7.41

3.28

3.89

5.34

7.47

3.00

3.21

3.94

3.56

4.86

6.05

3.34

3.21

3.33

5.91

3.73

4.89

2.92

3.90

3.66

7.01

3.04

2.80

4.26

2.77

3.05

6.03

3.00

4.05

3.62

4.17

5.70

3.59

3.44

5.05

4.84

3.31

3.86

3.37

4.03

3.47

3.43

4.82

6.48

4.01

EK

0.20

0.23

0. 15

0.23

0.26

0.24

0.21

0.15

0.28

0.25

0.26

0.22

0.22

0.20

0.21

0.19

0.23

0.23

0.15

0.15

0.23

0.24

0.21

0.21

0.18

0.15

0.23

0.23

0.23

0.15

0.21

0.20

0.22

0. 18

0.21

0.15

0.23

0.24

0.19

0.24

0.22

0.14

0.24

0.20

0.21

0.17

0. 14

0.21

0.21

0. 19

0.17

0.21

0.19

0.19

0. 19

0.19

0.19

0.17

0.12

0.20

GL

5.71

5. 59

9.23

3.84

2.82

4.31

6.14

5.60

6.00

3. 19

3.02

3.68

4. 10

5.07

5.09

9.77

5.62

3.96

7.80

5.99

3.20

4.38

4.53

3.95

7.72

5.28

4.77

4.69

3. 94

7.77

6.65

4.79

3.93

5.82

5. 17

8.63

3.70

3.45

4.13

4. 15

3.79

3.91

3.67

4.75

4.43

5.98

6.77

4.34

3.99

4.18

3.94

3.86

3.57

3.12

4. 18

3.63

3.68

18.75

7.19

3.74

EKL

0.15

0.17

0. 14

0.20

0.22

047

0.15

0.15

0.18

0.21

0.21

0.20

0.19

0.15

0.17

0.14

0.17

0.19

0.13

0.15

0.20

0.19

0.18

0.19

0.12

0.14

0.15

0.16

0.18

0.12

0.14

0.18

0.18

0.13

0.16

0.11

0.18

0.19

0.17

0.18

0.18

0.17

0.18

0. 14

0.17

0.13

0.12

0.16

0.17

0.16

0.17

0.17

0.17

0.18

0. 16

0.17

0.17

0.07

0.11

0.16

772-957 0-66—

105
VOTES WORD mAnn NOCC E

RELATE 839 5.92

4 OCCURR 1248 6.05

AGAI N 766 6.00

LEAST 766 6.00

THEREB 712 5.99

MUCH 693 5.99

13 JURISD 3056 6.00

HIMSEL 864 5.95

& ACK66 707 5.91

2 TIMES 751 5.95

OBVIOU 645 5.8 7

1 REV 1484 6.07

APPLY 806 6.00

LIKE 738 5.93

BECAME 734 5.81

HEARD 903 5.97

7 JUSTIF 885 5.90

i STILL 660 5.86

SUGGES 782 5.94

5 COMPAN 4677 6. 19

8 SERVIC 3855 6.04

PLACEO 781 5.88

WHOSE 655 5.89

OCCASI 742 5.95

READS 769 5.89

MENTIO 694 5.91

NOTED 710 5.88

HOW 739 5.93

4 EMPHAS 1012 5.96

8 RESPON 2872 5.94

COME 663 5.90

BEYOND 754 5.87

MERE 654 5.82

SEEMS 647 5.88

2 ESSENT 651 5.83

MAKES 565 5.73

PUT 719 5.88

FOREGO 626 5.73

1 STAT 1245 5.90

AMONG 579 5.83

FULLY 591 5.74

7 VALID 768 5.83

WHERE I 560 5.60

5 EMPLOY 6062 5.98

DOING 625 5.71

I APPROX 704 5.79

ALONE 536 5.73

DIFFIC 578 5.72

2 FILE 943 5.49

REACHE 539 5.63

2 QUOTED 591 5.60

1 CONCED 485 5.58

NONE 506 5.58

ALREAD 542 5.68

RELIED 487 5.62

3 CAREFU 453 5.42

2 WHOLE 651 5.74

1 DESIRE 507 5.38

ADDED 587 5.62

MOVED 49 2 5.61

Table VII. Sort

EL PZD

.12 20.04

21.78

19.32

19.40

19.02

19. 13

.10 29.67

.10 19.85

.10 16.98

.09 19.21

.09 18.23

.08 22.72

.08 19.63

.08 18.87

.08 18.61

.07 19.93

.07 19.85

.07 18.C8

06 18.68

.05 32.65

.05 29.63

.05 18.91

.04 17.70

0 3 18.38

.0 3 18.30

0 2 17.89

.02 18.04

.01 17.89

00 19.59

.00 29.21

.00 17.40

99 17.74

99 17.02

.98 16.87

98 16.76

.98 16.27

.96 17.40

.96 16.64

93 19.10

.93 15.81

93 16.00

.92 17.06

92 15.66

.89 32.50

.89 16.04

87 15.77

87 14.79

.87 15.06

.87 17.06

.86 14.91

85 15.13

.83 14.00

82 14.23

80 14.08

80 13.89

79 13.51

78 14.87

78 13.74

77 13.96

75 13.40

by EL

AVG

0.0233

0.0347

0.0209

0.0206

0.0192

0.0187

0.0812

0.0241

0.0 187

0.0201

0.0187

0.0446

0.0212

0.0198

0.0196

0.0241

0.0235

0.0176

0.0208

0. 1 180

0. 1114

0.0208

0.0179

0.0206

0.0220

0.0191

0.0182

0.0191

0.0246

0.0772

0.0173

0.0209

0.0170

0.0179

0.0173

0.0151

0.0197

0.0163

0.0383

0.0152

0.0159

0.0207

0.0155

0.1653

0.0167

0.0179

0.0152

0.0155

0.0265

0.0139

0.0149

0.0140

0.0136

0.0141

0.0134

0.0118

0.0169

0.0143

0.0144

0.0149

3.10

3.73

,64

98

22

,85

,48

5.07

3. 50

3.18

3.36

3.55

3.14

4.09

3.61

3.35

3.52

3.47

3.46

4.27

5.82

4.15

3.34

3.38

3. 56

4.96

3.47

3.23

3.16

6.24

3.24

3.35

3.36

4.19

3.67

3.28

3.40

3.55

3.51

3.05

4.28

3.58

4.62

5.38

3. 56

3.77

4.20

3.98

5. 51

4.07

3.88

3.43

3.70

3.49

4.43

3.79

3.54

4.09

4.33

3.94

EK

0.20

0.18

0.18

0.20

0. 19

0.19

0.14

0.17

0.19

0.19

0.18

0.18

0.19

0.17

0. 18

0.18

0. 18

0. 18

0.18

0.17

0. 13

0. 16

0.18

0.18

0.16

0. 16

0. 17

0.19

0.19

0. 12

0.18

0. 17

0. 17

0. 16

0.16

0.17

0.17

0. 16

0.15

0.17

0.14

0.16

0.13

0.11

0.15

0. 15

0.14

0.14

0. 10

0. 14

0.14

0.14

0.14

0. 14

0. 12

0.13

0.14

0. 12

0.13

0.13

GL

4.01

4. 81

3.29

3.43

3.25

3.99

6.50

3.60

3.35

3.80

2.92

9.27

4.78

3.62

3.09

5.06

4.41

2.94

3.55

10.01

7.29

4.20

3.38

5.02

3.85

4.13

4.48

3.80

5.19

11.25

3.88

3.90

3.95

3.41

3.52

3.07

5.70

3.70

6.23

3. 70

3.71

4. 77

3.89

7.48

5.74

4.01

3.50

3.51

4. 17

4.15

4.09

3.42

4. 14

4.07

4.02

3.84

5.73

3.97

3.95

4.21

EKL

0.16

0.15

0.17

o.n

o.ii

0.11

0.16

0.17

0. 16

0.18

0.12

0. 15

16

17

14

0.15

0.17

0.16

0.09

0.10

0.15

0.16

0.14

0.15

0.15

0.14

0.15

0.13

0.08

0.15

0.14

0.14

0.15

0.15

0.15

0.13

0.14

0.11

0.14

0. 14

0.12

0.14

0.08

0.12

0.12

0.13

0.13

0.12

0.13

0.12

0.13

0.12

0.12

0.12

0.12

0.10

0.12

0.12

0.11

106
VOTES WORD

OPPORT

SOLELY

MASS

DISSEN

NEVERT

ARGUED

HENCE

FAILS

ARGUES

STATIN

EVER

LIKEWI

HERETO

ABLE

SEEKS

ONCE

EXISTS

COMPAR

INSTEA

INSIST

RELIES

ALLEGI

QUITE

VIRTUE

NAMELY

WEYGAN

MATTHI

SOMEWH

DESMON

VOORHI

SOMETI

PECK

FULD

FROESS

NOGC

545

4687

751

370

396

447

426

443

385

481

404

498

416

374

375

376

418

328

368

301

320

307

322

316

251

249

236

230

209

237

216

208

209

5.53

5.50

5.77

5.48

5. 50

5.47

5.43

5.21

5.52

5.43

5.47

5.52

5.41

5.37

5.15

5.32

5.38

5.42

5.29

5.36

5.28

5. 18

5.32

5. 21

5.27

4.57

4.57

5.13

4.86

4.80

5.05

4.34

4.73

4.78

EL

5.7i

5.74

5.73

5.73

5.71

5.71

5.68

5.68

5.67

5.67

5.65

5.64

5.64

5.64

5.62

5.60

5.59

5.57

5.52

5.51

5.48

5.47

5.46

5.46

5.44

5.40

5.37

5.27

5.24

5.23

5.22

5.22

5.20

5.18

PZD

I 3. 70

12.87

16.98

13.43

II .92

12.15

12.26

12.15

12.23

11.77

12.23

11.70

12.60

11.77

11.32

11.02

10.94

11.09

10.07

10.41

9.62

9.66

9.39

9.55

9.36

8.79

8.64

7.73

7.47

7.32

7.39

7.43

7.09

6.98

AVG

0.0146

0.0118

0. 1483

0.0191

0.0096

0.0117

0.0118

0.0125

0.0136

0.0112

0.0127

0.0106

0.0121

0.0107

0.0117

0.0094

0.0104

0.0121

0.0088

0.0096

0.0090

0.0088

0.0083

0.0091

0.0080

0.0050

0.0049

0.0070

0.0065

0.0059

0.0068

0.0043

0.0057

0.0062

G

5. 13

4.03

3.41

3.84

3.19

3.88

4.38

4.84

4.75

4.44

4.47

3.26

3.70

4.69

4.95

3.70

4.09

4.96

4.25

3.68

4.16

4.31

4.11

4.56

4.71

6.09

6.34

4.87

4.60

4.32

5.15

7.17

4.57

4.96

EK

0.11

0. 12

0.12

0.12

0.13

0.12

0.11

0.10

0.11

0.11

0.11

0.12

0.12

0.11

0.10

0.11

0.11

0.09

0.10

0.10

0.09

0.09

0.09

0.09

0.09

0.05

0.05

0.07

0.07

0.07

0.07

0.04

0. 06

0.06

GL

4.15

4.06

4.36

3.90

3.20

3.34

3.85

3.68

3.96

3.86

4.27

4.45

6.07

4.20

3.75

3.77

3.84

4.20

3.97

4.72

3.91

4.05

3.74

3.99

4.09

3.57

4.17

4.12

4.06

3.98

4.18

4.22

4.05

3.98

EKL

0.11

0.12

0.10

0.11

0.12

0.12

o.ii

0.11

0.11

0.11

0.10

0.10

0.09

0.10

0.10

0.10

0.10

0.09

0.09

0.09

0.09

0.09

0.09

0.09

0.09

0.09

0.08

0.07

0.07

0.07

0.07

0.07

0.07

0.07

Table VII. Sorted by EL

107
JjLJiO WUJuJ ij THE 442506 7.87

AND 128355 7.83

THAT 89026 7.80

NOT 35835 7.75

FOR 45223 7. 73

WHICH 25522 7.70

AAAAAA 2649 7.07

THIS 29490 7. 66

WAS 56044 7. 69

WITH 21624 7.64

FROM 19879 7.62

HAVE 13825 7.53

BUT 9174 7. 48

BEEN 12072 7.50

THERE 12925 7.48

ANY 13855 7.47

ARE 13721 7.46

OTHER 8966 7.43

SUCH 18195 7.50

UPON 11816 7. 46

WERE 12911 7.43

HAS 10530 7. 36

1 ONE 9388 7. 39

1 ALL 902 1 7.36

3 CASE 1526 1 7.45

1 ONLY 6218 7.33

HAD 1545 1 7.43

MAY 9510 7. 37

WOULD 9678 7.34

ALSO 5230 7. 29

UNDER 10893 7.40

10 COURT 3302 1 7.45

MADE 7999 7.32

WHEN 6875 7.28

1 FOLLOW 6076 7.28

I TS 1 106 1 7.31

2 REASON 6845 7.17

MUST 5208 7.18

AFTER 6340 7. 24

WHETHE 5173 7.22

2 QUEST I 8776 7.25

HIS 19529 7. 32

DID 6224 7.24

WHERE 5794 7.19

SHOULD 5689 7.20

DOES 4264 7.09

BEFORE 5814 7.19

COULD 5096 7.16

8 CONS ID 5288 7.15

3 TIME 8254 7.17

WITHOU 4652 7.10

FURTHE 4546 7.11

THEREF 3871 7.01

HOWEVE 3333 7.09

2 LAW 9658 7. 23

3 PRESEN 5653 7.18

1 TWO 5 130 7.11

THESE 4753 7.11

THEN 4583 7.12

THAN 4378 7.11

Table VIII.

7.65

7.6 1 7.60

7.60

7.6 1 7.56

7.87

7.59

7»55

7.51

7.51

7.44

7.3 7 7.41

7.40

7.37

7.39

7.31

7.35

7.40

7.31

7.37

7. 31

7.26

7.36

7.31

7.30

7.30

7.23

7.23

7.31

7.41

7.29

7.24

7.24

7.20

7.25

7.22

7.2 1 7.19

7.28

7.22

7.17

7.16

7.20

7.20

7.23

7.11

7.14

7.20

7.17

7.13

7.18

7.11

7.20

7.20

7.11

7.07

7.07

7.10

Sorted by

EL PZD

99.99

9 9 . 7 3 98.15

96 .97

98.07

94.41

99 .99

96.67

95.73

92.03

92 . 18

85.99

78.89

83.76

84.25

83.12

84.37

76. 17

85.80

82.93

79.91

81.76

76.40

74.78

8 4. 74

72.14

82.44

76.70

73.12

67.15

80.44

93.58

74.51

69.87

69.38

75.34

72.48

66.70

68.47

66 . 13

77.08

78.63

66.70

65.26

66.59

63.30

68.55

61 .79

63.72

70.40

63.57

61 .94

62.21

55.90

74.29

68.25

60.51

59.79

59.19'

59.38

EK

AVG

12.1192

3.4562

2.4343

0.9798

1.2529

0.6984

0.0783

0.8106

1 . 5 6 30

0.5840

0. 5456

0.3 761

0.2485

0.3 306

0. 3545

0.3703

0.3766

0.2397

0.4817

0.3232

0.3486

0.2838

0.2540

0.2361

0.4182

0. 1693

0.4205

0.2605

0.2580

0. 1410

0.2937

0.9097

0.2213

0. 1866

0. 1661

0.2888

0. 1850

0.1412

0. 1745

0. 1408

0.2395

0.5396

0. 1665

0.1562

0.1511

0. 1 175

0.1612

0. 1383

0.1379

0.2237

0. 1274

0.1230

0. 1050

0.0923

0.2554

0. 1558

0.1408

0.1275

0. 1242

0.1198

G

-0.19

0.53

0.70

0.55

1.0 3 0.64

0.42

1.15

0. 52

1.15

1.25

1.17

0.84

1.41

1.30

1.29

1.56

1.18

1.49

1.37

1.43

1.34

1.61

1.45

.64

,57

,49

45

1.43

08

82

1.64

1.60

1.54

1.30

1.71

2.15

1.83

1.62

1.69

2.17

1.55

1.55

1.64

1.89

1.80

2.12

1.59

2.06

2.55

2.02

1.92

1.43

1.47

2.34

2.26

1.59

1.97

2.04

2.23

EK

41.17

15.25

9.48

6.95

5.00

4.89

4.32

4.02

3.68

3.46

3.01

2.52

2.21

1.96

1.87

1.87

1.85

1. 79

1.78

1.76

55

51

48

46

1.43

1.38

38

38

34

33

31

26

25

20

18

13

11

08

06

04

1.03

1.03

1.03

1.03

1.02

0.96

0.95

0.95

0.93

0.92

0.91

0.91

0.90

0.90

0.88

0.88

0.85

0. 83

0.82

0.81

GL

1.87

2.14

1.92

1.90

1.87

1.79

2.55

2.45

1. 78

2.15

1.83

2.53

2.06

2.07

2.17

2.37

55

45

91

83

EKL

1.93

1.57

2.67

2.41

2.40

3.34

2. 38

1. 88

2.68

2.50

2.49

1.95

2.98

3.97

1.97

2.24

44

49

2.86

2.79

2.27

2.57

4.30

2. 83

2.52

2.43

2.45

2.11

2.63

2.58

2.68

2.17

2.39

3.44

2.25

1. 76

3.39

3.49

2.47

3.27

2.60

2.63

54

56

1.59

1.38

31.32

1.41

1.33

1.16

1.19

0.97

0.89

0.95

0.91

0.83

86

76

74

95

70

83

75

64

0.80

0.82

0.69

0.72

0.64

0.71

0.69

0.76

0.76

0.69

0.69

0.54

0.64

0.64

0.65

0.61

0.62

0.60

0.59

0.58

0.63

0.67

0.66

0.54

0.56

0.62

0.62

0.53

0.65

0.62

0.54

0.58

0.55

0.48

0.51

0.54

108
VOTES WORD NOCC E

4 FACT 4658 7.06

5 DEFEND 25773 7. 20

WHO 5241 7. 1 1 4 AFFIRM 3897 6. 89

1 PART 4746 7. 12

THEY 7042 7. 14

SAME 4992 7.05

BECAUS 3553 7.00

2 BEING 3858 7.04

HELD 3978 7.04

2 REQUIR 6103 7.06

4 CONCUR 2290 6.65

CONTEN 3888 7.02

3 FIRST 4165 7.01

9 EVIDEN 12726 7. 10

3 OPINIO 4764 7.02

THEIR 6514 7.08

2 STATED 3698 6.99

1 CAN 2822 6.93

SOME 3394 6.97

HERE 3448 6.93

MORE 3050 6.94

OUT 4389 7.00

2 CERT A I 3069 6. 87

1 PROVID 5792 7.03

7 CONCLU 3665 6.95

4 DETERM 5030 7.02

2 PLAINT 20986 7.02

AGAINS 5725 7.04

9 ACCORD 2721 6.87

SINCE 2756 6.89

1 FACTS 4095 7.00

1 BOTH 2868 6.85

4 PERSON 6980 7.01

INTO 3583 6.93

CANNOT 2467 6.74

5 APPEAR 3855 6.95

5 EFFECT 3759 6.91

INVOLV 2933 6. 56

THEM 3505 6.92

BETWEE 3231 6.84

OUR 3179 6.80

9 JUDGME 10581 7.06

CASES 3896 6. 86

MAKE 2535 6.76

RESPEC 2579 6.80

4 FOUND 3608 6.91

MATTER 4313 6.91

NOR 2099 6.70

9 NECESS 3477 6.93

HIM 5613 6.91

HAV I NG 2006 6.67

WELL 2259 6.77

WHAT 2883 6.76

WITHIN 456 1 6.85

SAID 10747 7.07

GIVEN 2766 6.80

EITHER 2033 6.71

ALTHOU 1762 6.67

1 RESULT 3328 6.85

Table VIII.

EL

7. lu

7.12

7.03

7.23

7.09

7.08

7.07

7.11

7.08

7.02

7. 10

7.30

7.09

7.04

7.02

6.98

7.02

6.99

6.94

6.93

6.97

6.95

6.99

6.96

7.02

7.02

7.01

6.94

7.06

6.96

6.93

7.01

6.88

6.94

6.92

6.92

7.00

6.92

6.90

6.89

6.87

6.83

7.17

6.90

6.84

6.82

6.98

6.96

6.86

6.93

6.85

6.86

6.83

6.79

6.97

6.93

6.82

6.78

6.77

6.86

Sorted by

PZD

60.28

71 .19

59 .64

63.53

60 .62

64.47

60.73

57.19

57.41

55.34

63.98

63.91

57.11

57.15

65.64

58.85

61.75

54.77

49. 15

50.88

52.69

49.49

57.04

50.62

60.02

53.90

59.45

57.71

61.83

49.64

48.65

55.79

46.54

60.81

51.00

46.54

57.68

52.39

47.86

49. 37

47.45

47.98

73.19

51.41

43.94

44.43

53.68

55.19

43.14

52.20

54.24

42.09

43. 14

44.80

55.56

69. 15

45.07

40.20

38 .65

48. 50

EK

AVG

0. 1249

C . 7468

C. 1416

0. 1109

0. 1287

0. 1897

0.1299

0.0999

0. 1040

0. 1058

0. 1665

0.0643

0. 1094

0. 1116

0.3461

0. 1218

0. 1756

0.0975

0.0739

0.0897

0.0938

0.0822

0. 1164

0.0830

0. 1599

0. 1010

0.1314

0.6097

0. 1605

0.0745

0.0753

0.1137

0.0771

0.1897

0.0952

0.0694

0. 1045

0. 1018

0.0789

0.0943

0.0879

0.0833

0.3119

0.1062

0.0681

0.0678

0.1017

0.1166

0.0581

0.0937

0.1531

0.0548

0.0592

0.0725

0. 1294

0.2803

0.0744

0.0532

0.0487

0.0911

G

2.10

1.34

1.89

2.26

2.57

2.45

2.47

2.04

2.13

1.92

2.34

2.45

2.14

2.30

1.64

2.05

2.19

2.37

1.61

1.97

1.92

1.98

3.00

2.20

2.56

2.50

3.04

1.25

2.56

2.12

1.76

3.05

1.87

2.61

2.51

2.06

3.97

2.86

2.29

2.56

2.33

2.15

3.01

2.58

2.35

1.99

2.73

3.11

1.94

3.31

2.49

2.18

2.87

2.52

2.63

4.45

2.27

1.96

1.78

3.50

EK

0.80

0.79

0.79

0.78

0.78

0.77

0.76

0.75

0.75

0.75

0.74

0.73

0.71

0.71

0.71

0.71

0.70

0.68

0.67

0.67

0.66

0.66

0.65

0.65

0.64

0.64

0.64

0.64

0.63

0.62

0.62

0.60

0.59

0.57

0.57

0.57

0.56

0.56

0.56

0.56

0.55

0. 55

0.54

0.54

0.54

0.54

0.53

0.53

0.53

0.52

0.52

0.51

0.51

0.51

0.50

0.50

0.50

0.50

0.50

0.49

GL

2.40

2.43

3.51

2.61

2.85

3.52

3.32

2.28

2.89

2.83

4.53

2.51

2.24

3.27

3.09

4.63

3.29

3.69

2.68

4.84

3.12

2.76

6.13

3.90

3.62

2.52

3.95

2.24

3.13

2.92

2.78

2.90

2.81

5.09

3. 14

2.46

9.43

7.29

2.99

4.37

2.83

4.84

4.08

3.22

3.17

3.71

3.L6

4.12

2.78

4.91

6.64

2.07

3.49

3.76

3.59

6.83

3. 10

3. 10

2.66

3.97

EKL

0.54

0.53

0.44

0.70

0.52

0.45

0.48

0.58

0.52

0.47

0.47

0. 86

0.56

0.46

0.43

0.37

0.42

0.42

0.44

0.39

0.43

0.45

0.37

0.42

0.42

0.49

0.40

0.43

0.46

0.45

0.43

0.46

0.39

0.33

0.39

0.45

0.32

0.34

0.40

0.36

0.38

0.31

0.49

0.38

0.37

0.34

0.43

0.38

0.40

0.35

0.29

0.43

0.36

0. 32

0.41

0.27

0.35

0.35

0.37

0.34

109
VOTES WORD NOGC E

4 CIRCUM 254 3 6.75

EVEN 1964 6.64

2 REVERS 2857 6.66

5 STATUT 7283 6. 89

SEE 4704 6.93

6 RIGHT 5447 6. 76

4 GENERA 5262 6.87

5 SU8JEC 2855 6.70

NOW 2384 6.60

THOSE 2527 6.73

7 TRIAL 9898 6.97

SET 2964 6.71

4 SUFFIC 2484 6.72

2 PROVIS 4479 6.80

1 ESTABL 2947 6.74

5 DIRECT 5706 6.95

END 6422 6.81

5 CAUSE 4463 6.77

WHILE 2749 6.82

SHALL 6240 6.81

OVER 2622 6.72

ENTERE 2920 6.78

THEREO 2640 6.69

UNTIL 2347 6.65

3 I ND I C

1901 6.64

THUS 1622 6.58

6 RECORD 6093 6.91

2 PURPOS 4138 6. 76

2 PART IC 238 1 6.48

4 PRIOR 2379 6.69

L APP 4769 6.74

3 SUSTAI 2600 6.65

1 PROCEE 5021 6.79

2 ALLEGE 3766 6.72

I THREE 2437 6.70

6 ACTION 8248 6.94

2 STATE 9231 6.85

2 INCLUD 2632 6.71

ABOUT 3228 6.65

MIGHT 1734 6.57

UNLESS 1520 6.54

GROUND 2629 6.68

2 SECTIO 10226 6.83

I ENTITL 2141 6. 53

6 AUTHOR 4898 6.78

1 DENIED 2053 6. 30

TAKEN 2518 6.67

ANOTHE 1881 6.57

ITAL 11360 6.67

FOL 5682 6.67

3 SUBSTA 2527 6.62

HEREIN 2599 6.23

1 EACH 3332 6.68

DURING 2216 6.58

2 INSTAN 1867 6. 54

3 FIND 1954 6. 51

1 CONTAI 2096 6.55

ABOVE 1812 6.40

1 OWN 1857 6.53

2 BASED 1605 6. 38

EL

,75

75

93

80

88

86

82

6.81

6.80

6.77

6.98

6.84

6.81

6.77

6.72

6.92

6.71

6.90

6.85

6.73

6.71

6.87

6.75

6.70

6.70

.65

.98

.76

76

Table VIII.

6.74

6.72

6.89

6.84

6.81

6.73

6.92

6.80

6.76

6.65

6.63

6.63

6.77

6.76

6.69

6.81

6.77

6.76

6.65

6.57

6.57

6.71

6.70

6.69

6.62

6.60

6.66

6.65

6.63

6.60

6.56

Sorted by

PZD

41.94

38.80

46.96

53.15

5 5.00

54.24

52.92

45.48

43.29

42.43

62.85

46. 54

42.92

47.18

44.46

58.62

51 .86

54.28

46.31

49. 18

40.99

48 .58

41 .60

39.2 2 37.67

34.80

60.51

49. 30

42. 12

40.88

44.92

46.24

55.19

47. 86

41 . 18

64.55

62 .06

43.41

41 . 10

34.27

33.82

44. 16

55.75

38.42

52.32

40.39

43.07

36. 35

45.18

45 .18

41 .60

41.75

43.90

36.50

34.88

37.75

38.12

35.18

34.99

32.84

EK

AVG

0.0679

0.0509

0.0842

0. 1985

0. 1297

0. 1464

0. 1338

0.0784

0. 0629

0.0642

0.2884

0.0798

0.0708

0. 1251

0.0788

0. 1575

0. 1570

0.1255

0.0751

0. 1705

0.0701

0.0873

0.0697

0.0628

0.0499

0.0427

0. 1675

0. 1096

0.0625

0.0654

0. 1292

0.0753

0. 1373

0.1091

0.0677

0.2329

0.2417

0.0716

0.0882

0.0465

0.0418

0.0728

0.2858

0.0591

0.1319

0.0580

0.0697

0.0500

0.2755

0. 1378

0.0693

0.0670

0.0859

0.0609

0.0494

0.0519

0.0578

0.0483

0.0502

0.0431

G

2.08

2.09

2.65

2.26

2.95

2.91

3.11

2.72

2.79

3.12

2.75

3.36

2.35

2.55

3.00

5. 12

3.07

2.98

5.29

2.77

2.40

3.29

2.61

2.31

2.45

2.08

5.25

3.99

3.17

2.87

2.51

3.40

3.56

3.04

3.19

3.64

3.06

3.86

2.68

2.40

2.32

3.25

2.91

2.60

4.35

2.91

3.27

2.97

3.12

3.12

3.48

3. 17

4.53

2.73

2.58

3.11

3.35

2.94

2.91

2.60

EK

0.49

0.49

0.48

0.48

0.47

0.47

0.47

0.46

0.46

0.46

0.45

0.45

0.45

0.45

0.45

0.44

0.44

0.43

0.43

0.43

0.43

0.42

0.42

0.42

0.42

0.42

0.41

0.41

0.41

0.41

0.41

0.40

0.40

0.40

0.40

0. 39

0.39

0. 39

0.39

0.39

G.39

0.38

0.38

0.38

0.37

0.37

0.37

0.37

0. 37

0.37

0.36

0.36

0.36

0.36

0.36

0.35

0.35

0.35

0.35

0.35

GL

2.94

3.06

3.60

4. 39

3. 89

3. 87

5.01

3.64

3.10

3.52

2.96

3. 72

3.24

3.69

17.95

6.63

6. 84

4.08

4. 31

4. 34

3.50

4.02

3.06

3.46

3.59

2.88

4.95

6. 33

3.48

3. 12

3.31

2.63

6.15

3.37

3.87

4.77

4.64

3.68

3.45

2.78

2.95

5.73

4.29

3.68

4.61

2.72

4.04

3. 17

7.32

7.39

4.62

5.86

5.12

4.42

3.01

3.70

5.43

3.03

3.93

3.70

EKL

0.33

0.35

0.43

0.29

0.33

0.32

0.28

0.33

0. 34

0.33

0.41

0.35

0.36

0.30

0.18

0.29

0.22

0.34

0.35

0.27

0.29

0.34

0.33

0.30

0.31

0.31

0.35

0.25

0.32

0.32

0.29

0.41

0.26

0.33

0.30

0.31

0.25

0.31

0.27

0.30

0.30

0.29

0.27

0.30

0.28

0.35

0.31

0.29

0.19

0.19

0.27

0.25

0.25

0.26

0.28

0.28

0.25

0.29

0.27

0.26

110
VOTES

77

1

WORD

SPECIF

EXPRES

C ONCER

THOUGH

ILL

FILED

CLEAR

NOTHIN

CITED

SITUAT

ISSUE

ACT

SHOW

THEREA

REGARD

ORDER

NEW

RULE

SECOND

CALLED

YEARS

DECIDE

LATER

PAGE

APPEAL

MOTION

DECISI

THROUG

CONSTR

RELATI

APPARE

S IMILA

CLEARL

ERROR

PART I

BROUGH

DIFFER

VIEW

FAILED

GIVE

MANY

CONSTI

OHIO

MOST

SEC

RECEIV

AMOUNT

TAKE

OPERAT

APPLIE

NE I THE

THERE I MANNER

ITSELF

EXCEPT

WILL

FINDIN

CONDIT

BASIS

DAY

Table

NOCC

2900

2022

1797

1301

8605

5362

1537

1275

140 I 1358

3113

5147

1649

1342

1466

6773

4744

4090

2415

1618

2601

1409

1426

3218

9096

6621

3988

1954

3805

2530

1334

1243

1145

3841

3496

1534

1714

140 6

1442

1490

1117

4132

8519

1051

6808

2801

3110

1484

4207

1264

930

1068

1259

993

3589

7140

3437

2779

1500

2189

VIII,

E

6.65

6.51

6. 57

6.43

6.49

6.67

6.52

6.24

6.41

6.42

6.61

6.65

6.36

6.40

6.39

6.78

6.68

6.56

6.53

6.40

6.53

6.41

6.43

6.47

6. 80

6.71

6.52

6.52

6.58

6.54

6.43

6.38

6.31

6.56

6.55

6.50

6.46

6.35

6.29

6.32

6.27

6.41

6.49

6.25

6.65

6.52

6.49

6.38

6.52

6.25

6. 16

6.13

6.30

6.25

6. 58

6.84

6.56

6.46

6.41

6.41

EL

6.68

6.61

6.59

6.54

6.46

6.91

6.57

6.55

6.54

6.49

6.66

6.59

6.59

6.55

6.52

6.77

6.72

70

,61

57

56

50

6.47

6.45

7.06

6.84

6.69

6.56

6.55

6.53

6.53

6.46

6.45

6.66

6.59

6.59

6.55

6.48

6.48

6.45

6.38

6.49

6.35

6.31

6.62

6.57

6.52

6.47

6.45

6.40

6.38

6.38

6.37

6.33

6.82

6.74

6.59

6.47

6.47

6.46

Sorted by

PZD

42 .28

36.01

34.76

30 .46

32 . 88

55.26

33.48

30.65

30.95

29.40

42.88

45. 56

33.89

31.03

30.80

58.32

48.09

47. 18

38.50

32.76

37.10

29.89

29.48

33. 71

77.61

53.90

46. 58

34.61

40.50

37. 10

30.84

28.61

27.67

44. 80

41.71

33.74

33.14

30.95

30.31

29.78

25.82

42.99

34.39

24.95

49.60

39.10

37.56

30.35

39.56

27.63

24.87

25.70

27.29

24.38

49.79

62.55

41 .56

35.52

30.76'

34. 16

EK

AVG

0.0790

C .0546

C .0468

0.0340

0.2551

0. 1589

0.0425

0.0345

0.0390

0.0368

0.0831

0. 1370

0.0470

0.0389

0.0380

0.1918

0. 1295

0.1055

0.0656

0.0444

0.0687

0.0381

0.0387

0.0815

0.2637

0. 1942

0.1070

0.0531

0. 1054

0.0662

0.0364

0.0339

0.0304

0.1051

0.0960

0.0460

0.0466

0.0375

0.0414

0.0399

0.0286

0. 1058

0.2212

0.0273

0. 1929

0.0764

0.0869

0.0407

0.1145

0.0351

0.0252

0.0279

0.0329

0.0260

0. 1046

0. 1944

0.0995

0.0760

0.0412

0.0607

G

3.75

3.21

4.40

2.57

1.95

4.09

3.35

2.76

2.52

2.40

3.76

3.30

3.26

2.78

3.05

3.68

3.77

4.23

3.97

4.43

3.24

2.48

2.75

2.83

4.94

3.78

4.00

3.87

3.38

3.61

3.26

2.91

2.81

3.69

3.86

4.00

3.96

4.33

3.32

3.06

2.52

3.48

2.35

2.65

3.75

6.76

3.85

3.85

3.54

2.95

2.65

2.72

3.46

2.40

5.95

5.49

4.00

3.52

5.82

3.92

EK

0.34

0.34

0.34

0.34

0.34

0.33

0.33

0.33

0.33

0.33

0.32

0.32

0.32

0.32

0.32

0.31

0.31

0.31

0.31

0.31

0.31

0.31

0.31

0.31

0.30

0.30

0.30

0.30

0.30

0.30

0.30

0.30

0.30

0.29

0.29

0.29

0.29

0.29

0.29

0.29

0.29

0.28

0.28

0.28

0.27

0.27

0.27

0.27

0.27

0.27

0.27

0.27

0.27

0.27

0.26

0.26

0.26

0.26

0.26

0.26

GL

5.03

4. 18

3.67

2. 82

3.00

3.46

5. 39

2.84

3.08

3.07

4.96

6.21

3.21

2.92

3.05

11.48

4.33

12.48

5.63

3.42

4. 19

3.99

3.52

5.57

5.35

3.36

5. 57

4.00

4.65

5.77

3.32

3. 18

3.28

4.33

4.47

3.64

3.56

7.01

3.79

3.67

2.73

7.53

5. 51

6.00

4.50

5.74

3.75

3.52

4.52

3.46

2.44

3. 38

6.32

3.32

4.72

12.86

3.90

3.88

5.60

9.83

EKL

0.25

0.26

0.26

0.28

0.24

0.36

0.24

0.29

0.27

0.25

0.23

0.20

0.28

0.28

0.26

0.19

0.26

0.20

0.23

0.27

0.23

0.25

0.24

0.19

0.33

0.33

0.23

0.24

0.21

0.20

0.26

0.24

0.24

0.24

0.22

0.27

0.25

0.20

0.23

0.23

0.23

15

17

18

21

21

0.22

0.23

0.18

0.22

0.25

0.23

0.19

0.22

0.30

0. 15

0.23

0.21

0.21

0.17

111
VOTES WORD NOCG

5 SEVERA 1243

1 TRUE 1140

HOLD 1033

SAY 1088

MERELY 936

CONSIS 941

3 APPLIC 4168

2 ADDITI 1708

3 DUE 1937

1 POINT 1487

OTHERW 1095

9 ATTEMP 1404

7 TEST I M 3650

1 ENTIRE 1350

FORTH 1458

2 TERMS 1583

1 INTEND 1333

THERET 1022

DISCUS 1034

3 SUPPOR 3151

1 USED 2650

8 CHARGE 4622

4 COMPLE 1709

11 PRINCI 2158

3 REFERR 1309

5 FAILUR 1630

2 SUBSEQ 1263

SHOWN 1106

1 TESTIF 3484

BEL I EV 1176

10 JURY 5530

TOOK 1080

VERY 888

3 DISTIN 997

RATHER 917

2 CONTRO 2941

10 COUNTY 6245

7 CONTRA 8033

1 RENDER 1657

3 APPELL 14543

2 DATE 1983

6 REMAIN 1592

1 FAVOR 1249

SHOWS 1078

3 PROPER 5913

ORDERE 1180

5 ADM ITT 1667

LONG 1047

OBTAIN 1498

BECOME 1158

1 THINK 1035

SUPRA 2573

1 PAID 2316

5 RECOGN 1033

POSSIB 1018

3 COMPLA 3971

WAY 1771

5 ANSWER 3398

STATES 2343

MAKING 1060

EL PZD

6. 32

6.23

6.15

6.26

6.21

6. 19

6. 58

6.39

6,40

6.35

6. 14

6.05

6.42

6.30

6.25

6.33

6.29

6.05

6.22

6.65

6.45

6.48

6.30

6.46

6.24

6.16

6.25

6. 15

6. 35

6.22

6.41

6. 15

6. 15

6. 14

6. 15

6.48

6.62

6.56

6.30

6.53

6.31

6.35

6.22

6. 16

6.40

6.14

6.32

6.23

6. 18

6.07

6. 18

6.29

6.25

6. 10

,18

,40

,21

42

,38

36 27.25

36 26.23

35 24.61

34 25.44

32 23.78

31 23.66

6.60 47.37

6.49 32.12

6.47 32.08

6.42 29.48

6.42 27.18

6.42 29.18

6.41 34.65

6.41 28.53

6.40 28.80

6.39 28.46

6.39 27.63

6.35 24.95

6.31 24.34

6.67 46.35

6.58 38.16

6.47 40.69

6.45 31.40

6.43 34.61

6.43 28.65

6.43 30.16

6.37 26.99

6.36 25.74

6.35 31.74

6.34 25.67

6.31 34.2 7 6.28 24.46

,22 2L 93

22 22.68

21 22.00

55 39.93

52 52.43

6.49 52.96

6.45 31.74

6.44 50.16

6.41 31.37

6.38 30.46

6.37 26.87

6.35 25.25

6.34 36.91

6.33 26.23

6.32 28.87

6.32 24.80

6.30 27.40

6.30 25.36

6.28 23.53

6.25 29. 21

6.25 28

6.25 23

16

51

6. 19

,23 22.98

,45 37.44

,45 32.91

,41 39.33

,33 33.37

,33 25.14

AVG

0.0331

0.0309

0.0270

0.0294

0.0248

0.0260

0.1134

0.0453

0.0542

0.0407

0.0307

0.0376

0. 1010

0.0369

0.0391

0.0424

0.0361

0.0278

0.0267

0.0855

0.0734

0.1234

0.0455

0.0564

0.0341

0.0459

0.0363

0.0303

0.0969

0.0322

0. 1470

0.0302

0.0230

0.0265

0.0246

0.0849

0. 1787

0.2158

0.0464

0.3877

0.0555

0.0428

0.0364

0.0297

0. 1591

0.0324

0.0436

0.0280

0.0397

0.0320

0.0298

0.0636

0.0616

0.0261

0.0272

0. 1 136

0.0472

0.0913

0.0582

0.0282

3.47

3.33

2.49

94

46

47

97

5.06

4.13

4.43

4. 16

4.42

3.30

5.20

3.68

3.43

3.14

3.03

2.85

7.06

5.62

3.96

4.76

6.01

8.37

3.81

3.67

3.38

3.53

3.33

3.35

3.21

2.80

2.77

3.00

5.05

5.00

3.98

3.94

3.05

3.97

4.99

3.45

3.55

3.62

3.50

3.82

3.39

3.28

3.89

3.00

3.34

3.21

3.33

3.04

4.27

6.65

5.64

6.26

4.11

EK

0.26

0.26

0.26

0.26

0.26

0.26

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.24

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.22

0.22

0.22

0.22

0.22

GL

7.53

4.42

3.24

3.71

2.82

3.02

8.13

4.68

3.79

4.24

3.79

7.93

3.88

6.76

4.54

3.35

4.27

3.31

3. 19

9. 79

4.18

4.95

5.48

7. 85

5.55

4.43

3.97

3.23

3.72

3.34

4.31

4.38

3.45

4. 15

3.67

5.00

8.51

7.29

6.39

5.26

4.85

7. 12

4.09

3.06

5.71

6.13

5. 59

3. 84

5.62

3.96

3.20

4.77

4.69

3.94

3.70

4.90

10.08

9.44

8. 54

3.75

EKL

0. 18

0.20

0.22

0.21

0.22

0.21

0.16

0.22

0.22

0.21

0.23

0.19

0.20

0.20

0.20

0.21

0.21

0.22

0.21

0.18

0.23

0.18

0.20

0. 16

0.21

0.21

0.21

0.22

0.19

0.21

0.17

0.19

0.19

0.18

0.18

0.20

0.14

0.15

0.19

0. 16

0.19

0.16

0.21

0.22

0.15

0.18

0.17

0.20

0.17

0.19

0.20

0.15

0.16

0.18

0.18

0.19

0.16

0.13

0.13

0.21

Table VIII. Sorted by EK

112
VOTES

14

13

24

5

3

8

6

WORD

PREV I 0

NATURE

PURSUA

EVERY

PLACE

COURSE

CONTIN

ORIGIN

CQRREC

SOUGHT

HEARIN

DUTY

DONE

RAISEO

SUPREM

LANGUA

HOLDI

KNOWN

LESS

TOGETH

ARGUME

STATEM

GRANTE

RIGHTS

PARTY

INTERE

PUBLIC

FAR

HER

VARIOU

RELATE

LEAST

JUDGE

COMMON

PETITI

OVERRU

LEGAL

REFUSE

EXISTE

PREVEN

SHOWIN

NEVER

LATTER

WHOM

THEREB

MUCH

AGREE

TIMES

APPLY

HOW

EMPHAS

USE

CITY

OCCURR

AGAIN

OBVIOU

REV

BECAME

HEARD

JUSTIF

Table

NOCC

1043

1185

1039

922

1881

1500

2382

2053

1358

1132

2525

1873

1079

1050

1904

1492

1008

1083

923

861

1528

2732

1574

2108

2643

3637

4658

923

7548

815

839

766

4000

4042

7623

1644

1650

1286

1029

956

829

976

833

832

712

693

707

751

806

739

1012

3852

5969

1248

766

645

1484

734

903

885

VIII,

E

6. 16

6.16

6.08

6.11

6. 36

6.22

6. 37

6.23

6.14

6. 11

.28

,25

,09

,00

6. 16

6.22

6.05

6. 12

6.08

6.04

6.26

6.32

6.25

6.30

6.26

6. 36

6.33

6.11

6.30

5.99

5.92

6.00

6.52

6.46

6. 19

6.23

6.25

6. 14

6.06

6.00

5.78

6.01

6.04

6.00

5.99

5.99

5.91

5.95

6.00

5.93

5.96

6.29

6.24

6.05

6.00

5.87

6.07

5.81

5.97

5.90

EL

6.31

6.31

6.24

6.22

6.45

6.45

6.40

6.39

6.38

6.33

6.31

6.30

6.28

6.28

6.24

6.23

6.20

6.17

6.17

6.16

6.37

36

34

33

33

6.32

6.3C

6.24

6.2C

6.12

6.12

6.11

6.64

6.48

6.44

6.42

6.30

6.22

6.17

6.16

6.16

6.15

6.14

6.13

6.11

6.11

6.10

6.09

6.08

6.01

6.00

6.27

6.23

6.11

6.1 1 6.09

6.08

6.08

6.07

6.07

Sorted by

PZD

24.57

25.48

23.17

22.31

32.27

30.53

34. 35

32.01

28. 57

25.44

31.59

28.35

24.57

23.93

27.44

25.78

22.76

22.19

21.63

20.91

28.69

34. 16

28.35

30.38

31 .93

35.33

35.78

22.61

31.89

19.96

20.04

19.40

46.84

42.58

40. 39

30.46

28.57

24.49

22.08

2L.44

20.53

21.32

20.23

20.08

19.02

19. 13

18.98

19.21

19.63

17.89

19.59

36. 12

38.05

21.78

19.32

18.23

22.72

13.61

19.93

19.85

EK

AVG

0.0277

0.0313

0.0271

0.0244

0.0528

0.0421

0.0634

0.0558

0.0370

0.0316

0.0716

0.0506

0.0282

0.0290

0.0474

0.0411

0.0265

0.0285

0.0250

0.0222

0.0429

0.0720

0.0425

0.0581

0.0726

0.0944

0. 1226

0.0247

0.2095

0.0214

0.0233

0.0206

0. 1181

0.1171

0.2198

0.0456

0.0423

0.0351

0.0286

0.0265

0.0227

0.0254

0.0235

0.0228

0.0192

0.0187

0.0187

0.0201

0.0212

0.0191

0.0246

0. 1059

0. 1706

0.0347

0.0209

0.0187

0.0446

0.0196

0.0241

0.0235

G

3.93

3.80

2.92

3.05

6.46

6.86

5.85

4.38

4.35

3.80

4.03

3.82

3.94

3. 56

3.73

3.66

3.62

3.59

3.44

3.31

5.01

4.77

4.97

5.59

4.28

5.26

4.86

4.89

4.05

4.01

3.10

2.98

10.30

5.85

3.73

4.78

7.41

4.26

5.05

3.86

3.37

4.03

3.47

3.43

3.22

3.85

3. 50

3.18

3.14

3.23

3. 16

4.86

3.90

3.73

4.64

3.36

3.55

3.61

3.35

3.52

EK

0.22

0.22

0.22

0.22

0.21

0.21

0.21

0.21

0.21

0.21

0.21

0.21

0.21

0.21

0.21

0.21

0.21

0.21

0.21

0.21

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.19

0.19

0. 19

0. 19

0.19

0.19

0.19

0.19

0.19

0.19

0.19

0.19

0.19

0.19

0. 19

0.19

0.19

0.19

0.19

0. 18

0.18

0.18

0. 18

0. 18

0.18

0. 18

0.18

0.18

GL

3.68

4.10

3.93

3.79

5.21

4.36

10. 10

5.63

4.34

4.23

6. 14

5. 09

4.53

3.95

6.65

5.17

4.43

4.34

3.99

3. 86

4.22

5.32

5.70

4.76

5.91

5.71

5.07

4.79

4.75

3.74

4.01

3.43

6. 80

7.01

5.82

4.35

9.77

4.13

4.18

3.57

3. 12

4. 18

3.63

3.68

3.25

3.99

3.35

3. 80

4.78

3.80

5. 19

7.72

5.82

4.81

3.29

2.92

9.27

3.09

5.06

4.41

EKL

0.20

0.19

0.18

0.18

0.19

0.21

0.14

0. 18

0.20

0.20

0.15

0.17

0.18

0.19

0.14

0.16

0.17

0.16

0.17

0.17

0.19

0.16

0.17

0.17

0.16

0.15

0.15

0.18

0.14

0.16

0. 16

0.17

0.20

0.16

0.18

0.20

0.14

0.17

0.16

0.17

0.18

16

17

17

17

17

17

16

0.15

0.15

0.13

0.12

0.13

0.15

0.17

0.18

0.12

0.17

0.14

0.15

113
VOTES WORD NOCC E EL PZD

1 STILL 660 5.06 6.07 18.08

SUGGES 782 5.94 6.06 18.68

WHOSE 655 5.89 6.04 17.70

OCCASI 742 5.95 6.03 18.38

COME 663 5.90 6.00 17.40

5 PERMIT 2869 6.35 6.49 39.63

4 CODE 4152 6.21 6.18 29.55

CLAIME 921 5.97 6.17 21.44

7 OFFICI 4060 6.26 6.12 33.93

HI MS EL 864 5.95 6.10 19.85

LIKE 738 5.93 6.08 18.87

5 COMPAN 4677 6.19 6.05 32.65

NOTED 710 5.88 6.02 18.04

BEYOND 754 5.87 5.99 17.74

MERE 654 5.82 5.99 17.02

MAKES 565 5.73 5.98 16.27

PUT 719 5.88 5.96 17.40

AMONG 579 5.83 5.93 15.81

3 DISMIS 2755 5.96 6.48 35.90

6 COURTS 2033 6.28 6.36 31.21

PLACED 781 5.88 6.05 18.91

READS 769 5.89 6.03 18.30

MENTIO 694 5.91 6.02 17.89

SEEMS 647 5.88 5.98 16.87

2 ESSENT 651 5.83 5.98 16.76

FOREGO 626 5.73 5.96 16.64

7 VALID 768 5.83 5.92 17.06

2 RETURN 2074 6.24 6.32 31.48

5 OBJECT 2703 6.27 6.31 32.50

7 REVIEW 2347 6.02 6.30 32.72

5 REQUES 1941 6.11 6.29 29.44

9 COUNSE 3030 6.22 6.27 32.54

9 CLAIM 2565 6.24 6.24 32.27

5 EXAMIN 3117 6.19 6.23 35.56

1 STAT 1245 5.90 5.93 19.10

DOING 625 5.71 5.89 16.04

1 APPROX 704 5.79 5.87 15.77

1 DAYS 1500 6.05 6.22 24.99

13 NOTICE 2855 6.04 6.18 30.76

13 JURISD 3056 6.00 6.10 29.67

FULLY 591 5.74 5.93 16.00

ALONE 536 5.73 5.87 14.79

DIFFIC 578 5.72 5.87 15.06

REACHE 539 5.63 5.86 14.91

2 QUOTED 591 5.60 5.85 15.13

1 CONCED 485 5.58 5.83 14.00

NONE 506 5.58 5.82 14.23

ALREAD 542 5.68 5.80 14.08

2 WHOLE 651 5.74 5.78 14.87

8 SERVIC 3855 6.04 6.05 29.63

WHERE I 560 5.60 5.92 15.66

3 C ARE FU 453 5.42 5.79 13.51

ADDED 587 5.62 5.77 13.96

MOVED 492 5.61 5.75 13.40

NEVERT 370 5.50 5.71 11.92

8 ASSIGN 2654 6.00 6.12 29.82

8 RESPON 2872 5.94 6.00 29.21

RELIED 487 5.62 5.80 13.89

1 DESIRE 507 5.38 5.78 13.74

SOLELY 441 5.50 5.74 12.87

Table VIII. Sorted by EK

AVG

3. 0 1 76

0.0208

0.0179

0.0206

0.0173

0.0820

0. 1146

0.0261

0. 1032

0.0241

0.0198

0. 1180

0.0182

0.0209

0.0170

0.0151

0.0197

0.0152

0.0790

0.0553

0.0208

0.0220

0.0191

0.0179

0.0173

0.0163

0.0207

0.0589

0.0742

0.0676

0.0545

0.0868

0.0735

0.0831

0.0383

0.0167

0.0179

0.0447

0.0853

0.0812

0.0159

0.0152

0.0155

0.0139

0.0149

0.0140

0.0136

0.0141

0.0169

0.1114

0.0155

0.0118

0.0144

0.0149

0.0096

0.0715

0.0772

0.0134

0.0143

0.0118

G

3.47

3.46

3.34

3.38

3.24

6.17

4.17

4.84

4.82

5.07

4.09

4.27

3.47

3.35

3.36

3.28

3.40

3.05

5.16

9.19

4.15

3. 56

4.96

4.19

3.67

3. 55

3.58

8.81

8.66

5.34

7.47

6.05

5.91

7.01

3.51

3.56

3.77

6.03

5. 70

4.48

4.28

4.20

3.98

4.07

3.88

3.43

3.70

3.49

3.54

5.82

4.62

3.79

4.33

3.94

3.19

6.48

6.24

4.43

4.09

4.03

EK

0.18

0.18

0.18

0.18

0.18

0.17

0.17

0. 17

0.17

0.17

0.17

0.17

0. 17

0.17

0. 17

0. 17

0.17

0. 17

0. 16

0. 16

0.16

0. 16

0. 16

0.16

0.16

0.16

0. 16

0.15

0. 15

0.15

0.15

0. 15

0.15

0.15

0. 15

0.15

0. 15

0. 14

0.14

0.14

0.14

0. 14

0. 14

0. 14

0.14

0.14

0.14

0. 14

0. 14

0. 13

0. 13

0.13

0.13

0.13

0. 13

0. 12

0.12

0.12

0.12

0.12

GL

2.94

3. 55

3.38

5.02

3. 88

6. 36

5.98

3.94

18.75

3.60

3.62

10.01

4.48

3.90

3.95

3.07

5.70

3.70

5.01

5.77

4.20

3. 85

4.13

3.41

3.52

3.70

4.77

9.23

5.60

7. 80

5.99

5.28

7.77

8.63

6.23

5. 74

4.01

3.91

6.77

6.50

3.71

3. 50

3.51

4. 15

4.09

3.42

4. 14

4.07

5.73

7.29

3. 89

3. 84

3.95

21

20

19

EKL

0.17

0.16

0.16

0. 14

0.15

0.17

0.13

0.17

0.07

0.16

0.16

0.09

0.14

0.14

0.14

0.15

0.13

0.14

0.20

0.17

0.15

0.15

15

15

0.15

0.14

0.12

0. 14

0.15

0.13

0.15

0.14

0.12

0.11

11

12

0.12

0.17

12

11

11.25

4. 02

3.97

4.06

0. 14

0.13

0.13

0.13

0.12

0.13

0. 12

0. 12

0.10

0.10

0.14

0. 12

0.12

0.11

0.12

0.11

0.08

0.12

0.12

0.12

114
VOTES WORD NOCC E EL PZD

2

U A c c HO o 1 c3 • 77/ / c 77 lo.vo

4

n T c c C M 7cni1 c. A. w 3 • 1

C.3.(37-4 11U.H3A 7

a l? a i ic n 'JO i 3 > o

C. A.7 c. 7 1 3 . 1 1 1 A • 1 3

1 T v P i.r T A A A a c 3 3 • 3 AC A A 3 • O *t 11 7 <") uMtKt c d p t1 n

AQ Q C. A

1; A A 3 • 6 H 19 A A 1 A . O J

c3 cuni n

AAA 7 c. Qr c. a Q 73A7 C A • 3U

1 UrrUI\ flDDflOT1 C A C C\3i33C9 c. 7 c. 3 . / 3

1 7 7A 1 3 • f U ucwrc rltrMU t HHA A 71 C. A 5 3.00 c a a 17 "> A 1 A . A 6

ADTIlCC 4*» JC C 9 3 « 3 2C AT 3*67 12.23

C T A T T NJ O 1 ft 1 1 IN 7 a c. JOJ 1 A.7 3 • J

C A 7 3 • O / 11 77

c v n is AP 1 *to 1

C. A.7 3 • 1- ( c.3 • ASO 3 17 7 7

1 AQLC f\3 • 773 1 C. A A 1I

1L • 77I 1 3(3 C. 7 7 3 • 32 C A A 3 . O U

11 A 7

1 1 . U 2 cyrcTC t A 1 5 1 5 3 / O

c 7 a 3 • 3 O

C CO 3.37 1 A Q / I U • 74

Clip c. AQ c. a 7 3 • O / 17 A A

I I . U O

C A T 1 C r A l Li All *r26 C 7 1 3 . 2 1

C / 0 3.00 19 1 C

5 t C Is j 3/7 c. 1 c3 • 1 3

K A 7 3 . O A 11 7 7 11. 32

T M C T P A 1 INo 1 C A 3 a 0

c.7 . 7A Q7 3 • 3 A

11U.U A A 71 T M C ICT 7(LQ 30 0

C 7 A 3 • 36 3.31 1 A /. 1 1U . 4 i C

rU nU rlm dr Aa ur\ A.HID1 O c. A 7 3.31 11 A Q 1 1 . U 7

ii DPI IPC 3U 1

c. 7 a 3 • A O 3CT • AD*f O O A 7 7 . D 2

11A 1 1 PPT 7323 nJ R 1 S3 t% A 7 O A A 7 • O O UU 1 i t /

c. 7 7 R3itOA A Q7.3/7 0

2

If I DTI 1 C V 1 K 1 U t 73A0 92

7 1 3 • A 1 3«HOA A 7.33

M A MCI V iNAMtL T 7 1 A C.3.2/97 A A O7.307 A iUntWH O 1 A L JO C3.1319 3 . A771 71.(37 7 ZJU A Q A C. OA 3 • A *t 7 A 7

1 V UUKn 1 TAQ iU7 a a a 3 7 3 . A 3 7 7 7 ( . 3 A

o Uric 1 1 L3 7 7 J 1 c. a k 3 • U 3 3 . A77A 7( . 7Q37

rULU C. J a At 7 7 • 1 3

E.3

9 0 . A U 7/ • OQU 7 2uy A 7 a 4 • ( O

c;3 • la1 0

a o a O • 7 0

l1 ucvr A M a37 C. 11 AH.3l<i7 3

c; . inH u aO • 7QI 7 2 MATTHI 249 4.57 5.37 8.64

1 PECK 216 4.34 5.22 7.43

Table vrir. Sorted by EK

AVG G EK GL EKL

0 • I m u 3

"3 J \ A 1 "i H # 36 A0.101 A

1 Q /. on A 1 O U • 1 £O OA 3 • VU Au 11 . 1 I

1" A 1 1 7 O O Q

3 • O O A IO Urn LZ 3 • 3h A0.1217

l.ulUo 3*^6 A IO u • 1 Z

A A C A 1 A O • 1 J

r\ f> 1 *5 1 U. Ul<ll "T A A TO u « 1 1

A A "7 A AO 0 . 07

U • 1 6 -> 3 D * 5 0 A 11 0*11 "7 A Q a A a 0 • OB

A A 1 A A U • U i HO 3* 13 A 11 Well A 1 C A 11 0 • 1

0*0118 4.38 A 11 0 0 c A 11 0.11

A A i O A 4.75 0*11 3.96 A 11 A AIIO U • 0 1 1 £ 4. 44 A 11 0.11 7 O A 3. 86 A0.1111

r\ a 1 0 "7 U . U 1 £ ( A A 7 H • 4 ( A 11 0.11 y. 77 4. 2 f A0.101 A

A A 1 A "7 / A A 4.69 A 11 0*11 A 9 A 4.2 0A0.101 A

0 - 0 0 9h O 7 A 3.70 A 11 0.11 9 7 "7 3.77 A0.101 A

A A 1 A /. /4.09AO A 11 0.11 9 Q y. 3. 84 A0.101 A

A A O ^. C U . 0 2C 6 t> C CI 3 • 3 1 A0.101 A y 17 4.1/ A0.1217

0.0125 4 . 84 A 1 A 0. 10 9 AO 3.68 A 11 0.11

0.0117 / AC 4.95 A 1 A 0.10 9 "7 C 3.75 A 1 A 0.10

A A A O O U . 00 88 4.23 A 7 C 0.10 A 1 A 9 07 3. 9 f A0.09AO

A A A A A u • uuy6 3.68 1 A Q A 1 A 0. 10 A 7 9 4. 72 A AO 0 . 09

.~\ A 1 7 1 0.0121 A Q A 4 • V6 0.09 A A A y 9 A 4.20 A0.09AO

A A ^ Q A 4.16 A 1 A A AO 0 . OV 9 A 1 3.91 A0.09AO

'A A A O O 0 . 0088 4.31 A AO 0.09 4.05 y AC A AO 0 • 09

A A A Q 7 y. 11 4.11 A AO 0 . 07 9 7/. 3 • 1 4A0.09AO

A A A A 1 4.36 A AO 0. 09 9 00 3 . 99 A0.09AO

A A A Q A 0 . 00 80 y. 7 1 4. / 1 A AO 0 . oy A A A 4. 09 0.09 A AO

A0.00A A "7 A/ 0y 07 4. 8 f A A 7 0 • 0 ( y 19 4. 12 A A "7 0.0/

A0.0063 A A A C /.4.60A A A A "7 0.0 7

y. A a 4. 06 A A 7 0.07

A A A K O 0 . 0 0 3 7 4.32 A A 7 0 . 0 f 9 00 3.98 A A 1 0.0/

A A A A a U . U U 6 0

c. 1 c 3.13 A A 7 O • O ( a i a *r. 1 8 A0.0/A 7 A0.003/ A A C 7 A C 7 4. 3 ( A A A 0. 06 y AC 4. 05 A A 7 0 . 0 ( A0.0062 A A A 7 A OA 4. Vo A A A 0. 06 7 A Q 3. 98 A A "7 0.07

A A A C A 0 . OU 3U A6 • AOO 7 O.A ACU3 7 C 7 3.3/ A AO 0 • 09

A A A A O 0. 0047 A 7 A 6.34 A AC 0 • 05 A 1 *7 4. 1 / A AO 0 • 08

0.0043 7.17 0.04 4.22 0.07

115
VOTES WORD NOCC

AAAAA A 2649

THE 442506

FOR 45223

AND 128355

NOT 358 3 5 THAT 89026

TH1 n 1I J 29490

UU If U ? 5 5 7 ~> C J J C c. UAC n W w w w v ~ — FROM 19879

W I TH 21624

HAVE 13825

BEEN 12072

UPON 11816

THERE 12925

BUT 9174

ARE 13721

4 CONCUR 2290

ANY 1 3855

HAS 10 530

I ONLY 6218

3 CASE 1 526 1 OTHER 8966

1

COURT 330? 1 MADE 7999

1 ONE 9388

SUCH 18195

MAY 9510

ALSO 5230

WERE 129 1

4 AFF I RM 3897

HAD 1 545 1 UNDER 10893

WHEN 6875

I FOLLOW 6076

DOES 4264

BEFORE 5814

AFTER 6340

THEREF 3871

] ALL 902 1 WOULD 9678

5b REASON 6845

MUST 5208

should 5689

2 QUEST I 8776

T I ME 8254

Wn Ii THOU lit \j \j 4652

HOWFV3133

UHFTHF n i I L i ill* 5 17 3 his JL19529 7 ^ t 7 n i n 6224 U L £. ^ WHERE 5794

PRESEN 5653

BECAUS 355 3 8 CONS I

5288

CONTEN W W 11 1 La ( 3888

I TWO 5 130

ITS 1 106 1 COULD 5096

2 LAW 9658

Table IX.

E EL PZD

7.07 7.87 99.99

7.87 7.65 99.99

7.73 7.61 98.07

7.83 7.61 99.73

7.75 7.60 96.97

7.80 7.60 98.15

7.66 7.59 96.67

7.70 7.56 94.41

7.69 7.55 95.73

7.62 7.51 92.18

7.64 7.51 92.03

7.53 7.44 85.99

7.50 7.41 83.76

7.46 7.40 82.93

7.48 7.40 84.25

7.48 7.37 78.89

7.46 7.39 84.37

6.65 7.30 63.91

7.47 7.37 83.12

7.36 7.37 81.76

7.33 7.31 72.14

7.45 7.36 84.74

7.43 7.31 76.17

7.45 7.41 93.58

7.32 7.29 74.51

7.39 7.31 76. AO

7.50 7.35 85.80

7.37 7.30 76.70

7.29 7.23 67.15

7.43 7.31 79.91

6.89 7.23 63.53

7.43 7.30 82.44

7.40 7.31 80.44

7.28 7.24 69.87

7.28 7.24 69.38

7.09 7.20 63.30

7.19 7.23 68.55

7.24 7.21 68.47

7.01 7.18 62.21

7.36 7.26 74.78

7.34 7.23 73.12

7.17 7.25 72.48

7.18 7.22 66.70

7.20 7.20 66.59

7.25 7.28 77.08

7.17 7.20 70.40

7.10 7.17 63.57

7.09 7.11 55.90

7.22 7.19 66.13

7.32 7.22 78.63

7.24 7.17 66.70

7.19 7.16 65.26

7.18 7.20 68.25

7.00 7.11 57.19

7.15 7.14 £3.72

7.02 7.09 57.11

7.11 7.11 60". 51

7.31 7.20 75.34

7.16 7.11 61.79

7.23 7.20 74.29

Sorted by EKL

AVG G

0.0783 0.42

12. 1192 -0.19

1.2529 1.03

3.4562 0.53

0.9798 0.55

2.4343 0.70

0.8106 1.15

0.6984 0.64

1.5630 0.52

0.5456 1.25

0.5840 1.15

0.3761 1.17

0.3306 1.41

0.3232 1.37

0.3545 1.30

0.2485 0.84

0.3766 1.56

0.0643 2.45

0.3703 1.29

0.2838 1.34

0.1693 1.57

0.4182 1.64

0.2397 1.18

0.9C97 1.64

0.2213 1.60

0.2540 1.61

0.4817 1.49

0.2605 1.45

0.1410 1.08

0.3486 1.43

0.1109 2.26

0.4205 1.49

0.2937 1.82

0.1866 1.54

0. 1661 1.30

0. 1175 1.80

0.1612 2. 12

0. 1745 1.62

0. 1050 1.43

0.2361 1.45

0.2580 1.43

0. 1850 2.15

0.1412 1.83

0.1511 1.89

0.2395 2.17

0.2237 2.55

0.1274 2.02

0.0923 1.47

0. 1408 1.69

0.5396 1.55

0. 1665 1.55

0.1562 1.64

0.1558 2.26

0.0999 2.04

0.1379 2.06

0.1094 2. 14

0.1408 1.59

0.2888 1.71

0. 1383 1.59

0.2554 2.34

EK GL EKL

4.32 2.55 31.32

41.17 1.87 1.93

5.00 1.87 1.59

15.25 2.14 1.57

6.95 1.90 1.56

9.48 1.92 1.54

4.02 2.45 1.41

4.89 1.79 1.38

3.68 1.78 1.33

3.01 1.83 1.19

3.46 2.15 1.16

2.52 2.53 0.97

1.96 2.07 0.95

1.76 1.83 0.95

1.87 2.17 0.91

2.21 2.06 0.89

1.85 2.55 0.86

0.73 2.51 0.86

1.87 2.37 0.83

1.51 2.41 0.83

1.38 1.88 0.82

1.43 2.38 0.80

1.79 2.45 0.76

1.26 3.97 0.76

1.25 1.97 0.76

1.48 2.40 0.75

1.78 2.91 0.74

1.38 2.50 0.72

1.33 1.95 0.71

1.55 2.67 0.70

C.78 2.61 0.70

1.38 2.68 0.69

1.31 2.98 0.69

1.20 2.24 0.69

1.18 2.44 0.69

0.96 2.11 0.67

0.95 2.63 0.66

1.06 2.27 0.65

0.90 2.25 0.65

1.46 3.34 0.64

1.34 2.49 0.64

1.11 2.86 0.64

1.08 2.79 0.64

1.02 2.45 0.63

1.03 4.30 0.62

0.92 2.17 0.62

0.91 2.39 0.62

0.90 1.76 0.62

1.04 2.57 0.61

1.03 2.83 0.60

1.03 2.52 0.59

1.03 2.43 0.58

0.88 3.49 0.58

0.75 2.28 0.58

0.93 2.68 0.56

0.71 2.24 0.56

0.85 2.47 0.55

1.13 3.49 0.54

0.95 2.58 0.54

0.88 3.39 0.54

116
VOTES WORD

THAN

FACT

FURTHE

DEFEND

PART

BEING

THEN

CONCLU

JUDGME

THESE

SAME

HELD

REQUIR

FIRST

AGAINS

FACTS

THEY

MORE

ACCORO

CANNOT

WHO

CAN

EVIDEN

HERE

PLAINT

SINCE

FOUND

HAVING

REVERS

THEIR

STATED

CERT A

PROVID

WITHIN

TRIAL

SUSTAI

DETERM

INVOLV

NOR

SOME

BOTH

INTO

BETWEE

CASES

MATTER

OPINIO

OUT

MAKE

ALTHOU

THEM

WELL

SUFFIC

FILED

NECESS

GIVEN

EITHER

EVEN

SET

WHILE

RECORD

NOCC

4378

4658

4546

25773

4746

3858

4583

3665

10581

4753

4992

3978

6103

4165

5725

4095

7042

3050

2721

2467

5241

2822

12726

3448

20986

2756

3608

2006

2857

6514

3698

3069

5792

4561

9898

2600

5030

2933

2099

3394

2868

3583

3231

3896

4313

4764

4389

2535

1762

3505

2259

2484

5362

3477

2766

2033

1964

2964

2749

6093

Table IX.

E

7.1

7.06

7. 1 L 7. 20

7. 12

7.04

7. 12

6.95

7.06

7.11

7.05

7.04

7.06

7.01

7.04

7.00

7. 14

6.94

6.87

6.74

7.11

6.93

7. 10

6.93

7.02

6.89

6.91

6.67

6.66

7.08

6.99

6.87

7.03

6.85

6.97

6.65

7.02

6.56

6.70

6.97

6.85

6.93

6.84

6.86

6.91

7.02

7.00

6.76

6.67

6.92

6.77

6.72

6.67

6.93

6.80

6.71

6.64

6.71

6.82

6.91

Sorted

EL PZD

7.10 59.38

7.10 60.28

7.13 61.94

7.12 71.19

7.09 60.62

7.08 57.41

7.07 59.19

7.02 53.90

7.17 73.19

7.07 59.79

7.07 60.73

7.02 55.34

7.10 63.98

7.04 57.15

7.06 61.83

7.01 55.79

7.08 64.47

6.95 49.49

6.96 49.64

6.92 46.54

7.03 59.64

6.94 49.15

7.02 65.64

6.97 52.69

6.94 57.71

6.93 48.65

6.98 53.68

6.86 42.09

6.93 46.96

7.02 61.75

6.99 54.77

6.96 50.62

7.02 60.02

6.97 55.56

6.98 62.85

6.89 46.24

7.01 59.45

6.90 47.86

6.86 43.14

6.93 50.88

6.88 46.54

6.92 51.00

6.87 47.45

6.90 51.41

6.96 55.19

6.98 58.85

6.99 57.04

6.84 43.94

6.77 38.65

6.89 49.37

6.83 43.14

6.81 42.92

6.91 55.26

6.93 52.20

6.82 45.07

6.78 40.20

6.75 38.80

6.84 46.54

6.85 46.31

6.98 60.51

by EKL

AVG

0. 1198

0. 1249

1. 1230

0. 7468

0.1287

0. 1040

0. 1242

0.1010

0.3119

0.1275

0. 1299

0. 1058

0.1665

0.1116

0. 1605

0.1137

0.1897

0.0822

0.0745

0.0694

0. 1416

0.0739

0. 3461

0.0938

0.6097

0.0753

0. 1017

0.0548

0.0842

0. 1756

0.0975

0.0830

0. 1599

0.1294

0.2884

0.0753

0. 1314

0.0789

0.0581

0.0897

0.0771

0.0952

0.0879

0.1062

0.1166

0.1218

0.1164

0.0681

0.0487

0.0943

0.0592

0.0708

0.1589

0.0937

0.0744

0.0532

0.0509

0.0798

0.0751

0.1675

G

2.23

2.10

1.92

1.34

2.57

2.13

2.04

2.50

3.01

1.97

2.47

1.92

2.34

2.30

2.56

3.05

2.45

1.98

2.12

2.06

1.89

1.61

1.64

1.92

1.25

1.76

2.73

2.18

2.65

2.19

2.37

2.20

2.56

2.63

2.75

3.40

3.04

2.29

1.94

1.97

1.87

2.51

2.33

2.58

3.11

2.05

3.00

2.35

1.78

2.56

2.87

2.35

4.09

3.31

2.27

1.96

2.09

3.36

5.29

5.25

EK

0.81

0.80

0.91

0.79

0.78

0.75

0.82

0.64

0.54

0.83

0.76

0.75

0.74

0.71

0.63

0.60

0.77

0.66

0.62

0.57

0.79

0.67

0.71

0.66

0.64

0.62

0.53

0.51

0.48

0.70

0.68

0.65

0.64

0.50

0.45

0.40

0.64

0.56

0.53

0.67

0.59

0.57

0.55

0.54

0.53

0.71

0.65

0.54

0.50

0.56

0.51

0.45

0.33

0.52

0.50

0.50

0.49

0.45

0.43

0.41

GL

2.63

2.40

3.44

2.43

2.85

2.89

2.60

2.52

4.08

3.27

3.32

2.83

4.53

3.27

3.13

2.90

3.52

2.76

2.92

2.46

3.51

2.68

3.09

3.12

2.24

2.78

3.16

2.07

3.60

3.29

3.69

3.90

3.62

3.59

2.96

2.63

3.95

2.99

2.78

4.84

2.81

3. 14

2.83

3.22

4.12

4.63

6. 13

3.17

2.66

4.37

3.49

3.24

3.46

4.91

3. 10

3.10

3.06

3.72

4.31

4.95

EKL

0.54

0.54

0.53

0.53

0.52

0.52

0.51

0.49

0.49

0.48

0.48

0.47

0.47

0.46

0.46

0.46

0.45

0.45

0.45

0.45

0.44

0.44

0.43

0.43

0.43

0.43

0.43

0.43

0.43

0.42

0.42

0.42

0.42

0.41

0.41

0.41

0.40

0.40

0.40

0.39

0.39

0.39

0.38

0.38

0.38

0.37

0.37

0.37

0.37

0.36

0.36

0.36

0.36

0.35

0.35

0.35

0.35

0.35

0.35

0.35

117
VOTES WORD NOGC

DURING 2216

2 BASED 1605

7 EXPRES 2022

1 CONCER 1797

REGARD 1466

1 NEW 4744

1 APPARE 1334

2 PURPOS 4138

2 STATE 9231

HEREIN 2599

1 EAGH 3332

1 CONTAI 2096

7 SPECIF 2900

2 SITUAT 1358

DECIDE 1409

DIFFER 1714

NEITHE 930

4 ILL 8605

4 CLEAR 1537

LATER 1426

THROUG 1954

2 SIMILA 1243

CLEARL 1145

6 ERROR 3841

5 ISSUE 3113

SECOND 2415

2 YEARS 2601

2 DECISI 3988

FAILED 1442

2 GIVE 1490

MANY 1117

TAKE 1484

THERE I 1068

3 FINDIN 3437

OTHERW 1095

1 USED 2650

END 6422

5 PART I E 3496

4 AMOUNT 3110

APPLIE 1264

ITSELF 993

HOLD 1033

MERELY 936

2 ADD I T I 1708

3 DUE 1937

THERET 1022

SHOWN 1106

SHOWS 1078

4 CONSTR 3805

1 SEC 6808

RECEIV 2801

7 CONDIT 2779

5 BASIS 1500

SAY 1088

CONSIS 941

1 POINT 1487

2 TERMS 1583

i INTEND 1333

DISCUS 1034

3 REFERR 1309

Table IX.

EL PZD

6.58

6. 38

6.51

6. 57

6.39

6.68

6.43

6.76

6.85

6.23

6.68

6. 55

6.65

6.42

6.41

6.46

6. 16

6.49

6.52

6.43

6.52

6. 38

6.31

6. 56

6.61

6.53

6.53

6.52

6.29

6. 32

6.27

6. 38

6. 13

6.56

6.14

6. 45

6.81

55

49

25

25

15

21

6. 39

6.40

6.05

6. 15

6. 16

6.58

6.65

6.52

6.46

6.41

6.26

6. 19

6.35

6.33

6.29

6.22

6.24

Sorted

6.62 36.50

6.56 32.84

6.6 1 36.01

6.5 9 34.76

6.5 2 30.80

6.72 48.09

6.53 30.84

6.76 49.30

6.80 62*06

6.70 41.75

6.69 43.90

6.65 38.12

6.68 42.28

6.49 29.40

6.50 29.89

6.55 33.14

6.38 24.87

6.46 32.88

6.57 33.48

6.47 29.48

6.56 34.61

6.46 28.61

6.45 27.67

6.66 44.80

6.66 42.88

6.61 38.50

6.56 37.10

6.69 46

6.48 30

6.45 29.78

6.38 25.82

6.47 30.35

6.38 25

6.59 41

6.42 27

6.58 38.16

6.71 51.86

6.59 41.71

6.52 37.56

6.40 27.63

6.33 24.38

6.35 24.61

6.32 23.78

6.49 32.12

6.47 32.08

6.35 24.95

6.36 25.74

6.35 25.25

6.55 40.50

6.62 49.60

6.57 39.10

6.47 35.52

6.47 30.76

6.34 25.44

6.31 23.66

6.42 29.48

6.39 28.46

6.39 27.63

6.31 24.34

6.43 28.65

by EKL

58

31

,70

,56

18

AVG

0.0609

0.0431

0.0546

0.0468

0.0380

0. 1295

0.0364

0. 1096

0.2417

0.0670

0.0859

0.0578

0.0790

0.0368

0.0381

0.0466

0.0252

0.2551

0.0425

0.0387

C.0531

0.0339

0.0304

0.1051

0.0831

0.0656

0.0687

0. 1070

0.0414

0.0399

0.0286

0.0407

0.0279

0.0995

0.0307

0.0734

0. 1570

0.0960

0.0869

0.0351

0.0260

0.0270

0.0248

0.0453

0.0542

0.0278

0.0303

0.0297

0.1054

0. 1929

0.0764

0.0760

0.0412

0.0294

0.0260

0.0407

0.0424

0.0361

0.0267

0.0341

2.73

2.60

3.21

4.40

3.0 5 3.77

3.26

3.99

3.06

3.17

4.53

3.35

3.75

2.40

2.48

3.96

2.65

1.95

3.35

2.75

3.87

2.91

2.81

3.69

3.76

3.97

3.24

4.00

3.32

3.06

2.52

3.85

72

00

16

62

07

86

85

95

40

2.49

2.46

5.06

4.13

3.03

3.38

3.55

3.38

3.75

6.76

3. 52

5.82

2.94

2.47

4.43

3.43

3.14

2.85

8.37

EK

0. 36

0.35

0.34

0.34

0.32

0.31

0.30

0.41

0. 39

0.36

0.36

0.35

0.34

0.33

0.31

0.29

0.27

0.34

0.33

0.31

0.30

0.30

0.30

0.29

0. 32

0.31

0.31

0.30

0.29

0.29

0.29

0.27

0.27

0.26

0.25

0.24

0.44

0.29

0.27

0.27

0.27

0.26

0.26

0.25

0.25

0.25

0.24

0.23

0.30

0.27

0.27

0.26

0.26

0.26

0.26

0.25

0.25

0.25

0.25

0.24

GL

4.42

3.70

4.18

3.67

3.05

4.33

3.32

6.33

4. 64

5.86

5.12

5.43

5.03

3.07

3.99

3.56

2.44

3.00

5.39

3.52

4.00

3.18

3.28

4.33

4.98

5.63

4. 19

5. 57

3. 79

3.67

2.73

3. 52

3.38

3.90

3.79

4. 18

6. 84

4.47

3.75

3.46

3.32

3.24

2.82

4.68

3. 79

3.31

3.23

3.06

4.65

4.50

5.74

3.88

5.60

3.71

3.02

4.24

3.35

4.27

3.19

5. 55

EKL

0.26

0.26

0.26

0.26

0.26

0.26

0.26

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.25

0.24

0.24

24

24

24

0.24

0.24

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.23

0.22

0.22

0.22

0.22

0.22

0.22

0.22

0.22

0.22

0.22

0.22

0.22

0.21

0.21

0.21

0.21

0.21

0.21

0.21

0.21

0.21

0.21

0.21

0.21

118
VOTES

5

2

2

16

24

17

1

4

2

1

3

5

3

WORD

FAILUR

SUBSEQ

BEL I EV

FAVOR

MAKING

COURSE

ACT

RULE

RELATI

VIEW

TRUE

TESTIM

ENTIRE

FORTH

COMPLE

CONTRO

LONG

THINK

PREVIO

CORREC

SOUGHT

JUDGE

OVERRU

DISMIS

ITAL

FOL

ORDER

PAGE

MANNER

ATTEMP

TESTIF

TOOK

VERY

RENDER

DATE

BECOME

COMPLA

NATURE

PLACE

RAISED

ARGUME

EST ABL

MOST

OPERAT

SEVERA

SUPPOR

CHARGE

DISTIN

RATHER

ORDERE

RECOGN

POSSIB

PURSUA

EVERY

ORIGIN

DONE

FAR

PETITI

SHOWIN

OBVIOU

NOCC

1630

1263

1176

1249

1G60

1500

5147

4090

2530

1406

1140

3650

1350

1458

1709

2941

1047

1035

1040

1358

1132

4000

1644

2755

11360

5682

6773

3218

1259

1404

3484

1080

888

1657

1983

1153

3971

1185

1881

1050

1528

2947

1051

4207

1243

3151

4622

997

917

1180

1033

1018

1039

922

2053

1079

923

7623

829

645

E

6. 16

6.25

6.22

6.22

6.19

6.22

6.65

6.56

6. 54

6.35

6.23

6.42

6.30

6.25

6.30

6.48

6.23

6. 18

6. 16

6. 14

6.11

6. 52

6.23

5.96

6.67

6.67

6.78

6.47

6.30

6.05

6.35

6. 15

6.15

6.30

6.31

6.07

40

16

36

00

EL PZD

43 30. 16

37 26.99

34 25.67

37 26.87

33 25.14

6.45 30.53

6.26

6.74

6.25

6. 52

6.32

6.65

6.48

6. 14

6. 15

6. 14

6. 10

6.18

6.08

6.11

6.23

6.09

6.11

6.19

5.78

5.87

56

18

10

6.59 45

6.7C 47

6.53 37,

6.48 30.95

6.36 26.23

6.41 34.65

6.41 28.53

6.40 28.80

6.45 31.40

6.55 39.93

6.32 24.80

6.28 23.63

6.31 24.57

6.38 28.57

6.33 25.44

6.64 46.84

6.42 30.46

6.48 35.90

6.57 45.18

6.57 45.18

6.77 58.32

6.45 33.71

6.37 27.29

6.42 29.18

6.35 31.74

6.28 24.46

22 21.93

45 31.74

41 31.37

30 25.36

6.45 37.44

6.31 25.43

6.45 32.27

6.28 23.93

6.37 28.69

6.72 44.46

6.31 24.95

45 39.56

36 27.25

67 46.35

47 40.69

22 22.68

6.21 22.00

6.33 26.23

6.25 23.51

6.23 22.98

6.24 23.

6.22 22,

6.39 32.

6.28 24.57

6.24 22.61

6.44 40.39

6.16 20.53

6.09 18.23

.17

,31

01

AVG

0.0459

0.0363

0.0322

0.0364

0.0282

0.0421

0. 1370

0.1055

0.0662

0.0375

0.0309

0. 1010

0.0369

0.0 391

0.0455

0.0849

0.0280

0.0298

0.0277

0.0370

0.0316

0.1181

0.0456

0.0790

0.2755

0. 1378

0.1918

0.0315

0.0329

0.0376

0.0969

0.0302

0.0230

0.0464

0.0555

0.0320

0. 1136

0.0313

0.0528

0.0290

0.0429

0.0788

0.0273

0.1145

0.0331

0.0855

0. 1234

0.0265

0.0246

0.0324

0.0261

0.0272

0.0271

0.0244

0.0558

0.0282

0.0247

0.2198

0.0227

0.0187

G

3.81

3.67

3.33

3.45

4.11

6.86

3.30

4.23

3.61

4.33

3.33

3. 30

5.20

3.68

4.76

5.05

3.39

3.00

3.93

4.35

3.80

10.30

4. 78

5.16

3.12

3.12

3.68

2.83

3.46

4.42

3.53

3.21

2.80

3.94

3.97

3.89

4.27

3.80

6.46

3.56

5.01

3.00

2.65

3.54

3.47

7.06

3.96

2.77

3.00

3.50

3.33

3.04

2.92

3.05

4.38

3.94

4.89

3.73

3.37

3.36

EK

0.24

0.24

0.24

0.23

0.22

0.21

0.32

0.31

0. 30

0.29

0.26

0.25

0.25

0.25

0.24

0.23

0.23

G.23

0.22

0.21

0.21

0. 19

0.19

0.16

0.37

0.37

0.31

0.31

0.27

0.25

0.24

0.24

0.24

0.23

0.23

0.23

0.22

0.22

0.21

0.21

0.20

0.45

0.28

0.27

0.26

0.24

0.24

0.24

0.24

0.23

0.23

0.23

0.22

0.22

0.21

0.21

0.20

0.19

0.19

0.18

GL

4.43

3.97

3.34

4.09

3.75

4. 36

6.21

12.48

5. 77

7.01

4.42

3. 88

6.76

4.54

5.48

5.00

3. 84

3.20

3.68

4. 34

4.23

6.80

4.35

5.01

7.32

7.39

11.48

5. 57

6.32

7.93

3. 72

4. 38

3.45

6.39

4. 85

3.96

4.90

4. 10

5.21

3.95

4.22

17.95

6. 00

4. 52

7. 53

9.79

4. 95

4. 15

3.67

6. 13

3.94

3. 70

3. 93

3.79

5.63

4.53

4. 79

5.82

3. 12

2.92

EKL

0.21

0.21

0.21

0.21

0.21

0.21

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.20

0.20

19

19

19

19

0.19

0.19

0.19

0.19

0.19

0.19

0.19

0.19

0.19

0.19

0.19

0.19

0.19

0.18

0.18

0.18

0.18

0.18

0.18

0.18

0.18

18

18

0.18

0.18

0.18

0.18

0.18

0.18

0.18

0.18

0.18

Table IX. Sorted by EKL

119
VOTES

2

5 10

5

6

1

15

6

13 11

36

1

2

1I9

3

3

2

WORD

OHIO

DAY

JURY

ADKITT

OBTAIN

DUTY

HOLD I

LESS

TQGETH

GRANT E RIGHTS

LEAST

REFUSE

PREVEN

LATTER

WHOM

THEREB

MUCH

AGREE

AGAIN

BECAME

STILL

PERMIT

CLAIME

COURTS

DAYS

APPLIC

PRINCI

APPELL

REMAIN

PAID

WAY

LANGUA

KNOWN

STATEM

PARTY

VARIOU

RELATE

COMMON

EXISTE

NEVER

TIMES

SUGGES

WHOSE

HI MS EL

LIKE

CONSTI

WILL

CONTRA

PROPER

SUPRA

HEARIN

INTERE

PUBLIC

APPLY

HOW

OCCURR

JUSTI

COME

MAKES

Table

NOOC

8519

2189

5530

1667

1498

1873

1008

923

861

1574

2108

766

1286

956

833

832

712

693

707

766

734

660

2869

921

2033

1500

4168

2158

14543

1592

2316

1771

1492

1083

2732

2643

815

839

4042

1029

976

751

782

655

864

738

4132

7140

8033

5913

2573

2525

3637

4658

806

739

1248

885

663

565

IX.

6.49

6.41

6.41

6. 32

6. 18

6.25

6.05

6.08

6.04

6.25

6.30

6.00

6. 14

6.00

6.04

6.00

5.99

5.99

5.91

6.00

5.81

5.86

6.35

5.97

6.28

6.05

6. 58

6.46

6.53

6.35

6.25

6.21

6.22

6.12

6.32

6.26

5.99

5.92

6.46

6.06

6.01

5.95

5.94

5.89

5.95

5.93

6.41

6.84

6.56

6.40

6.29

6.28

6.36

6.33

6.00

5.93

6.05

5.90

5.90

5.73

Sorted

EL PZD

6.3b 34.3 9 6.46 34.16

6.31 34.27

6.32 28.87

6. 30 27.40

6.30 28.35

6.20 22.76

6.17 21.63

6.16 20.91

6.34 28.35

6.33 30.38

11 19.40

22 24.49

16 21.44

14 20.23

13 20., 0 8 11 19.02

11 19.13

10 18.98

11 19.32

6.08 18.61

6.07 18.08

6.49 39.63

6.17 21.44

6.36 31.2 1 6.22 24.99

6.60 47.37

6.

6.6.

6,

6,

6,

6.

43 34.61

44 50.16

38 30.46

25 28.16

45 32.91

23 25.78

17 22.19

6.36 34.16

6.33 31.93

6.12 19.96

6.12 20.04

6.48 42.58

6.17 22.08

6.15 21.32

6.09 19.21

6.06 18.68

6.04 17.70

6.10 19.85

6.08 18.87

6.49 42.99

6.74 62.55

6.49 52.96

6.34 36.91

6.25 29.21

6.31 31.59

6.32 35.33

6.30 35.78

6.08 19.63

6.01 17.89

6.11 21.78

6.07 19.85

6.00 17.40

5.98 16.27

by EKL

AVG

0.2212

C.0607

C . 14 70

0.0436

0.0397

C.0506

0.0265

0.0250

0.0222

0.0425

0.0581

0.0206

0.0351

0.0265

0.0235

0.0228

0.0192

0.0187

0.0187

0.0209

0.0196

0.0176

0.0820

0.0261

0.0553

0.0447

0. 1134

0.0564

0. 3877

0.0428

0.0616

0.0472

0.0411

0.0285

0.0720

0.0726

0.0214

0.0233

0. 1 171

0.0286

0.0254

0.0201

0.0208

0.0179

0.0241

0.0198

0.1058

0. 1944

0.2158

0. 1591

0.0636

0.0716

0.0944

0.1226

0.0212

0.0191

0.0347

0.0235

0.0173

0.0151

2.35

3.92

3.35

3.82

3.2 8 3.82

3.62

3.44

3.31

4.97

5. 59

2.98

4.26

3.86

3.47

3.43

3.22

3.85

3. 50

4.64

3.61

3.47

6.17

4. 84

9.19

6.03

4.97

6.01

3.05

4.99

3.21

6.65

3.66

3.59

4.77

4.28

4.01

3.10

5.85

5.05

4.03

3.18

3.46

3.34

5.07

4.09

3.48

5.49

3.98

3.62

3.34

4.03

5.26

4.86

3.14

3.23

3.73

3.52

3.24

3.28

EK

0.28

0.26

0.24

0.23

0.23

0.21

0.21

0.21

0.21

0.20

0.20

0.20

0.19

0.19

0.19

0. 19

0. 19

0.19

0.19

0.18

0.18

0. 18

0.17

0.17

0.16

0. 14

0.25

0.24

0.23

0.23

0.23

0.22

0.21

0.21

0.20

0.20

0.20

0.20

0. 19

0.19

0. 19

0. 19

0.18

0.18

0.17

0.17

0.28

0.26

0.23

0.23

0.23

0.21

0.20

0.20

0.19

0.19

0. 18

0.18

0.18

0.17

GL

5.51

9.83

4.31

5.59

5.62

5.09

4.43

3.99

3. 86

5.70

4. 76

3.43

4.13

3.57

3.63

3.68

3.25

3.99

3.35

3.29

3.09

2.94

6.36

3.94

5.77

3.91

8. 13

7.85

5.26

7.12

4.69

10.08

5. 17

4.34

5.32

5.91

3.74

4.01

7.01

4. 18

4. 18

3.80

3.55

3.38

3.60

3.62

7.53

12.86

7.29

5.71

4.77

6.14

5.71

5.07

4.78

3. 80

4.81

4.41

3. 88

3.07

•JtSKL 0.17

0.17

0.17

0.17

0.17

0.17

0.17

0.17

0.17

0.17

0.17

0.17

0.17

0.17

0. 17

0.17

0.17

0.17

0.17

0.17

0.17

0.17

0. 17

0.17

0.17

0.17

0. 16

0. 16

0. 16

0.16

0.16

0.16

0. 16

0.16

0.16

0.16

0.16

0.16

0.16

0.16

0.16

0.16

0.16

0.16

0. 16

0.16

0. 15

0.15

0.15

0.15

0.15

0.15

0.15

0.15

0.15

0.15

0. 15

0.15

0.15

0.15

120
16

32

VOTES WORD NOCC E EL J PLACED 781 5.88 6.0b 18

READS 769 5.89 6.03 18

MENTIO 694 5.91 6.02 17

SEEKS 64 7 5.88 5.98 16

2 ESS E.NIT 651 5.83 5.98

5 OBJECT 2703 6.27 6.31

5 RE QUE 5 1941 6.11 6.29 29

10 COUN"n 6245 6.62 6.52 52

4 CQNTIN 2382 6.37 6.4Q 34

1 SUPREM 1904 6.16 6.24 27

HER 7548 6.30 6.20 31

1 LEGAL 1650 6.25 6.30 28

HEARD 903 5.97 6.07 19

OCCASI 742 5.95 6.03 18

NOTED 710 5.88 6.02 18

BEYOND 754 5.87 5.99 17

MERE 654 5.82 5.99 17

AMONG 579 5.83 5.93 15

FOREGO 626 5.73 5.96 16.

2 RETURN 2074 6.24 6.32 31.

9 COUNSE 3030 6.22 6.27 32.

FULLY 591 5.74 5.93 16.

WHERE I 560 5.60 5.92 15.

5 ANSWER 3398 6.42 6.41 39.

STATES 2343 6.38 6.33 33.

4 EMPHAS 1012 5.96 6.00 19.

5 CITY 5969 6.24 6.23 38.

4 CODE 4152 6.21 6.18 29.

PUT 719 5.88 5.96 17.

7 REVIEW 2347 6.02 6.30 32.

ALONE 536 5.73 5.87 14.

DIFFIC 578 5.72 5.87 15.

REACHE 539 5.63 5.86 14.

1 CONCED 485 5.58 5.83 14.

3 USE 3852 6.29 6.27 36.

1 REV 1484 6.07 6.08 22.

7 VALID 768 5.83 5.92 17.

9 CLAIM 2565 6.24 6.24 32.

DOING 625 5.71 5.89 16.

1 APPROX 704 5.79 5.87 15.

13 NOTICE 2855 6.04 6.18 30.

2 QUOTED 591 5.60 5 . 8 'j 15.

NONE 506 5.58 5.82 14.

ALREAD 542 5.68 5.80 14.

3 CAREFU 453 5.42 5.79 13.

ADDED 587 5.62 5.77 13.

NEVERT 370 5.50 5.71 11.

RELIED 487 5.62 5.80 13.

1 DESIRE 507 5.38 5.78 13.

SOLELY 441 5.50 5.74 12.

ARGUED 396 5.47 5.71 12.

2 FILE 943 5.49 5.87 17.

5 EXAMIN 3117 6.19 6.23 35.

1 STAT 1245 5.90 5.93 19.

13 JURISD 3056 6.00 6.10 29.

MOVED 492 5.61 5.75 13.

8 ASSIGN 2654 6.00 6.12 29.

4 DISSEN 751 5.48 5.73 13.

1 OPPORT 545 5.53 5.75 13.

HENCE 447 5.43 5.68 12.

Table IX. Sorted by EKL

'Z.u .91

. 30

.89

.87

. 76

.50

.44

.43

.35

.44

.89

.57

.93

.38

.04

.74

.02

.81

.64

.48

.54

.00

.66

.33

.37

.59

.05

.55

.40

.72

. 79

.06

.91

.00

. 12

.72

.06

.27

.04

.77

.76

.13

.23

.08

.51

.96

.92

.89

.74

.87

. 1 5 .06

.56

10

67

40

82

43

70

26

A v 0

0. 0208

0. 3220

0.0191

0.0179

0. 0173

0.0742

0.0545

0. 1787

0. 0 6 34

0.0474

0.2095

0.0423

0.0241

0.0206

0.0182

0.0209

0.0170

0.0152

0.0163

0.0589

0.0868

0.0159

0.0155

0.0913

0.0582

0.0246

0. 1706

0. 1146

0.0197

0.0676

0.0152

0.0155

0.0139

0.0140

0.1059

0.0446

0.0207

0.0735

0.0167

0.0179

0.0853

0.0149

0.0136

0.0141

0.0118

0.0144

0.0096

0.0134

0.0143

0.0118

0.0117

0.0265

0.0831

0.0383

0.0812

0.0149

0.0715

0.0191

0.0146

0.0118

G

4.15

3.56

4.96

4.19

3.67

8.66

7.47

5.00

5. B5

3.73

4.05

7.41

3.35

3. 38

3.47

3.35

3.36

3.05

3.55

8.81

6.05

4.28

4.62

64

26

16

90

17

3.40

5.34

4.20

3.98

4.07

3.43

4.86

3.55

3.58

5.91

3.56

3.77

5.70

3.88

3.70

3.49

3.79

4.33

3.19

4.43

4.09

4.03

3. 88

5.51

7.01

3.51

4.48

3.94

6.48

3.84

5.13

4.38

EK

0. 16

0.16

0. 16

0.16

0.16

0.15

0. 15

0.23

0.21

0.21

0.20

0. 19

0.18

0.18

0. 17

0.17

0.17

0.17

0. 16

0.15

0. 15

0.14

0.13

0.22

0.22

0.19

0.18

0.17

0. 17

0.15

0. 14

0. 14

0. 14

0.14

0.18

0.18

0. 16

0.15

0.15

0. 15

0. 14

0. 14

0.14

0. 14

0.13

0.13

0. 13

0.12

0. 12

0.12

0. 12

0. 10

0. 15

0. 15

0. 14

0.13

0. 12

0.12

0.11

0. 11

GL

4.20

3.85

4.13

3.41

3.52

5.60

5.99

8. 51

10.10

6.65

4. 75

9. 77

5.06

5.02

4.48

3.90

3.95

3.70

3. 70

9.23

EKL

0.15

0.15

0.15

0.15

0.15

28

71

89

44

8.54

5.19

5. 82

5.98

5.70

7.80

3. 50

3.51

4.15

3.42

7.72

9.27

4.77

7. 77

5. 74

4.01

6. 77

4.09

4. 14

4.07

3. 84

3.95

3.20

4.02

3.97

4.06

3. 34

4. 17

8.63

6.23

6. 50

4.21

7. 19

3.90

4. 15

3. 85

15

15

0.14

0.14

14

14

0.14

0.14

0.14

0.14

0.14

0. 14

0.14

0.14

0.14

0.14

0.14

0.14

0.13

0.13

0.13

0.13

0.13

0.13

0.13

0.13

0.13

0.13

0.13

0.12

0.12

0.12

0.12

0.12

0.12

12

12

12

0. 12

0.12

0.12

0.12

0.12

0.12

0.12

0.12

0.12

0.1

0.1

0.1

o.i:

0.1

0.1

0.1

o. i:

121

772-957 0-66—
VOTES WORD NOGC E EL PZD

ARGUES 44 3 5.52 5.67 12.23

STATIN 385 5.43 5.67 11.77

FA I LS 426 5.21 5.68 12.15

2 WHOLE 651 5. 74 5.78 14.87

8 S ERV I C 385 5 6.04 6.05 29.63

2 MASS 4687 5.77 5.73 16.98

L I KEW I 404 5.52 5.64 11.70

EVER 48 1 5 .47 5.65 12.23

1 ABLE 416 5.37 5.64 11.77

ONCE 375 5.32 5.60 11.02

EX ISTS 376 5.38 5.59 10.94

SEEKS 374 5.15 5.62 11.32

5

f\ II i\ A 11 COMPAN 4677 6.19 6.05 32.65

HERETO 498 5.41 5.64 1 2 . 60

I NSTE A 328 5.29 5.52 10.07

INSIST 368 5.36 5.51 10.41

2 COMPAR 418 5.42 5.5 7 11.09

i RELIES 30 1 5.28 5.48 9.62

1 ALLEG320 5.18 5.47 9.66

QU I TE O A "J 30 7 5.32 5.46 9 . 39

2 VIRTUE 322 5.21 5.46 9.55

NAMELY 316 5.27 5 . 44 9.36

1

111- V/ /"* AVI WEYG AN 251 4.57 5.40 8.79

8 R E S P 0 N 2872 5.94 6.00 29.21

5 EMPLOY 6062 5.98 5.89 32 . 50

2 MATTH

249 4.57 5.37 8 . 64

7 OFF ICE 4060 6.26 6.12 33.93

S OME WH 236 5.13 5.27 7.73

DESMON 230 4.86 5.24 7.47

I VOORH209 4, 80 5.23 7.32

SOMET

237 5.05 5.22 7.39

FULD 208 4.73 5.20 7.09

FROESS 209 4.78 5.18 6.98

I PECK 216 4.34 5.22 7.43

X. die TY-LA • by EKL

AVG G EK GL EKL

0

0136 4.75 0.11 3. 96 0.11

0 . 0112 4. 44 0.11 3. 86 0. 1

f,ua i •> c 4. 84 0.10 3. 68 0.11

a A 1 A 0 169 3.54 0.14 5.73 0.10

A I i 1

5.82 0.13 7.29 0.10

A 14 8 3 3.41 0.12 4. 36 0.10

A0 . 0 106 3.26 0.12 4. 45 0.10

0 . 0127 4.47 0.11 4.27 0.10

0

0107 4.69 0. 11 4*20 0.10

A U U Vh 3. 70 0.11 3. 77 0.10

A0 . A 1 A U L U4 4.09 0.11 3. 84 0.10

A A 1 1 "7 011/ 4.95 0. 10 3.75 0.10

0 .

1 1 o A

1 180 4.27 0.17 1 0. 01 0.09

A 0 121 3.70 0. 12 6. 07 0. 09

A(J o A A O O 0 0 o o 4.25 0.10 3. 97 0.09

A 0096 3.68 0.10 4. 72 0.09

A 0 1 £ 1 4. 96 0.09 4.20 0.09

A A A A A 0090 4.16 0.09 3. 91 0.09

A

Q o A A a o 0088 4.31 0. 09 4. 05 0.09

Q o A A Q 1 008 3 4.11 0. 09 3. 74 0.09

0 m A P. A 1 0 091 4.56 0. 09 3. 99 0.09

U o A A O A 0 U oO 4.71 0. 09 4. 09 0. 09

u © A "> C A 00 DO 6.09 0.05 3. 57 0.09

0 • A ~7 "7 O 6.24 0.12 11.25 0.08

A iOD j 5.38 0.11 7.48 0.08

O • An/, o 6. 34 0.05 4.17 0.08

Q • 10 32 4.82 0.17 18. 75 0.07

0 o A A "7 A 00 f 0 4.87 0. 07 4. 12 0. 07

a A A £ a 4.60 0.07 4. 06 0.07

Ao • A A A 00 1> 9 4. 32 0. 07 3. 98 0.07

A A A L O 006 o 5.15 0. 07 4. 18 0 . 07

Ao • A A C "7 00 3/ 4.57 0. 06 4.05 0. 07

0 * A A / 1 0062 4. 96 0. 06 3. 98 0.07

0. 0043 7.17 0. 04 4. 22 0 . 07

122
1

1

10

3

9

WORD NOCC E EL PZD AVG

THE 442506 7. 87 7.6b 99 .99 12. 1192

AAAAAA 2649 7. 07 7.87 99 .99 0.0783

WAS 56044 7. 69 7.55 95 . 73 1. 5630

AND 128355 7. 83 7.6 1 99 . 73 3.4562

NOT 35835 7. 75 7.60 96 .97 0.9798

WHICH 25522 7. 70 7.56 94 .41 0.6984

THAT 89026 7. 80 7.60 98 .15 2.4343

BUT 9174 7. 48 7.3 7 78 .89 0.2485

FOR 45223 7. 73 7.6 1 98 *07 U2529

ALSO 5230 7. 29 7.23 67 .15 0.1410

THIS 29490 7. 66 7.59 96 .67 0. 8106

WITH 21624 7. 64 7.51 92 .03 0.5840

HAVE 13825 7. 53 7.44 85 .99 0.3761

OTHER 8966 7. 43 7.31 76 .17 0.2397

FROM 19879 7. 62 7.51 92 . 18 0.5456

PLAINT 20986 7. 02 6.94 57 .71 0.6097

ANY 13855 7. 47 7.37 83 . 12 0.3703

THERE 12925 7. 48 7.40 84 .25 0.3545

FOLLOW 6076 7. 28 7.24 69 .38 0. 1661

HAS 10530 7. 36 7.37 81 . 76 0.2838

DEFEND 25773 7. 20 7.12 71 .19 0.7468

UPON 11816 7. 46 7.40 82 .93 0. 32 32

BEEN 12072 7. 50 7.41 83 .76 0. 3306

WERE 12911 7. 43 7.31 79 .91 0. 3486

WOULD 9678 7. 34 7.23 73 .12 0.2580

THEREF 3871 7. 01 7.18 62 .21 0. 1050

MAY 9510 7. 37 7.30 76 .70 0.2605

ALL 9021 7. 36 7.26 74 . 78 0.2361

HOWEVE 3333 7. 09 7.1 1 55 .90 0.0923

SUCH 18195 7. 50 7.35 85 .80 0.4817

HAD 15451 7. 43 7.30 82 .44 0.4205

WHEN 6875 7. 28 7.24 69 .87 0. 1866

HIS 19529 7. 32 7.22 78 .63 0. 5396

DID 6224 7. 24 7.17 66 .70 0. 1665

ARE 13721 7. 46 7.39 84 .37 0. 3766

ONLY 6218 7. 33 7.31 72 . 14 0. 1693

COULD 5096 7. 16 7.11 61 . 79 0.1383

TWO 5130 7. 11 7.11 60 .51 0. 1408

MADE 7999 7. 32 7.29 74 .51 0.2213

ONE 9388 7. 39 7.31 76 .40 0.2540

CAN 2822 6. 93 6.94 49 .15 0.0739

AFTER 6340 7. 24 7.2 1 68 .47 0. 174-5

COURT 33021 7. 45 7.41 93 .58 0.9097

CASE 15261 7. 45 7.36 84 .74 0.4182

EVIDEN 12726 7. 10 7.02 65 .64 0.3461

WHERE 5794 7. 19 7.16 65 .26 0.1562

WHETHE 5173 7. 22 7.19 66 .13 0. 1408

ITS 11061 7. 31 7.20 75 .34 0.2888

SINCE 2756 6. 89 6.93 48 .65 0.0753

ALTHOU 1762 6. 67 6.77 38 .65 0.0487

DOES 4264 7. 09 7.20 63 .30 0. 1175

UNDER 10893 7. 40 7.31 8 0 .44 0.2937

MUST 5208 7. 18 7.22 66 .70 0.1412

BOTH 2868 6. 85 6.88 46 .54 0.0771

SHOULD 5689 7. 20 7.20 66 .59 0.1511

WHO 5241 7. 11 7.0 3 59 .64 0. 1416

FURTHE 4546 7. 11 7.13 61 .94 0. 12 30

HELD 3978 7. 04 7.02 55 .34 0. 1058

HERE 3448 6. 93 6.97 52 .69 0.0938

NOR 2099 6. 70 6.86 43 . 14 0.0581

G

•0.19

0.42

0.52

0.53

0.55

0.64

0.70

0.84

1.03

1.08

1.15

1.15

1.17

18

25

25

29

1.30

1.

1,

1.1.

1.

1.

1,

1,

,30

,34

,34

.37

,41

,43

,43

,43

1.45

1.45

1.47

1.49

1.49

1.54

1.55

1.55

56

57

1.59

1.59

1.60

1.61

1.61

1.62

1.64

1.64

64

64

69

71

76

78

80

82

83

87

89

89

92

92

92

1.94

EK GL EKL

41.17 1.87 1.93

4.32 2.55 31.32

3.68 1.78 1.33

15.25 2.14 1 .57

6.95 1.90 1.56

4.89 1. 79 1.38

9.48 1.92 1.54

2.21 2. 06 0.89

5.00 I . 87 1 .59

1.33 1.95 0.71

4.02 2.45 1.41

3.46 2.15 1.16

2. 52 2.53 0.97

1.79 2.45 0.76

3.01 1.83 1.19

0.64 2.24 0.43

1.87 2.37 0.83

1.87 2.17 0.91

1.18 2.44 0.69

1.51 2.41 .0.83

0.79 2.43 0.53

1.76 1.83 0.95

1.96 2.07 0.95

1.55 2.67 0.70

1.34 2.49 0.64

0.90 2.25 0.65

1.38 2.50 0.72

1.46 3.34 0.64

0.90 1. 76 0.62

1.78 2.91 0.74

1.38 2.68 0.69

1.20 2.24 0.69

1.03 2.83 0.60

1.03 2.52 0.59

1.85 2.55 0.86

1.38 1. 88 0.82

0.95 2.58 0.54

0.85 2.47 0.55

1.25 1.97 0.76

1.48 2.40 0.75

0.67 2.68 0.44

1.06 2.27 0.65

1.26 3.97 0. 76

1.43 2.38 0.80

0.71 3.09 0.43

1.03 2.43 0.58

1.04 2.57 0.61

1.13 3.49 0.54

0.62 2.78 0.43

0.50 2.66 0.37

0.96 2.11 0.67

1.31 2. 98 0.69

1.08 2.79 0.64

0. 59 2.81 0.39

1.02 2.45 0.63

0.79 3.51 0.44

0.91 3.44 0.53

0.75 2.83 0.47

0.66 3. 12 0.43

0.53 2. 78 0.40

Table X. Sorted by G

123
VOTES WORD

4 ILL

EITHER

THESE

SOME

MORE

RESPEC

W I THOU

THEN

BECAUS

OPINIO

CONSID

CANNOT

CIRCUM

THUS

EVEN

FACT

BEFORE

ACCORD

BEING

CONTEN

REASON

OUR

QUESTI

HAVING

THEIR

CERTAI

THAN

PRESEN

AFFIRM

STATUT

GIVEN

I NVOLV

FIRST

UNTIL

UNLESS

BET WEE

LAW

REQUIR

MAKE

SUFFIC

OHIO

STATED

OVER

MIGHT

SITUAT

ITSELF

THEY

CONCUR

INDICA

MERELY

SAME

CONSIS

DECIDE

HIM

HOLD

CONCLU

INTO

APP

WHAT

CITED

Table

0605

2033

4753

3394

3050

2579

4652

4583

3553

4764

5288

2467

2543

1622

1964

4658

5814

2721

3858

3888

6845

3179

8776

2006

6514

3069

4378

5653

3897

7283

2766

2933

4165

2347

1520

3231

9658

6103

2535

2484

8519

3698

2622

1734

1358

993

7042

2290

1901

936

4992

941

1409

5613

1033

3665

3583

4769

2883

1401

49

71

1 1 97

94

80

10

7.12

7.00

7.02

7.15

6.74

6.75

6. 58

6.64

7.06

7. 19

6.87

7.04

7.02

7. 17

6.80

7.25

6.67

7.08

6.87

7. 11

7. 18

6.89

6.89

6.80

6. 56

7.01

6.65

6. 54

6.84

7.23

7.06

6.76

6.72

6.49

6.99

6.72

6. 57

6.42

6.25

7. 14

6.65

6.64

6.21

7.05

6. 19

6.41

6.91

6.15

6.95

6.93

6.74

6.76

6.41

X. Sorted

SL

6.46

6.78

7.0 7 6.9 3 6.95

6.82

7.17

7.07

7.1

6.98

7il4

6.92

6.75

6.65

6.75

7.10

7.23

6.96

7.08

7.09

7.25

6.83

7.28

6.86

7.02

6.96

7.10

7.20

7.23

6.80

6.82

6.90

7. OA

6.70

6.63

6.87

7.20

7. 10

6.84

6.81

6.35

6.99

6.71

6.63

6.49

6.33

7.08

7.30

6.70

6.32

7.07

6.31

6.50

6.85

6.35

7.02

6.92

6.72

6.79

6.54

by G

PZD

32.88

40.20

59. 79

50.8 8 49 .49

44.43

63.57

59.19

57.19

5R.85

63.72

46 . 54

41 .94

34.80

38.80

60.28

68.55

49.64

57.41

57.11

72 .48

47.98

77.08

42.09

61.75

50.62

59.38

68.25

63.53

53.15

45.07

47.86

57.15

39.22

33.82

47.45

74.29

63.98

4 3.94

42.92

34.39

54.77

40.99

34.27

29.40

24.38

64.47

63.91

37.67

23.78

60.73

23.66

29.89

54.24

24.61

53.90

51 .00

44.92

44. 80

30. 7 5

AVG

C.2551

0.0532

0. 1275

0.0897

0.0822

0.0678

0. 1274

0. 1242

0.0999

0.1218

0. 1379

0.0694

0.0679

0.0427

0.0509

0. 1249

0.1612

0.0745

0. 1040

0. 1094

0. 1850

0.0833

0.2395

0.0548

0. 1756

0.0830

0. 1198

0. 1558

0.1109

0. 1985

0.0744

0.0789

0. 1116

0.0628

0.0418

0.0879

0.2554

0. 1665

0.0681

0.0708

0.2212

0.0975

0.0701

0.0465

0.0368

0.0260

0. 1897

0.0643

0.0499

0.0248

0. 1299

0.0260

0.0381

0. 1531

0.0270

0.1010

0.0952

0. 1292

0.0725

0.0390

G

.95

.96

1.97

1.97

1.9 8 1.99

2.02

2.04

2.04

2.05

2.06

2.06

2.08

2.08

2.09

2.10

2.12

2.12

2.13

2.14

2.15

2.15

2.17

2. 18

2.19

2.20

2.23

2.26

2.26

2.26

2.27

2.29

2.30

2.31

2.32

2.33

2.34

2.34

2.35

2.35

2.35

2.37

2.40

2.40

2.40

2.40

2.45

2.45

2.45

2.46

2.47

2.47

2.48

2.49

2.49

2.50

2.51

2.51

2.52

2.52

EK

0.34

0. 50

0.83

0.67

0.66

0.54

0.91

0.82

0.75

0.71

0.93

0.57

0.49

0.42

0.49

0.80

0.95

0.62

0.75

0.71

1.11

0.55

1.03

0.51

0.70

0.65

0.81

0.88

0.78

0.48

0. 50

0.56

0.71

0.42

0.39

0.55

0.88

0.74

0.54

0.45

0.28

0.68

0.43

0.39

0.33

0.27

0.77

0.73

0.42

0.26

0.76

0.26

0.31

0.52

0.26

0.64

0.57

0.41

0.51

0.33

GL

3.00

3.10

3.27

4. 84

2.76

3.71

2. 39

2.60

2.28

4.63

2.68

2.46

2.94

2.88

3.06

2.40

2.63

2.92

2.89

2.24

2. 86

4.84

4. 30

2.07

3.29

3.90

2.63

3.49

2.61

4.39

3.10

2.99

3.27

3.46

2.95

2.83

3.39

4.53

3.17

3.24

5.51

3.69

3.50

2. 78

3.07

3.32

3. 52

2.51

3.59

2.82

3.32

3.02

3.99

6.64

3.24

2. 52

3. 14

3.31

3.76

3.08

EKL

0.24

35

48

39

45

0.34

0.62

0.51

0.58

0.37

0.56

0.45

0.33

0.31

0.35

0.54

0.66

0.45

0.52

0.56

0.64

0.31

0.62

0.43

0.42

0.42

0.54

0.58

0.70

0.29

0.35

0.40

0.46

0.30

0.30

0.38

0.54

0.47

0.37

0.36

0.17

0.42

0.29

0.30

0.25

0.22

0.45

0.86

0.31

0.22

0.48

0.21

0.25

0.29

0.22

0.49

0.39

0.29

0.32

0.27

124
VOTES WORD

MANY

TIME

PRQVIS

AGAINS

PROVID

THEM

PART

THOUGH

CASES

INSTAN

ENTITL

BASED

PERSON

THEREO

WITHIN

REVERS

MOST

NE I THE

ABOUT

SUBJEC

THERE I FOUND

DURING

TRIAL

LATER

NOTHIN

SHALL

DISTIN

THEREA

NOW

VERY

CLEARL

PAGE

DISCUS

EFFECT

WELL

PRIOR

SECTIO

R I GHT

DENIED

OWN

SIMILA

PURSUA

ABOVE

SAY

SEE

APPLIE

ANOTHE

CAUSE

LEAST

OUT

ESTABL

THINK

RATHER

JUDGME

THERET

DETERM

ALLEGE

POSSIB

FACTS

NOCC

1117

8254

4479

5725

5792

3505

4746

1301

3396

1867

2141

1605

6980

2640

4561

2857

1051

930

3228

2855

1068

3608

2216

9898

1426

1275

6240

997

1342

2384

888

1145

3218

1034

3759

2259

2379

10226

5447

2053

1857

1243

1039

1812

1088

4704

1264

1881

4463

766

4389

2947

1035

917

10581

1022

5030

3766

1018

4095

E

6.27

7. 17

6. 80

7.04

7.03

6.92

7. 12

6.43

6.86

6.54

6.53

6.38

7.01

6.69

6.85

6.66

6.25

6. 16

6.65

6.70

6.13

6.91

6.58

6.97

6.43

6.24

6.81

6.14

6.40

6.60

6.15

6.31

6.47

6.22

6.91

6.77

6.69

6.83

6.76

6.30

6.53

6.38

6.08

6.40

6.26

6.93

6.25

6.57

6.77

6.00

7.00

6. 74

6. 18

6. 15

7.06

6.05

7.02

6.72

6. 18

7.00

Table X. Sorted

EL

6. 38

7.2C

6.77

7.06

7.02

6.89

7.09

6.54

6.90

6.60

6.69

6.56

6.94

6.75

6.97

6.93

6.31

6.38

6.65

6.8 1 6.38

6.98

6.62

6.98

6.47

6.55

6.73

6.22

6.55

6.80

6.22

6.45

6.45

6.31

6.92

6.83

6.74

6.76

6.86

6.77

6. 60

6.46

6.24

6.63

6.34

6.88

6.40

6.65

6.90

6.11

6.99

6.72

6.28

6.21

7.17

6.35

7.01

6.81

6.23

7.01

by G

PZD

25.82

70.40

47.13

61.83

60.02

49. 37

60 .62

30.46

51.41

34.88

38.42

32.84

60.81

41.60

55.56

46.96

24.95

24.87

41 . 10

45.48

25. 70

53.68

36.50

62.35

29.48

30.65

49. 18

22.68

31.03

43.29

21.93

27.67

33.71

24.34

52. 39

43.14

40.88

55.75

54.24

40 . 39

34.99

28.61

23. 17

35.18

25.44

55.00

27.63

36.35

54.28

19.40

57.04

44.46

23.63

22.00

73. 19

24.95

59.45

47.86

22.98

55.79

AVG

0.0286

0.2237

.1251

1605

1599

0943

1287

0.0340

0.1062

0.0494

0.0591

0.0431

0. 1897

0.0697

0. 1294

0.0842

0.0273

0.0252

0.0882

0.0784

0.0279

0.1017

0.0609

0.2884

0.0387

0.0345

0. 1705

0.0265

0.0389

0.0629

0.0230

0.0304

0.0815

0.0267

0.1018

0.0592

0.0654

0.2858

0. 1464

0.0580

0.0502

0.0339

0.0271

0.0483

0.0294

0.1297

0.0351

0.0500

0. 1255

0.0206

0.1 164

0.0788

0.0298

0.0246

0.3119

0.0278

0. 1314

0. 1091

0.0272

0.1137

G

2.52

2.55

2.55

2. 56

2.56

2.56

2.57

2.57

2.58

2.58

2.60

2.60

2.61

2.61

2.63

2.65

2.65

2.65

2.68

2.72

2.72

2.73

2.73

2.75

2.75

2.76

2.77

2.77

2.78

2.79

2.80

2.81

2.83

2.85

2.86

2.87

2.87

2.91

2.91

2.91

2.91

2.91

2.92

2.94

2.94

2.95

2.95

2.97

2.98

2.98

3.00

3.00

3.00

3.00

3.01

3.03

3.04

3.04

3.04

3.05

EK

0.29

0.92

0.45

0.63

0.64

0. 56

0.78

0.34

0. 54

0.36

0.38

0.35

0.57

0.42

0.50

0.48

0.28

0.27

0.39

0.46

0.27

0. 53

0.36

0.45

0.31

0.33

0.43

0.24

0.32

0.46

0.24

0.30

0.31

0.25

0. 56

0.51

0.41

0.38

0.47

0.37

0.35

0.30

0.22

0.35

0.26

0.47

0.27

0.37

0.43

0.20

0.65

0.45

0.23

0.24

0.54

0.25

0.64

0.40

0.23

0.60

GL

2.73

2.17

3.69

3.13

3.62

4.37

2. 85

2. 82

3.22

3.01

3.68

3.70

5.09

3.06

3. 59

3.60

6. 00

2.44

3.45

3.64

3.38

3.16

4.42

2.96

3.52

2. 84

4.34

4.15

2.92

3. 10

3.45

3.28

5.57

3. 19

7.29

3.49

3.12

4.29

3.87

2.72

3.93

3.18

3.93

3.03

3.71

3.89

3.46

3.17

4.08

3.43

6. 13

17.95

3.20

3.67

4.08

3.31

3.95

3.37

3. 70

2.90

EKL

0.23

0.62

0.30

0.46

0.42

0.36

0.52

0.28

0.38

0.28

0.30

0.26

0.33

33

41

43

18

0.25

0.27

0.33

0.23

0.43

0.26

0.41

0.24

0.29

0.27

0.18

0.28

0.34

0.19

0.24

0.19

0.21

0.34

0.36

0.32

0.27

0.32

0.35

0.27

0.24

0.18

0.29

0.21

0.33

0.22

29

34

0.17

0.37

0.18

0.20

0.18

0.49

0.22

0.40

0.33

0.18

0.46

125
VOTES WORD NOCC EL PZD

1 10

4

APPELL

REGARD

EVERY

AMONG

STATE

GIVE

END

RELATE

MATTER

GENERA

FIND

ITAL

FOL

THOSE

INTEND

APPLY

EMPHAS

PARTIC

HEREIN

TIMES

THREE

NEVERT

EXPRES

PAID

TOOK

THERE8

HOW

YEARS

COME

GROUND

SHOW

APPARE

LIKEWI

TAKEN

OBTAIN

MAKES

ENTERS

ACT

TESTIM

NECESS

TOGETH

FAILED

TRUE

BEL I EV

RECOGN

SUPRA

WHOSE

CONTAI

JURY

CLEAR

HEARD

BEYOND

SET

OBVIOU

MERE

SHOWIN

CONSTR

SHOWN

OCCASI

LONG

Table

14543

1466

922

579

9231

1490

6422

839

4313

5262

1954

11360

5682

2527

1333

806

1012

2381

2599

751

2437

370

2022

2316

1080

712

739

2601

663

2629

1649

1334

404

2518

1498

565

2920

5147

3650

3477

861

1442

1140

1176

1033

2573

655

2096

5530

1537

903

754

2964

645

654

829

380 5 1106

742

1047

X.

6.53

6. 39

6. 1 I 5.83

6.85

6. 32

6.81

5.92

6.91

6.87

6.51

6.67

6.67

6.73

6.29

6.00

5.96

6.48

6.23

5.95

6.70

5.50

6.51

6.25

6.15

5.99

5.93

6.53

5.90

6.68

6.36

6.43

5.52

6.67

6. 18

5.73

6.78

6.65

6.42

6.93

6.04

6.29

6.23

6.22

6.10

6.29

5.89

6.55

6.41

6.52

5.97

5.87

6.71

5.87

5.82

5.78

6.58

6.15

5.95

6.23

Sorted

6.44

6.52

6.22

5.93

6.80

6.45

6.71

6.12

6.96

6.82

6.66

6.57

6.57

6.77

6.39

6.08

6.00

6.76

6.70

6.09

6.73

5.71

6.61

6.25

6.28

6.1

6.01

6.56

6.00

6.77

6.59

6 . 5 3 5.64

6.76

6.30

5.98

6.87

6.59

6.41

6.93

6.16

6.48

6.36

6.34

6.25

6.25

6.04

6.65

6.31

6.57

6.07

5.99

6.84

6.09

5.99

6.16

6.55

6.36

6.03

6.32

by G

50. 16

30.80

22.31

15.81

62.06

29.78

51.86

20.04

55.19

52.92

37.75

45. 18

45.18

42.43

27.63

19.63

19.59

42.12

41.75

19.21

41.18

11.92

36.01

28. 16

24.46

19.02

17.89

37.10

17.40

44. 16

33.89

30.84

11 .70

43.07

27.40

16.27

48.58

45.56

34.65

52.20

20.91

30.31

26.23

25.67

23.51

29.21

17.70

38.12

34.27

33.48

19.93

17.74

46.54

18.23

17.02

20.53

40.50

25.74

18.38

24.80

AVG

0. 3877

0.0380

0.0244

0.0152

0.2417

0.0399

0. 1570

0.0233

0. 1166

0.1338

0.0519

0.2755

0. 1378

0.0642

0.0361

0.0212

0.0246

0.0625

0.0670

0.0201

0.0677

0.0096

0.0546

0.0616

0.0302

0.0192

0.0191

0.0687

0.0173

0.0728

0.0470

0.0364

0.0106

0.0697

0.0397

0.0151

0.0873

0.1370

0.1010

0.0937

0.0222

0.0414

0.0309

0.0322

0.0261

0.0636

0.0179

0.0578

0. 1470

0.0425

0.0241

0.0209

0.0798

0.0187

0.0170

0.0227

0. 1054

0.0303

0.0206

0.0280

3.05

3.05

3.05

3.05

3.06

3.06

3.07

3.10

3.11

3.11

3.11

3.12

3. 12

3.12

3.14

3.14

3.16

3.17

3.17

3.18

3.19

3.19

3.21

3.21

3.21

3.22

3.23

3.24

3.24

3.25

3.26

3.26

3.26

3.27

3.28

3.28

3.29

3.30

3.30

3.31

3.31

3.32

3.33

3.33

3.33

3.34

3.34

3.35

3.35

3.35

3.35

3.35

3.36

3.36

3.36

3.37

3.38

3.38

3.38

3.39

EK

0.23

0.32

0.22

0.17

0.39

0.29

0.44

0.20

0.53

0.47

0.35

0.37

0.37

0.46

0.25

0.19

0.19

0.41

0.36

0. 19

0.40

0.13

0.34

0.23

0.24

0.19

0.19

0.31

0.18

0.38

0.32

0.30

0.12

0.37

0.23

0.17

0.42

0.32

0.25

0.52

0.21

0.29

0.26

0.24

0.23

0.23

0.18

0.35

0.24

0.33

0.18

0.17

0.45

0.18

0. 17

0. 19

0.30

0.24

0.18

0.23

GL

5.26

3.05

3.79

3.70

4.64

3.67

6. 84

4.01

4. 12

5.01

3.70

7.32

7.39

3.52

4.27

4.78

5.19

3.48

5.86

3.80

3.87

3.20

4. 18

4.69

4.38

3.25

3.80

4.19

3.88

5.73

3.21

3.32

4.45

4.04

5.62

3.07

4.02

6.21

3.88

4.91

3.86

3.79

4.42

3.34

3.94

4.77

3. 38

5.43

4.31

5.39

5.06

3.90

3.72

2.92

3.95

3.12

4.65

3.23

5.02

3.84

EKL

0.16

0.26

0.18

0. 14

0.25

0.23

0.22

0.16

0.38

0.28

0.28

0. 19

0.19

0.33

0.21

0.15

0.13

0.32

0.25

0.16

0.30

0.12

0.26

0.16

0.19

0.17

0.15

0.23

0.15

0.29

0.28

0.26

0.10

0.31

0.17

0.15

0.34

0.20

0.20

0.35

0.17

0.23

0.20

0.21

0.18

0.15

0.16

0.25

0.17

0.24

0.14

0.14

0.35

0.18

0. 14

0.18

0.21

0.22

0.14

0.20

126
VOTES

3

2

2

1

1

5

1

6

3

WORD

SUSTAI

PUT

MASS

TERMS

WHOM

CONCED

LESS

FAVOR

MANNER

SUGGES

SEVERA

LATTER

STILL

NOTED

CONST I SUBSTA

ALREAO

RESULT

ORDERE

AGREE

STAT

CONDIT

JUSTIF

TESTIF

OPERAT

WHOLE

SHOWS

REV

FOREGO

PROCEE

RAISED

READS

DOING

VALID

KNOWN

RELATI

BECAME

PROPER

HOLDIN

ACTION

LANGUA

SUBSEQ

ESSENT

ORDER

FORTH

INSIST

ERROR

NONE

HERETO

ONCE

PETITI

SUPREM

OCCURR

SEC

SPECIF

ISSUE

NEW

APPROX

MOTION

CAREFU

NOGC

2600

719

4687

1583

832

485

923

1249

1259

782

1243

833

660

710

4132

2527

542

3328

1180

707

1245

2779

885

3484

4207

651

1078

1484

626

5021

1050

769

625

768

1083

2530

734

5913

1008

8248

1492

1263

651

6773

1458

368

3841

506

498

375

7623

1904

1248

6808

2900

3113

4744

704

6621

453

Table X.

E

6.65

5.88

5.77

6.33

6.00

5. 58

6.08

6.22

6. 30

5.94

6.32

6.04

5.86

5.88

6.41

6.62

5.68

6.85

6. 14

5.91

5.90

6.46

5.90

6.35

6. 52

5.74

6.16

6.07

5.73

6.79

6.00

5.89

5.71

5.83

6.12

6.54

5.81

6.40

6.05

6.94

6.22

6.25

5.83

6.78

6.25

5. 36

6.56

5.58

5.41

5. 32

6. 19

6. 16

6.05

6.65

6.65

6.61

6.68

5.79

6.71

5.42

Sorted

EL

6.89

96

73

39

13

5.83

6.17

6.37

6.37

6.06

6.36

6.14

6.07

6.02

6.49

6.71

5.8 0 6.86

6.33

6. 10

5.93

6.47

6.07

6.35

6.45

5.78

6.35

6.08

5.96

6.84

6.28

6.03

5.89

5.92

6.17

6.53

6.08

6.34

6.20

6.92

6.23

6.37

5.98

6.77

6.40

5.51

6.66

5.82

5.64

5.60

6.44

6.24

6.11

6.62

6.68

6.66

6.72

5.87

6.84

5.79

by G

PZD

46.24

17.40

16.98

28 .46

20.08

14.00

21.63

26.87

27.29

18.68

27.25

20.23

18.08

18.04

42.99

41.60

14.08

48.50

26.23

18.98

19.10

35.52

19.85

31 .74

39.56

14.87

25.25

22.72

16.64

55.19

23.93

18.30

16.04

17.06

22. 19

37.10

18.61

36.91

22.76

64.55

25.78

26.99

16.76

58.32

28.80

10.41

44.80

14.23

12.60

11.02

40.39

27.44

21.78

49.60

42.28

42.88

48.09

15.77

53.90 ' 13.51

AVG

0.0 753

0.0197

0.1483

0.0424

0.0228

0.0140

0.0250

0.0364

0.0329

0.0208

0.0331

0.0235

0.0176

0.0182

0. 1058

0.0693

0.0141

0.0911

0.0 324

0.0187

0.0383

C.0760

0.0235

0.0969

0.1145

0.0169

0.0297

0.0446

0.0163

0. 1373

0.0290

0.0220

0.0167

0.0207

0.0285

0.0662

0.0196

0.1591

0.0265

0.2329

0. 0411

0.0363

0.0173

0. 1918

0.0391

0.0096

0. 1051

0.0136

0.0121

0.0094

0.2198

0.0474

0.0347

0. 1929

0.0790

0.0831

0. 1295

0.0179

0.1942

0.0118

G

3.40

3.40

3.41

3.43

3.43

3.43

3.44

3.45

3.46

3.46

3.47

3.47

3.47

3.47

3.48

3.48

3.49

3.50

3.50

3.50

3.51

3.52

3.52

3.53

3.54

3. 54

3.55

3.55

3.55

3.56

3.56

3.56

3.56

3.58

3.59

3.61

3.61

3.62

3.62

3.64

3.66

3.67

3.67

3.68

3.68

3.68

3.69

3.70

3.70

3.70

3.73

3.73

3.73

3.75

3.75

3.76

3.77

3.77

3.78

3.79

EK

0.40

0.17

0.12

0.25

0.19

0. 14

0.21

0.23

0.27

0.18

0.26

0. 19

0.18

0.17

0.28

0.36

0.14

0.49

0.23

0. 19

0.15

0.26

0. 18

0.24

0.27

0.14

0.23

0.18

0. 16

0.40

0.21

0. 16

0.15

0. 16

0.21

0.30

0.18

0.23

0.21

0.39

0.21

0.24

0. 16

0.31

0.25

0.10

0.29

0.14

0.12

0.11

0. 19

0.21

0. 18

0.27

0.34

0.32

0.31

0.15

0.30

0.13

GL

2.63

5.70

4.36

3.35

3.68

3.42

3.99

4.09

6.32

3.55

7.53

3.63

2.94

4.48

7. 53

4.62

4.07

3.97

6. 13

3.35

6.23

3.88

4.41

3. 72

4.52

5.73

3.06

9.27

3.70

6. 15

3.95

3.85

5.74

4.77

4.34

5.77

3.09

5.71

4.43

4. 77

5. 17

3.97

3. 52

11.48

4.54

4.72

4.33

4. 14

6.07

3.77

5. 82

6.65

4.81

4.50

5.03

4.98

4.33

4.01

3.36

3.84

EKL

0.41

0.13

0.10

0.21

0.17

0.13

0.17

0.21

0.19

0.16

0.18

0.17

0.17

0.14

0.15

0.27

0.12

0.34

0.18

0. 17

0.11

0.21

0.15

0.19

0.18

0.10

0.22

0.12

0.14

0.26

0.19

0.15

0.12

0.12

0.16

0.20

0.17

0.15

0.17

0.31

0.16

0.21

0.15

0. 19

0.20

0.09

0.24

0.12

0.09

0.10

0.18

0.14

0.15

0.21

0.25

0.23

0.26

0.12

0.33

0.12

127
VOTES

I

556

4

4

WORD

NATURE

SOUGHT

FAILUR

A 0 M I T T DUTY

DISSEN

AMOUNT

TAKE

MUCH

INCLUD

PART I E PREVEN

THROUG

QUOTED

ARGUED

BECOME

CITY

DAY

PREVIO

RENDER

DONE

MOVED

CHARGE

DIFFER

APPEAR

SECOND

DATE

CONTRA

DIFFIC

PURPOS

DECISI

FIND IN

BROUGH

VARIOU

HEARIN

NEVER

SOLELY

HEK

REACHE

FILED

LIKE

DESIRE

EXISTS

MAKING

QUITE

DUE

PLACED

OTHERW

RELIES

CODE

SEEMS

ALONE

RULE

INSTEA

REFUSE

COMPLA

COMPAN

PARTY

FULLY

ALLEGI

NOGG

1 1«5

1 1 32

1630

1667

1873

751

3110

1484

693

2632

3496

956

1954

591

396

1158

5969

2189

1040

1657

1079

492

4622

1714

3855

2415

1983

8033

578

4138

3988

3437

1534

815

2525

976

441

7548

539

5362

738

507

376

1060

307

1937

781

1095

301

4152

647

536

4090

328

1286

3971

4677

2643

591

320

Table X.

E

6. 16

6. 1 I 6.16

6.3 2 6.25

5.48

6.49

6. 38

5.99

6.71

6. 55

6.00

6. 52

5.60

5.47

6.07

6.24

6.41

6. 16

6.30

6.09

5.61

6.48

6.46

6.95

6.53

6.31

6.56

5.72

6. 76

6.52

6. 56

50

99

28

01

50

6. 30

5.63

6.67

5.93

5.38

5.38

6. 19

5. 32

6.40

5.88

6. 14

5.28

6.21

5.88

5.73

6.56

5.29

6.14

6.40

6. 19

6.26

5.74

5. 18

Sorted

EL

6.3 1 6.33

6.43

6.32

,30

73

52

47

11

6.76

6.59

16

,56

,85

.7 1 ,30

6.23

6.46

6.31

6.4 5 6.28

5.75

6.47

6.55

7.00

6.6 1 6.41

6.49

5.87

6.76

6.69

6.59

6.59

6.12

6.31

6.15

5.74

6.20

5.86

6.9 1 6.08

5.78

5.59

6.33

5.46

6.47

6.05

6.42

5.48

6.18

5.98

5.87

6.70

5.52

6.22

6.45

6.05

6.33

5.93

5.47

by G

PZD

2 5 . , 8

2 5 . <i 4

30. 16

2 8.87

28.35

1 3. A 3 37. 56

30.3 5 19.13

43.41

41.71

21.44

34.61

15.13

12.15

25.36

38.05

34. 16

24. 57

31 .74

24.57

13.40

40.69

33. 14

57.68

38.50

31 . 37

52.96

15.06

49. 30

46.58

41 .56

33.74

19.96

31 . 59

21 .32

12.87

31 .89

14.91

55.26

18.87

13.74

10.94

25.14

9.39

32.08

18.91

27.18

9.62

29.55

16.87

14.79

47. 18

10.07

24.49

37 .44

32.65

31.93

16.00

9.66

AVG

0.0313

0.0316

0.0459

0. 0436

0.0506

0.0191

0.0869

3.0407

0.0187

0.0716

0.0960

0.0265

0.0531

0.0149

0.0117

0.0320

0. 1706

0.0607

0.0277

0.0464

0.0282

0.0149

0. 1234

0.0466

0. 1045

0.0656

0.0555

0.2158

0.0155

0. 1096

0. 1070

0.0995

0.0460

0.0214

0.0716

3.0254

0.0118

0.2095

0.0139

0. 1589

0.0198

,0143

,0104

,0282

,0083

0.0542

0.0208

0.0307

0.0090

0.1 146

0.0179

0.0152

0. 1055

0.0088

0.0351

0. 1136

0. 1180

0.0726

0.0159

0.0088

G

3.80

3.80

3.81

3.82

3.82

3.84

3.85

3.85

3.85

3.86

3.86

3.86

3.87

3.88

3.88

3.89

3.90

3.92

3.93

3.94

3.94

3.94

3.96

3.96

3.97

3.97

3.97

3.98

3.98

3.99

4.00

4.00

4.00

4.01

4.03

4.03

4.03

4.05

4.07

4.09

4.09

4.09

4.09

4.11

4.11

4.13

4.15

4.16

4. 16

4.17

4. 19

4.20

4.23

4.25

4.26

4.27

4.27

4.28

4.28

4.31

EK

0.22

0.21

0.24

0.23

0.21

0. 12

0.27

0.27

0.19

0.39

0.29

0. 19

0.30

0. 14

0.12

C.23

0.18

0.26

0.22

0.23

0.21

0. 13

0.24

0.29

0. 56

0.31

0.23

0.23

0.14

0.41

0.30

0.26

0.29

0.20

0.21

0. 19

0. 12

0.20

0. 14

0.33

0.17

0. 12

0.11

0.22

0.09

0.25

0. 16

0.25

0.09

0. 17

0.16

0. 14

0.31

0.10

0. 19

0.22

0.17

0.20

0. 14

0.09

GL

4.10

4.23

4.43

5.59

5.09

3.90

3.75

3.52

3.99

3.68

4.47

3.57

4.00

4.09

3.34

3.96

5. 82

9. 83

3.68

6.39

4.53

4.21

4.95

3.56

9.43

5.63

4. 85

7.29

3.51

6.33

5. 57

3.90

3.64

3. 74

6. 14

4. 18

4.06

4. 75

4. 15

3.46

3.62

3.97

3. 84

3.75

3.74

3. 79

4.20

3.79

3.91

5.98

3.41

3.50

12.48

3.97

4.13

4.90

10.01

5.91

3.71

4.05

EKL

0.19

0.20

0.21

0. 17

0.17

0.11

0.22

0.23

0.17

0.31

0.22

0.17

0.24

0.12

0.12

0.19

0.13

0.17

0.20

0. 19

0.18

0.11

0.18

0.25

0.32

0.23

0.19

0.15

0.13

0.25

0.23

0.23

0.27

0.16

0.15

0.16

0.12

0.14

0.13

0.36

0.16

0.12

0.10

0.21

0.09

0.22

0.15

0.23

0.09

0.13

0.15

0.13

0.20

0.09

0.17

0.19

0.09

0.16

0. 14

0.09

128
VOTES

14

6

3

5

19

13

12

3

3

6

10

3

2

2

2

5

1

3

16

8

7

57

WORD

VOORHI

VIEW

ADDED

AUTHOR

CORREC

ORIGIN

HENCE

CONCER

ATTEMP

CALLED

POINT

RELIED

STATIN

SAID

EVER

JURISD

EACH

VIRTUE

FULD

DESMON

WHERE I AGAIN

ABLE

NAMELY

ARGUES

COMPLE

STATEM

OVERRU

OFFICE

CLAIME

FAILS

USE

PUBLIC

SOMEWH

FAR

APPEAL

SEEKS

ME NT I 0

COKPAR

FROESS

APPLIC

GRANTE

REMAI

COUNTY

ARGUME

CONTRO

EXISTE

ADDITI

HIMSEL

DIRECT

OPPORT

SOMETI

DISMIS

ENTIRE

RECORD

INTERE

WHILE

REVIEW

EMPLOY

WILL

NOCC

209

1406

587

4898

1358

2053

447

1797

1404

1618

1487

487

385

10747

481

3056

3332

322

208

230

560

766

416

316

443

1709

2732

1644

4060

921

426

3852

4658

236

923

9096

374

694

418

209

4168

1574

1592

6245

1528

2941

1029

1708

864

5706

545

237

2755

1350

6093

3637

2749

2347

6062

7140

E

4. 80

6.3b

5.62

6.78

6. 14

6.23

5.43

6.57

6.05

6.40

6.35

5.62

5. 43

7.07

5.47

6.00

6.68

5.21

4.73

4.86

5.60

6.00

5.37

5.27

5.52

6.30

6.32

6.23

6.26

5.97

5.21

6.29

6. 33

5. 13

6.11

6.80

5.15

5.91

5.42

4.78

58

25

35

62

26

48

Table X.

6.06

6. 39

5.95

6.95

5.53

5.05

5.96

6. 30

6.91

6. 36

6.82

6.02

5.98

6.84

Sorted

EL

5.23

6.48

5.77

6.8 1 6.38

6.39

5. 68

6. 59

6.42

6.57

6.42

5.80

5.67

6.93

5.65

6. 10

6.69

5.46

5.20

5.24

5.92

6.11

5.64

5.44

5.67

6.45

6.36

6.42

6.12

6.17

5.68

6.27

6.30

5.27

6.24

7.06

5.62

6.02

5.57

5.18

6.60

6.34

6.38

6.52

6.37

6.55

6.17

6.49

6.10

6.92

5.75

5.22

6.48

6.4 1 6.98

6.32

6.85

6.30

5.89

6.74

by G

PZD

7.32

30.95

13.96

52 . 32

28.57

32.01

12.26

34.76

29.18

32. 76

29.48

13.89

11 . 77

69. 15

12.23

29.6 7 43.90

9.55

7.C9

7.47

15.66

19.32

11.7 7 9.36

12.23

31 .40

34. 16

30.46

33.93

21.44

12.15

36. 12

35.78

7.73

22.61

77.61

11.32

17.39

11.09

6.9 8 47.37

28.35

30.46

52.43

28.69

39.93

22.08

32. 12

19.85

58.62

13.70

7.39

35.90

28.53

60.51

35.33

46.31

32.72

32.50

62.55

AVG

0.0059

0.0375

0.0144

0.1319

0.0370

0.0558

0.0118

0.0468

0.0376

0.0444

0.0407

0.0134

0.0112

0.2803

0.0127

0.0812

0.0859

0.0091

0.0057

0.0065

0.0155

0.0209

0.0107

0.0080

0.0 136

0.0455

0.0720

0.0456

0.1032

0.0261

0.0125

0. 1059

0.1226

0.0070

0.0247

0.2637

0.0117

0.0191

0.0121

0.0062

0. 1 134

0.0425

0.0428

0. 1787

0.042 9

0.0849

0.0286

0.0453

0.0241

0. 1575

0.0146

0.0068

0.0790

0.0369

C . 1675

0.0944

0.0751

0.0676

0.1653

0. 1944

G

4.32

4.33

4.33

4.35

4.35

4.38

4.38

4.40

4.42

4.43

4.43

4.43

4.44

4.45

4.47

4.48

4.53

4. 56

4.57

4.60

4.62

4.64

4.69

4.71

4.75

4. 76

4.77

4.78

4.82

4.84

4.84

4.86

4.86

4.87

4.89

4.94

4.95

4.96

4.96

4.96

4.97

4.97

4.99

5.00

5.01

5.05

5.05

5.06

5.07

5.12

5.13

5. 15

5.16

5.20

5.25

5.26

5.29

5.34

5.38

5.49

EK

0.07

0.29

0. 13

0. 37

0.21

0.21

0.11

0. 34

0.25

0.31

0.25

0. 12

0.11

0.50

0. 11

0. 14

0.36

0.09

0.06

0.07

0.13

C. 18

0. 11

0.09

0.11

0.24

0.20

0.19

0.17

0.17

0.10

0. 18

0.20

0.07

0.20

0.30

0.10

0. 16

0.09

0.06

0.25

0.20

0.23

0.23

0.20

0.23

0. 19

0.25

0.17

0.44

0. 11

0.07

0.16

0.25

0.41

0.20

0.43

0. 15

0.11

0.26

GL

3. 98

7.01

3.95

4.61

4.34

5.63

3. 85

3.67

7.93

3.42

4.24

4.02

3. 86

6. 83

4.27

6.50

5. 12

3.99

4.05

4.06

3. 89

3.29

4.20

4.09

3.96

5.48

5.32

4.35

18.75

3.94

3.68

7.72

5.07

4. 12

4. 79

5.35

3. 75

4. 13

4.20

3.98

8. 13

5. 70

7.12

8.51

4. 22

5. 00

4. 18

4.68

3.60

6. 63

4. 15

4.18

5.01

6. 76

4.95

5.71

4.31

7.80

7.48

12. 86

EKL

0.07

0.20

0.12

0.28

20

18

11

26

19

0.27

0.21

0.12

0.11

27

10

11

25

0.09

0.07

0.07

0.14

0. 17

0.10

09

11

0.20

0.16

0.20

0.07

0.17

11

12

0.15

0.07

0.18

0.33

0.10

0.15

0.09

0.07

0.16

0.17

0.16

0.14

19

20

16

22

16

29

11

0.07

0.20

20

35

15

0.35

0.13

0.08

0.15

129
VOTES WORD NOCC E EL PZD AVG EK uii

2 F I LE 94 3 5.49 5.87 1 7. 06 0.0265 5.51 0.10 4. 17 0.12

6 RIGHTS 2108 6.30 6.33 30. 33 0.0581 5. 59 0.20 4.76 0.17

1 USED 2650 6.45 6.58 38. 16 0. 0734 5.62 0.24 4. 18 0.23

5 ANSWER 3398 6.42 6.41 39. 33 0.0913 5.64 0.22 9.44 0.13

13 NOT ICE 2855 6.04 6.18 30. 76 0 .0853 5. 70 0.14 6. 77 0.12

5 BAS IS 1500 6.41 6.47 30. 76 0.04 12 5. 82 0.26 5.60 0.21

8 SERV I

3855 6.04 6.05 29. 63 0.1114 5.82 0. 13 7.29 0.10

3 COMMON 4042 6.46 6.48 42. 58 0.1171 5.85 0.19 7.01 0.16

4 CONTIN 2382 6.37 6*40 34. 35 0.0634 5.85 0.21 10. 10 0.14

9 CLAIM 2565 6.24 6.24 32. 27 0.0735 5.91 0. 15 7.77 0.12

6 EXCEPT 3589 6.58 6.82 49. 79 0. 1046 5.95 0.26 4. 72 0.30

1

PRINCI 2158 6.46 6.43 34. 61 0. 0564 6.01 0.24 7.85 0.16

1 DAYS 1500 6.05 6.22 24. 99 0. 0447 6.03 0.14 3.91 0.17

9 COUNSE 3030 6.22 6.27 32. 54 0.0868 6.05 0. 15 5.28 0. 14

1 WEYGAN 251 4.57 5.40 8. 79 0.0050 6.09 0.05 3.57 0.09

5 PERMIT 2869 6.35 6.49 39. 63 0. 0820 6.17 0.17 6.36 0.17

8 RESPON 2872 5.94 6.00 29. 21 0.0772 6.24 0. 12 11. 25 0.08

STATES 2343 6.38 6.33 33. 37 0.0582 6.26 0.22 8. 54 0.13

2 MATTHI 249 4.57 5.37 8. 64 0.0049 6. 34 0. 05 4. 17 0.08

3 PLACE 1881 6. 36 6.45 32. 27 0.0528 6.46 0.21 5.21 0.19

8 ASS IGN 2654 6.00 6.12 29. (32 0.0715 6.48 0. 12 7. 19 0.11

WAY 177 1 6.21 6.45 32. 91 0.0472 6. 65 0.22 10.08 0. 16

RECEI

2801 6. 52 6.57 39. 10 0.0764 6.76 0.27 5. 74 0.21

2 COURSE 1500 6.22 6.45 30. 53 0.0421 6.86 0.21 4. 36 0.21

5 EXAMI N 3117 6.19 6.23 35. 56 0.0831 7.01 0.15 8.63 0. 11

3 SUPPOR 3151 6.65 6.67 46. 35 0.0855 7.06 0.24 9.79 0.18

1 PECK 216 4. 34 5.22 7. 43 0. 0043 7.17 0. 04 4.22 0.07

1 LEGAL 1650 6.25 6.30 28. 57 0.0423 7.41 0.19 9.77 0.14

5 REOUES 194 1 6.11 6.29 29. 44 0.0545 7.47 0.15 5. 99 0.15

3 REFERR 1309 6.24 6.43 28. 65 0.0341 8.37 0.24 5. 55 0.21

5 OBJECT 2 70 3 6.27 6.31 32. 5C 0.0742 8.66 0. 15 5.60 0.15

2 RETURN 2074 6. 24 6.32 31. 48 0.0589 8.81 0.15 9.23 0.14

6 COURTS 2033 6.28 6.36 31. 21 0.0553 9.19 0.16 5. 77 0.17

5 JUDGE 4000 6.52 6.64 46. 84 0.1181 10.30 0.19 6.80 0.20

Table X. Sorted by G

130
WOrUJ t,L HOWEVE 3333 7.09 7.1 1 WAS 56044 7.69 7.55

WHICH 25522 7. 70 7.56

FROM 19879 7.62 7.51

UPON 11816 7.46 7.40

FOR 45223 7.73 7.61

THE 442506 7.87 7.65

1 ONLY 6218 7.33 7.31

NOT 35835 7.75 7.60

THAT 89026 7.80 7.60

ALSO 5230 7.29 7.23

MADE 7999 7. 32 7.29

BUT 9174 7.48 7.37

BEEN 12072 7.50 7.41

HAVING 2006 6.67 6.86

DOES 4264 7.09 7.20

AND 128355 7.83 7.61

WITH 21624 7.64 7.51

THERE 12925 7.48 7.40

3 TIME 8254 7. 17 7.20

2 PLAINT 20986 7.02 6.94

WHEN 6875 7.28 7.24

CONTEN 3888 7.02 7.09

THEREF 3871 7.01 7.18

AFTER 6340 7.24 7.21

BECAUS 3553 7.00 7.1

ANY 13855 7.47 7.37

3 CASE 15261 7.45 7.36

WITHOU 4652 7. 10 7.17

1 ONE 9388 7.39 7.31

4 FACT 4658 7.06 7.10

HAS 10530 7.36 7.37

5 DEFEND 25773 7.20 7.12

WHERE 5794 7. 19 7.16

1 FOLLOW 6076 7.28 7.24

NE I THE 930 6. 16 6.38

THIS 29490 7.66 7.59

OTHER 8966 7.43 7.31

SHOULD 5689 7.20 7.20

CANNOT 2467 6.74 6.92

1 TWO 5130 7.11 7.11

WOULD 9678 7.34 7.23

MAY 9510 7.37 7.30

CONCUR 2290 6.65 7.30

DID 6224 7.24 7.17

7 CONCLU 3665 6.95 7.02

HAVE 13825 7.53 7.44

AAAAAA 2649 7.07 7.87

ARE 13721 7.46 7.39

WHETHE 5173 7.22 7 19

COULD 5096 7. 16 7.11

THEN 4583 7.12 7.07

4 AFFIRM 3897 6.89 7.23

BEFORE 5814 7. 19 7.23

THAN 4378 7.11 7.10

3 SUSTAI 2600 6.65 6.89

ALTHOU 1762 6.67 6.77

WERE 12911 7.43 7.31

HAD 15451 7.43 7.30

CAN 2822 O . y *f Table XI. Sorted by GI

5 5.90

95.73

94.41

92. 18

82.93

98.07

99.99

72. 14

96.97

98.15

67. 15

74.51

78.89

83.76

42.09

63.30

99.73

92.03

84.25

70 .40

57.71

69.87

57.11

62.21

68.47

57. 19

83.12

84.74

63.57

76.40

60.28

81.76

71 . L9

65.26

69.38

24.87

96.67

76. 17

66. 59

46.54

60.51

73.12

76.70

63.91

66.70

53.90

85.99

99.99

84.37

66.13

61.79

59. 19

63.53

68.55

59.38

46.24

38.65

79-. 91

82.44

49.15

AVG

0.0923

1.5630

0.6984

0. 5456

0.3232

1.2529

12. 1192

0. 1693

0.9798

2.4343

0. 1410

0.2213

0.2485

0.3306

0.0548

0. 1175

3.4562

0.5840

0.3545

0.2237

0.6097

0. 1866

0. 1094

0. 1050

0. 1745

0.0999

0.3703

0.4182

0. 1274

0.2540

0. 1249

0.2838

0.7468

0. 1562

0.1661

0.0252

0. 8106

0.2397

0.1511

0.0694

0. 1408

0.2580

0.2605

0.0643

0. 1665

0.1010

0.3761

0.0783

0.3766

0.1408

0. 1383

0.1242

0. 1109

0.1612

0.1198

0.0753

0.0487

0.3486

0.4205

0.0739

G

1.47

0.52

0.64

1.25

1.37

1.03

0.19

1.57

0.55

0.70

1.08

1.60

0.84

1.41

2.18

1.80

0. 53

1.15

1.30

2.55

1.25

1. 54

2.14

1.43

1.62

2.04

1.29

1.64

2.02

1.61

2.10

1.34

1.34

1.64

1.30

2.65

1.15

1.18

1.89

2.06

1.59

1.43

1.45

2.45

1.55

2.50

1.17

0.42

1.56

1.69

1.59

2.04

2.26

2. 12

2.23

3.40

1.78

1.43

1.49

1.61

EK

0.90

3.68

4.89

3.01

1.76

5.00

41. 17

1.38

6.95

9.48

1.33

1.25

2.21

1.96

0.51

0.96

15.25

3.46

1.87

0.92

0.64

1.20

0.71

0.90

1.06

0. 75

1.87

1.43

0.91

1.48

0.80

1.51

0.79

1.03

1.18

0.27

4.02

1.79

1.02

0. 57

0.85

1.34

1.38

0.73

1.03

0.64

2.52

4.32

1.85

1.04

0.95

0.82

0. 78

0.95

0.81

0.40

0.50

1.55

1.38

0.67

GL

1.76

1.78

1.79

1.83

1.83

1.87

1.87

1.88

1.90

1.92

1.95

1.97

2.06

2.07

2.07

2.11

2.14

2.15

2.17

2.17

2.24

2.24

2.24

2.25

2.27

2.28

2.37

2.38

2.39

2.40

2.40

2.41

2.43

2.43

2.44

2.44

2.45

2.45

2.45

2.46

2.47

2.49

2.50

2.51

2.52

2.52

2.53

2.55

2.55

2.57

2.58

2.60

2.61

2.63

2.63

2.63

2.66

2.67

2.68

2.68

EKL

0.62

1.33

38

19

95

1.59

1.93

0.82

I .56

1 .54

0.71

0.76

0.89

0.95

0.43

0.67

1.57

1.16

0.91

0.62

0.43

0.69

0.56

0.65

0.65

0.58

0.83

0.80

0.62

0.75

0.54

0.83

0.53

0.58

0.69

0.25

1.41

0.76

0.63

0.45

0.55

0.64

0.72

0.86

0.59

0.49

0.97

31.32

0.86

0.61

0.54

0.51

0.70

0.66

0.54

0.41

0.37

0.70

0.69

0.44

131
VOTES

8

1

WORD

C O.MS ID

DENIED

MANY

MORE

S INCE

NOR

MIGHT

MUST

BOTH

MERELY

THOUGH

HIS

HELD

BETWEE

NOTHIN

PART

REASON

THUS

BEING

FACTS

SUCH

ACCORD

THEREA

OBVIOU

CIRCUM

STILL

UNLESS

TRIAL

UNDER

INVOLV

ILL

I NS TAN

CONSIS

ABOVE

REGARD

EVEN

THEREO

SHOWS

S ITUAT

MAKES

CITED

EVIDEN

BECAME

EITHER

GIVEN

NOW

HERE

PRIOR

SHOWIN

AGAINS

I NTO

FOUND

MAKE

ANOTHE

SIMILA

DISCUS

THINK

NEVERT

SHOW

CASES

NOGC

5288

2053

1117

3050

2756

2099

1734

5208

2868

936

130 I 19529

3978

3231

1275

4746

6845

1622

3858

4095

18195

2721

1342

645

2543

660

1520

9898

10893

2933

8605

1867

941

1812

1466

1964

2640

1078

1358

565

140 1 12726

734

2033

2766

2384

3448

2379

829

5725

3583

3608

2535

1881

1243

1034

1035

370

1649

3896

E

7. 15

6. 30

6.27

6.94

6.89

6. 70

6.57

7. 18

6.85

6.21

6.43

7.32

7.04

6.84

6.24

7„ 12

7. 17

6.58

7.04

7.00

7.50

6.87

6.40

5.87

6.75

5.86

6.54

6.97

,62

.48

Table XI.

40

56

49

54

19

6.40

6.39

6.64

6.69

6. 16

6.42

3.73

6.41

7. 10

5.81

6.71

6.80

6.60

6.93

6.69

5.78

7.04

6.93

6.91

6.76

6.57

6. 38

6.22

6. 18

5. 50

6.36

6.86

Sorted

EL PZD

7.14 63.72

6.7 7 40. 39

6.38 25 . 82

6.9^ 49.49

6.93 48.65

6.86 43.14

6.63 34.27

7.22 66. 70

6.88 46.i>4

6.32 2 3.78

6.54 30.46

7.22 78.63

7.02 55.34

6.87 47.45

6.55 30.65

7.09 60

7.25 72

6.6 5 34.80

7.08 57.41

7.01 55.79

7.35 85.80

6.96 49.64

6.55 31.03

6.09 18.23

6.75 41.94

6.07 18.08

6.63 33.82

6.98 62.85

7.31 80.44

6.90 47.86

6.46 32.88

6.60 34.88

6.31 2 3.. 66

6.63 35.18

6.52 30.80

6.75 38.80

6.75 41.60

6.35 25.25

6.49 29.40

5.98 16.27

6.54 30.95

7.02 65.64

6.08 18.61

6.78 40.20

6.82 45.07

6.80 43.29

6.97 52.

6.74 40.

6.16 20.53

7.06 61.83

6.92 51.00

6.98 53.68

6.84 43.94

6.65 36.35

6.46 28.61

6.31 24.34

6.28 23.63

5.71 11.92

6.59 33.89

6.90 51.41

by GL

,69

,88

AVG

0. 1379

0.0580

0.0286

0.0822

0.0753

0.0581

0.0465

0.1412

0.0771

0.0248

0.0340

0.5396

0. 1058

0.0879

0.0345

0. 1287

0. 1850

0.0427

0. 1040

0.1137

0.4817

0.0745

0.0389

0.0187

0.0679

0.0176

0.0418

0.2884

0.2937

0.0789

0.2551

0.0494

0.0260

0.0483

0.0380

0.0509

0.0697

0.0297

0.0368

0.0151

0.0390

0.3461

0.0196

0.0532

0.0744

0.0629

0.0938

0.0654

0.0227

0.1605

0.0952

0. 1017

0.0681

0.0500

0.0339

0.0267

0.0298

0.0096

0.0470

0. 1062

G

2.06

2.91

2.52

1.98

1 .76

1.94

2.40

1.83

1.Q7

2.46

2.57

1.55

1.92

2.33

2.76

2.57

2.15

2.08

2.13

3.05

1.49

2.12

2.78

3.36

08

47

32

75

82

2.29

1.95

2.58

2.47

2.94

3.05

2.09

2.61

3.55

2.40

3.28

2.52

1.64

3.61

1.96

2.27

2.79

1.92

2.87

3.37

2.56

2.51

2.73

2.35

2.97

2.91

2.85

3.00

3.19

3.26

2.58

EK

0.93

0.37

0.29

0.66

0.62

0.53

0.39

1.08

0.59

0.26

0. 34

1.03

0.75

0.55

0.33

0.78

1.11

0.42

0.75

0.60

1.78

0.62

0.32

0.18

49

18

39

45

31

56

34

0.36

0.26

0.35

0.32

0.49

0.42

0.23

0.33

0.17

0.33

0.71

0.18

0.50

0.50

0.46

0.66

0.41

0.19

0.63

0.57

0.53

0.54

37

30

25

23

GL

2.68

2. 72

2.73

2.76

2.78

2.78

2.78

2.79

2.81

2.82

2.82

2.83

2.83

2.83

2.84

2.85

2.86

2.88

2.89

2.90

2.91

2.92

2.92

2.92

2.94

2.94

2.95

2.96

2.98

2.99

00

01

02

03

05

06

0.13

0.32

0.54

3.06

3.06

3.07

3.07

3.08

3.09

3.09

3.10

3. 10

3. 10

3.12

3.12

3.12

3. 13

3.14

3. 16

3. 17

3. 17

3.18

3.19

3.20

3.20

3.21

3.22

EKL

0.56

0.35

0.23

0.45

0.43

0.40

0.30

0.64

0.39

0.22

0.28

0.60

0.47

0.38

0.29

0.52

0.64

0.31

0.52

0.46

0.74

0.45

0.28

0.18

0.33

0. 17

0.30

0.41

0.69

0.40

0.24

0.28

0.21

0.29

0.26

0.35

0.33

0.22

0.25

0.15

0.27

0.43

0.17

0.35

0.35

0.34

0.43

0.32

0.18

0.46

0.39

0.43

0.37

0.29

0.24

0.21

0.20

0.12

0.28

0.38

132
VOTES WORD

SHOWN

SUFFIC

HOLD

THERE B THESE

FIRST

CLEARL

THEIR

AGAIN

APP

THERET

ITSELF

SAKE

APPARE

ALL

BEL I EV

ARGUED

TERMS

AGREE

MOTION

ALLEGE

THERE I WHOSE

LAW

SEEMS

CONCED

CALLED

LEAST

FURTHE

ABOUT

VERY

UNTIL

APPLIE

FILED

PARTIC

ITS

PRESEN

WELL

OVER

ALONE

WHO

DIFFIC

THEY

LATER

THOSE

ESSENT

TAKE

SUGGES

DIFFER

PREVEN

WEYGAN

INDICA

WITHIN

REVERS

HIMSEL

PROVID

LIKE

LATTER

SUBJEC

BROUGH

NOCC

1 106

2484

1033

712

4753

4165

1145

6514

766

4769

1022

993

4992

1334

9021

1176

396

1583

70 7 6621

3766

1068

655

9658

647

485

1618

766

4546

3228

888

2347

1264

5362

2381

11061

5653

2259

2622

536

5241

578

7042

1426

2527

651

1484

782

1714

956

251

1901

4561

2857

864

5792

738

833

2855

1534

E

6. 15

6.72

6. 15

5.99

7.1

7.01

6.31

7.08

6.00

6.74

6.05

6.25

7.05

6.43

7.36

6.22

5.47

6.33

5.91

6.71

6.72

6.13

5.89

7.23

5. 88

5.58

6.40

6.00

7. 11

6.65

6.15

6.65

6.25

6.67

6.48

7.31

7.18

6.77

6.72

5.73

7.11

5.72

7. 14

6.43

6.73

5.83

6. 38

5.94

6.46

6.00

4.57

6. 64

6.85

6.66

5.95

7.03

5.93

6.04

6.70

6.50

EL PZD

6.36 25.74

6.8 1 42.92

6.35 24.61

6.11 19.02

7.07 59.79

7.04 57.15

6.45 27.67

7.02 6 1.75

6.11 19.32

6.72 44.92

6.35 24.95

6.33 24.38

7.07 60.73

6.53 30.84

7.26 74.78

34 25.67

71 12.15

39 28.46

10 18.98

84 53.90

6.81 47.86

6.38 25.70

6.04 17.70

7.20 74.29

5.98 16.87

5.83 14.00

6.57 32.76

6.11 19.40

7.13 61.94

6.65 41.10

6.22 21.93

6.70 39.22

6.40 27.63

6.91 55.26

6.76 42 . 12

7. 20 75.34

7.2 0 6 8.25

6.8 3 43.14

6.71 40.99

5.87 14.79

7.03 59.64

5.87 15.06

7.08 64.47

6.47 29.48

6.77 42.43

5.98 16.76

6.47 30.35

6.06 18.68

6.55 33.14

6.16 21.44

5.40 8.79

6.70 37.67

6.97 55.56

6.93 46.96

6.10 19.85

7.02 60.02

6.08 18.87

6.14 20.23

6.8 1 4 5.48

6.59 33.74

AVG

0.0303

0.0708

0.0 270

0.0192

0. 1275

0. 1 116

0.0304

0. 1756

0.0209

0. 1292

C.0278

0.0260

0. 1299

0.0364

0.2361

0.0322

0.0117

0.0424

0.0187

0. 1942

0.1091

0.0279

,0179

,2554

,0179

,0140

0.0444

0.0206

0. 1230

0.0882

0.0230

0.0628

0.0351

0. 1589

0.0625

0.2888

0.1558

0.0592

0.0701

0.0152

0. 1416

0.0155

0. 1897

0.0387

0.0642

0.0173

0.0407

0.0208

0.0466

0.0265

0.0050

0.0499

0. 1294

0.0842

0.0241

0. 1599

0.0198

0.0235

0.0784

0.0460

G

3.38

2.35

2.49

3.22

1.97

2.30

2.81

2.19

4.64

2.51

3.03

2.40

2.47

3.26

1.45

3.33

3.88

3.43

3.50

3.78

3.04

72

34

34

19

43

4.43

2.98

1.92

2.68

2.80

2.31

2.95

4.09

3.17

1.71

2.26

2.87

2.40

4.20

1.89

3.98

2.45

2.75

3. 12

3.67

3.85

3.46

3.96

3.86

6.09

2.45

2.63

2.65

5.07

2.56

4.09

3.47

2.72

4.00

EK

0.24

0.45

0.26

0. 19

0. 83

0.71

0.30

0. 70

0.18

0.41

0.25

27

76

30

46

0.24

0.12

0.25

0. 19

0.30

0.40

0.27

0. 18

0. 88

0. 16

0. 14

0.31

0.20

0.91

0. 39

0.24

0.42

0.27

0.33

0.41

1.13

0.88

0.51

0.43

0. 14

0. 79

0. 14

0.77

0.31

0.46

0. 16

0.27

0.18

0.29

0.19

0.05

0.42

0.50

0.48

C. 17

0.64

0. 17

0. 19

0.46

0.29

GL

3.23

3.24

3.24

3.25

3.27

3.27

3.28

3.29

3.29

3.31

3.31

3. 32

3.32

3.32

34

34

34

35

35

3.36

3.37

3.38

3.38

3.39

3.41

3.42

3.42

3.43

3.44

3.45

3.45

3.46

3.46

3.46

3.48

3.49

3.49

3.49

3. 50

3.50

3.51

3. 51

3.52

3.52

3. 52

3.52

3.52

3.55

3.56

3.57

3.57

3.59

3. 59

3.60

3.60

3.62

3.62

3.63

3.64

3.64

EKL

0.22

0.36

0.22

0.17

0.48

0.46

0.24

0.42

0.17

0.29

22

22

48

0.26

0.64

0.21

0.12

0.21

0. 17

0.33

0.33

0.23

0.16

0.54

0.15

0.13

0.27

0.17

0.53

0.27

0.19

0.30

0.22

0.36

0.32

0.54

0.58

0.36

0.29

0.13

0.44

0. 13

0.45

0.24

0.33

0.15

0.23

0.16

0.25

0.17

09

31

0.41

0.43

0.16

0.42

0.16

0.17

0.33

0.27

Table XI. Sorted by GL

133
WUKD NOCC E EL PZD AVG G EK GL EKL

RATHER 917 6. 15 6..2 1 22 .00 0 .0246 3. 00 0..24 3.67 0 .18

GIVE 1490 6. 32 6,.45 29 . 78 0 .0399 3. 06 0. 29 3.67 0 .23

C ONCER 1797 6. 57 6..59 34 .76 0..0468 4. 40 0, 34 3.67 0..26

ENTITL 2141 6. 53 6..69 38 .42 0..0591 2. 60 0.38 3.68 0 .30

WHOM 832 6. 00 6..13 20 .08 0..0228 3. 43 0. 19 3.68 0 . 17

INCLUD 2632 6. 71 6.,76 4 3 .41 0..0716 3. 86 0. 39 3.68 0..31

PREVIO 1040 6. 16 6. 3 1 24 .57 0..0277 3.93 0. 22 3.68 0..20

FAILS 426 5. 21 5. 68 12 .15 0,.0125 4. 84 0. 10 3.68 o .11

STATED 3698 6. 99 6. 99 54 .77 0. 0975 2. 37 Ot 68 3.69 0 .42

PROVIS 4479 6. 80 6. 77 47 .18 0..1251 2.55 0. 45 3.69 0..30

BASED 1605 6c 38 6. 56 32 .84 0. 0431 2. 60 0. 35 3.70 0..26

POSSIB 1018 6. 18 6. 23 22 .98 0. 0272 3. 04 0. 23 3.70 0..18

AMONG 579 5, 83 5. 93 15 .81 0- 0152 3. 05 0. 17 3.70 0.. 14

FIND 1954 6, 51 6. 66 37 .75 0. 0519 3. 11 0. 35 3.70 0..28

FOREGO 626 5. 73 5. 96 16 .64 0. 0163 3. 55 0. 16 3.70 0. 14

RESPEC 2579 6. 80 6. 82 44 .43 0. 0678 1. 99 0. 54 3.71 0.,34

SAY 1088 6c 26 6. 34 25 .44 0, 0294 2. 94 0. 26 3.71 0..21

FULLY 591 5. 74 5. 93 16 .00 0, 0159 4. 28 0. 14 3.71 0. 14

SET 2964 6 = 71 6. 84 46 .54 0. 0798 3. 36 0. 45 3.72 0. 35

TEST I F 3484 6. 35 6 - 35 31 .74 0. 0969 3. 53 0. 24 3.72 0. 19

VARIOU 815 5. 99 6. 12 19 .96 0. 0214 4. 01 0. 20 3.74 0, 16

QUITE 30 7 5. 32 5. 46 9 .39 t. 0083 4. 11 0. 09 3.74 0. 09

AMOUNT 3 I 1

6. 49 6. 52 37 .56 c

0869 3. 85 0. 27 3.75 0. 22

MAKING 1060 6. 19 6, 33 25 . 14 C. 0282 4. 11 0. 22 3.75 0. 21

SEEKS 374 5. 15 5. 62 11 .32 0. 0117 4. 95 0. 10 3.75 0. 10

WHAT 2883 6. 76 6. 79 44 .80 0. 0725 2. 52 0. 51 3.76 0. 32

ONCE 375 5. 32 5. 60 11 .02 0. 0094 3. 70 0. 11 3.77 0. 10

EVERY 922 6. 11 6 = 22 22 .31 0. 0244 3. 05 0. 22 3.79 0. 18

FAILED 1442 6o 29 6 c 48 30 .31 0. 0414 3. 32 0. 29 3.79 0. 23

DUE 1937 6o 40 6. 47 32 .08 0. 0542 4. 13 0. 25 3.79 0. 22

OTHERW 1095 6. 14 6. 42 27 .18 0. 0307 4. 16 0. 25 3.79 0. 23

TIMES 751 5. 95 6o 09 19 .21 0. 0201 3. 18 0. 19 3.80 0. 16

HOW 739 5 C 93 6. 01 17 .89 0. 0191 3. 23 0. 19 3.80 0. 15

LONG 1047 6. 23 6. 32 24 .80 0. 0280 3. 39 c. 23 3. 84 0. 20

CAREFU 453 5. 42 5. 79 13 .51 0. 0118 3. 79 0. 13 3.84 0. 12

EXISTS 376 5 S 38 5. 59 10 .94 0. 0104 4. 09 0. 11 3.84 0. 10

READS 769 5. 89 6. 03 18 . 30 0. 0220 3. 56 0. 16 3. 85 0. 15

HENCE 447 5. 43 5a 68 12 .26 0. 0118 4. 38 0. 11 3.85 0. 11

TOGETH 861 6. 04 6. 16 20 .91 0. 0222 3. 31 0. 21 3.86 0. 17

STATIN 385 5. 43 5. 67 11 .77 0. 0112 4. 44 0. 11 3.86 0. 11

RIGHT 5447 6. 76 6. 86 54 .24 0. 1464 2. 91 0. 47 3.87 0. 32

THREE 2437 6. 70 6. 73 41 .18 0. 0677 3. 19 0. 40 3.87 0. 30

COME 663 5. 90 6. 00 17 .40 0. 0173 3. 24 0. 18 3.88 0. 15

TESTIM 3650 6. 42 6. 41 34 .65 0. 1010 3. 30 0. 25 3.88 0. 20

CONDIT 2779 6. 46 6. 47 35 .52 0. 0760 3. 52 0. 26 3.88 0. 21

SEE 4704 6. 93 6. 88 55..00 0. 1297 2. 95 0. 47 3. 89 0. 33

WHERE I 560 5. 60 5. 92 15 .66 0. 0155 4. 62 0. 13 3.89 0. 14

CERTAI 3069 6. 87 6. 96 50.62 0. 0830 2. 20 0. 65 3.90 0. 42

BEYOND 754 5. 87 5. 99 17 .74 0. 0209 3. 35 0. 17 3.90 0. 14

DISSEN 751 5. 48 5. 73 13 .43 0. 0191 3. 84 0. 12 3.90 0. 11

FIND IN 3437 6. 56 6. 59 41 .56 0. 0995 4. 00 0. 26 3.90 0. 23

RELIES 301 5. 28 5. 48 9 .62 0. 0090 4. 16 0. 09 3.91 0. 09

DAYS 1500 6. 05 6. 22 24..99 0. 0447 6. 03 0. 14 3.91 0. 17

OWN 1857 6. 53 6. 60 34..99 0. 0502 2. 91 0. 35 3.93 0. 27

PURSUA 1039 6. 08 6. 24 23..17 0. 0271 2. 92 0. 22 3.93 0. 18

RECOGN 1033 6. 10 6. 25 23..51 0. 0261 3. 33 0. 23 3.94 0. 18

CLAIME 921 5. 97 6. 17 21..44 0. 0261 4. 84 0. 17 3.94 0. 17

DETERM 5030 7. 02 7. 01 59..45 0. 1314 3. 04 0. 64 3.95 0. 40

MERE 654 5. 82 5. 99 17..02 0. 0170 3. 36 0. 17 3.95 0. 14

RAISED 1050 6. 00 6. 28 23..93 0. 0290 3. 56 0. 21 3. 95 0. 19

Table XI. Sorted by GL

134
OTES WORD NOCC

AUUCU a nn p n R a t DCUUnC 11 JO

ArvuUc o

L.L. X •T*t 3

11 AU J"X JUL 1 A 9 11

11 1 3 u 3 3 A o

A

ci idc c r\ 1 Deo I Kc C "7

T M C T C A

I NSTt A 328

1 \;nnou VUUKn T

J, o a n 20 v eonccc rKUC JO 3 A o, ncrinp 1 AA O

1(.Coo FCC nil 7 A 3

Ml If Ul AOS O 7 3

A \JVIMT Q T 1UC1 F "X 3 9 Tup ni 1 r 1731

Dr\ CIC AT F L A 1 C O 3 7

1X Mrappro* r i\UA 1 U *T P NT PR P C IN 1 C r\ C £ V A U ACL r p i i

1 Cp nU AHofl 7( T1 AA KIn CFNIN '31u

1 A 1 1 P r I

•3. 3 A

pi ii n lUo

c. ni pi v AA1111 DFCMnM uconui'i 9 "X n

A 1 R F A n CAT 31i

K3 U AUo c

AAA, ^ q7 1U301

11 1 A V U r\ ltt J 9A ni inT p n 3 7 1 N A M F 1 V 3 1 O

11

NI ATI 1 R F 1 1 o CC 1 1U3

MATTPR PI A 1 1 C r\ A '3 1 O 1313

o UriL nn l3u 9 2 A A R F Fl 1 C F 1 9 fl A i AO O nM Fc MTin i T

I nu

A Q A

IN UIN C

C A A DUO

3 niCTIM Ul J 1 1 IN a77o. 71 D F A C W F con 3 3v

i1 noono

Ur rUI\ 1 c A c 3 1

A rlLC p r i p 7-+

a ri A 1 1 nl A

9 aH a7

7 cvbdfc C A r l\ C o dU3 A v 9 A A

IN C V C l\ 07 A

A p V I C T F C A 1 o 1 C

1 A 3 Q

c nMFT T 3 "I 7 A 3 f

1 1UoIC CUFO. 3 A A AO 3U

&

V F A D C T C AKo AO3 A A 1 J 1 PI ATFn 7lolfl 1

1 A

A DQ 1 p L C A 1 A

A CUUIflM1 Pr AA t\R A 1 fl nuv mhv/fcun AQ 3 A KuU n c i. 3 A o

11 rCUIv ncr k A9 1 A 1 O

cni ir u t 113A

1 oni r mt U 1 IN 1 1lnoAQ7f 1

T1 MTIN 1 FC MIN nU

i "3. 3 "3 13 3 3

C V C t\ Afl 1 oc CCTT T n o fcO 1 I U

1 A 9 9 A 1 U A A O

A9 QUESTI O "7 7 A 8 7 76

10 JURY 5530

Table XI.

E EL

5.62 5.77

Z ZN -» 6.07 6.30

5.52 5.67

7.45 T Z 1 7.41

6.85 Z. O Z 6.86

6.25 6.37

5.38 5.78

5. 29 5.52

4.80

y » ft 4.78 5.18

6.41 6.50

6.08 6.17

5.99 6.11

5.21 C Z z 5 .46

z c n 6.52 z c z 6.56

5.92 6.12

5 • 79 5.87

/ TO 6.78 Z O "7 6.87

5.62 5.80

6.67 Z "7 Z 6.76

5.18 5.47

4.7 3

C **> J". 3 . 3U

4.86 5.24

5.68 5.80

6» 7

Z nA 6.90

7.06 7 17 7.17

6.22 6.37

C A A 5 . 60 c 0 c 3.85

5.27 5.44

A 1 A 0 . L 0A "3 7 O. 31

A O 1 6.91 A A A 6.96

5.13 5.27

6. 14 Z TO 6.22

5.91 6.02

5.58 5.82

6.14 6.22

5.63 5.86

5.53 5.75

5.49 5.87

4. 57 5.37

6.51 6.61

6.01 6.15

6.06 6.17

5.05 5.22

6.45 6.58

6.53 6.56

5.88 6.05

5.37 5.64

5 . 42 5.5/

C Z 1 5.61 5.75

6.26 6.37

4. 34 5.22

6.11 6.33

6.35 6.42

6.29 6.39

5.47 C Z c 5.65

z O "> 6.8 3Z T Z 6.76

7.25 7 00 7.28

6.41 6.31

Sorted by

PZD AVG

13.96 0.0144

25.36 0.0320

12.23 0.0136

93.58 0.9097

48.50 0.0911

26.99 0.0363

13.74 0.0143

10.07 0.0088

7.32 0.0059

6.98 0.0062

29.89 0.0381

21.63 0.0250

19.13 0.0187

9.55 0.0091

34.61 0.0531

20.04 0.0233

15.77 0.0179

48.58 0.0873

13.89 0.0134

43.07 0.0697

9.66 0.0088

7.09 0.0057

12.87 0.0118

7.47 0.0065

14.08 0.0141

54.28 0.1255

73.19 .0.3119

26.87 0.0364

15.13 0.0149

9.36 0.0080

25.48 0.0313

55.19 0.1166

7.73 0.0070

24.49 0.0351

17.89 0.0191

14.23 0.0136

22.68 0.0265

14.91 0.0139

13.70 0.0146

17.06 0.0265

8.64 0.0049

36.01 0.0546

21.32 0.0254

22.08 0.0286

7.39 0.0068

38.16 0.0734

37.10 0.0687

18.91 0.0208

11.77 0.0107

11.09 0.0121

13.40 0.0149

28.69 0.0429

7.43 0.0043

25.44 0.0316

29.48 0.0407

27.63 0.0361

12.23 0.0127

55.75 0.2858

77.08 0.2395

34.27 0.1470

GL

G EK

4.33 0.13

3.89 U» 2 3 4.75 0.11

1.64 1.26

3. 50 0 . 49

A 7 3.67 0.24

4.09 0.12

4.25 0.10

4*32 0.07

4.96 0.06

2.48 0.31

3. 44 0.21

3.8 5 0.19

/ C A 4.36 A A A 0.09

3.87 A OA 0.30

3.10 0.20

0 77 3.77 0.15

3.29 A AO 0 . 42

4.43 Zt *1 -> 0.12

3.2 7 0.37

4.31 0. 09

4.57 0.06

4.03 0. 12

/ A A 4. 60 A A 7 0.07

3.49 A 7 A 0.14

2.98 A / T> 0.43

O A 1 3.01 A C /. 0. 34

3.45 0.23

3.8 8

/"» 1 Z 0. 14

H. 71 A A A 0. 09

3 « 0 U A "7 "7 0.22

3.11 A C "3 0.5 3 4 • 0 1 A A 7 0. 07

4.26 0.19

4 . 96 0.16

3.70 0.14

2.77 0. 24

/ z*\ -7 4.07

5.13 0.11

c CI 5.51 0.10

6.34 0.05

3.21 0. 34

4.03 0.19

5.05 0.19

5.15 0.07

C Z *T» 5 . 62 0. 24

3.24 0.31

4.15 0.16

4.69 0.11

4. 96 0. 09

3.94 0.13

5.01 0.20

7.17 0. 04

3.80 0.21

4.43 0.2 5 3.14 0.2 5

/ j 7 4.4 7 A 17 0.11

2.91 0. 38

2.17 1.03

3.35 0.24

GL

3. 95 0 . 12

3.96 0.19

3. 96 0.11

3. 97 0.76

3. 97 0.34

3.97 0.21

3.97 0.12

3. 97 0.09

3.96 0.07

3.98 0. 07

3. 99 0.25

3. 99 0.17

3. 99 0.17

3. 99 0 . 09

4. 00 0.24

4.01 0.16

4.01 0.12

4.02 0 . 34

4. 02 0.12

4. 04 0.31

4.05 0.09

4. 05 0.07

4. 06 0.12

4. 06 0.07

4.07 0.12

4.08 0. 34

4.08 0 . 49

4.09 0.21

4. 09 0.12

4. 09 0.09

4.10 0.19

/ 1 "1 4. 12 0*38

4.12 0.07

4.13 0.17

4. 13 0.15

4.14 0.12

4.15 0.18

4.15 0.13

4.15 0.11

4. 17 0.12

4.17 0.08

4.18 0.26

4.18 0.16

4.18 0.16

4.18 0.07

4. 18 0.23

4. 19 0.23

4. 20 0.15

4.20 0.10

4.20 0. 09

4.21 0.11

4.22 0.19

4.22 0.07

4.23 0.20

4. 24 0.21

4. 27 0.21

4.27 0.10

4.29 0.27

4.30 0. 62

4.31 0.17

135
VOTES

6

1

13

2

2

WORD NCCG E EL PZD AVG G EK GL EKL

WHILE 2749 6. 82 6 . 3 5 'i 6 . 3 1 0

0751 5. 29 0. 43 4. 31 0. 35

ERROR 384 1 6. 56 6 .66 44 .80 0. 1051 3. 69 0. 29 4. 33 0. 24

NEW 4744 6. 68 6 .72 4 8 .09 0. 1295 3. 77 0. 31 4. 33 0. 26

SHALL 6240 6. 8 1 6 . 7 3 49 . 1 8 0. 1 705 2. 77 0. 43 4. 34 0. 27

KNOWN 108 3 612 6 .17 22 . 19 0. 0285 3. 59 0. 21 4. 34 0. 16

CORREC 1358 6o 14 6 .38 28 .57 0. 0370 4. 35 0. 21 4. 34 0. 20

OVERRU 1644 6. 23 6 .42 30 .46 0. 0456 4. 78 0. 19 4. 35 0. 20

MASS 4687 5. 77 5 .73 16 .98 0. 1483 3. 41 0. 12 4. 36 0. 10

COURSE 1500 6. 22 6 .45 30 .53 0. 0421 6. 86 0. 21 4. 36 0. 21

THEM 350 5 6. 92 6 .89 49 . 37 0

0943 2. 56 0. 56 4. 37 0. 36

TOOK 1080 6. 15 6 .28 24 .46 0. 0302 3. 21 0. 24 4. 38 0. 19

STATUT 7283 6. 89 6 .80 53 . 15 0. 1985 2. 26 0. 48 4. 39 0. 29

JUST I885 5. 90 6 .0 7 19 .85 0. 0235 3. 52 0. 18 4. 41 0. 15

DUR I NG 2216 6. 58 6 .62 36 .50 0. 0609 2. 73 0. 36 4. 42 0. 26

TRUE 1140 6. 23 6 .36 26 .23 0. 0309 3. 33 0. 26 4. 42 0. 20

HOLD I

1008 6o 05 6 .20 22 . 76 0. 0265 3. 62 0. 21 4. 43 0. 17

FAILUR 1630 6. 16 6 .43 30 . 16 0. 0459 3. 81 0. 24 4. 43 0. 21

LIKEWI 40 4 5. 52 5 .64 1 1 . 70 0. 0106 3. 26 0. 12 4. 45 0. 10

PART I3496 6 .= 55 6 .59 41 . 71 0. 0960 3. 86 0. 29 4. 47 0. 22

NOTED 710 5. 88 6 .02 18 .04 0 . 0182 3. 47 0. 17 4. 48 0

14

SEC 6808 6. 65 6 .62 49 .60 0. 1929 3. 75 0. 27 4. 50 0. 21

OPERAT 4207 6. 52 6 .45 39 .56 0. 1 145 3. 54 0. 27 4. 52 0. 18

REOU I

6103 7; 06 7 . 1C 63 .98 0. 1665 2. 34 0. 74 4. 53 0. 47

DONE 1079 6. 09 6 .28 24 . 57 0. 0282 3. 94 0. 2 1 4. 53 0 . 18

FORTH 1458 6. 25 6 .40 28 . 80 0. 0391 3. 68 0. 25 4. 54 0. 20

AUTHOR 4898 6. 78 6 .81 52 .32 0. 1319 4. 35 0. 37 4. 61 0. 28

SUBSTA 2527 6. 62 6 .7 1 41 .60 0. 0693 3. 48 0. 36 4. 62 027

OPINIO 4764 7. 02 6 .98 58 .85 0. 1218 2. 05 0. 71 4. 63 0 . 37

STATE 9231 6 . 85 6 .80 62 .06 0. 2417 3. 06 0. 39 4. 64 0. 25

CONST R 3805 6. 58 6 .55 40 . 50 0. 1054 3. 38 0. 30 4. 65 0. 21

ADDIT

1708 6. 39 6 .49 32 .12 0. 0453 5. 06 0. 25 4. 68 0. 22

PAID 2316 6. 25 6 .25 28 .16 0 . 0616 3. 2 1 0. 23 4. 69 016

I NS I ST 368 5. 36 5 . 5 1 10 .41 0. 0096 3. 68 0. 10 4. 72 0. 09

EXCEPT 3589 6o 58 6 .82 49 . 79 0. 1046 5. 95 0. 26 4. 72 0. 30

HER 7548 6o 30 6 .20 31 . 89 0. 2095 4. 05 0. 20 4. 75 0. 14

RIGHTS 2 108 6. 30 6 .33 30 . 38 0. 0581 5. 59 0. 20 4. 76 0 . 17

SUPRA 2573 6o 29 6 .25 29 .21 0. 0636 3. 34 0. 23 4. 77 0. 15

VALID 768 5o 83 5 .92 17 .06 0. 0207 3. 58 0. 16 4. 77 0. 12

ACT I ON 8248 6. 94 6 .92 64 .55 0

2329 3. 64 0. 39 4. 77 0. 31

APPLY 806 6. 00 6 .08 19 . 6 3 0. 0212 3. 14 0. 19 4. 78 015

FAR 923 6. 16 .24 22 .61 0. 0247 4. 89 0. 20 4. 79 0. 18

OCCURR 1248 6. 05 6 . 12 1 .78 0. 0347 3. 73 0. 18 4. 81 0. 15

SOME 3394 6. 97 6 .93 50 .88 0. 0897 1. 97 0. 67 4. 84 0 . 39

OUR 3179 6. 80 6 .83 47 .98 0. 0833 2. 1 5 0. 55 4. 84 0 . 31

DATE 1983 6. 31 6 .41 31 . 37 0. 0555 3. 97 0. 23 4. 85 0. 19

COMPLA 3971 6. 40 6 .45 37 .44 0. 1 136 4. 27 0. 22 4. 90 0 . 19

NECESS 3477 6. 93 6 .93 52 .20 0. 0937 3. 31 0. 52 4. 91 0 . 35

CHARGE 4622 6. 48 6 .47 40 . 69 0. 1234 3. 96 0. 24 4. 95 0 . 18

RECORD 6093 6. 91 6 .98 60 .51 0. 1675 5. 25 0. 41 4. 95 0

35

ISSUE 3113 6. 61 6 .66 42 .88 0. 0831 3. 76 0. 32 4. 98 0 . 23

CONTRO 2941 6. 48 6 .55 39 .93 0. 0849 5. 05 0. 23 5. 00 0 . 20

GENERA 5262 6« 87 6 .82 52 .92 0. 1338 3. 11 0. 47 5. 01 0

28

DISMIS 2755 5. 96 6 .48 35 .90 0. 0790 5. 16 0. 16 5. 01 0. 20

OCCAS I 742 5. 95 6 .03 1 8 .38 0. 0206 3. 38 0. 18 5. 02 0 . 14

SPECIF 2900 6. 65 6 .68 42 .28 0. 0790 3. 75 0. 34 5. 03 0

25

HEARD 90 3 5. 97 6 .0 7 19 . 9 3 0. 0241 3. 35 0. 1 8 5. 06 0 . 14

PUBLIC 4658 633 6 . 30 35 .78 0. 1226 4. 86 0. 20 5 . 07 0

15

PERSON 6980 7. 01 6 .94 60 .81 0. 1897 2. 61 0. 57 5. 09 0. 33

DUTY 1873 6. 25 6 .30 28 . 35 0. 0506 3. 82 0. 2 1 5. 09 0 . 17

EACH 3332 6. 68 6 .69 43 .90 0. 0859 4. 53 0. 36 5. 12 0. 25

Table XI. Sorted by GL

136
VOTES

2

4

3

3

9

19

4

14

2

3

2

5

5

5

8

1

11

2

5

1 13

5

1

1 13

5

WORD

LANGUA

empha:

PLACE

APPELL

COUNSE

STATEM

APPEAL

CLEAR

CONTAI

COMPLE

OHIO

REFERR

PAGE

DECIS I ADM ITT

BASIS

OBJECT

OBTAIN

SECOND

ORIGIN

PUT

GRANT E PROPER

INTERE

GROUND

WHOLE

DOING

RECEIV

RELATI

COURTS

PETITI

CITY

HEREIN

PARTY

CODE

REQUES

MOST

HERETO

OUT

ORDERE

HEARIN

PROCEE

ACT

STAT

MANNER

PURPOS

PERMIT

RENDER

JURISD

DIRECT

HIM

SUPREM

ENTIRE

NOTICE

JUDGE

SAID

END

VIEW

COMMON

REMAIN

NOCC

1492

1012

1881

14543

30 30

2732

9096

1537

2096

1709

8519

1309

3218

3988

1667

1500

270 3 1498

2415

2053

719

1574

5913

3637

2629

651

625

2801

2530

2033

7623

5969

2599

2643

4152

1941

1051

498

4389

1180

2525

5021

5147

1245

1259

4138

2869

1657

3056

5706

5613

1904

1350

2855

4000

10747

6422

1406

4042

1592

22

96

36

53

22

32

6.80

6.52

6.55

6. 30

6.49

6.24

6.47

6.52

6. 32

6.41

6.27

6. 18

6.53

6.23

5.88

6.25

6.40

6. 36

6.68

5.74

5.71

6. 52

6. 54

6.28

6.19

6.24

6.23

6.26

6.21

6. 11

6.25

5.41

7.00

6.14

6.28

6.79

6.65

5.90

6.30

6.76

6.35

6.30

6.00

6.95

6.91

6. 16

6.30

6.04

6. 52

7.07

6.81

6.35

6. 46

6. 35

EL PZD

6.23 23.78

6.0 0 19.39

6.45 32.27

6.44 50.16

6.27 32.54

6.36 34.16

7.06 77.61

6.57 33.48

6.6-i 38.12

6.45 31.40

6.3 5 34.39

6.43 28.65

6.45 33.71

6.69 46.58

6.32 28.87

6.47 30.76

6.31 32.50

6.30 2 7.40

6.61 38.50

6.39 32.01

5.96 17.40

6.34 28.35

34 36.91

32 35.33

77 44.16

78 14.87

89 16.04

6.57 39.10

6.53 37. 10

6.36 31.21

6.44 40.39

6.23 38.05

6. 70 41.75

6.33 31.93

6.18 29.55

6.29 29.44

6.31 24.95

5.64 12.60

6.99 57.04

6.33 26.23

6.31 31. 59

6.84 55.19

6.59 45.56

5.93 19.10

6.37 27.29

6.76 49.30

6.49 39.63

6.45 31.74

6.10 29.67

6.92 58.62

6.85 54.24

6.24 27.44

6.41 28.53

6.18 30.76

6.64 46.84

6.93 69. 15

6.71 51.86

6.48 30.95

6.48 42.58

6.38 30.46

AVG

C.0411

0.0246

0.0528

0.3877

0.0B68

0.0720

0.2637

0.0425

0.0578

0.0455

0.2212

0.0341

0.0815

0. 1070

0.0436

0.0412

0.0742

0.0397

0.0656

0.0558

0.0197

0.0425

0.1591

0.0944

0.0728

0.0169

0.0167

0.0 764

0.0662

0.0553

0.2198

0. 1706

0.0670

0.0726

0. 1146

0.0545

0.0273

0.0121

0. 1 164

0.0324

0.0716

0. 1373

0. 1370

0.0383

0.0329

0. 1096

0.0820

0.0464

0.0812

0.1575

0. 1531

0.0474

0.0369

0.0853

0.1181

0.2803

0. 1570

0.0375

0.1171

0.0428

G

3.66

3.16

6.46

3.05

6.0 5 4.77

4.94

3.35

3.35

4.76

2.35

8.37

2.83

4.00

3.82

5.82

8.66

3.28

3.97

4.38

3.40

4.97

3.62

5.26

3.25

3.54

3.56

6.76

3.61

9.19

3.73

3.90

3.17

4.28

4.17

7.47

2.65

3.70

3.00

3.50

4.03

3.56

3.30

3.51

3.46

3.99

6.17

3.94

4.48

5.12

2.49

3.73

5.20

5.70

10.30

4.45

3.07

4.33

5.85

4.99

EK

0.21

0. 19

0.21

0.23

0. 15

0.20

0.30

0.33

0.35

0.24

0.28

C.24

0. 31

0. 30

0.23

0.26

0. 15

0.23

0.31

0.21

0.17

0.20

0.23

0.20

0.38

0.14

0.15

0.27

0.30

0. 16

0.19

0.18

0. 36

0.20

0.17

0.15

0.28

0. 12

0.65

0.23

0.21

0.40

0.32

0.15

0.27

0.41

0.17

0.23

0. 14

0.44

0.52

0.21

0.25

0. 14

0.19

0.50

0.44

0.29

0.19

0.23

GL

5.17

5.19

5.21

5.26

5.28

5.32

5.35

5.39

5.43

5.48

5.51

5. 55

57

57

59

60

5.60

5.62

5.63

5.63

5. 70

5.70

5.71

5.71

5.73

5.73

74

74

77

77

5.82

5. 82

5. 86

5.91

5.98

5. 99

6.00

6.07

6. 13

6.13

6.14

6.15

6.21

6.23

6.32

6.33

6. 36

6. 39

6. 50

6.63

6.64

6.65

6. 76

6.77

6. 80

6. 83

6. 84

7.01

7.01

7. 12

EKL

0.16

0.13

0.19

0.16

0.14

0.16

0.33

0.24

0.25

0.20

0.17

0.21

0.19

0.23

0.17

0.21

0.15

0.17

0.23

0.18

0.13

0.17

0.15

0.15

0.29

0.10

0.12

0.21

0.20

0.17

0.18

0.13

0.25

0.16

0.13

0.15

0.18

0.09

0.37

0.18

0.15

0.26

0.20

0.11

0.19

0.25

0.17

0.19

0.11

0.29

0.29

0. 14

0.20

0.12

0.20

0.27

0.22

0.20

0.16

0. 16

Table XI. Sorted by GL

137

772-957 0-66— 10
UAon WUrtJJ NUGC EL

8 ASSIGN 2654 6. 00 6.12

5 EFFECT 3 759 6

9 1 6.92

7 CONTRA 80 3 3 6. 56 6.49

8 SERV I38 5 5 6. 04 6.0 5

I T AL 11 360 6 . 67 6.57

F OL 5682 6 . 67 6.57

5 EMPLOY 6062 5. 98 5.89

5 S EVER

1243 6. 32 6.36

6 CONST I 4132 6. 41 6.49

3 USE 3852 6 . 29 6.27

9 CLAIM 2565 6 . 24 6.24

7 REVIEW 2347 6» 02 6.30

1

PR I NC

2158 6. 46 6.4 3 9 ATTEMP 1404 6 . 05 6.42

3 APPL I

4168 6* 58 6.60

10 COUNT Y 6245 6. 62 6.52

STATES 2343 6. 38 6.33

5 EXAM I3117 6 . 19 6.23

2 RETURN 2074 6 . 24 6.32

1 REV 1484 6. 07 6.08

5 APPEAR 3855 6. 95 7.00

5 ANSWE R 3398 6. 42 6.41

1 L EGAL 1650 6. 2 5 6.30

3 SUPPOR 3151 6 65 6.67

5 DAY 2189 6. 41 6 .46

5 COMPAN 4677 6. 19 6.05

WAY 1771 6. 26.45

4 CONT I

2382 6o 37 6.40

8

r* r* r> & RES PON 2872 5 o 94 6.00

3 ORDER 6773 6. 78 6.77

AAAAA

2649 7. 07 7.87

1 ESTA8L 2947 6. 74 6.72

7 OFFICE 4060 6. 26 6.12

PZD

29.82

52.39

52.96

29.63

4 5.18

45.18

32.50

27.25

42.99

36.12

32.27

32.72

34.61

29. 18

47.37

52.43

33. 37

35.56

31.48

22.72

57.68

39

28

46,

33

5 7 35

34.16

32.65

32.91

34.35

29.21

58.32

99.99

44.46

33.93

AVG

0.0715

0. 1018

0.2158

0.1114

0.2755

0. 1378

0. 1653

0.0331

0. 1058

0.1059

0.0735

0.0676

0.0564

0.0376

0.1134

0. 1787

0.0582

0.0831

0.0589

0.0446

0.1045

0.0913

0.0423

0.0855

0.0607

0. 1180

0.0472

0.0634

0.0772

0. 1918

0.0783

0.0788

0.1032

G EK GL EKL

6.48 0.12 7.19 0.11

2.86 0. 56 7.29 0.34

3.98 0.23 7.29 0.15

5.82 0.13 7.29 0.10

3.12 0.37 7.32 0.19

3. 12 0.37 7.39 0.19

5.38 0.11 7.48 0.08

3.47 0.26 7.53 0. 18

9.48 0.28 7. 53 0.15

4.86 0.18 7.72 0.12

5.91 0.15 7.77 0.12

5.34 0.15 7.80 0.13

6.01 0.24 7.85 0.16

4.42 0.25 7.93 0.19

4.97 0.25 8.13 0.16

5.00 0.23 8.51 0.14

6.26 0.22 8.54 0.13

7.01 0.15 8.63 0.11

8.81 0.15 9.23 0.14

3.55 0.18 9.27 0.12

3.97 0.56 9.43 0.32

5.64 0.22 9.44 0.13

7.41 0.19 9.77 0.14

7.06 0.24 9.79 0.18

3.92 0.26 9. 83 0.17

4.27 0.17 10.01 0.09

6.65 0.22 10.08 0.16

5.85 0.21 10.10 0.14

6.24 0.12 11.25 0.08

3.68 0.31 11.48 0.19

0.42 4.32 2.55 31.32

3.00 0.45 17.95 0.18

4.82 0.17 18.75 0.07

Table XI. Sorted by GL

138
TABLE XII. PROGRAMMING STEPS TO

ACCOMPLISH PHASE II

Step Input Nature of Processor Output

1) concordance FORTRAN: deletes words

2) list of having EK >. 30 and/or GL

16, 000 word ^ 4. 0 (about 350 word types)

types with

statistics

1) purged alpha concord'

ance (1, 225, 000 words)

2) thesaurus word list

(15, 780 types)

purged

concordance

SORT: orders by document

number, paragraph number,

and alpha word

concordance by

doc -par- word- - 3 reels

concordance by

doc-par- word

FORTRAN: generates word word pair list — 18 reels

pairs within paragraphs,

sampling via random number

generator for words appearing in more than 253 paragraphs

word-pair list SORT: orders alphabetically alpha word-pair list

by word-pair

alpha word-pair

list

FORTRAN: counts cooccurrences, writes out

insignificant cooccurrences

with applicable statistics

on second file

1) summary of insignificant

cooccurrences - -6 reels

2) summary of potentially

significant cooccurrences

- - 3 reels

potentially

significant

cooccurrences

FORTRAN: edits and

eliminates to produce

readable report

"significantly"

cooccurring words

139
TABLE XIII. THESAURUS SETS. The pages following are a

sample extracted from the computer printout of the thesaurus

sets. The full printout contains about 7000 sets.

The word at the far left is the head-word; it is followed

immediately by the number of paragraphs in which the head-word

appears. The words grouped to the right are the associated

words, arranged in descending order of standard deviation

units

For example, the word abutti appears in 94 paragraphs

and is associated with 23 other words, the first of which is

egress. The number of standard deviation units measured

for the abutti-egress association is 34 and egress appears in

82 paragraphs in the total file.

140
\ HIGHW l 

inA7 TR

A VER in 3 T AR

T . 94 HAP 23 6 IN&R ES 2 2 69 EGKE SS 20 82

IB AA7r 17 1470 15 771 PRQPER

J

<y 39 1 1 crqece i* fl? TNCHF 5 tn . _ 7ft 2 5 I j AREA 2 5 842 E A SEME 22 226

PL AT 'J OA PI ATTF 70 no i ur 19 245 CENTER 1 6 1 HPROV 1 8 566 MAINS 1 / 35 PROPER 391 1 SOUTH I 7 635 ALLEY 16 149 C OMP AC 16 2 2 L 1 GH T | I 6 62 TR AFF 

I 6 580 l 5 4 7 H I GHW 

15 106 7 LOTS I 5 299 PR 1 VAT 1 5 48 0 V I L

L

A r. 1 5 730

T AMTH 

i w

C UR

R I 

F AC 4 1 C OL

L EG 2 17 ARTS 22 26 SC IE NC 2 1 49

MnnT 70 70 51 ft! TF AC

H 1 70 5 4 S TUDEN I 9 10 1 T ENURE 1 8 68 DEATH j 7 166 7 LE ARN 

1 5 P R 0 R A T 15

*^ RE SFAR 15 - CAS * 79 54 L ONDON 79 38 ACT IDE 2 7 1415 MASS 26 1504 HUT 2 3 62 pni^r v _ DEC REF 1 9 i n!o 1 fl 1466 0 1 ' 34 1 INS \ 7 321 DOC TR 1 16 536

1ft CONPEN : RF AO 1 1 5 1 39

pi nw it lit P A

VH E N 11 1774 10 OFPRFC 79 122 OVER FL 25 22 AG&RA

V 2 3 8 1 777 77 Al MA TIIR 45 MILES 19 37 3 flCCURR 1 8 1 096 RATE 1 8 52 5 of pi f r 95 01 SEAS 17 145 TC Q 17 A | R P 0 R 1ft 7 6 BRAKES 1 6 121 HO UR 16 406 m 1 T 1 F T J ?R CORONA :_ 14 ACfFPT 1 no OFFER

Pi* All 51* 774 __ 1 6 1 I DAY 2 7 17 12 PURCHA 26 1487 VAL 1 DA 2 5 2 7 C I T V _ %5?9 24 . _ STA TF M 24 2014 FIXED 2 3 459 L ANGUA 2 3 1252 REFUSF 22 1134 2 2 APPEAL 2 i 6176 DRAWEE EXCUSE 2 1 170 PERIOD 2 i 1516 CASH 20 345 COMPL 1 20 6 51 0 I S HON 20 40 MASS 20 1 504 REJECT 20 346 AOV ANC 1 9 447 COMM I 

1 1317

08 S E R

1 

r nic i 

19 1 9 47flf 1 

2010 8 ID 1 8 79 CLIENT 1 

188

rn MMnr 1 

CON S I v> CONTRA 1 

rnuNSF 1 

2111 PL

F ADS 1 8 29 REV 1 

1216

Mi l|P ^ E _ . 1 7 1 flSI f ON TIN 1 

1917 0FP05 

I 7 69 7 HEREBY I 7 4 55 HUN I 7 1 5 SE ATS M

TO AOMI NI lft99 HRF AfH 1 

DEVOT 16 1 6 EL ECT I 6 63 3 LITTLE 404 P OR T 1 

i A 947 PR 1 NT 1 1 

175 1 _ 16 15 7 7 SETTLE 1 

90 3 C I R 1 5 3 7 4

cr\m ni i 

LU TMJUUL IS 1 40ft c n o n in 15 40 ni \r ha 77 1 FRANC 1 GIFT 1 5 241

RF MMT ^ 142 7 IS*

j: 147A

iz e 9 III5T

rF 15 RIO Idrouv nw CUR c no 15 814 REL IG I 1 

137 15 19 1 <J *_ 19 toi n 15 79ft WIT HDR 1 5 5 31 ......

HI

C HWA io KF YSTO 24 22 23 B2

L IM I T E 22 782 AP PUR2 j 48 INGRES _ . _ H F AN S 2 

845 IJSFO 2 

2130

F AS E HE 20 22 6 FREEW

19 26 RE BU I 1 35

ORA I N

A 1 R 1 09 FEET 9 3 3 7 ON I NC CONGES I 7 5 2 PR I VAT I 7 4 80 S TK 1 

17 1 34 1 7 7ft9ft C ON TRO ift 2

?Ift TRACT 16 295 URRAN 1 

2 5 ABUT S 1 5 1 7 ACRO S S I 5 325

jw ns , ,

fl . _ HA

Y 1 

15 30 ... ._ 1415 INJURY 51 ?9fl f (INTRA 47A1 COHPAN 0 1 S AB 

*. 5 24 7 INJUR t 45 1125 1 N SURE i, d, 868 HAPPEN 42 CAR zz 1222 OC C UR R 39 1096 3 8 25 6 38 1 3 36 3 7 1712 C AUS E 0 3 6 91ft OR 1 VEN 15 281 35 6 79 ROAD 3 2 72 2 ROURK E 30 23 S t CKNE 30 52

PL ACE 1 5R1 PR 0 VOC 79 RFC OVF 1577 ADD I T 28 OR I V ER 2 R 728 2 8 565 7 7 17 n A M A fl F 77 ?09fl IFF 77 2 7 l *86 2 7 546 PHVSlIf 2 7 702 RACING 27 ft 1 _ 1917 DAYLIG 76 I 8 0 1 S E AS 14 5 HANG I 

26 1 8 MI c HAP 26 I 7 throh 

26 17 ANGLE^ 25 50 E MPL 0 

75 3101 MEDF OR 2 5 1 9 2 5 1 9 THANKS 25 19

ART

F

R I 24 2 1 OR t V I 

24 526 74 20 24 10 6 INS 24 321 AUTOMO 2 3 1513

15 30 468 OF PUT 1* 2 3 LOS S 22 2 3 STRAIN 22 6 2 STREET 22 1420 HHE EL AUT 0 2 I 12 8 POL ICY 2 

128 8 BOD I LY 20 109

20 1188

R E AC HE 20 S TR UCK 70 372 TRAUMA 20 2 9 T RUCK S 20 109 UNA VO 

20 50 VEHKL 5 52 19 COMP EN 19 H t GH WA LEFT 19 108 3 p ermI 

19 472 ROUTES 1 9 32 SEC 19 406 5 S E V E R

A 19 1113 I 9

101*7

rnr a^

L *tft TR IP 19 1 14

A UC

B Jf n*sr ar 1 8 1 5 FHRrp 1* 1 

CH1HNF

J jj 15 1 fl C ORON 

n f r f n f 1 fl Qftft 1ft 599 FRONT 447 HAYnFN 18 16 18 406 I MPIII 5 1 fl 15 IIIQV in Ifll 0 MATHFW ift PP. FT j 1 fl }

ft 1 PR INC I I 6 17 53 REMOVE 1 fl 6 31 R I GGS 1 fl . , 1 R - 5 F 0 AN 18 _ . 174 SUFFER 599 U 1 1 H IN I 8 hi nnn j _

fl? BOTTL 

} _ 1 H RRFATH j 7 18

r FRFflR j. Ifl DECORA 17 EL EVA T | 7 142 EVE NTU j ? 63 H AZ ARO 23 3 I MM F n 

050 7. } 7 I 504

M 1 DW E S l j 18M I LE S I 7 37 3 OCC UR S 1 7 101 1 7 14 1 on i r

k i ]_ 7 in SC

F NF 14 3 j 7 j 7 5 YMP TO 8R I DGE EXCEPT 2937 TAtfF ift 7 HYpFRT L 6 20 ILL I NO 16 136 3 I NC A P 7ft 

1 1 54 OPE R AT 1 6 295 6 PERFOR 1 6 1 392 P IFCF id PORT ER I g 72 R E S T I 

16 SIXTY 1 

159 SL I GHT 1 6 212T

R AC TO 16 105 TRA 1 1 F ift 1 54 TRIVIA 1 6 1285 2 1 AGG R AV BOS TON BR A K t S 15 1 7 1 COLA 15 2 1 CON SE 0 1 

704 0 I N NfR 1 

2 1 1 5 2 1 ENGAGE 15 705 15 1 70 FIVE 1 

1 195 F OH TB 

15 2 1 H AMPSH HEHORR 15 4ft 1 

R5 INC HE 

1 5 1 TO LOOSE 1 5 48 HON THS I 5 709 MOR N I 

1 5 2flfl NATUR 1 5 44ft NFARFS 15 45 DM N 1 R 1 

15 7 1 REACH! 1 

RIDE 1 SKID 48 S PONTA 2 1 C

T AD TC 1 

791 TOTALL 15 77 TRACKS 1 

1 70 TYPE 1 

4 39 UP SE T 15 48 WOODW 

1 

2 j ACCOMM 58 EV 1 C T I 28 35 R EN T I 

25 1 5 BUSINE 2 I 1 909 OPE RAT 19 2956 HOUS IN 1 

1 7ft

_ . . 1301 10 EXCUS E 1 5 1 70 ACCOMP 615 SHE 32 22 50 COUN SE 31 2111 DESIGN 29 107 5 INTENT 1ft I 4 UNCORR 79 . .... A SHL A N 75 1 5 OF F I C E 2 5 2598 24 APPFAI ... 6

fl51

C OHMON 795ft 7 1 1917

nfi^H 3 974

^984 UHRK 1 119 1 OF

NT 

20 19 110 7 rnonnH 19 74 1ft 79

HE AN ( 

1 8 AT TORN 1 

1 893 BOARD j 7 3543 NOTICE I 7 1938 srunnf

.pen SUSP 17 ftfl TR AN S P I 7 4 81 CR ED I B 1 6 269 PASSBO 1 

70 PH INC I 16 175 3 i ft ... 1161 . I 15 ft5 I OAT E 1 5 1611 OU 1 CKL 1 

ACCOUN 1267 BANK 7 5 1 129 SUR

V I V 52 4ft6 S A

V I NG 50 392 JUNE 46 1080 PART I 44 7741 in int 41 ASSETS 40 4 26 CHECKS 38 22 6 DEC E MR 36 104 2 C AROS 34 1ft S I GNED 34 757 r RFnr t A71

DE POS I 33 697W I THOR 32 531 PA S SB 0 3 j 30 48 NET ?1fl pa in 79 I 747

OEL I VE 28 854 CONVER 2 7 592

F UNDS 2 7 595 OPENED 2 7 17 7 2 7 14R5 RFCF 1 V 7770 TELLER 27 26 VERAC I 27 26 E M I L 26 1 

2 5 CITY 1579 STOCK 75 51 n HER 24 3428 LEDGER 24 1 9 PARTY 24 1838 VOE GE L 2 4 1 9 APP ELL 2 3 7099 BORROW 7 1 1 io OE AT H 23 166 7 EMPLOY 2 3 3101 TOTAL 23 54 2 ANNE X E 2 2 17 7 671 BA L A NC 77 50fl OONAT 

22 2 1 22 172 PROF I T 22 311 2 1 SET TL I 7 1 IRAN SA 31 A40 VAULT 2 1 24 LAST 20 819 HONE Y 20 1274 PERC EN 20 816 RENTS 20 9ft 70 7750

SL IPS 20 25 WR I T IN 20 432 APPROV 19 1201 CARRIE 1 9 740 CLOS I 

19 1 35

C ON SIC 19 79 LOAN 1 9 389 MAKER 1 

4 9 RE F 6 RR 1 9 1197 SHOW E 0 1 

379 79A 5POKF 19 77 SUMS 19 146 AUD I TO 1 

359 1 

6 73 l<"in cash*

T 18 145

r (INT IN 1 fl 917 COST 1 8 4 86 IN VO IC 1 B 52 PAROL 1 

56

R E I NSU 1 RFTA IN 1 491 1 fl AO 5 SHARE IB 355 SUPPLE 18 365 TUR NED 1 8 2 79 UNI M PO 1 8 30 ABSENC 91 1 RClNnS 17 Si a BOOK 1 7 168 BOOK S X 7 213 BOXES j 7 36 C OUR T S 1 

16 5 5 DAT EO 1 

445 nAv 1717 DECREE 1 7 1 700 OE PR EC 1 7 122 DONE 1 7 1 

117 FORM 17 92 1 I NUR E 17 15 ITALIA I 7 j 5 1 7 955 KNOW L E 96 9 LEASES J 7 no MARCH 1 7 9B 

NAME 1 7 988 OR

V I LL i 7 1 5 PR I NC 

1 7 1753 RE PR I N 1 

1 5 SHARE S j 7 WORTH 1 AL LOC A I I 1 

65 CARD 16 6 8 COMPLE 1 

145 3 OAUGHT DR

A

W E R fl INC OME 525 INOEBT 16 1 90 KE EP IN 1 6 135 L OA NS 16 96 ME RC H 

1 260

rpa/

ES 1 6 ftl rtlj itHD Ift 8 OCTOBE 16 808 OFF SE T 1 f J 

1 8 OVE RDR 16 ^ 7 PE RH A P 1 6 19 3 57fl 1ft 1 7 TOLD 1 

796 TROOP 16 1 8 TRUST 1 6 1546 BASIS 1 

RATH 15 70

C 0 MM I 

15 1317

COMPLA 15 2823 OEOUCE 1 5 1 9 E XPEN 

I 5 784

rf ruf

I 5 l 49 1 15 TO HU

TCH 

15 19 l 5 3 36 1 iii SHEET 

1 c WAGE 15 45 ACCRED 22 COLL EG 36 217 CURR IC 2 9 31 TEACH I 29 54 YF AO ?R J GR AOUA 7ft ft5 f « »^ J ' 2 

6 33 2 

1161 STUOEN 2 1 101 APPOI

20 1097 in 47 1 P R OCRA 18 131 TR

A I N I 17 83 PERFOR 1 

I 392 ACCRUA 29 AC CRUE 39 205 DE A TH 2 2 I 667 OA T£ 2 j 16 1 

REPORT 20

C OUNT

Y 19 4 129 E XC EP

T 19 2937

OE CR E A 1 7 64 MAP I 7 61 PROOUC 1 7 1040 DEEMED ^ 5 4 1A ACCRUE 205 AC CRU A 39 29 ACCRU 

32 5 8 P A YHE

N 20 1 724 EXECUT 19 ...

1611 COHPE

N 16 1069 ACCUMU 1 5 125 EXCEPT 1 

2 9 3 7 UNP A I 

I 5 140 ACCAUt 58 ACCRUE 32 205 VOUC HE 3 

1 

KELLY 2 i 8 5 COST 19 EXCEPT 1 9 CERT IF 1 

1358

FARE 1 7 34

R E SPON 16 1 964 BE NEF I 1 

1816 I HPR OV SftA PAYMEN 15'

jj 1734

AC CU MU 125 CAPITA ilni* 2 31 20 471 0 1 V IOE 19 347 SURPLU I 8 115 17 221 E MP LO Y 17 INT ATT 17 4 3 SL I PPE 1 

103 ART I F 

1 6

FRO/. EN 1 6 16 MEN TOR 16 I 7 S T

A I R

W 1 64 ACCRUE 15 205 CESSAT 15 19 I N TERE 1 5 2557

LIFE 15 9 30 ACCURA 198 STATEM 19 2014 DE SCR I 16 1067 CORREC 15 1221 ACCUSA 62 HAL FE 

33 15 SAZAMA 33 15 E QUI VO 32 25 ACCUSE 24 382 INFORM 20 1107 DURHAM 19 44

INCR IM 18 50 ME TC 18 26 EVASI

17 30 INDICT 17 848 POL ICE 16 1241 JEOPAR 15 105 141
accuse 382 PEOPLE A] 1850 CRIME 40 850 GUILT 38 345 Oi-l-tNS 38 69 8 CUNV Id 3 , « IV ( INO IC 

2 f 148 ACCUSA 2* 62 FUGITI 24 31 JURY 24 3610 OUASHE 21 34 AOMISS 22 812 PROOF 2| 1220

CR IN IN 20 1063 GUILTY 20 1427 COMMI

19 1317 CONFES 19 267 REPRES 19 1337 OE t ENS 18 1159

INTENT IS 161ft EXCUSE 17 170 EXTRAD IT 60 IMPART 16 130 SUSPIC 16 68 BRAND 15 24 CREDIB 15 269 OUTY 15 1485 SPEEDY 15 69 ACCUST 15 ITENI2 36 29 WASN 24 64 BUI LOI 20 1593 WAY 19 1530 PREMIS 18 1327 REMAIN 10 1370 LIVING IT 295 CARE 16 1060 SUPPOS 15 163 ACHIEV 66 ORAST 

17 29 BETTER 16 225 PROSSE 16 57 GOVERN 15 1154 ACHOR UO LANOIS 103 12ft B0BB1T 93 137 ARTERB 90 122 1LLNES 85 126 JACKSO 55 279 ACKNOM 239 NOTARY 65 59 BURTON 37 74 DEAN 29 97 NAME 28 988 RECEIP 25 394 AFF1XE 24 36 SICNAT 23 298 SIGNEO 23 752 OEEO 22 705 VIRGIL 22 32 PEAK 21 37 SEAL 21 65 GRANTO 20 133 DORR 19 24 ATTEST IS 73 FICTIT 18 39 DEEDS 17 238 BEATTI 16 16

PA I 0 16 17ftT SMITH 16 545 UNTO 16 33 ACME 10 GOOORI 84 36 RICE 59 39 BALDWI 54 46 HEATER 52 49 LINCOL 34 90 POULTR 14 16 GRAIN 51 15 BEAM 30 21 PRICE 26 586 CHICKE 24 13 MERGER 24 1) CORONA 21 14 BRAKE J 1 41 BUFFAL 21 42 INC 20 1466 SEC 20 4065 CORPOR 1} 1916 OISTRI is 1949

ST AT EH 15 2014

AC9UAI 69 SOUNON 19 21 OAV 18 1712 WATCH 17 4* ASKEO 16 • 78 OISCNE 19 1190 POLICE 11 1241 OUASHE * 9 34 WITNES 15 2175

ACSUIE 101 CONTRA 24 4781 KNOWLE 23 969 EMPLOY 19 3101 PARI IE 18 2743 LACHES If 70 CON SEN IS 111 ARTIFI 6 7 WATFR 10 6 36 CHANNF 28 77 NA T UR A 27 446 WAT T RC 26 6 It SHFPAfc 22 1 7 LAGOON 21 19

N€ CL I 2 1 2129 THROW 18 46 MASS 1 7 1 504 ACCUMU 16 125 GUARD 15 9 9 LOANS 1 5 96 WATERS 15 103 Mrs 26 S£ fPNC »2 49 CR AF TS 51 15 SABBAT »1 21 REVERE 30 4 1 FTHICS 28 27 IEACHI 27 54 tEARN 

26 56 TAUGHT 26 11 PRO&RA 25 131 RtL 1 GI 25 137

A ILMFN 24 JR GRAOUA 24 65 GOO 21 40 AC AOEM 22 4 1 BRANCH 21 181 PUBL IC 20 3129 SUR&ER 20 55 STUDEN 19 101 TOUNG IB 1 7ft MA IN TA 1 J 979 AONI SS 16 832 COLLEG 16 21 t LEGISl 16 1845 MED IC 1 15 H9 ascfrt 427 CONTRA 22 47RI CON T IN 21 1912 LFG1 SL 21 1B45 TEST AT 21 54 7 CONSTR 1 t 2758 INuUlR 17 4 74 PART IE 1 7 2743 STERN 17 21 I NTf RE 16 255 7 NAPFRV 16 2 3 USED 15 21 30 Wl TNES 15 21 75 ASCRfB 5' 

LI BELO 29 36 ME AN 1 

26 9S4 CONSTR I 7 275R OPERA! 17 2956 USEO 15 2130

ASHL AN I 5 KENTIJC 26 5 5 ACCOMP 25 615 HIGHWA 16 106 7 HALL 15 154 ASHLEY 1"> CORR IC 7") 29 WESLEY 53 19 TABLF 31 75 DOUG LA 32 82 WASHIN 30 179 KN IFE 2ft 50 SAW 22 628 EXAMIN 20 2219 REMARK 1 7 173 1 MPART 15 130 SOMEON 15 126

ft SHIP 6)0 SE

TT I N 53 346 VACATE 3H 611 RENOER 15 1456

F RAUD 29 57 1 EXCUSA 27 21 NAUGHT 2 7 22 PET tr I 27 4506 VOIR 2 7 58T A T TORN 26 1891 CIRCUI 26 1 189 WE l&HT 23 7B9 COMMI 

22 3056 EXCUSE 22 1 70 OEFAUL 21 309 BRUSHE 19 15 CONF IR 18 279 L INORO 18 IT NOT ICF 18 1918 ACTUAL I 7 1 165 ALLEGA 1 7 1171 DECREE 1 7 1700 F 11 ING 16 100 3 INO 16 1615 MAN 1 F E 16 54 2 PR I DE 16 21 RETURN 16 1641 STA T 16 1042 WALGRE 16 21 AOM 1 T T 15 1419 INTERE 15 2557 MOVED 15 451 PAR T 1 T 15 125 PREMI S 15 1327 REMAND 15 915 SETTLE 15 903

ASK I B<> AT TORN t 7 1R93 YOUR 1 7 551 ROOM 16 457 ANSWER 15 2411 YOU 15 1471 ASKEO Ill RE PL I E 56 141 SHE 50 2250 F XAM1 

49 2219 WANTED 46 236 KNOW 33 649 TOLD 33 796

Wl TNFS 32 2175 COHPAN 31 2887 ANYTH! 10 466 DON 29 175 POL ICE 29 1241 WHY 28 408 INFORM 2f 1107 CALL 26 409 JUDGE 25 2919 PURV 1 

25 8 1 CAME 24 700 DAY 24 1712 MEMORY 2ft 48 OFF ICF 24 2598 RECALL 24 156 TALKEO 24 132 ANSWER 23 2431 GET 23 485

GOT 2 J 2 79 HOME 23 1091 SERVIC 23 2498 WON 23 37 BRY SON 22 27 HANO 22 715 LEAVE 22 7 I 7 TALK 22 76 WENT 22 733 WASN 2 1 64 DION 20 222 HANDED 20 92 JUST 20 842 AUNT 19 20 BEER 19 130 GAVE 19 768 GOING 19 545 TELL 19 205

APPEAL 18 6176 RUILDI 1 n 1593 KNOWLE 1 8 969 PRETTY 18 22 REMARK 18 173 SHOES 18 39 TELEPH 18 491 WATCHI 1 8 21 YES IB 365 8EDR00 I 7 43 CONVER 17 592 IN0U1R 17 4 74 PUT 1 7 641 REPRES 1 7 1337 RUL I NG 17 881 SAW 17 628 WANT 17 439 ACUUA 

1 6 69

BACK I 6 793 COUNSE 16 2111FR I F NO 1 6 320 JUROR 16 104 JURORS 16 228 KNIFE 16 50 MRS 16 739 PARKED 16 181 PRE JUD 16 II 74 STAND 16 392 SURPR 1 16 45 WALKED 16 105 CORR 10 15 29 COUK TR 15 53 CROSS 1 5 1074 DOUGL A 15 82 OR INK 15 51 FOREHA 15 81 MAROLO I 5 55 HEAR IN 15 1935 HELPED 15 53 LAWYER 15 245 MAIN 15 330 NEXT 15 674 OBJECT I 5 1963 PARTY 1 5 1888 PLACED 15 713 PROBAB 15 470 RFCONV 15 52 WHERE U 15 117 ASKING i n PET IT 1 1 7 4506 ANSWER 16 2411 DECLAR 15 1 381 ASKS 92 APPELL 23 7099 ASLEEP 2} SLEEP 52 25 EXCUSA 34 21 FALLI 

32 94 BU1CK 29 52 OR INK I 27 93 FELL 26 244 WHEEL 26 63 REOUES 21 1534 BLACK 22 141 DROVE 20 238 PARK EO 19 181 VERDIC 18 2067

MEN 16 423 AUTOMO 15 1513 COLLIS 1 5 681 KNEW 15 619 MINUTE 15 375 PLACE 15 1583 ASPECT 10 3 FAVORA 31 361 L INORO 25 1 7 MOST 24 974 SEC 23 406 5 WALGRE 23 21 MASS 19 1504 PROPER 18 391 

CR IMIN 16 1063 RELEVA 16 417 ASPHAL 23 PAVING 48 30 L IMEST 40 15 OUANTU 29 50 AREA 17 842 OR IVEW 17 141 PLANT 17 297 PLAY I 7 R6 CONCRE 16 158 SLIPPE 15 103

ASS 201 BLOG se 70 AMER IC 23 635 INC 22 1466 MEMOR 

22 59 COMMER 21 527 CEMETE 20 74 CIR 20 3 74 COM 19 162 CIVIC 1 8 22 POLISH 18 15 SAVING 18 392 BANKER 1 7 25 LOAN I 7 389 MASS 1 7 1504 APPLIC 16 3117 OIV 16 34 1 LOWELL 16 58 SUPRA 16 2012 ASSOC I 1 5 671 CH1CAG 1 5 1176 MAYWOO 15 20 ASSAIL 45 STRUGC 30 25 WORE 27 10 GRARBE 26 18 HAT 26 IB ROBBED 25 34 MEN 24 423

DARK 2 I 75 Wl INES 19 21 75 PEOPLE 18 1850 NECK 1 7 4 3 ROBBER 17 302 ASSAUL 16 191 ALLEY 1 5 149 DR INK 15 51 ASSAUl 191 BATTER no 69 PROVOC 40 25 MURDER 32 195 H0M1CI 29 62 I NFL IC 2 7 97 INTENT 27 1614

KEOZ I E 27 16 BODILY 25 109 PALMER 24 45 HAMMER 23 4 7 SODOMY 23 15 VIOLEN 23 1 32 WEAPON 23 72 CANDY 22 42 SEXUAL 21 B2 MALICI 20 14 7 NEGRO 20 19 AGGRES 19 30 BEAT 19 30 COMMIT XI 1317 STABBI 1 ') 21 BEATEN IB 16 K ILL IB 49 CHARGE 17 3341 FUROR 17 18 GUILTY 1 t 1427 OUICKl 17 38 ASSAIL 16 45 AC TUAT 15 23 CR IME 15 850 INDICT 15 848 HANSLA 15 89 ASSEHB 426 SENATE 33 64 ART1CL 30 973 APPOIN 27 1097 ENACT 27 78 SESS 10 24 170 ROADS 23 138 LAYING 22 64 L EG I SL 21 1845 POWER 21 1853 H0L01N 1 7 924 OHIO 17 5161 UNI TS 17 64 DELEGA 16 143 ENACTE 16 381 HOUSE 16 829 LAWS 16 1072 UN 1 FOR 16 264 AMENDE 15 1484 CORONE 15 28 MUNICI 15 1382 ASSENT 59 PARKS 21 36 PURCHA 1 9 1487 ADVANT 16 198 COMPET 16 67 1 WEBB 16 34 LI TEM 15 71 PR INC 1 1 5 1753 ASSERT 1112 APPEAL 52 6176 PERMIT 37 2261 PEOPLE 34 1850 SEC 33 4065 SUPPOR 33 2618 HER 31 3428 CONST I 30 3193 NOT ICE 29 1938 DAMAGE 24 2098 DECLAR 24 1 38 1 CLAIM 23 1938 POSITI 23 1276

REL IEF 23 902 RELATE 22 785 COURTS 21 1655 RECOVE 21 1577 SUE 21 133 ES IOPP 20 222 EXERCI 2 0 1541 PART IE 20 2743 OUl TTI 20 23 WAIVED 20' 472 INSTRU 19 2295 JURISD 18 21 38 VALUE i a 1229 APPOIN 17 1097 BAR 17 1061 HIDDIN 17 3 1 COUNT 1 7 495 ISSUEO 17 1086 SEVERA 1 7 1113 WRONG 17 275 ADVANC 16 447 ELECT 1 16 633 EX ISTE 16 919 PROCEO 16 993 PROPER 16 3911 OUASHE 16 34 ROOT 16 15 SLOCHO 16 1 6 THEORY 16 597 URGE 16 123 aooiti 1 5 1490 AGREEO 15 805 ARGUE 15 244 BOARD 15 354 3 CR 1 

15 36 GLENN 15 36

INC 15 1466 INOEEO 15 276 LEGAL 15 1431 PETROL 15 39 PRACT 15 1577 REAL 15 1528 SUBROG 15 103 TENDER 15 366 UNCERT 15 189 UNSUPP 15 6 3 ASSESS 823 TAKES 50 498 PROPER 4'1 391 1 VALUAT 48 173 TAX 45 1334 REASSE 35 31 EQUAL I 32 47 REAL 31 1528 ROLL 31 65 OANVI 

30 21 LEVIED 30 136 SEC 29 4065 EXCISE 27 65 MAPS 25 20 TRUSTE 25 1033 ABA TE M 24 126 TRAILW 24 22 VALUE 24 1229 TAXABL 23 136 TAXAT I 23 2 30 COST 22 486 LEVY 22 200 REVISE 22 2493 REMISS 21 15 COLLEC 20 670

INHAB 1 20 139 LEVYIN 20 31 TAXING 20 88 IHPROV 19 566 SALES 19 572 APPRA 

18 247 BOSTON IB 409 C0MM1S 1 8 3056 EXCEED 18 405 LIBRAR i e 102 APPEAL 17 6176 01 SPRO 17 41 ESTATE 1 7 2561 HENORI I 7 40 INC 1 7 1466 LAND 1 ! 1389 SENATE 17 64 VENDOR 17 148 ACREAG 16 26 PERCEN 16 816 TANG 1 

16 94 DAMAGE 15 2098 IMPOSE 15 777 PAVING 15 30 SKETCh 15 30 WILL 1 5 4823

ASSET 47 MCLEAN 29 38 ENOUIR 27 46 ASSETS 26 426 CONTRA 18 478 1 BONIS 17 39 OECEDE 17 966

HER 15 3428

L 1 ST IN 15 51 ASSETS 426 ESTATE 45 2561 ACCOUN 40 1267 COLBY 38 18 CORPOR 15 1916 INVENT 32 172 PAR TNE 30 259 EKECUT 29 1970 TRUST 29 1546 TRUSTE 29 1033 PAY 28 1485 SALE 27 1338 ASSET 26 47 BONIS 26 39 SHARES 26 298 LIFE 25 930 L IOUIO 25 10 3 STOCK 25 510 SMYTH 24 25 TRANS 

2 3 837 DECEDE 22 966 INCOHE 22 525 REAL 22 1528 APPOIN 21 1097 BENEF 

21 1816

BUS INE 21 1909 DEL I VE 21 854 SHARE 21 355 OINER 20 16 INTERE 20 2557 PAVMEN 20 1724

PR INC I 20 1753 SHAREH 20 115 VALUE 20 1229 DEBTS 19 147 APPL IC 18 3117 CENTRA 17 373

142
En ou I 

1 

4A?t E X I STE 1 

'*a ^A \ £ TnritM 1 

174 7 17

1*1*0 PROBA

T WILL ifct 1 

it PIVEtFC 15? it AHM IT COUNS DnrJ v it it e L 5 2111 D I SSOL zz 185 nf uici 1 5 PURCHA 14A 7 ioi 1 in 1 A WE B6 E R in74 -,?5 ASS 1 GN 20 1 

ERROR cd no c t H K UK > Vt\ APBFFM AS

X PROPER CHIC AG 7A

£?£,.,,, 5? c ^ 50 1 120 CL A IMS 2 8 1015 A MC ««r r. 2 8 ooo

7*7

77

\, AR GUM E 2 5 1 7QO

D 1 M 1 NU 7SW22 PERM J T 2 5 226 

OEN I AL 24 545 PRE JUD 24 1174 TITLE 24 118 3 HEAR IN 2 3 19 35 30 2 3 30 DAY 22 1712 22 1798 2 2 f f *f ?^

BR I EE 71 COURT S 2 jj 1 16 55 D I SPOS 21 881 DR I LLE 2 

16 I T Al" I A inn 2 1 I 5 R EF IL E 2 

1 7 R 1 GHT S 1 AAI FILE 20 792 MANI FE 20 542 OVE RR U 20 1460 I Al F 20 50 7 8AS I 

1 

1 334 Gl V ING 10 731 POSTPO 1 

P UR POR 7?* uPua 

a 1 

43 1330 coorn* , nTi 1 A 1 1 50 1 

1129 IN SUR

A 1 a 1 8 1188 NUMB E R 1 8 ' it? 1 e u 1 A J

j* 17 1 S SUE 

IT 1101

J i it

N

ARR

A 

it 7 A RAPE ifc OA

A P PROP 15 LEASE 1 it 

70 BAB 1 A07

zjl

r ratlie e ?t 7AS KUU 1 IN 1 5 ASSIST 5 20 ^1 J* ?572 iTIOBN

r?5?

a in

C DUNS c 7111 Uf TNf 

2 

tfFRn rr 20 ,551nftTA IN 10 f 1 

DAY

J* 171? 1 A 7S0H rntiNTY 17|Z fl

FC Ci

rnwMi \ 22 TO

ri aim

?2

IN CDCf 1 Ait RES PON 10A4 unoiiaiu UHUKf] 1 A 1 A \ , COMMI 

15 3054

r - " t FQ R T UN 1 5 20 pcno

C 1850 P UB LIC 1 

3129 REMOVE 1 5 631T OL 0 1 796

AS SOC 671 f

.*

n 1b li in* fl 389

r no ana L UK r UK 1916 '2 UN 1 NLU 3 3 30 GR I E VA 3T SUPER I 30 C EME TE 29 REC E 1 

2 9 AUD I TO 2 5 359 25 1 274 RE I NSU 77 30

AC

C OU

N 7AT

C ON P A

N 7AA7 MORT

G

A PAY ABL 339 77 1 3 3ft

F* Im"in 17 5 3 m i Van lit

BUI LD 

fa 1 593 PROV OC 7ft 2 5 LUNUUL 1 

555

EN JO 1 

i a 423

10 1 363 1 TM PROPER 1 o 391 1 J*p*J' 1 

2 30 VOL UNT i a 422

HI THDR 10 ATM FT 1 A _7? 1141 1 A 6 6 1 

5-2 SOC 1 E i 295

on o qnu IT| i in rwAP 17J* CHARGE 1 T ?5i

17 "It

CONNEC |' 1142 01 SPUT ITJ J 761 J*

7n IN 1 

1 i n

a if? jJ9I : _ **, 1 AUN una

u nnUU 441

U BL

J AR

T 1CL 16 1 NT FAN 1 

24 1 L AWS 1 A m77 1072 DAD PAR TMC TNE 1 A16 23*

US E 16 2696

V I OLA

T •1 

1561 ASS 15 201 AV EL LO IS 24 BASIS •15 1 334 CLARA

ft 23

o°or 1 5 2* 1 

1 54 LAND 1 1309 LEGAL 1 

1431 LUCY 15 24 HUN I C 

15 1 382 1 

1 A 8 7 R E GUL A 1 

1333 SALLE 1 42 SPELL 1 

24 Sup E R1 5 335 TR ANSA 15 640 A^MlMP *| * 7 17 r 

l tti B 1 1 

C 24 9 26 1327 r * r 293 7 U A N G C H 74 sn* f i ir a a 71 f .cor 

?, i in!

71fn ADO I T 

2 2 1 490 CA S T S 77 1 

7 1 114 Uu In IN 71 7QA DHI NT i !« f APF V 1 o 18 E xC USA i a 21

PA YH EN 10 1 774* 19 , DUTY 1 A PAAT 1 A 774 1 if

PER IOD 1 A 1516 1 A 158 3 1 fl S UP P OR 1 fl i in

17 nFF I AN IT

L I AB 1 

17J 7 1336

L IGHT 17 015 rnwra « 1 Ait A7BI ,Z5i 1 Ait if?. 01 SCRE 1 

1 1 90 I TFNn 16 117 5 UNDER T 16 287 A T TORN 1093

OESCR I 1067 1 n ikuu 601 l * 241 . 15 6 7 5 NULL 1 F 1 

S7 PMVSIC ** T02

PO

w E R15 ASSUMI 18% UN F I r N I 1 1 7 DEC IDI 15 1 70 ASSUME 212 RISK 36 2A9 ASSUME 35 094 PROPER If 391 

ASSURA S3 DONE 19 9AA SOLVE 1 7 22 6ARNET 16 26 ASSURE 211 DIST AN 45 365 Excuse 31 170 AHE AO 29 147 SMILEY |S D1SCER 22 FOG l« 20

SEWER 1 6 154 DRA IN

1 1 09 PLACE 1 5 150 3 REPORT 1 5 1 4 ?ft WAT FR 636

CAT EGO 1 A 3 FALL 20 3A0 FALL S ^« 1 70 1 5 2^3 7 CATHER 3 

AGNE S 3 1 I 9 L EO 29 22 ALICE 26 FOLF

V 2 MU R R A Y 1 9 50 1 7 1 09 3 61 76 BR? FN 15 NFVER 1^ 852 CATHOt 36 ROMAN 5A 15 Lf BL ON 42 16 MfRini 38 19 CHURCH 37 349 ARCHB 

33 40 ATHENA 26 40

PL AY GR 21 34 TE AC HE 21 192 OHIO 19 5161 ZONING 18 78 3 HILLS 1 7 53 INSTI

17 756

SC HOOL 1 6 1161 UN I VER 1 6 229C 1 NC I N 1 5 34 1 GR

A DUA 1 5 6 5 CAT TLE 3 3 L 1 V ES T 52 35 SANI TA 20 16 7 PROG R A 1 

1 31 PI SEAS 18 14 5 C EEK . AN 1 A AA66 C AUGHT 92 27 2 7 GEAR 25 I 7 GLOVE 2 5 1 7 Hl.lR N e n 2 2 6 3 C UR BIN HAND 2 1 715 thro wn 2 1 68 APPF L L I Q 7090 M I R R C A 1 

2 9 CMAPF 19J 79 HEARD 1 A ITS ROOMS 1 0 55 FOO T 1 7 257 HOLE 1 7 66 SIDE I 7 722 S IDEWA J I 18 5 1 7 64 INSIDE 1 6 106 WEAK I N 16 42 COAT I 5 4<. DEBRIS I 5 4 7 L OOS E 1 

48 NO I SE 1 5 4 3 PIECES 1 5 * 7 PLACE 1 5 150 3 CAUSA 1 

MOR

T IS 20 6 1 7 OONOR 5 I HO GIFT 5 0 24 1

r! ~ 46 5 0 GIF TS 29

VE ST I 29 A4 INTER 26 153 TITLE 26 118 3 [) AM

A

GF 20 209 8 *rr inF t A 14 15 5 1 CAUSAL 1 25 CONNEC * 7 1142 I N JUR Y 40 1 298 0 I S Aft I 36 24 7 H Y PO T H 3 ? 10 8 C OR ON 

30

W

PRO* IH 70 CAUSAT 2A 30 E XER T I 24 2 1 0 1 SE A S f o >o 6 PR OVOC 2 2 4 it

<,

y p\

p

T

Q IT SHOCK 20 A 2 T R A UM A 2 0 2 9 1 9 1 069 EXPERT 286 I ! nnc [no IIUL 1 UK ID 111 I 7 l ?n J J CTD A 1 M 1

il tn

TtionuQ 1 HK 

16 HVPFOT ?n HAN IF E S47

M c D

F OR 1 

1 

NA THAN «7 OCCLUS *' CAUSAT 30 CAUS E D 27 9 36 r* a in** 74 125 r nn ntt a 2 3 TI^A 1

MA

A GGR

A

A AC

C IDE 0 1 41 * INJURY 19 1298 CHA IN 1018 VIEW 1 fl 128 5 CLAI MA 116A >5 3 EXAM IN 116A 2219 7710 HEDICA 116A 546 ABSENC 1 

911 CAUSED "36 L I AB I 

A 0 13 36 PRO VOC 40

w

DAMAGE 38 1415 I NJ UR

Y ??2

RE COV E fa 33 1577 CAUSA T 2 7 30 EMPLOY 2 7 3101 ERROR 77 LOSS 2 7 1 1 ^ I R A 1 J M A VER D I 

2 7 206 7 WORK 26 1319 L I A8LE 25 66 I USE \ \ 269 6 PROX I 

24

iln 4 72 MA S S 23 1504 PER IOD 2 3 1516 SUF F E R ? 3 599 ROD I L 

2 2 109 FALL 22 -»55 1 N JUR E 5 5 \% 6 J' PROPER 2 2 3911 SC I N T I 22 2 9 22 1 

CAUS 1 N 2 1 269 CITY 2 

2 1 117 INJUR I 20 1125 QUA SHE 20

Vl

A

GG

R

A

V 1 9 5i CARE j*5? COMPAN 19 7BA7 rnHDi

a ???? COUR T S 19 1655 FALLEN 19 38 UNSK 1 

1 9 22 ID 148 5 SHOCK 1 

45 SUPPOR 1 A 5J2

W I ND 1 

23 BLA ST I 17 2 7 HE MOR

R 1 7 46 IMCTOI IT 229 5 MED I C

1 546 Un ItHV 1 7 BIO

RE

C E IV 1 

2270

5f*y*}f! 1 7 1 6 

WALLS 1 7 \ ABAT

F 1 A

it

J * 1 HI 3 73

£X POSU 1 

30 FR AC TU 1 50 HANARO 16 233 MA I WC AO CFUFQ

1 

un nil/1 c 1 A 386 CANAL L 5 34 COLL I S 15 681 C OMPE N 1 

1069 nFT CO . z 1 

3 3 vunui KNOWLEF 15 OAO 969 OWEN S 1 5 '*

PA

V EM E 1 

89 PUT 15 641

C AUS E 

3*1 GR OUP E 28 1 PE

T I T I 1 

4506 ASSIGN 1 6 2010 OASIS 16 1334 E X CUS E 16 1 70 VE RO I 1 6 206 7 COMPL A 1 5 2823 CONTRA 15 4 781 SUPPOR 1 

2618

C AUS I 269 PR OVOC 35 25 NEGL IG 30 2129 I N JUR 

26 1125 I N JU

RY 2 3 1290 PL AC E 1 583

C

A USED 2 

936 DEATH 2 

1667 COLL 10 20 141 FALL 1 340 LEFT 1 

100 3 PUR

V I 1 

8 

THR OWN 1 

68

J'J ^ ! t7

® f 7099

no I^FD 16 3 5 1 

AA5 442 1 

62 1 

147 ir no ice 1 A 2 4 USE 116A 2696 OR I VER 1 

728 FELL L 5 244 uc ah 15 227 K NOC K 

1 

4 1 R I D I NG 1 5 22 2 CAUTIO 1 24 CARE 3 3 1060 SUSP IC 30 68 2 7 2175 EXERC 

26 154 1 GU I L

T

Y 25 1427 SAFE

T

20 569

JUR

Y 19 3810 UTMOST 1 

2 2 NFGL I G I 8 2129 GREAT I 7 528 1 NFER E 17 64 8 M 1 NDS 1 164

UN COR

R 16 29 1 

133 1 

R E PR 0 0 29 38 ELL I0T 58 REC OVE 1 

1577 COMH | * 1317 COM PUT 1 192 I LL I NO 1 7 136 3 DE PREC 1 6 122 ER R FO 15 103 3 CEASE 6 A EK CUS E 21 1 70 R E M ARR 20 0 

DEATH CAT 1 9 1667 REVERT 19 40 CEASED 103 DAT E 26 1 6 1 

J ANUA P 22 116 7 EXIST I 9 300 C ONT R A 1 7 4 781 DEC EM8 1 1042 AGREEM 1 

1853

EX ECUT 1 5 1970 HER 1 3428

r f 1 i m 42

2f?'f¥? 2 5 2 

26 FALLEN 43 3 1 PAINT 2 4 W |R I NG 37 1 

SL EEP 34 30 SPR INK 32 20 COMH U 

50\ ei tiid 2 8 28 36 1 FLOORS 2 0 26 UN TEN

28 46 FIVE 2 5 1 195 L 0 D G I H 24 33 nnnoc 2 2 Id

RES 1ST 22 7 2 BU IL

D 2 

159 3 PARTI

21 125 Pit-" 5 onnF

52

305W I NDOW

i° 137

L 1 GHT 1 815 LEAK 45 27 WATER 37 636 HOUSE 36 82 9 CEMENT 34 6 5 ELLA 34 29 WALL S 33 71

EX PLOD 31 35 PUMP 31 35 GAS 28 387 PENETR 27 2 6 SHUT 2 4 57 BURNED 2 3 FL EM IN 22 38 SWITCH 22 68 BASEME 21 114 FLOOR 20 36 1 STORED 20 46 BU1L0 19 94 EXCEPT 19 2937 STEWAR t9 92 WALL 19 138 ORY 18 60 INS10E 10 106 OFF 18 637 CURS 17 112 DRAIN 17 67 EXPLOS 1 7 117 CARELE 16 122 AOV 15 85 PURVIE 15 01

143
CEMENT 65 CONCRE 47 158 POR TLA 47 16 CELLAR

BUILOI 23 1593 SIOEWA 22 185 LAWN

MIX 20 21 COHMER 19 527 WALLS

BLOCK 16 1 7B INGRED 16 31 STONE PUMP 1 5 35 SL IPPE 15 103 SOIL CEHETE 74 BUR 1 AL 77 52 MONUME 60 24 GROVE LAND 29 1389 TRACTS 27 56 QUI TTI OAK 21 67 SALES 21 572 ASS OUASHE 20 34 REESE 20 19 HALF

CONT IN 18 1912 CORPOII 1 8 1916 EXEMPT

PLATTE I 7 26 REL 1 7 1028 ACOUIR

SALE 15 1338 CENSUR 25 EXACT I 35 18 DI SC IP 31 91 PROFES

01SMIS 18 Z222 FEES 16 463 LEGI SL CENSUS 1 t POPULA 55 122 MANUAL 1 1 34 CRI 1ER CITY 16 3529 COUNT 1 16 131 YEAR. CENT 72 YORK 4 1 904 AOMX 21 17 BAIRD

SCHNE 1 16 50 CHESAP 15 3? CHIC AG CENTER 331 SHOPP

84 53 LANE 46 238 ROAD

WEST 34 642 ACROSS 33 325 FEET

TRUCK 24 565 H1GHWA 23 1067 HOUR NORTH 22 694 PROPER 22 3911 EGRESS

BUILOI 20 1593 CAR 20 1222 COLLI

SOUTHS 20 258 TRAVEL 20 462 AIRWAY

APPELL 16 7099 APPROA 18 546 CORNER

NORTHS 1 A 35 SK 10 IB 4fl ANGLE COHHER 16 527 CONCOR 16 18 EASTBO WIDTH 16 128 BLOCK 15 178 ONCOMI OUANTU 15 50 ROUTE 15 232 S10RM VILLAG 15 7 30 WIDE 15 209 CENTRA 373 YORK 36 904 KENNET 35 511 PROPER CLEVEL 24 52 1 WORCES 23 95 SI STER

RA tLRO 19 627 SUB SCR 19 130 MASTER

ASSETS 17 426 RE INSU 17 30 CAUSCD

DIOCES 15 25 INFERE L5 648 SUPPflR CENTS 94 DOLLAR 29 441 HUNORE 28 415 CUBIC

RATE 20 525 1AXAHL 19 136 PAY

TEN 1 7 610 THOUSA 1 7 218 BROOKS

CENTUR 78 STANIS 104 17 SPEARS 43 16 AGO

DICT 10 19 13 LOADIN 16 111 LI A8I

CEREBR 38 HEHORK l?l 46 HYPER T 101 20 BRAIN

THRnHB 49 17 HEART 47 106 HE ADAC OECEOE 34 966 HYPO TH 11 108 01 SEAS DRESS 25 23 HER 25 342 8 HOSPI

OIED 24 498 DEPRES 23 50 SUFFER DOCTOR 20 333 PAIIEN 20 147 ORE A Til WOUND 18 42 ACC IDE 17 1415 SMALL STRAIN 15 62 CEBEHO 19 ALFREO 52 44 COHAB 1 39 20 GLOUCE

AL IVE i ) 47 MAR GAR 29 100 BIRTH

WENT 16 733 TRANSA 17 640 DEAD

01E0 15 498 L IVE 15 130 CERT1F 1338 COUNTY 47 4129 CLERK 14 847 COPY OFFICE 29 259R PLUMRE 27 59 OOARO

SCHOOL 24 1 16 1 APPEAL 23 6176 AMENDE TUESDA 21 26 VEHICL 20 1 1 60 THANSC REGIS1 i 8 REVIEW 1 6 179B SEC JUVENI 1 i 184 PLUHU

17 92 OUICKL

ENTR1E 16 107 I ICENS 16 7B2 MUNIC

VOIICHE 16 18 WORK 1 0 1319 I ON ING JANUAR 15 1167 NUMBER 15 1167 PREPAI CERTIO 202 wr i r 44 1096 APPEAL 40 6176 C IR MARION DMA ClJC SUPREH I 1 1622 ILL INO 16 1 363 UNI TED CESSAT 19 Si oppa' 40 19 STRIKE 20 498 OPE RAT

CESTUI 25 OUASHE 121 34 ENR ICH 44 32 FIDUCI AGREEM 25 1853 UNJUST 23 1 1 7 LEGAL RATES 20 30 8 PROPER I 9 391 1 RFCE 1 V PR 1 NC 1 I 6 175 3 RE SC IN 1 6 H4 RFC OVE CHAIN 53 PROVOC 2 7 25 CHEM 1

26 2 7 SPEC I M LUCV 24 SOLOMO 3 1 24 CUSH 28 30 GIBSON

BENEF I 16 1816 ASSOCI 15 671 AUTOMO

HA SONR

OU ICKl 20 38 YARD 19 137 1 NC LUMHUS 20 nowo in 31 1ND 27 161 5 PR I NC

LUMP 32 SUM 39 1032 PERIOO 26 1516 SETTLE

PAT ERN 19 4 7 RATE 19 525 REMARK USED 1 5 2130 LUNCH 41 GLANOE 47 25 0 I NNER 25 21 MASS

CONT IN 1 9 1912 DAY 1 A 1712 ROARO

LUTHER 30 CL AR A 28 23 CL INE 27 26 ESTATE

ERROR 19 3093 AMERIC 17 635 CHURCH

LYING LSI SAW 33 628 IMHEDI 24 850 SOUTH CAME 18 100 LIO 1 A 20 ROAD

ALLEY 16 149 CORNER 16 234 FEE T FENDER 15 16 LEGS 15 41 LOGAN

WHEEL I 5 63 GRANT

LYNN IB BOSTON 16 409 LOWELL 16 58 LYONS 31 BUCHAN 54 25 OAVI s 11 255 CAOILL

MABEL 22 EVERET 92 34 BOWMAN '10 '.3 SALAR1 DONALC 34 120 GROSSL 31 4 7 GRATUI MCCAST 20 62 ME E TIN 19 331 SHARES RESIGN 1 I S3 VEP. IF I 17 221 COMPOS

WILL 15 4623 MACHIN 388 PINOAL 52 17 REBUIL 49 35 SEWING

DEV ICE 42 221 PIN 41 27 PL A YE R CORN 30 27 TRIMON .10 26 PICKER VACUUM 28 29 ADVERT 2 6 257 SA2AMA

GEAR 23 17 VOT ING 23 129 CLEANE MANUFA 20 415 BUCKET 19 16 GAM1 NG TOOL 19 37 WON 19 )7 AUTOMA

KNOWLE 1 ( 969 OPCRAT 17 2956 PREC I N ROLLER It 21 SELL IN 16 215 TOOLS OESCR

15 1067 01 '.PAR 15 26 1 NC TELEVI 15 57 VOTEO 15 91

34 29 S AND 32 7 3 POURED 27 21 BUCKE T 23 16

21 35 MARBLE 21 20 PAT 10 21 19 HOARD 20 3543

1 8 71 WATER 18 636 w iNiinw 18 137 FELL 17 244

16 124 GARAGE 15 18 7 PLACED 15 713 PLEASA 15 38

15 38 39 67 MARKER 34 1 8 LOTS 33 299 ASSOC I 29 671

24 23 EVILS 23 26 PEOPLE 23 1850 EXTEND 21 72 2 20 201 DEDUCT 20 200 GRAVES 20 19 NONPRO 20 32 PROPER

IB 406 PLOT IB 23 ACRES 17 217 OPERAT 17 2956

16 610 FUNDS 16 595 INTTRM 16 80 MA1NTE 15 388

23 290 CLIENT 22 188 ENT IRE 20 1234 LAWYER 19 245

16 1845 RESPON 16 1964 RECOMM 15 372 UNCONS 15 400

25 53 PROPOS 23 1053 ENUMER 20 146 FEUERA 18 678

16 1185 INHABI 15 1 39 RESIDE 15 1079 UN 1 TED 15 10B0

21 31 COAST 18 24 CORP IB 542 POWER 18 1853

15 1 176 PIERCE 15 33

42 722 LINE 38 69 5 S IDE 34 722 SOUTH 34 635

30 933 LEFT 30 1083 EAST 27 653 INTERS 25 643

23 406 PER 23 B6 1 COLL ID 22 141 MA INTA 22 9 79

2 1 B2 FRONT 21 442 IMPACT 21 1 16 INGRES 21 69 20 681 LANFS 20 60 MILES 20 373 SLOWED 20 38

19 22 OUTER 19 32 TRAFF I 19 580 ABUIT1 IB 94

18 234 CURB 18 112 OR IV IN 16 526 FENOER 18 16

17 50 VEHICL 17 HBO APPROX 16 613 80UN0E 16 41 213 1

1

15 22 PA I NT E 15 34 P ARK WA 15 60 POINT 15 1301

15 35 STRUCK 15 372 TURN 15 429 USE 15 2696

28 391 I ALBERT 27 109 BROTHE 25 369 SHARES 25 298

21 272 UNION 21 566 COMPLA 20 2823 STOCK 20 510

1 A 604 QUOTES 18 5 1 RUSS IA 18 50 STOCKH 18 171

16 936 MAINE 16 4 9 NORTH 16 694 REVOLU 16 22

1 5 2618

27 23 E IGHTY 25 65 FIFTY 23 236 SAV 22 32

i n 1485 SIXTY 18 159 G 1 BOON 17 20 PER 17 061

16 39 FIVE 16 1195 SALES 16 572 THIRTY 15 399

33 86 INOEMN 29 21 1 R 1 TCHE 27 18 MIKE 20 18

15 1336 ORIHOD 15 32

75 43 SYMPTO 54 3 7 PRESSU 53 84 SHOCK 50 42

38 1 B 61 000 36 182 ARTER

35 21 LACERA 35 22

30 145 WEAKNE 30 29 PATHOL 29 32 LARGER 27 80 25 731 SKULL 25 2 3 VESSEL 25 40 0 I AGNO 24 72

I 9 3B DAY 19 1712 MEOICA IB 546 RECOVE 18 1577

1 7 324 PROGRE 16 157 CIRCUL 15 117 MCCART 15 62

23 53 WIFE 22 1141 MARR I E 19 208 LAWSUI 18 89

16 116 FAITH 16 33H KNEW 16 619 PERFOR 16 1 392

34 456 ALLOWA 3? 39 7 COPIES 31 173 ESTATE 29 2561

26 3543 TITLE 25 118 3 AUTHEN 24 53 LEVY 24 200

22 14B4 NEGAT22 184 ANSWFR 21 2431 JOURNE 2 1 44

19 4B0 ACCRUI 1 fl 58 DAT F D 18 445 PAYMEN 18 1724

1 B 4065 SURREN 18 89 A D 0 1 T I 17 1490 BIRTMA 17 17

I 7 3B RETURN 1 7 164 3 AUGUST 16 604 CODE 16 3149

16 1 382 PAR 16 796 RECOUN 16 41 HE 1 SSU 16 18

16 7B3 AUCT 10 15 2 1 BR 1 T IS 15 22 HAH 1 L

15 323

15 21 SUPPDR 15 261 8 YEAR 15 1185

39 374 2 ON 1 NG 36 783 PETIT 1 34 4506 REVIEW 24 1798

2 i V A R 1 A N J UR ISO I j

1 6 1080

1 9 2956 BAS IS 16 1334 REGULA 16 1333 ACCUMU 15 125

39 196 H AH R 36 I / RCSC IS 35 50 SETTLO 30 71 22 1431 TRUS T 22 1546 REPUDI 21 51 CONSTK 20 2758

1 8 2270 THE R EU 1 8 539 FRE 1GH 1 7 135 BENEF16 1816

1

1577

2 3 35 HOC K £ R 2 1 2 3 P 1 T 19 4 7 C AUSA T 1 B 30

22 49 DAUGHT 17 286 REMARR 17 81 SURVI

17 486

15 1513 OWNER PROPER 3911 01IA SUE 22 34

18 1466 STORED 15 46

26 1753 LEAD 22 155 LIENS 17 97 MASS 16 1504

26 90 3 LIEU 25 75 JURY 20 3B10 01 SABI 19 247

19 81 COMPEN 16 1069 DIVEST 16 69 CLOTHI 15 74

23 1504 PERIOD 23 1516 TELLER 23 26 RESTAU 20 186

) r 3543 CODE 16 3149 COMPAN 15 2887

26 2561 BLDG 22 70 RUSSEL 21 115 VIGO 20 44

16 349 HEIR 15 79 23 635 TENEKE Z 1 15 WEST 21 642 TRACT 20 295 AVENUE NOR THW 1 j 127

16 933 HELPED 16 53 HIGHLA 16 38 LANOS 16 316

L OOK E ft 1 5 105

152

28 40 OUICKL 22 38 COUNTY 20 4129 CAR 18 1222

46 65 SHAREH 45 115 SHOE 41 43 ALLEN 40 211

1 A 296 TEM POR 18 46 5 VOT ING IB 129 PERCEN 17 816

16 93 DAY 16 1712 HOWARD 16 101 INJUNC 16 684

47 21 COIN 46 35 GAMES 45 47 SLOT 43 20

41 37 GAMBL39 76 VENOIN 39 16 SWI TCH 35 66 29 22 OALL 28 80 DECEPT 28 48 FAMOUS 28 17 25 15 USEO 25 2130 OUANTU 24 50 COFFMA 23 18 22 38 SCORE 22 19 SK ILL 22 74 EOUIPM 20 366

1 9 16 LOCK 19 60 OUASHE 19 34 SELL 19 514

18 176 CITY 18 3529 COINS 18 17 SALES 18 572

1 7 58 SALE 17 1338 DISPAT 16 34 POOL 16 85

16 86 USE 16 2696 BLUE 15 56 COLUMN 15 37

15 1466 MERC HA 15 260 OFF 15 637 PARTS 15 378
MACK 22 SHOE 32 43 REVOLV 3

75 BAG

2

641 RENTED 20 65 KE Y DRUGS 1 7 8 I PEOPLE I 7 1650 RAY MACON 29 DEC AT u 4 7 35 LOUB 47 1 b C I RC Ul 20 1 84 BUM GL

I 7 177 JUR I SO NAD I SO 115 LE AGU E 38 30 DINER 2 3 1 (, F LORE S

1

BERTHA 76 2 l B ARNE T 6 8 2 6 HARRY

MAfAZ

4 2 58 1 6 MODERN 40 1 34 I LL IC

MARK 24 92 PUHL I

24 3129 CONFES

PU B 1 7 79 RE TURN 11643 CHANGE

COVER 1 5 208 HAND 15 715 NEWS

HAG 1 ST 75 POL ICE 39 1241 PEOPLE 2 8 1 050 SE SSI 0 PEACE 19 175 CI TY 1 8 3529 CRI MI

BOND 15 528 CLERK 15 847 MAGNIT 1

US E 1 9 2696 COMPLE 1 7 1453 MAHON

42 VALLEY 49 63 SAN I TA 2 3 167 YOUNG S

MA I139 REGIST 54 434 HAIL IN 44 39 POSTAL

OR AM EE 2 6 36 HAILS 2 7 1 5 ADORE S PI EPER 22 1

MAKER 2

49 NOTI F

ENVELO 136 FILING 1 7 1003 RECE I P SUPPOR 1

2618

MAI LEO 62 MA I L I N 35 39 BEAL S 34 1 6 COPY

ENVELO 1 6 36 CHECK 17 404 JANUAR COMPAN 1 5 2887 REAL 1

1528

MA 1 L I39 MAIL 1 39 HA I L ED 35 82 NOTICE LETTER 2 3 60 1 ENVELO 20 36 D I SHON REGIST 16 4 34 SEND 16 53 COMPAN

MAILS I 5 POSTAL 49 16 D 1 SHON 3

40 MAIL

MA I N 330 ESCAP

30 22 SWITCH 2 0 6 6 L I NE

AL ON G 20 426 INCH 20 94 INTERS TRAIN 1 8 320 E XPLOS 1 7 117 L I GHT

TRAINS 1 6 90 A SK ED 1 5 876 C OMPAN TRACKS 1 5 170

MAI HE 49 BOSTON 69 409 FAY 56 1 5 MASS

EMPTY 1 8 33 LEGI SL 1 7 1 845 MA S SAC

Cl E AN I 1 6 4 1 COMPAR 1 & 392 VOR K MA 1 NLV 36 HER 1 8 3426 MAINS 35 PIPES 7 2 50 LEAK 4 1 2 7 WATER

INSTAL 24 4 39 SYSTEM 24 596 E XPLOS BASEME 1

114 PROPER 19 3911 SEWERS

MAI NT A 979 CONSTR 39 2758 MAINTE 29 388 CHARGE

PUBL IC 2 7 3129 PUHP 25 35 ADD 1 T

ROADWA 24 76 MEMOR

23 59 CENTER

REV 2 1 1216 ER EC T 20 77 1 N TRUS LEWDNE 1

2 3 PROSSE 1

57 C LE AN PERFOR 18 1392 PROTEC 1

924 TAX EXCEED 1

405 OBTA IN 1

1 296 PAR HIRES I 7 75 AGREEM 16 1853 A PPOI

HIGHWA 11067 I NC I DE 16 474 PARTY

W A RN 1 N 16 196 SYSTEM is 596 MAINTE 3B8 SUPPOR 32 26 18 MA 1 NTA 29 979' E XPENS

S I CKNE 2 l 52 20 471 POWER

CH ILOR 1 r 910 IN SURA 18 1188 LEVIED

L I BRAR 16 102 PR INC I 16 1753 AUT OHO

PAR 1

796

MA I TLA 37 P I LGR144 30 GREENB 101 49 TAYLOR LARCEN 48 122 HARVAR 44 32 CREDI

OEFRAU 23 77 SHORE 23 80 CONSPI BORROW 10 130 DI SCOU 1 8 79 BASES

OBT A I

1 7 1296 BOSTON 1

409 C OMMDN

LOAN 15 389 SERVIC 12498 MAJOR 97 GOO 16 40 L IMA 16 2 1 I NTRAS

KAJCRI 588 APPEAL 30 6176 VOTE 26 306 TERRI

VOTES I 7 90 CONTIN 1

1912 GREAT

MAKER 49 DRAWEE 54 36 PAYEE 42 47 NOTE INSTRU 26 2295 SOL VEN 24 1

CHARGE

ACCOUN 1

1 267 BANK 1 8 1129 DRAWER

CHECK I 1 5 48 HER 1 5 3428 MAKERS 26 HOLL I 0 33 20 MAKER 20 49 JOI NT INSTRU I 7 2295 PAYHEN 1

1724 MALOEN 25 OOWL IN 5 1 24 FLYNN 3

42 BOYLE

COV ERN 19 1154 INTERE I 9 2557 TERM

MALE 32 FE MAI84 55 GIRLS 24 29 RE F OR M PEN I T E 1 8 207 PROSSE 1 7 57 REVI SE MALFEA 15 ACCUSA 33 62 NEGLEC 20 24 8 L I AB 1

POL ICE I 7 124 1 CHARG

1 & 236 FEES

MAL ICE 120 MAL 1 C 1 54 147 GIST 45 44

K 1 LL I N 28 77 SL ANOE 2 0 3

MURDER

JU S T I F 24 810 24 8 9 PUN 1 SH FALS 1

19 33 L 1 B6L0 1 9 36 W 1 LF UL RECOVE 16 1577 WANTON 16 195 DENOTE

MALICI 147 WILLFU 60 107 MAL ICE 54 1 20 wl LFUL BANKRU 26 1 30 WAN TON 26 195 BATTER

INJURY 19 1298 UNL A WF 1

5 30 FAL SE PKOBAB 1

4 70 COMPLA 1

202 3 CR 1 ME MALLEV 20 AY ER 53 18 HENDER 35 4

BROKER

NE GOT

2 1 256 MALONE 15 COUNS E 20 2111 LAWS 16 1072 MALPRA 42 SuRGEO 38 83 DAMAGE 34 2098 B LOOHB FEASOR 19 65 SURG IC 1 7 46 TREATM

MAN 64 1 HEN 45 423 HAT 35 1 8 WEN T CAR 2B 1222 PLACE 28 1583 SHOUTE AROUNC 26 355 GET 26 485 PAIR CREEK 1 9 5 t nf.C URR 1 8 1096 SE wr R OATH I 31 SW OR N 29 I 7 1 AN A $ T A SOLEMN | 7 1 5 STATES 1 7 1 H63 TMUL

Y^ OATHS 24 JURORS 40 22R , s IMW TY AFOR F S 1 9 256 OFF ICE 19 2596 FURM

OBEDIE 21 INTERS 2 1 64 3 S TOP 20 161 E * POSE

FA I L UR I 5 OBEY 24 ORDINA 1 8 2 34 1 PUhER 1

1 85 J OHETEO 19 STATES 24 1863 UNEUUI 22 103 OI F I N 1 CITIJE 15 2R5 FOU 1 TY 1 5 544

OOI TER 15 DICTUM 78 34 DICTA 54 21

24 45 GUN 24 124 NARCOT 22 201 POL ICE 22 1241

19 66 PART 1

19 125 W I TNES 19 2175 DI ST IN 18 890

1 7 89 HOWARD 16 101 PAPER 15 200 PURCHA 15 1487

35 1389 REPLEV 29 65 HAYES 21 73 CARROL 20 84

16 2136 CORPOR 15 1916 IMHED

15 650 NIGHT 15 282

16 16 JUR 1 SO 15 2136

22 110 FORMER 16 785 LEE 15 128 36 18 NEWSPA 29 171 PUBL IS 29 246 STORIE 25 21 22 267 TITLES 22 48 TRADE 18 218 UPPER 18 71

16 1383 UNFAIR 16 130 BOOKS 15 213 COMMON 15 2956

1 5 56 WORD 15 659

26 170 PROC 22 28 HISDEM 20 124 RECEIV 20 2270

17 1063 SIT 17 46 ARREST 16 614 JUSTIC 16 876

1 9 97 COUNTY 18 4129 JURY 17 3810 WA TER 15 636

37 16 NOTICE 31 1936 LETTER 30 601 OELIVE 28 B5*

26 347 COPY 26 456 SENT 25 245 PROPER 23 391

2 I 303 DI SHON 20 40 NOT I FY 18 107 PREPA

18 21

17 394 SENO 17 53 DEFERR 15 47 POSTIN 15 44

22 456 NOTICE 22 1938 LETTER 20 601 WR I TTE 20 1019

1 7 1167 NOT AT I 17 65 DATE 16 1611 ADORES 15 347

34 1938 AODRES 32 347 HEALS 30 16 POSTAL 30 16

I 9 40 DEL 1 VE 18 854 PROPER 18 391

PUBLIC 16 3129

15 2887

27 139 ACCEPT 1 7 1150

24 695 TRACK 22 173 GAS 21 387 WEST 21 642

20 643 NORTH 20 694 RUNS 19 95 ODOR 18 46

16 815 PIPE 16 167 PLANTS 16 55 STREE16 1420

1 5 2887 OISTAN 15 365 PR INC 1 1 5 1 753 SOUTHW 15 131

26 1504 Wl LM IN 24 36 COMHON 21 2956 CONTRO 19 2 3 74

1 7 307 PARTY 1 7 1888 CAUSED 16 936 CENTRA 16 373

16 904 CHANGE 15 1 38 i CON DUC 15 1406 CROSS 1 15 32T

33 636 GAS 32 38 7 PUMP 28 35 SEWER 27 154 23 I 1 7 EXPLOO 2 1 35 LOCAT E 2 1 707 BENEA T 20 41 19 45 EXT ENS 1 8 342 ABUT T 1 I 7 94 TRANSH 16 157 28 3341 CONST I 26 3193 L IABLE 28 661 NU1 SAN 28 184

24 1490 FUNDS 24 595 PROVOC 24 25 REAL 24 1528

22 331 CAREY 2 1 1 9 PEOPLE 2 1 1850 POWER 21 1653 20 21 OPERAT 20 2956 PRINC1 20 1753 OUA SHE 20 34

18 44 JOIN 1 B 89 LOCUS 18 63 OVERAL 18 24

18 1334 URBAN 16 25 APPURT 17 48 ARTS 17 26

1 7 796 REMEDY 1 7 545 S INGLE 17 571 USED 1 7 2130

16 1097 EXISTI 16 660 GOVCRN 16 1 154 GRAOES 16 53

16 1 868 PLEA01 16 103 7 ROAD 16 722 SUPRA 16 2012

26 784 REPAIR 25 464 PROPER 24 3911 TAXES 22 498

20 1853 HUSBAN 19 93 1 PAYMEN 19 1724 OUA SHE 19 34

1 7 136 HENEFI 16 1816 OIVORC 16 578 HO SPIT 16 731

15 1513 CEMETE 15 74 GRANOP 15 26 LEGALL 15 289

66 190 LOANS 59 96 INDICT 52 648 COLLAT 49 203

33 673 FICT IT 26 39 LENOIN 26 23 COMPAN 25 2687

22 2 1 7 MONEY 22 1274 NOTES 22 218 CHECKS 21 226

I 7 46 CHARGE 17 334 1 GENU IN 17 93 NINETY 1 7 92

16 2956 MONEYS 16 10 3 TRANSA 16 640 WR 1 TIN 16 432

1 5 41 V 1 LL AG 15 730

1 8 293 DEADLO 17 1 6 JUST IC 17 676 MEMBER 1 7 1268

15 528 PROPOS 15 1053 REAL 15 1526 TENOR 15 20

30 726 MAKERS 26 26 POST IN 27 44 OEFERR 26 47 23 3341 PAYME N 22 1724 MAIL 21 139 OEFRAU 20 77

I 7 38 FORGER 1 7 65 CHtCKS 16 226 MATURI 16 45

23 585 COGNOV 22 4 5 PAYEE 2 1 47 CONTRA I 7 4781

28 28 JORDAN 26 3 3 EXCEPT 22 2937 POINT 21 1 301

19 1149 OWNED 16 842 BLOCK 15 178 OAY 15 1712

23 90 WOMEN 20 1 1 9 SENTEN 19 817 COOE 1 6 3149

16 2493

18 13 36 OFF ICE 18 2598 PERFOR 18 1392 OONALD 1 7 120

16 46 3 43 2 3 OEFAMA 36 40 ACTUAL 32 1165 CAPIAS 32 1 8 27 195 PROVOC 27 25 K ILL 26 49 PREMEO 26 28

23 240 KECKLE 23 98 L IBEL 22 49 PROBAB 20 470

18 249 PROSEC 17 1143 ACTEO 16 297 DEROGA 16 49

15 2 1 37 249 MISCHI 36 26 PROSEC 32 1143 RECKLE 27 98

24 69 JURY 22 3810 SLANOC 22 31 ASSAUL 20 191

18 316 OFF ICE 18 2598 L lul l 17 49 PUI SON 16 41

15 650 FELONI 15 62

32 147 FRANKS 29 33 LAWMEN 25 125 WOOO 23 90

26 19 APPL IC 24 3117 AGGRAV 21 81 SURGE W 20 55

17 268 EMPLOY 15 310 1 35 733 WITNES 32 2175 TOLO 3 1 796 GUN 30 124 28 20 SWUNG 28 19 VICT IM 28 108 COLORE 27 29

2* 23 SAW 25 628 HAND 24 715 PORVIE 24 81

1 5 21

1 5*

7? 1 1 1 NUU 1

20 4 1 4 HMNU 11 52H ME HHE H 1 7 1268

\ 5 19

1 868 VOIR 25 66 0 1 1< E 24 TO mr.NI T 22 46

1 8 921 PUT 1 5 64 1 11 73 STOPPE 19 522 ACTS 15 1271 APPKUA 15 548

19 864 1 NJUNC 1 7 68'. CONOUC 16 14„6 WAHUIN 16 116

145
OB JFC

T 1 96 3 APPELL 59 7099 Hi TNE S 4 1 2175

C OUNSE 40 2 111

C OMPLA 3 7 28 2 3 PERMIT 3 3 2261 RECE 1 33 2270

AOM I S S 30 832 CRIME 30 850 LEG 1 SL 30 1 845 ERROR 29 309 3 OU ASHE 29 34 AN S WE

R 26 24 31 OFFICE 26 259B 0 Vf RRU 26 1 460 PRQFF E 26 64 R E NDE R 2 6 1456 BENCH 2 5 66 INS TR U 25 2295

OR 0 1 N

2 5 2 341 PQ ( n

y 2 5 1301 OEF

FN S 24 1159 RED I R

G 2 4 2 B CROSS 2 3 1 074 FAIR 2 j 709

HE AR 1 N 23 19 35 I NO 1 1 1 2 3 PUR

V I 

23 8 1 2 ) 69 7 V 1 LL AG 2 3 7 30 MO

V EO 22 451 2 2 32 OF F ER 

22 66^ CIVIL 2 i 956 DATE 2 l loll FUMES 2 I 1 5 PAYERS 2 1 1 5 PU HP 2 l 35 REL E VA 1 1 4 1 7 BE R NH 

20 I 7 DI SCER 20 39 I NTERP 20 860 QU 1CKL 20 38

RUL I N C- 20 88 3 THEA

T S 20 I 7 TRUST 20 1546 OR AWN 1 9 469 JUO 1 C 

19 800 MOUTH 1 9 4 1 PE RFOR 1 9 1392 PR EM 1 

1 9 1 32 7 S I X I 9 74 5 T OLO 1 9 796 HE EK EN 19 19 WEN I 1 9 733

AL IKE 1 8 ^ 9 BEHALF 1 8 6 89 BOUN

T

Y 1 8 2 1 CH 1 C AG I 8 1176 CONSTR 1 8 2 758 HOL LAN 1 

22 INTROC 18 6 8 1 NUMBER 1 8 1167 PROOF 1 8 1220 PURSUE ia 17 3 SUPPOR 1 8 26 1 8 UNWOR T 1 

22 WANT 1 B ii 39 CL A SS 

I 7 3 78 I SSUE S I 7 1101 MAS T ER l 7 60 4 OPEN 1 7 597 PL E AD 1 1 7 1 037

PR ES E R 1 / 2 51R E F USF I 7 1134 SOLE | 7 6 38 CHE

M I 

16 2 J DES I RA I 6 96 E MBRAC 1 6 136 FAVOR I 1 b 26 [ N I EH V 1 6 54 3 ME THOD 1 6 559 NEGL I G I 6 2129 OHNE R S I 6 913 QU 1 T 16 54

RE AO I 

1 

139 SOUND 1 334 TOOK 1 6 978

T YPIC

1 6 26 ASKED 1 5 678 CAP I TA 1 

221

CH 1 L DR 1 5 9 10 COME S 15 328 CREAM 1 5 29U I S TR 

1 5 194 5 EVAS IV 1 5 30 EXPEC

T 1 5 290

EX POS

I 5 30 GU I SE 1 

3 \ HE

A

R SA 1 5 66 1 NO I CT 15 64 8 I r em 1 i 1 5 29 JUS

T 1 

1 810

NOT ICE 1 5 19 38 PA

V I NG 15 30 STRIKE 1 5 498

081 IGA 978 PROPER 3 J 3911 I N SURE 25 86B 01 SCH

A 2 4 771 1 NCURR 24 2 3 7 ENFORC 2 3 745 NEGL IG 22 2129

AG RE EM 2 i 185 3 BU S I NE 2 

1 909 1 NOE

BT 2 

1 90

F 1 NANC 20 54 3 I MPL I E 20 431 PA YM£

N 20 1724

RE

L I E V 20 208

C

R ED 1 1 9 6 73

L ANDLO 1 9 185 LIABLE 19 66 1 QU I CKL 19 38 CREATE 1 

852

En cum

h 1 R 93

L

A NO I H I 389 PFRFOR 1 g t 3*72 RF PRES 18 133 7 DAMAGE 1 7 2098 E XONER 1 7 28 INJURY 1 7 1298 IN TFND 1 7 1175 L ANGUA I 7 1252 LESSEE 1 7 178 MORTGA 1 7 574 EX I ST I 16 660 LEG 1 SL 1 6 1845 LE VY IN 16 31 MANNE R 1 6 1161 NOTE 16 726 RE AO 16 697 BONOS I 5 316 FUKN I S 1 5 915 L E SSOR 1 

119 SF

R VE I 5 319 SOVE RE 1 60 TCNANC 1 5 125

OBL ICE 1 00 PROPER 3 i 3911 OBSCEN 90 LEWD 104 55 I NDCC E 8S

L ASC 1 

68 34 PROF AN 48 1 5 PAHPHL 42 25 SPEECH 4 1 59 POSSES 3 8 114 3 FILM 37 33 P 1 C TUR 37 202 I F HONE 3 3 2 3 L I T ER A 3 1 105 BOOKS 30 213

SC I EN

30 9 1 BOOK 2 8 168 ROTH 25 1 8 PROSSE 24 5 7 BROADC 22 36 FR EEOO 22 1 io GU I L T Y 22 142 7 PRINT 1 

29 STRICT 19 357 CONT RO 18 2374

D I SORD 18 76 1 NFR IN 1 

50

L I AB I L I 0 13 36 I MMOR

1 7 36

R AC 1 NG 16 61 STATES 16 1 86 3 CONS

T 

1 5 3193 OR AW 1 

15 69

EV I LS 1 26 KNOW I 

1 

232 KNOHLE 1 5 969 MATURE I 5 26 OBSCUR 60 FOG 58 20 SOL ON 28 2 

K E AL Y 23 1 8 V 1 S I B I 22 34 PL ACE 19 1583 TRAVEL 1 462 6E0F0R 1 5 40 0B5ERV 8 10 SUNOAY 3 7 177 DEMEAN 25 HILL 31 482 3 SABBAT 29 2 3 FORM 2 7 9~2i HOL Y 26 1 8 HE I GHT 25 789 S I GHT 2 3 49 F A I TM 2 0 338 NEGL 1 

20 2129 ST AT EM 20 201 4 VIEW 20 1 285 ACCEPT I 9 1 1 50 APPELL 19 7099 F LORE S 1 9 1 

FUROR 1 9 I 8 S AH 19 628 ENFORC 1 8 745

REL I G I 18 137 HEN

T I 8 733 CAUSED 1 7 936 GOO 17 40 MAN 1 7 64 1 OFF ICE 1 7 2598

op por

r \f 500 ROBH 1 

1 7 2 3 SUPPOR 1 7 2618 ADM 1 S S 16 83 2 CHRIST 1 & 1 30 GOLF 16 26

AL MO S T 1 5 29 3 Al

h e AD 1 507 AMP

L E I 5 1 46 8ES 1 DE I 5 7 i CAME 1 5 700 CAR 1 5 1222

D I S T AN 15 365

D 1 S IR I 1 5 I 94 5 DRIVER 1 

728 MANAGE 15 532 MEMORY 1 48 NUMBER 1 5 1167

OC CUR

R 1 ^ 1096 OU AN

r

U 1 5 50

R I NG 15 47 SHAPE 1 5 29W 1 TNES 1 5 2175 OBSOlE 20 ACRE AG 33 26 DEPR EC 10 1 22 CARDS 2 8 36 VALUES 2 3 9 4 REPL AC 22 398 OWNER S 1 9 91 

BU ILD I l i 1593 COMH 1 15 1117 SCHEOU 1 2 I 1 OBSTAC I 7 JU R ISO 2 I 2138 SUPR EM 1 9 1 622 POSSE S 1 

1143 OBST AN ! 9 VEREO 

2 32 I 7 NON 54 389 OVE RRU 26 1 460 VERO 1 22 206 7 INJURY 16 129B OBSTRU L99 HI NDER 5 1 3 I TR ACK S 29 I to HEEOS 29 1 8 CROSS I 28 32 7 APPROA 2 7 548 UN OB S 

26 30

V I EH 26 \ 285 TREE S 25 40 I N TERF 24 399 TRAVEL 24 46 2 V I S I ON 19 52 Wl NOOW 1 

137 0 I ST AN 16 365 FENCE I 8 7 1 ACROSS 1 7 325 CONS T 

I 7 319 J PER JUR 1 

48 PILED 1 7 16 POL E S 1 ? 27 PUMP 1 7 35 TR AC K I 7 17 3 H AR N I N I 7 196 EVAS IV 1 & 30 FLOW 1 6 1 34 MOTOR I 1 6 54 HE S I I 6 642 BARR I 

1 20 JUST I C 1 5 876

L AWFUL 1 435 MANNER 1 5 1161

NU I SAN 1 5 184 SURF AC 15 22 7 TRUTHF 1 33

OB

TA 1 N 1296 INFORM i, j 1107 PRE TEN 4 3 MCNEy 15 1274 FALSE 3 3 316 TRANSA 32 640 PUMP 29 35 COAST 2B 24 NA TURE 2 8 1 064 AOVANT 26 198 CREO I 

26 67 3 PETERS 23 102 PROPOS 23 1053

ADDI

T 

22 1490 PURCHA 22 1487

A SSUHE 2 1 894 QU I T

T 2 1 2 3 CHECK 20 404 CONTRO 20 2 3 74 0 I SCOV 20 417 REPR E S 20 1337 A SSI ST 1 g 520 Dfc'FR AU 19 7 j SUMMON 19 369 YEAR 19 1185 CLAIM 1 n 1938 COLLEC L 8 670 CON T I 

I 8 1912 COUNTY 18 4129 ENR I CH 18 32 LEGI SL 1 8 1845

NOT ICE I 8 19 38 PARTY 1 

1088 PR I NC I 1 8 175 3 SERVED 18 650 BU IL

D 1 1 7 1593 CLAY 1 I fy 01 VORC 1 

578 OR AFT I 7 6 3 F RAUO 1 571 HUNDR E 1 7 415 INOUC E 1 7 227 HA 1 NTA 1 979

MA I TLA 1 I 37R I GHTS 1 7 1681 RULl NG 1 

BO 3 USER I 7 3 5 ARR ANG 16 364 BAL ANC 1 6 508 COOL I 

16 1 7 CREATE i 6 852 06

L I VE 16 854 dH

L I g e 16 190 EP I SCO 16 18

F I NANC 16 543

GU I LT

Y 1 6 1427 HEL PLE 16 I ft KNOH I 

1 6 2 32 L60GER 16 19 PAY 16 1485 UNL AUF 16 530 CARRY I 5 304 CONSTR 15 2 758 E PS TE 

1 2 I NOT E 1 5 726 REV 1 5 1216 SEPARA 15 1230 S I GNEO 1 

752 S I X 15 74 5 , TRY I 5 1 54V I S I T

A 1 5 20 WENT 1 

733

O0VI AT 32 OBJECT 2 2 19* 3 AMENDM I 5 90 3 PROBL E 15 481 0CCAS1 6 78 PROPER 38 39X 1 S I X 2 8 7 4 5 REF E RR 25 1197 CROS S 23 107 4 T EMPER 2 3 43 DOWN 22 761 MONEY 2 2 1274 PUR V I 

2 2 8 I F I VE 2 1 1195 PR 1 NC 1 2 I 175 3 SE VERA 20 1113 MEN 19 423

SU I C I 0 I 9 16

H I TNE S 1 9 2175 BUS I NE I 8 1 909 CONT IN 1 

1912 MCC ANT 18 1 8 TABLET 1 8 17 WEOO I 

! 8 ! 8 HORK 1 8 1319 BONE SE 1 

20 GRAVES 17 1 9 RUGGL E 1 7 20 S I GNl1 7 449 STICK 1 7 20 TOLD 1 7 796 HAN T 1 7 439 COMP AN 16 288 7 COMPL A 16 2823 FURNAC 16 22 1MPRE S 1 6 j 79 NUMERO 1 6 304 PROPOS 16 1053 V I C I OU 16 2 1 VISIT 16 84 WEN T 16 733 BABY 1 5 40 OEL 1 VE 1 5 854 ENTER 1 

610 HAB I TS 1 5 2 4 JUMPED 1 5 25 NAMEO I 5 646 NEVER 1 5 852 REPR E S 1 

13 37 SA

T | SF 1 

628 STREET 1 5 1420 OCCLUS 19 CORONA [ 9 9 34 THROMB 70 I 7 HEAR T h 7 106 HULL 60 I 5 ART FRY 52 20 AU TUP 

43 28

AT TACK 42 427 EMOT 10 4 

32 na than 4 I 32 PATHOL 4 1 3 2 chest 37 38 PREC I 

37 61

STRA I N 36 62 VE SSEL 36 40 0 1 AGNO )4 72C AUS AT 31 30 thrown 28 68 GOO 27 40

ACC I DE 2 5 1415 FR AC TU 24 50 I N JUR

2 4 1298 HLOOD 2 i 18 2 TRUCK 2 1 565 E X AM I 20 2219

JULY 20 955 SUDDEN 1 9 I it PA VEME ! fl 89 FATAL i 6 115 FORCE 1 

599 SUFFER 16 599 CAUSAL I

'5 125 MED 1C

I 5 546 WEE KS 1 5 22 1 OCCUPA 01 SEAS 4 0 145 BUS I NE 35 1 909 E NGAGE 2 8 705 TENANT 28 47 7 S I L ICO 2 7 20 BU 1 LD 

26 1593 ZON INC 25 783 LANDLO 23 185 L 1 CENS 22 782 L I Mb 22 30 RENT 22 257 COMPEN 2 1 1069

LE HONE 2 1 2 3 OCCUP 

2 

246 0 1 SA6L 2 0 91 PKEM I S 20 132 7 SM IPP 20 3* OWNER 1 9 1163

EX POSU 1 8 30 PAY 1 8 1485 RE SPON 1 I 964 BE NE

F I I I 1616 ENDANG I 7 46 EXCEP

T 1 7 2937

LIFE I 7 9 30 PUR SUE I 1 1 73 RE TAI

| 7 232 WAGE 1 

4 5 WORK I 7 1319 POSSE S 1 a 1143

ME AN I N I 5 984 PROSSE I 5 57 VOC

A

T 

I 5 28 OCCUP 

2*6 OHNE 0 35 842 AP AR TM 2 7 352 LAND 2 6 1 389 ROOMS 25 5 5 SPACE 24 146 BU I LD 

23 1593

RES I OE 2 3 10 79 FLOOR 2 J 16 l F LOOR S 2 1 26 OCCUPA 2 i 422 PROP E 

20 3911 SH I NNE 20 1 6 L ANOLO I 9 185 RENT 19 2 5 7 RENTAL 1 9 259 TENANT 1 

47 7 LEASE 1 4 79 PR £ M 1 

1 g 1327 8ASEME 1 7 114 LOCA TE I 7 707 E V 1 C T 1 16 35 QUAN T U 1 

50 SHE 1 5 2250 OCCUPY 8 2 PR EM I S I 8 1327 S I OE 16 722 PAY 15 1485 RENT 1 

257 USE 1 5 2696 OCCUR 126 UH ICHE I 8 24 OCCURR 16 1096 GCCURR 1096 ACC I OE 39 14 15 COL L I 

37 681 TRUC

K 34 565 1 HRACT 29 116 MOTORC 29 20 SAW 29 628 UNEXPE 2 7 35 1MME0 

26 850 PLACE 26 158 3 WEST 26 64 2 CONT RO 25 23 74 DR I V I N 24 526 EVENTS 2 A 168 GU IL TY 2<* 142 7 HAPPEN 24 355 HE AD 2 4 22 7 JURY 24 3810 M 1 RROR 24 29

NEGL 1 G 2 *. 2129V 1 EH <> 3 1285 HA TCH 23 48 APPL 1 

2 2 3117C

H I C AG 2 2 1 1 76 CORONA 22 34 ROOBEO 22 34 E X TEND 21 722 MONTH S 2 

709 PASSEN 2 1 42 3 ROAD 2 i 722 S 1 OE 2 1 722 JULY 20 955 USE 20 2696 BOY I 'i 9 3 OR A K E S 19 1 2 1 OR [ v EN 1 9 2 8,1 I P SA 1 9 65 LOOU 1 I I 'i 6 4 OFF 19 6 37 PAVE ME 1 9 09 POINT 19 130 1 PROVOC 1 9 25 SHOR T L 1 

2 34

SK 1 DOE 1 'J 25 SPOT 19 43 S TREET 1 1420 ACC E L 

1 Q 7 2 AL UN GS 1 8 28 CONTRA 1 S 4781 MEN I 8 42 3 OAK HOO 18 26 PROOF I 8 1220 SK | 0 I a S T 0 P P E 1 

322 1 7 BL OCK S I 7 76 C AR E 1 7 1060 CARRIE 1 7 740 ENDE 0 1 7 5 2 LEFT 1 7 108 3 NAT UR E I 7 1 064

PET I T I 1 / 4 506 QUAN TU 1 f 50 S TOP I 7 36 1 HA

L

K I N 1 

8 I ACTOR 1 6 1 ADM I N 

16 I 699

8R E AK 16 92 BROK EN 16 1 1 9 C OL LAP 16 35

C OMM E N I 6 100 I F ENOER 1 (, 1 

1 1 6 HULL 16 15 INTERS 16 643 MAN 1 6 64 I MOV I NG 16 21 Li OC CUR 1 6 ROBBER 1 6 302 ROPE I 6 1 5 SEATED 16 56 SPARK 1 6 1 5 AWAY I 5 38 9 BELT 1 5 1 6 H 1 C YC L I 5 1 g CARS 15 240 GL A SSE I 5 17 GOT 15 279 HER 15 3420 I DENT I I 5 853 LAP SE 15 67 LETT IN 15 36 LOSING 15 17 MARKS 15 100 M 1 SHAP 15 I 7 MORN IN 15 288 OHSERV 15 610 RECOVE 15 1577 RE IKOA 15 103 RUNNI 

15 244 STRUCK 15 37 2 SUDDEN 15 136 THROMB 15 17 THROHN 15 68 TON 15 I 8 HAY 1 5 1530 WHEEL 15 6 3

146
OCCURS 101 WHICHE 30 24 vacanC 24 104 ACC I OF I 1 1 415 TERH1N

OCEAN 16 CARGO 46 17 EXPORT 43 19 INLAND 39 23 SHI >PE vessel 30 40 CORPOR 21 1916 FREIGH 21 135 CARRIE

CHART e 18 179 COURSE 18 1362 CORP 17 542 HOVEHE OCIOBE 808 KANK AK 3H 26 STILES 35 16 JULY 31 955 J ANU Aft DAY 28 1712 DATE 27 161 KAY 27 17 SIGNEO

ANSW Eri 22 Z431 NOVfcHB 22 814 OVERKU 21 1460 SHE aooit i 17 1490 MARCH 1 7 98 ACCQUN 16 1267 AUGUST

cess tc 15 29 CONTRA 15 478 FOUR 15 1 173 PRfjAHB ODCR 46 8R EATh 54 38 E SCap 1 47 22 C API TO 44 18 ALCOHO

NO ISE 34 43 VI SUAL 25 19 8 URL N 23 72 PRINKI

HESTER IB 243 WE STWA IB 13 ACCOHH 58 OAIGHT 17 DRY 1 7 60 HIDDEN 17 36 LAWN

_ 35 LETT IN 1 7 KCC0N6 17 16 NOR THW 17 127 OCEAN 16 OWCN 17 RUNS 1 7 95 SOUNDEr 17 37 SU1C 10 1 7 16 TERRA 1 17

BR I EF 16 R68 COMPLE 16 1453 DR 1 VE W 141 FELL 16 GUESTS 16 41 HANCOC 16 IB HER 3428 INC 16 SIDING 16 19 SOUTHS 16 258 STOPPA n STOPP ! 16 HHATEV 16 386 ARGUME 13 1299 HLOCKS 76 BUMPER IS COHPAN 15 2887 DARK 15 75 OOUBTE 20 FASTER 15 LANGUA 1 5 1252 LINES 15 330 HA TER

1222 MOV 1 NG 19

Pr-OTOC I 5 158 POSt TI 15 12 76 REFERF 1350 SEAT 1 5 WARNED 15 42 WORK 15 1319 POINTE 473 NANCY 37 21 JESSIE 29 19 WILL 4 R2 3 HOL OUP 20 CHANGE 16 1393 L INEUP 16 16 MEN 423 STAT 16

»0 [ NT 1 45 PIN 29 27 V [ EW 25 1285 DATE 16 1 1 ANGLE 1

POINTS 330 SUPPOR 1 B 2618 MEMORY 15 48 POISON 41 TIFFIN 51 15 FOOD 35 174 HAH 20 ADUL TE 3 3 SYMPTO 25 37 INFECT 23 26 SICK

_ 73 TEA 2 I AT L ANT 19 66 CARBON 19 38 GROWTH ! 38 HUH AN I 9 TEMPER 18 43 UNF I T 18 39 BOARD 3543 HAL | C 1 16 POLE S3 CURB 32 1 12 HOLE 21 C OR NER 5ft 234 20 HIT 1 8 120 UT IL IT 18 370 722 s i'gn^ 1 7 MILES 15 373 STOP 15 361 TOWFR ' 44 POLES 27 WIRES 67 75 CONDUI 63 1

CABLE S 3 5 1 7 WEEDS

TREES 23 40 ERECT 22 77 TOWER 2 2 APPL I A 20

TR ansm I 9 157 1 L L UN I 1 8 63 HEAT 1

7 i OBSTRU 1 1 V 1 LL AG 1 5 7 30 POL ICE 124 1 OF F ICE 135 2598 ARREST 62 614 CHIEF 443 CAR 44

DE PART 39 1230 HAG 1 ST 39 75 PATROL ' 76 MAYOR 38 WENT 34 733 ORD INA 33 2341 RF.CE I V 2270 APPREH 32 CO MM I T 30 1317 PEOPLE 30 1850 POWER JO 1853 STOPPE 30 SERGE A 29 35 CR 1 ME 28 850 PAC K AG 2 B 88 ANSWFR 27 FIRE 26 554 L INEUP 26 16- RE L 2

1026 ROOM 2 6 CITY 25 3529 CONFES 25 267 DANCE 2 5 17 SALE 23 8URGL A 24 177 CH ICAG 24 1176 OE TEC T 2 4 71 FOUR 24 COMM I

23 3056 OK INK 23 51 HEN 2 j 423 SOU AD 23

MACK 22 22 SCENE 22 143 ABSENC 2 1 911 CADI LL 2 I RACING 2 \ 6 I SAW 21 62 8 STORE 2 J 366 WATCH! 2 I GU ILT 20 345 GUN 20 124 HANDEO 20 92 IHMEOI 20 SIREN 20 25 STAIRS 20 ' 45 TICKE T 20 42 DISHIS 19 MEMBER 19 1268 HORN IN 19 288 PHONE 1 9 27 VEHICL 19 WITCH 19 48 ALCOHO 18 154 APPOI

1 8 1097 BULLET IS DEL ICT IB 53 GONE IB 154 JACOBS I 0 51 NARCOT 18 STOLEN 18 146 ACROSS 17 325 BEATEN I 7 16 BREAK 17 OR I NK I 17 93 EVER 17 439 FILLED 1 7 119 HOLD IN 17 PENS 10 17 124 PROTEC 17 924 PUMP 1 7 35 SOUTH 1 7 TRY ING 17 92 VOICE 17 34 ACC USA 62 ADDICT 16

OR IV IN 16 526 GOT 16 279 HANG II 6 18 HOUR 16

PI NBAL 1 6 17 P 1 STOL 16 38 PLACE I 6 1583 REVEAL 16 ST ART E 16 293 SUPPRE 16 99 SWI NCI I 6 18 TALKIN L6 AC QUA I 1 5 69 BLOCKS 15 76 BRUTAL 1

20 COAT 15 OR IVER 15 728 FASTER 15 20 FUNCTI 1 5 494 GAVE 15 INSIDE 15 106 LEARNE 15 ISO HERRI

1 5 20 MONEY 15 POWERS 15 499 PUBL IC 13 3129 RFGULA I 5 1333 REVOLV 15 TELL 15 205 TREASU 15 249 VICINI 15 114 VISUAL 1 5 POLICI 183 POL ICY 56 1288 INSURA 51 use PREHIU 4 3 187 INSURE 33 EXCISE 29 65 INS 25 321 COVERA 2 256 FRATER 20 BUS INE 17 1909 LAPSED 17 18 NET 1 7 238 SICKNE \ 1 LTO 16 32 HA TERN 16 20 REINSU 1 6 30 SOVIET 16 POLICT 1288 INSURE 92 868 INSURA 83 1 188 COVERA 63 256 POLICI 56 INTENT 40 1614 COHPAN 3B 2887 ATLAS 3 6 34 CLAUSE 33 SICKNE 28 52 AUTO 27 128 FIOELI 2 1 75 AMER I

25

AT TORN 24 1893 PREMIU 24 187 AMS TER 2 3 20 EVENT 23 APPEAL 22 6176 ACC IDE 21 1415 BEN 1 1 42 BUCK E

21 METER 20 25 600ILY 19 109 CAS 1 9 79 DAMAGE 19 SICNED 19 752 VAL ID 19 712 HILL 1 9 4623 CASUAL 16 INOORS 18 55 ISSUED 18 1086 LEG 1 SL I 8 1845 LOSS 1 8 UNLOAD 18 113 CANCEL 1 7 223 01 STIN 898 EOUI T

I 7 PROPER 1 7 3911 RECOVE 17 1577 SUPREM 1 7 1622 ACC 16 CONST

1 6 3193 01 SAB1 16 247 LAPSED 1H PAY 16 SCRAP 16 39 SUBROG 16 103 TOUCHI I 6 4 1 BULL 15 DECL AR 1 5 1 381 EX TENT 15 727 JERRY I 3 21 OPERAT 1 5 POLISH 15 WASHIN 19 179 ASS 18 201 CH1CAG 1 7 1 176 LOAN I 5 POLITI 139 AFF IL I 46 55 PATR I A 34 57 ANAS T A 32 37 RUSSIA 30 SUOVER 28 58 CANDIO 27 138 DEHOCR 2 7 23 GOVERN 27

RE VOLU 23 22 ROOY 22' 491 MORAL 22 106 AUTONO 2 I LOYAL T 17 37 CUR 1AM 16 82 PHI LOS 16 27 GOOO 15 PART 1 £ 15 2743 REPUBL 15 46 POLLUT 32 STREAM 50 AO AIR 45 lie SEWAGE 4 0 60 PLYHOU 30 INCREM 26 22 WA TER 26 636 SAN 1 T

24 167 CITY 22

AB AT EM 19 126 BOARO 19 3543 WEIGH! 17 57 CONTRO 16 POhO 49 DAM 76 3 i LAND 4 1 1389 MILLAR 15 LOCUS 36 HA Y DEN 27 16 BROOK 26 24 173 WAY 21 1530 SITES 20 27 BRUSH 19 30 CAMP 19 ATTRiC 1 8 56 FLOW IB 134 HILL 1 B 63 ACRE 17 TURNP

1 7 40 PROPER 16 391 1 STREAH 16 60 INC 1 5 PQhTI

23 KL INE 54 15 BROADS 39 29 DRESS 33 HISS 27 RESEMB 22 50 BUICK 21 52 SAW 20 628 CHEVRO 19

RQ AO 16 722 STANDI 16 403

FOOL 85 Sw I MM J 89 38 WELLS 40 91 SHAKER 36 27 POOLRO 34 RECREA 26 68 MCCANT 25 18 G1080N 24 20 ROOM 22 TOOK 19 978 JANE 1 8 19 MAR | JU 1 6 53 SKILL 18 PAMPHL 16 25 RETURN 16 1643 HARREL 15 27 GAHBL1 15 TABLES 1 5 29 PQOLRO 16 MCCANT 74 IB SALOON 44 18 HAR | JU 43 53 IOLE 35

BACK 27 793 GET 27 485 HANDEO 26 92 WILLU 26

1712 WALKEO 1 ft 105 HONEY 17 ALLEY 15 149 BUY 15 140 GOOD 1 5 1033 POOR 72 CARED 18 24 WEAK 16 29 ACTS 15 1273 PAUPER 15

1 7 694 PROPER 1 6 39 11 : 1 LL 1

32 60 AMER IC 3 1 635 SH I P 31 20 740 FOREIG 19 167 SPEC I A 1 9

1 7 113 PO INT I 7 1301 \Rd 1 TR 15 30 1167 OECEHO 29 1042 SEP TCH 29

2 7 152 EMPLOY 25 3101 AID 23

21 225*> DA T ED 19 445 :ohmon 1 8

1 6 804 CLINIC 16 46 IALPH 16

15 27

38 154 SHORE 3l> 40 J K 1 NK S 34

2 3 93 GAS 2 2 367 DR 1 NK 20

3 7 CARGO 17 17 CLOSE L *' 96

36 LITTLE 17 404 MARK 1 7 92

I 7 PLACE 17 1583 OUA SHE 1 7 34

16 THICK 17 I 7 TRACK S 1 7 1 70 244 F IFTEE 16 195 GUARDS 16 1

1466 MOMENT 16 142 PEOR 1 A 16 66 TON 16 18 TURNED 16 2 79 20 CENTER 15 331 COLLIS 13 681 20 HANDL E 1 5 160 INC ioe 1

^44 216 NORTH 15 117 SOUNO 15 334 UNCOLL 15 21

I 7 CORREC IB 122 1 RE V IE W 16 1 796 1042 DEF INI 15 864 HOLDI

I 5 924

50

5 1 DEL ETE 30 59 RE FR I

25 39

53 FEED 20 33 HEALTH 20 515

106 proouc 19 1040 AARON 18 43

147 PAC 1 F16 91 44 POL E S 20 27 S 1 DE WA 20 I B 5 330 CROS S

16 327 FRONT 16 442

1 6 VOL TAG 27 29 OVERHE 24 37 86 POL E 20 53 ALONG 19 426

199 T ELEPH 16 491 UNDERG 1 5 83

1222 TOLD 43 796 SEARCH 40 319 248 F IREHE 37 83 ROBBER 36 302 88 TAVERN 32 184 A TTORN 30 1 893

322 ASKED 29 878 ROHBCD 29 34 243 I APARTM 26 352 FENDER 26 1 6 457 SAT. AHA 26 15 VI LLAG 26 730 1338 SERIAL 25 29 ALLEY 24 149

117 3 PARKED 24 181 STA T 10 24 490

34 r.OING 22 545 HEAL TH 22 515

40 HAN 21 641 PUHCHA 21 1407

23 BELONG 20 325 FORCE \20 599 850 KEYS 20 26 PERFOR 20 1 392 2222 HEA00U 19 28 KN I FE 19 1180 VICT |M 19 ri*nr«°

19 *7 BUS INE 18 1909 1*9

20 1 SANDER 16

?«

«i nt i a

r 5 ??

92 CLOSET 17 15 CUR TA

1 7 *f 924 KNOW 17 649 MALFE a 1

1 5 635 TAMPER 17 1 5 TOOK 1

978

36 BOARO 16 3543 CANADA 16 38 406 H | DN 1

16 39 M | SSIN 16 64 404 SHOES 16 39 SHOTS 16 38 67 THREAT 16 183 VIOLAT 16 1561

44 COHMON 15 2956 CONDUC 1 5 1406 768 HEARD 15 775 HER S 15 20

1274 HUNIC1 15 1 382 PE TTY 15 19 75 SCREW IS 20 SHOE 1 5 43

19 86 8 LIFE 30 930 COHPAN 29 2BB7

66 ANNUIT 19 34 AE TNA I 7 4

52 AMS T ER 16 20 I NSUR176

30 CASUAL 15 257

18 3 L 1 AB IL 46 I 336 I N SUR

44 76 54 1 INS 30 321 CO VER E . 28 385 635 INJURY 25 129B LIMIT 25 2 71 72 3 EXCESS 23 559 PA 1 D 2 3 1 74 7 6 6 I N T ERE 21 2557 NAMED 2 1 646 2098 FULL 19 1 120 GULF 19 48 257 COLL IS 18 681 OE S IGN 1 8 1075

65 2 NAT ION 18 6 70 PROSSE 1 6 57 399 INOEMN 17 213 MANUAL 1 7 34

1 7 APPL IC 16 3117 Can AD I 16 1 7 148 i PUBL IC 16 3129 KENTED 16 65

20 CHANGE 15 1383 CON STR 1 5 2758 2956

389

50 LEADER 29 34 KONIGS 28 28 1154 SOVIET 27 30 SUBOI

26 391

1 7 ORTHOO 19 32 OVERTH IB 23

103 3 NOMINA 15 126 OFF ICE 15 2598

35 TEST IN 30 54 WA TER S 30 103 3529 SAMPLE 22 37 WA STE 22 62 2374

6 3 SW1MMI 35 38 WA TER 35 636

BO DOCK 22 23 F I SH 21 25 32 PERM IT 19 2261 PREHIS 19 1327

149 ADJ ACE 17 203 SUR VE17 152 1466 POOL 15 85 TREE 1 5 5 1 9 1 REVERE 24 43 PARKEO 23 181 69 CORNER 17 234 IOENT

1 7 853

16 LEASEH 33 53 PIN 26 27 457 BALL 21 80 IDL E 20 28

74 HALL 17 154 HACHIN 16 388

78 HEIGHT 15 153 PUNO 15 49

POOL 34 696 GOT 24 279 TOOK 23 978

1274 FOUR 16 1173 WANTED 16 236

33 SHE 15 2250 V1SIBI IS 34

147
POPE 1 5 CR AH FO 1 7 1 1

OE S I GN POPOV it Jit «? it 1 68 CHRY SL 9 8 56 MUTUAL J

FLOOR W 1NST0 *? PLAN

65

1/,

7 17 MOR TO A AU T OHO ii PUR C H

LOANED 10

nc

PL

c V

L ER 75^ HONOR t 77 5* 1 1 160 23 740 2 J 63

P YHEN 2 3 TR ANSA mi ctrNN HA NO FA . RFM I I 1 9 MONE V 20 1 2 74

ST A T EM 20 20 1 4 0 1 S Trtfc i a > »

J A NU A R i a

i /! I? 1466 TITLES 1 ? 48 CAVE iit1 »

[ 6 '

<

e r»°

' J * Asn C 1* e THAT If i % POSSES L 5 I 1*3

, II

j i in POPULA 122 CENSUS 5 5 1

LEG I SL 1 c

I NHAB

74 1 io CHAP 761 CITIES i 9 a t T J 10/ 19 71fl SF y AGE 17 60 16 249B STAT 16 1042

AS T | C L

\

, PUB 7Q rprtiNi 1 PORCH Z 3 ENCL OS

a?

: .... _ . ... _ 366 K 1 TC HE 19 64

L A NOL 0

[ I PflNAGt "l M t r NI

i r 50 1 . ..

1 7 879 . ISA OOVLE ^5 102

l 9 11 ft 79 PUT j PORT 30 CL INTO

f 7 MAR OOR 59 . .. 29 AO VESSEL 29 AO LOAOIN 26 111

6 n

c 77 ... rhuNTV 20

... ERIE 1

9 3 FAC 1 L 1 1296

tecce l B

i nr a Toledo 1 IS PORTE i 9 in? ?nn NOR MAN f 1 Rtll 1 ft .. . POM TEA 1 ? HAHOW

ze JO A 1 1 A

_ -

l fi _._ Own I BU 1

2 3

. , ^ j 30)6

ACC 1 OE 1 IRQ ANSUf r *15 2431

POH T 1PAVEO .1 3 > i Mr T JUNC

_ ... APP uh T_

48

Afl p A 2 1 7\ MO ct 868 63 7 SOUTH 635 2 I 64 2 COHM 1

2 o 30 56 19 *95 LOCATE 1 9 707 OUICKL 19 3 8 ROAD 19 722 S I Ok 1

722 SOLON nnr* ft f FNf r j 7 1 NT f RP I 7 IRON 1 1 126 RE FER6 1 7 L 350

IT URBAN 1 7 2 5 WHEELS I 1 7 i WILL , - 482 3 ACCEPT 16 1 150 NAKK A T 16 28 NOHTH _ KATE 16 '25 TOUNSH 16 378 DAH 3 I FRONT is 442 CRASS is Jl

r . 'KUSTE 10J3 HF S IE R 1 5 ?*> 3 WIPE 1 5 20 9 POR TLA i6 CEMENT 47 65 HA NHOL 1 1 COMHEA 22 527 SUPREM 1

1622

1 B 1 00 LOCAL 1 S 602 I LL I NO 1 1 1363 FMANKL 16 2T2 NOR T HU 16 12T POSTS* 52 phoroc 24 158 MAP 17 61

'2 PORTSH 50 ENUU IS 68 46 CLOSIN 31 I 35 CALLAG 31 19 OHOINA 2341 CIRCLE 3 36 C INC IN 19 341

t t All IL 19 1 3>6 H I TC HE 1 1 86 ftAL *VNC 1* 50P r i t »s e 5?9 15 5<H ifNOWN* 16 '»'. ft

148
Latent Class Analysis as an Association Model for Information Retrieval

Frank B. Baker

Laboratory of Experimental Design

University of Wisconsin, Madison, Wis. 53706

The lack of mathematical models for classification of documents and information retrieval systems

has resulted in a research for models existing in other fields which can be applied to information

retrieval. The similarity of using key words in documents to classify the documents and that of clas- sifying human subjects according to their responses to questionnaires led to the application of latent class analysis to the former. Other statistical association methods relate individual key words to the documents by means of association indices; however, latent class analysis associates on a probabilistic basis a pattern of key words to an underlying storage category. A number of statistical associ- ation techniques are based upon the correlation coefficient or a variant thereof: these methods depend

fundamentally upon 2-tuples of key words to elicit relationships. The mathematical model of latent class analysis is based upon tuples of key words up to n-tuples, and hence more nearly approximates

the relationships involved within the patterns of key words. Latent class analysis and certain other

statistical association models show a common dependence upon matrices and the methodology of matrix

algebra. The memory capacity of present computers restricts one to matrices of size 200 or less which

is insufficient for a usable information retrieval system. The memory capacity of computers, coupled

with the difficulty of maintaining numerical accuracy when matrix size is large, would appear to limit the usefulness of statistical models involving matrices to scientific exploration rather than yielding

generally useful retrieval systems. Matrix techniques for manipulating sparse matrices could ameliorate this situation somewhat. As a mathematical model latent class analysis opens some interesting avenues of exploration into automatic classification of documents and the design of information retrieval systems.

1. Introduction

Despite the considerable increase in computer

power in the past few years, the computerized so- lution to the "library problem" has continued to be

elusive. A small but energetic group of researchers has been exploring this problem area and I must

admit I view this activity somewhat from the sidelines. Because of my relation to the field, the contents of the present paper are more speculative

than the results of extensive research in the field. In reviewing the restricted sample of literature on

this field which was available to me, I was struck

with two impressions, one the existence of a certain amount of similarity within all of the statistical association techniques, and second, the broad range

of interests encompassed by the term information

retrieval. In the case of the latter I shall use in- formation retrieval to include such topics as indexing of documents, classification of documents, as well as retrieval. The underlying uniformity in technique stems from the computer imposed re- quirement of data reduction. The typical computers such as the 7090 or 1604 have a rather limited

storage capacity and even the large computers such

as the Control Data 3600 or 6600 have small memories in relation to the volume involved in the "li- brary problem." Because of the limitations of

computer memory, data reduction is a necessity

and the existing statistical association methods

rely upon the use of key words 1 as an abbreviated means for representing the content of a document.

Despite the criticism leveled at key words, there

does not seem to be any obvious technique which

will perform a similar function in computer-based

retrieval systems. Given the key word vector representation of a document, one needs to make

certain assumptions to proceed mathematically.

The assumption of statistical independence of key

words has been employed explicitly or implicitly in several association models.

As a case in point such an assumption underlies

Maron's [l] 2 automatic indexing system. In Maron's system:

P(Cj | WiWk ) is the probability that if the ah and Ath word occur

in a document, the document belongs to the 7th category.

P(Cj) is the a priori probability that an arbitrary document will be indexed under the jth category. P{Wi\Cj) is the conditional probability that if a document is indexed under the ;'th category it will contain word W\.

Then the following relation holds:

PiCj\W,Wk )- p{W^ p(WAW^

(1)

At this point Maron states

Assuming that relative to a given category any two clue words

are independent (1) reduces to:

P(Cj I Wx Wt ) = kP{ Cj) • Pi W, I Q) P( W, I Cj) (2)

where clearly this independence assumption is false in the sense

that P(Wk \CyWj)^P{Wk \Cj), (3)

1 No distinction shall be made between key words and index terms or tags in the present paper, though the author is aware of their differences.

2 Figures in brackets incidate the literature references at the end of the paper.

nevertheless to facilitate (although degrade) the computations we make the independence assumption.

The paragraph above illustrates why one invokes

the independence assumption, as without doing so one cannot easily proceed mathematically. Clearly

149
the assumption does not agree with the real world,

but insistence upon such congruence would se- verely hamper mathematical model building. Such

an assumption simplifies the mathematics but does

not necessarily make sense linguistically. For

example, the word pair teach computer and computer

teach should lead to different areas of interest, but

under this assumption they are equivalent. Doyle

[2, 3] in reference to this point has indicated that when the number of key words in the dictionary is large, such reversals are rare, and hence are unimportant idiosyncrasies. Although such a sweep of

the hand disturbs some, perfection is not our goal.

Rather, the goal is a reasonably good level of per- formance, and we can five with a certain amount

of idiosyncratic behavior in achieving that goal.

Hence, for the time being, the independence of

key word assumption is an integral part of most

statistical association methods.

Statistical association methods are attempts to exploit the vector of key words which represents

the document; hence it is important that the role of key words be clarified. Once key words have

been obtained by any of a number of existing techniques, their usage within statistical association

methods varies considerably. In some methods

(Maron, [1]; Baker, [4]) the mere existence of the

key word in the document is recorded by means

of one and its absence by a zero; Borko [5] records

the number of times the word appears in the document; and in probabilistic indexing (Maron and

Kuhns, [6]) the degree of relevance of the key word

to the document is recorded, but there seems to be

little appreciation of the quite different meanings

of these numbers. There appears to be a lack of concern by those developing statistical association

methods for the numerical representation of key

words within their method, yet a significant inter- action could exist between what the numbers used

in place of the key words represent and the effec- tiveness of the association model. For example, it is implicit in Borko's model [5] that relative fre- quency of a word within a document is equivalent to relevance as defined by Maron and Kuhns [6J. It would be an interesting experiment to replicate

Borko's [5] analysis using relevance numbers rather

than frequency. The discussion of the proper role of key words and their representations need not be

prolonged, as it is a major topic in its own right.

2. Latent Class Model

The intuitive appeal of key words is so great

that it seems that there must be something we can

do with them, and I am forced to admit that latent class analysis is another attempt to "do something"

with them. Latent class analysis was developed

by Lazarsfeld [7] during World War II to provide a means for categorizing soldiers according to their

attitudes towards selected topics. In the original context the responses made by the soldiers to the

items of a questionnaire were used to group the

soldiers into categories along an ordered continuum,

say from unfavorable to favorable. Since that time

latent class analysis has been the subject of considerable research and the current rationale is that the analysis "partitions the total population of people into m-homogeneous classes such that within

any single class the items are independent" (Tor- gerson, [8], p. 365). The capability to partition a population, coupled with the similarity between

responding yes or no to a question and the presence

or absence of a key word, attracted me to latent class analysis as an information retrieval model (Baker,

[4]). In the latter context the document replaces

the subject and the key word replaces the questionnaire item. Derivation of storage categories

for documents from the information contained in their vectors of key words is a fundamental part of

information retrieval and it is exactly that task

which latent class analysis performs.

Maron [1] had said, "Instead of stating that either a document belongs to a given category or not it would be more realistic to recognize that a document

can belong to a category to a degree (i.e., with a weight). Once we allow a weight to be associated

with an index the road is cleared for a radically

improved interpretation of the entire problem."

A feature of latent class analysis is that it accomplishes exactly what Maron had desired; namely,

latent class analysis associates the documents with

a storage category on a weighted basis. From the

above it is clear that latent class analysis has a number of features which make it a highly plausible

model for information retrieval. The field of infor- mation retrieval has been marked by a paucity of

mathematical models, and the basis of present

operational computer retrieval systems is essentially heuristic in design. Because of the lack of

existing models one looks about for models from

other fields which might provide a steppingstone

into mathematical models unique to information

retrieval. There is no guarantee that a model such

as latent class analysis, factor analysis, or anything

else borrowed from another field will meet the de- mands of its new context; however this should not dissuade one from investigating such plausible

models. With this disclaimer in mind, the derivation of the latent class model is presented in ab- breviated form in the paragraph below.

The latent class model assumes that each document is represented by an A^-valued vector of l's and O's, where a 1 indicates that the key word ap- pears in the document and a 0 indicates that the

word was absent. The probability of a document

possessing key word / is denoted by 11;, of key words

/ and J by IL; and of /, J, and K by Uijk . With N

key words there are 2" ITs, which is equivalent to saying there are 2" different possible response

patterns. The latent class model further assumes

150
the population can be divided into m (mutually

exclusive) subpopulations (classes) where a denotes

a subpopulation. The conflict between the real world and that which can be manipulated mathematically arises again with the assumption of mutually exclusive classes. It is obvious that a document can be classified into a number of dif- ferent categories; however to allow such within a mathematical model is extremely complicated.

This assumption was also invoked by Maron [1] in order to facilitate computation. The choice of

a value for m rests with the investigator, subject

to certain restrictions; in general, the restriction

is that the inequality (n+l)/2>m can be met. Let Va be the probability of a document being ran- domly drawn from the ath subpopulation (class),

a=l, 2, . . ., M. Let be the probability of a document from the ath subpopulation possessing

the 7th word, where Ara = 1 — V denotes the probability of not possessing the word. The probability

that a document drawn from the ath class will pos- sess both words I and J is given by . It should be

noted, however, that the model assumes independ- ence of key words; i.e., A.a — \

a

A.a. u i j The probability of obtaining a given key word

pattern for a document is the sum of the products

of the probability of belonging to a latent class V" and the probability of possessing the word, A.a

; thus, the response patterns represented by the ITs are functions of the Ps and \'s. The relation- ships existing among the ITs, Vs, and \'s are ex- pressed in a system of equations known as the ac- counting equations, several of which are given below

for illustrative purposes.

m

n, = £ va

x.f

m ny =X^g i*j (i) a=l

m Uijk= 2 Va

hjk, i k, j * k. a=l

If one denotes those key words which a document

possesses by the subscript z, where z is the subset

of the integers 1, 2, . . ., TV, the accounting equam

tions can be summarized as IL. = ^ Vak%. Latent

class analysis is fundamentally the problem of

solving the accounting equations for the estimates

of the Vs and Vs using approximations for the

ITs. Because the II' s are unavailable manifest

parameter values, they must be replaced by the

corresponding observed P's. The original mathematical computations given by Lazarsfeld [7] were

extremely laborious and difficult to implement;

hence more tractable methods based upon matrix

algebra were soon developed (Anderson [9, 10];

Gibson [11, 12]; Green [13]; Madansky [14]. At the

present time we are writing a FORTRAN program

for Green's method of solving the accounting equations. The solution of the matrix equations yields a mXn matrix, illustrated in table 1, of X's, which

express the probability of key word (j) having been

possessed by documents belonging to latent class m(i) and a vector of Vs which specify the proportion of the total group of documents which belong

in each of the m classes. The relation of documents

to the mathematically derived storage categories

(latent classes) is determined by computing ordering ratios which are composed of the products of

the probabilities of key words present and absent

in a particular pattern of key words.

VWjJXf

m£ (UVtNXf)

a=l

where Xf — \f when the document possesses key

word j

Xf=l — \f when the document does not pos- sess key wordj.

Table 1. Estimated latent structure

Latent class Probability class Probability of possessing the key word

1 2 3 n

1 V XI X! ' AS

2

yi

3

m ym K

The ordering ratio is associated with a particular

pattern of key words and can be interpreted as the probability that a particular pattern of key words

would be possessed by the documents in a particular latent class. The inverse interpretation

is used to associate documents with a latent class. The key word pattern of the document is used to compute m ordering ratios, and the latent class which has the highest probability of generating

such a pattern is the one to which the document

is assigned. The possibility exists of key word

patterns yielding identical ordering ratios for several classes, but the mutually exclusive assumption

indicates the document should be assigned to one

class. From a practitioner's point of view, I doubt

if after-the-fact violation of the assumption and

multiple assignment of doubtful documents would

degrade the system. An important feature of

151
latent class analysis is that the ordering ratio is a function of the pattern of key words and involves

terms corresponding to both the presence and ab- sence of a key word in a document. Several pre- vious statistical association methods utilize only

the fact that a key word is present (Maron, [1]; Borko, [5]), and Maron's [1J automatic indexing

scheme breaks down when a key word for a category was absent, necessitating the use of 0.001 in place

of zero in the index calculations.

To summarize briefly, latent class analysis provides

a method for mathematically deriving storage cate- gories based upon the information contained in the vector of key words representing documents. The

model utilizes the data provided by 2-tuples, 3-tuples

up to n-tuples of key words rather than being restricted only to 2-tuples as are other models (Borko,

[5]; Stiles, [15]; Salton, [16]). The key word patterns

are associated with underlying storage categories on a probabilistic basis rather than on an absolute

basis. Key words in latent class analysis are designated

by a 1 if they are present in the document, which

is equivalent to giving them a relevance of 1 in Maron and Kuhn's [6] system. Maron and Kuhns

[6] found 70 percent more answer documents were

retrieved when they switched from l's to relevance

numbers for representing key words. Hence, use of

relevance numbers in latent class analysis might also

effect a significant improvement in deriving appropriate classes, etc.

3. Comparison of Latent Class Analysis and Factor Analysis

The statistical association model which bears the

closest resemblance to latent class analysis is the

factor analytic scheme due to Borko [5]. Factor

analysis is another attempt to do something with

key words. What it does is to reduce the rc-dimensional index space of the key word dictionary to a fewer number of dimensions. In Borko's application, the orthogonal axes of the reduced index space

correspond to storage categories. Thus to assign

a document to a storage category one computes

its location in this reduced space and assigns the

document to the closest axis. The assignment is accomplished by computing a vector of factor scores and the largest factor score determines the category

to which the document is assigned. Latent class analysis has a somewhat similar system except

that the calculation of the ordering ratio includes

terms for both the presence and absence of key

words and it yields a probability value rather than

a correlational value. Borko and Bernick [17, 18] reported approximately 50 percent success in assigning documents

to categories in an experiment which was a replication of Maron's [2] earlier work with the exception

of the classification technique employed. Borko

and Bernick [17, 18] used the key word vectors from Maron's 247 computer documents to derive

factor-analytically 21 storage categories. A sec- ond sample also obtained from Maron [1] was then

classified by means of the key word factor loadings

derived from the first sample. There are several

points in the procedure which should be elucidated.

First, such a two-sample procedure is contrary to the rationale underlying factor analysis and latent class analysis. With a scheme such as factor analysis one should not attempt to derive a replacement for the Dewey Decimal System which will then be used to categorize all subsequent documents entering the library. Rather what one does

is to derive a classificatory system which is optimal

for the documents already in the library. That this

is the case is shown by Borko's data, where 63 per- cent of the first sample documents were classified

properly and only 50 percent of the second sample were classified correctly. The two-sample proce- dure leads to some horrendous sampling problems

which could never be adequately resolved, and

samples of size 247 and 85 do not provide a very

good basis for resolving them. Both latent class analysis and factor analysis yield derived storage

categories which are valid only for th^ documents

upon which they were calculated. If one wishes

to add additional documents to the library, their key

words must be assigned from the same dictionary

and addition of any sizable numbers of new documents requires a rederivation of the storage cate- gories and possibly an expansion of the key word

dictionary.

Second, factor analysis depends upon 2-tuples

of key words and hence the 90 X 90 matrix consists

of all possible correlations of 90 words taken two

at a time. Maron's data [1] showed that as the

number of key words used conjunctively to identify a document increases, the probability of correct

classification increases. To take the conjunction

of say n key words and fractionate it into all possible 2-tuples seems to be a backward step. Human

indexers employ the total (or at least a large part)

combination of key words to assign a document.

For example, given computer teaching devices one would not break it up into computer teaching, computer devices, teaching devices, teaching computers,

devices teaching, and devices computer and then use

the six pairs to assign the document. If you restrict human indexers to independent knowledge of

six 2-tuples rather than the whole patterns, I would

suspect that they would do a poor job of classification. The rationale underlying the 2-tuple approach

is that words which appear often in company will form clusters which show a high intracorrelation

and a low correlation with words not in the cluster,

hence the original key word conjunction will reappear. In this respect I feel that latent class analysis

offers a significant advantage over factor analysis

in that the mathematical model of the former in- volves all possible tuples of key words, not just

152
2-tuples as in the latter. Green's [13] method for solving the accounting equations consists essentially of factor analyzing the matrix of 2-tuples and

rotating the structure until it fits the 3-tuple data,

whereas factor anaylsis merely rotates the structure

until it fits the 2-tuple matrix. Hence, latent class analysis should reflect the 3-tuples. whereas factor

analysis cannot do so. Other solutions of the ac- counting equations take into account the higherorder n-tuples but I would not want to try to write

the computer programs to implement them. An

investigation needs to be performed to determine

the relative frequency of the possible tuples of key

words in a corpus, as there is probably a value of

n beyond which n-tuples are too rare to be of any

value. Third, how one compares the effectiveness of

two different statistical association models is a very

sticky problem. Maron [1], Borko [5], and Borko

and Bernick [17. 18] have attempted to evaluate

their procedures by means of comparing the de- rived document assignments against existing clas- sifications of the same documents. I would suspect

such a comparison is foredoomed due to the sample

not being a miniature of the population and due to peculiarities of the existing system. I would rather

evaluate the systems in terms of their ability to yield

documents relevant to a request. hen I send

an assistant to the library to search for books re- lated to a topic I couldn't care less as to how the

librarian has categorized them. My interest is in the relevance to the original request of the books

the assistant brings back and in this regard I would

not anticipate the categories derived by latent class analysis or factor analysis to correspond closely to any existing scheme.

Despite their differences latent class analysis

and factor analysis share two common problems,

communalities and the number of classes to be

derived. The communality problem arises out of

the necessity to express the relationship of the key

word with itself, i.e., what are the diagonal terms

in the correlation matrix. This perplexing problem

has essentially been solved for factor analysis by means of Guttman's [19] image analysis (Harris,

[20]: Kaiser, [21], and in our latent class analysis

computer program we will incorporate the image

analysis approach to resolve the communality problem. How many storage categories to derive re- mains a rule-of-thumb procedure in both latent

class analysis and in factor analysis and no really

good solution is in sight. The lack of a definitive rule for determining the number of storage categories is rather embarrassing in the context of infor4. Problems Involving Matrices

mation retrieval, as the effectiveness of the system

is highly dependent upon the number of categories

employed. Borko [5] does not state what rule was

employed to ascertain that 21 rather than 20 or 30

categories should be employed. In the case of

latent class analysis, McHugh [22] has provided a chi-square goodness of fit test which enables one

to compare how well the corpus has been partitioned for different numbers of classes. One must

however reanalyze the corpus for each different

set of classes to obtain the data necessary for the

test and such an iterative approach is extremely

expensive.

If one derives m underlying storage categories

by means of latent class analysis or factor analysis,

documents can be assigned to these classes on

the basis of their ordering ratio or factor scores. Within these derived classes the documents are stored in descending order of these weighting numbers. Retrieval in such a system is performed by

reading a key word vector as a request, computing

the vector of factor scores or ordering ratios, and

the largest value determines the appropriate class. Once the storage category is found, those documents having a high probability of belonging to the storage category or factor score are retrieved

and now we are in a trap. Such a procedure means

that there are only m possible sets of documents

retrieved. The length of these m lists varies with

the cutoff number set by the request but nonetheless are the same m lists. This is useless of course

but Baker [4], at least, did not appear to have been

aware of this trap: one should not employ the same

scheme to categorize the documents and then re- trieve them. In the case of latent class analysis we are looking at the possibility of retrieving not those documents which have a high probability of

belonging to the category, but those which have a

probability of belonging similar to that of the re- quest. Such a system would at least yield dif- ferent sets of documents for different requests, but

would need to be checked out carefully as it is only

a guess at present.

The trap described above w7 as not realized until

I reread Stiles' [15] description of his method for searching the corpus for key word profiles which

in essence generates storage classes unique to each

request. These storage classes are then investi- gated in more detail for the desired documents.

Definition of sets of documents peculiar to the

words in the request leads to a large amount of magnetic tape spinning which can be avoided by a struc- tured library: hence the latter is to be preferred.

in Statistical Association Models

Statistical association methods such as latent

class analysis are essentially problems in matrix

algebra: factor analysis and latent class analysis

involve taking the eigenvalues, eigenvectors of the

nXn index space and manipulating some matrices

of order mX n. ^ ith present computer capabilities

(7090, 1604), matrices of order 200 are about maximum and yet maintain reasonable running times.

A more serious problem is that of computational

accuracy in the matrix algebra calculations (Freund

153

772-957 0-66— 11
[23]). It is well known that inverses of matrices of

size 50 or greater are highly suspect unless matrix

improvement schemes are employed. The single

precision floating point arithmetic of 7090

FORTRAN yields 27 bit mantissas and I doubt if this is sufficient accuracy for matrices of order 200. The double precision floating point of the Control

Data 3600 has a mantissa of 84 bits which should

improve accuracy considerably but if it is sufficient for matrices of order 1,000 is a moot point. In addition to the storage requirements and accuracy

problems, the sheer mechanics of manipulating

matrices of sufficient size to accommodate the key

word dictionary of a reasonably sized library is a problem and I do not believe that conventional

techniques will prove adequate. If one can demonstrate that the index space is sparse when large

dictionaries of key words are used, where sparse

means a large number of cells are empty, then some

newer techniques are available. The inverse of a

large sparse matrix has been presented by Steward

[24] and the eigenvalues of a large matrix can be

obtained by the graph theoretic technique due to Harary [25]. At the present time we are rapidly

approaching the upper limit of our capability for manipulating matrices and yet are dealing with

unrealistically small dictionaries of key words.

One needs to look at restricted matrix size in its proper context; I do not believe any of the authors

of statistical models involving matrices advocate

attempting to implement such models as operational

systems. Rather, they intend to implement them

in order to study the structure of a corpus of documents and to explore various other avenues of

research.

5. Information Retrieval and Correlational Indices

Inspection of the published statistical association methods reveals that many of them are based

entirely upon the product moment correlation coefficient or variants thereof (Borko [5]; Maron and

Kuhns [6]; Stiles [15]; Salton [16]). The product

moment correlation coefficient is a very peculiar

descriptive statistic and improperly used leads one

into a number of unusual activities. Parker-Rhodes

[26], for instance, states that the product moment

correlation coefficient is a predictive statistic, which

is a new twist for one of the classical descriptive

statistics. The recent paper by Salton [16] which

presents a statistical association technique is a prime example of the type situations into which the

product moment correlation coefficient leads. He

established a number of correlation matrices of terms (it was only after 8 pages of text that he ad- mitted his cosine index of association was in fact the product moment correlation coefficient) and

then proceeded to compare these matrices by computing correlation coefficients using the correlation

coefficients of these matrices as the data. What

meaning can be attached to the correlation of cor- relation coefficients is not easily elicited. The

intent was to compare matrices to determine if they were significantly different. A number of legitimate statistical techniques exist (Anderson,

[9, 10]; Federer [27]) for this purpose, but to cor- relate correlation coefficients and then test the

supercorrelation for significance is not one of them.

In the behavioral sciences we have already been

through the major portion of our correlational period

and the educational, psychological literature is resplendent with similar inappropriate applications

of the correlation coefficient. It seems as if each

developing science is compelled to discover the

correlation coefficient and this is most unfortunate.

The excursion into the blind alley of the correlation

coefficient set educational psychology back 50

years; let's profit from their example and not do

the same for information retrieval.

6. Summary

The lack of mathematical models for information

retrieval has resulted in borrowing from other dis- ciplines models and techniques which appear to have promise in the information retrieval context. The introduction of such borrowed models does

not imply that they will resolve existing problems,

but rather it is hoped that they might provide the

steppingstones to mathematical models unique to information retrieval. In order to proceed in the

development of mathematical models, one must of

practical necessity introduce certain assumptions

which are at variance with the real world such as independence of key words and mutually exclusive

sets of documents. The implications of such as- sumptions cannot be ignored, yet one usually cannot proceed smoothly without such assumptions.

The latent class model embodies features of a number of existing techniques in one compact

package, which makes it an attractive model to study in the information retrieval context. It satisfies Maron's desire for an approach which yields an indication of the relationship of a document on a storage category and does it on a probabilistic

basis. It should be noted the probability actually

involved is that of the documents in a given latent class possessing a specific pattern of key words.

The calculation of these probabilities, i.e., ordering

ratios, employs terms corresponding to both the

presence and absence of key words, whereas pre- vious models have been concerned only with terms

154
representing the presence of a key word. The

mathematical model of latent class analysis involves

all of the possible n-tuples of key words in its ac- counting equations rather than dealing only with

2-tuples such as in factor analysis, however. The

particular solution of the accounting equations

presently being developed into a computer pro- gram (that due to Green, [13]) involves only 2-tuples

and 3-tuples. The solution of the accounting equations involves matrix algebra with its accompanying problems of numerical accuracy, matrix size,

and utility. Although the requirement for such

matrix calculations is a disadvantage, I feel this can be overcome. If experiments with a corpus

of documents indicated latent class analysis per- forms well in the information retrieval context, it

would be a relatively straightforward task for mathematicians to derive approximation techniques

for realistically large key word dictionaries.

The lack of a really good corpus of say 10,000

documents key worded from a dictionary of 1,000

words is severely hampering research. A common

corpus such as this would be of incalculable benefit

to research workers, as would some objective cri- terion for comparing various techniques for manipulating such a corpus. As a final comment I would like to reiterate my

distaste for the product moment correlation coeffi- cient and its variants. This descriptive statistic can lead one far from the goal and should be

studiously avoided.

7. References

[1] Maron, M. E., Automatic indexing: An experimental inquiry,

J. Assoc. Comp. Mach. 8, 404-417 (1961).

[2] Doyle, L., The microstatics of text, Information Storage

Retrieval 1, 189-214 (1963).

[3] Doyle, L., Semantic road maps for literature searchers, J. Assoc. Comp. Mach. 8, 553-578 ( 1961).

[4] Baker, F. B., Information retrieval based upon latent class

analysis, J. Assoc. Comp. Mach. 9, 512-521 (1962).

[5] Borko, H., The construction of an empirically based mathematically derived classification system, Proc. 1962 Spring

Joint Computer Conf., Palo Alto, Calif., 279-285(a)

(National Press, 1962).

[6] Maron, M. E., and J. L. Kuhns, On relevance, probabilistic indexing and information retrieval, J. Assoc. Comp. Mach.

17, 216-244 (1960).

[7] Lazarsfeld, P.F., Latent structure analysis, Ch. 10 and 11 of the American Soldier, Vol. 4, Measurement and Prediction, ed. S. A. Stouffer» Princeton Univ. Press, Princeton,

N.J. (1950).

[8] Torgerson, W. S., Theory and Methods of Scaling (John Wiley & Sons, New York, 1958. 460 pages).

[9] Anderson, T. W., On estimation of parameters in latent structure analysis, Psychometrika 19, 1—10 (1954).

[10] Anderson, T. W., Introduction to Multivariate Statistics, ch. 1 (John Wiley & Sons, New York, 1958).

[11] Gibson, W. A., Extending Latent class solutions to other

variables, Psychometrika 27 , 73-81 (1962).

[12] Gibson, W. A., An extension of Anderson's solution for the

latent structure equations, Psychometrika 20, 60-73

(1955).

[13] Green, B. F., A general solution for the latent class model

of latent structure analysis, Psychometrika 16, 151-166

(1951).

[14] Madansky, A., Determinantal methods in latent class analysis, Psychometrika 25, 183-198 (1960).

[15] Stiles, H. E., The association factor in information retrieval,

J. Assoc. Comp. Mach. 8, 271-279 (1961).

[16] Salton, G., Associative document retrieval techniques using

bibliographic information, J. Assoc. Comp. Mach. 10, 440-457 (1963).

[17] Borko, H., and Myrna Bernick, Automatic document classification, J. Assoc. Comp. Mach. 10, 151-162 (1963),

[18] Borko, H., and Myrna Bernick, Automatic document classification: Part II. Additional experiments, System De- velopment Corp. TM-771/001/000, 33 pages (Oct. 1963).

[19] Guttman, L., Image theory for the structure of quantitative

variates, Psychometrika 18, 277-296 (1953).

[20] Harris, C. W., Some Rao-Guttman relationships, Psy- chometrika 27, 247-263 (1962).

[21] Kaiser, H. F., Image analysis, Proc. Social Sci. Res. Council

Conf. on Measuring Change, ed. C. W. Harris (Univ. of Wisconsin Press, 1963).

[22] McHugh, R. C, Efficient estimation and local identification

in latent class analysis, Psychometrika 21, 331-347

(1956).

[23] Freund, R. J., A warning of roundoff errors in regression, Am. Statistician 17, 13-15 (Dec. 1963).

[24] Steward, D. V., On an approach to techniques for the

analysis of the structure of large systems of equations,

SIAM Rev. 4, 321-342 (1962).

[25] Harary, F., A graph theoretic method for the complete

reduction of a matrix with a view toward finding its eigenvalues, J. Math. Phys. 38, 104-111 (1959/60).

[26] Parker-Rhodes, A. F., Contributions to the theory of clumps:

the usefulness and feasibility of the theory, ML-138,

Cambridge Language Research Unit (Mar. 1961).

[27] Federer, W. T., Testing proportionality of covariance

matrices, Ann. Math. Statist. 22, 102-196 (1951).

155

Problems of Scale in Automatic Classification

Roger M. Needham

University of Cambridge

Cambridge, England

One of the problems of automatic classification for information retrieval is the number of terms which need to be handled. It is not difficult to construct and use association matrices between,

say, two or three thousand terms. However, even "controlled" vocabularies are often larger than

this, and part of the object of automatic classification is to lessen the need for careful vocabulary control. The paper will discuss some approaches to the problem of scale, specifically involving:

1. Techniques for constructing partial matrices, or sample matrices.

2. Some techniques at present under experiment which implicitly make use of associations, but

avoid constructing a matrix at all. It is hoped that some preliminary results will be available. The paper will conclude with some arguments in favor of using a classification technique rather than using a matrix of associations directly for reference purposes, even if the latter were technologically convenient.

157

A Nonlinear Variety of Iterative Association Coefficients

Robert F. Barnes, Jr.

University of California Berkeley, Calif. There are in existence a number of different systems of association coefficients, which may be

characterized and compared in several different ways. A framework that seems especially fruitful treats each set of coefficients as elements of a linear vector space of dimension /V 2 (where N is the

size of the object population at hand). Then any given set of coefficients can be viewed as the image

under some vector-space transformation of a certain canonical set of coefficients. From this point

of view, many of the properties of the resulting coefficients can be related to corresponding properties

of the generating transformation. For one type of association coefficient, which we term an iterative association coefficient, the generating transformation is best viewed as the limit of the set of iterations of a second transformation. Such iterative coefficients can take into account higher-order relationships of co-occurrence, which are generally neglected by simple coefficients but which may be of considerable significance. Where

the iterated transformation is non-linear, the theory of such coefficients becomes quite complicated;

however, analytic and empirical studies of one such variety of coefficient have revealed certain properties of some interest and have indicated certain kinds of retrieval situations in which these coeffi- cients might prove useful.

159

The Measurement of Information from a File

Robert M. Hayes

University of California at Los Angeles Los Angeles. Calif. 90014

Many of the problems of measuring the responsiveness of a file can be approached by appropriate extension of communication theory: (1) by introducing the parameter of relevancy into the entropy

function: (2) by allowing the output of multiple signals as a method of handling error: and (3) by combining these with the methods of sequential decoding for analyzing file indexing procedures.

At the risk of being boring and perhaps obvious,

I am going to present a technical approach to the

statistical view of information storage and re- trieval, one which is somewhat different from that with which we are concerned this week and yet one which very clearly relates to it. I hope that another dose of mathematics will not be too in- digestible, but I offer you this opportunity to steal quietly away.

To introduce this approach. I would like to raise

three questions, two of which I won't pursue much

further and the third of which is the concern of

my talk this evening.

The first question involves the relation between

the value of an information system and the re- sponse time from it. I propose that this relation- ship is characterized by a logistic decay function

based on a single parameter — its half-life — and I suggest that virtually all of the characteristics of an information system are a function of that single

parameter. I therefore raise the question, "Can we define the appropriate relation between time

and value and determine that parameter?""

The second question involves the relation between

the value of an information system and the cost of

it. I suggest that the obvious criterion is the

economist's dictum — "cost equals value" — but

that is apparently not valid. -Ail too many systems

have been designed with virtually no concern for

their cost. I therefore raise the question, "Can we define the appropriate relation between cost and value?'' The third question involves the relation between

the value of an information system and the information derived from it. I propose that this relationship

is characterized by a logistic growth curve as a function of the amount of information provided.

This obviously raises the question, "Is this the

relationship?", but more fundamentally, it raises

the question, "How do we measure the information

from a file system?"

I raise these three questions for two reasons:

First, I believe that the efficiency of an information

system is expressible as a function of the three

parameters, T, C. and A with which these questions

are concerned, and second, I wish to suggest some

approaches to the study of the third — the measure- ment of information from a file. The obvious approach — so obvious in fact that one might wonder why the question is raised at

all — is to apply information theory. So let's try

it. Picture a file system as though it were a communication channel with an associated decoder.

As input we have requests and as output we have

the file records for relevant documents — perhaps

including selected content from the document

itself. Can we characterize the information

characteristics of such a channel?

Consider a file of F bits consisting of items, x, each of N bits. Suppose a request y is matched

against each item in the file over a specified n bits of the A . and the item which matches most closely

is output. I am concerned with measuring the

information from the file, in response to r, as a function of F, N, and n. I want to consider it in four parts:

1. Assuming that the search process is noiseless.

2. Assuming that the significance is dependent

upon the relevancy of the information.

3. Assuming that the search process is noisy

due to error in the request, the items, or the match

process.

4. Assuming that the search process is noisy due

to the imposed indexing structure. Consider the 2

V possible ac's. Assume that they

are equally likely and consider any one of them,

say x. If we measure the relevancy, or degree

of match, between x and y by the number of bits of the n over which x and y agree, we can formulate

the total number of files from which x might be

the response and, therefore, the probability of x

given y. The measure of information provided by

such a communication channel with this probability

distribution is traditionally given by the entropy

function

H{xly) = — ^p (x/y) logp (*/}')•

We can bound this and derive the not unexpected

result that the information is approximately

H(xly)^N -log —

Thus, given the file as a communication channel

to which requests are input, the output consists

F

of sets of N bits, of which log — are in some sense N

already "known"" and the remainder are essentially new information.

However, in some very important senses, this seems counterintuitive. For instance, one feels

161
that the "information" from a file should increase

as the size of the file increases, but the standard measure of information states the opposite.

Secondly, and perhaps more importantly, this measure completely ignores the extent to which

the output is actually responsive to the request.

In this respect, a file is not simply a communication

channel, and disparity between input and output is not solely a result of noise. Thus, as we increase

F

jy, the number of file items, we increase the likeli- hood of finding a good match, but we decrease the

traditional measure of information in communication

theory.

Communication theory normally confines itself to models that are statistically defined so that the

only significant feature of the communication is its predictability. I wish to extend this to include,

as an equally significant feature, the relevancy of the information received — determined, for example,

by its degree of similarity to a request input to the

file. I therefore define the concept of "significance"

as a function of both the probability of x,p (x) and

the relevancy of x, r(x). Under the most straightforward assumptions of

additivity with respect to both parameters, we can

define the significance of a selection x as the

product — r{x) log p (x)

and the average significance as S(X) = -^p(x) r(x) logp(x)-

x

In the special case of a noiseless communication

channel, r(x)=l and we have the usual entropy

function.

Returning now to the importance of finding a good match, if the relevancy of x is measured, for example, by the number of bits of agreement be- tween x and y, the average significance from a file is a convex function of the size of the file. Intuitively,

it has the properties which I think such a measure

should have, and I suggest that it be considered not only in the context of a file, but in other situations

where value to the receiver is significant.

The nature of the characteristics of a file as a communication channel is particularly felt in the

effects of error. Again, in normal communication

theory, where one expects to get out of the channel what one puts into it, the effects of a probability

of error in a single bit can be counteracted simply

by increasing the number of bits of match. In

fact, the probability of erroneously decoding the

output is an exponentially decreasing function of

the length of the identifier, n. Unfortunately, this

is just not true of a file operation, since we are deal- ing at potentially correct points in the coding

lattice near which the number of possible alter- natives is enormously greater. In fact, there is a

size of identifier beyond which the probability of error must increase.

How then can we combat the effects of error, if increasing the length of the identifier is at best a stopgap? The answer is obvious, once it is recog- nized—we must output not just one response but

a set of potential responses to reduce the probability

of erroneously missing the correct one. Then,

the probability of error becomes an exponentially

decreasing function of the number of items output.

However, error in file operation as we have de- fined it will not be due solely to the type of noise

resulting from an error in single bits of the request,

or the file items, or the comparison process. A

highly significant source of error arises from the

failure even to consider the file item which matches

the request over the maximum number of bits; such

an error can arise whenever an indexing structure

is imposed upon the file. In fact, the type of process

I have just described — the output of several items

in response to a request — represents the character

of such an indexing structure. For example, an

index might be constructed by establishing a "sequence of significance" on the identifying bits and using successive groups of bits as index

criteria; a match on some fewer number of identifying bits then requires selecting not only the closest index term but a set of them.

This problem can now be analyzed by an approach

similar to that of Wozencraft in his Sequential

Decoding procedure, but including the additional

complexities which I have discussed.

In summary, I suggest that many of the problems

in measuring the responsiveness of a file can be

approached by appropriate extension of communication theory and in particular first by introducing the parameter of relevancy into the entropy

function; second, by allowing the output of multiple

signals as a method of handling error; and third,

by combining these with the methods of sequential

decoding for analyzing file indexing procedures.

162
Vector Images in Document Retrieval

Paul Switzer

Arthur D. Little, Inc. Cambridge, Mass. 02138

The paper describes a model for generating a term-term association matrix. The model, based on

co-occurrence frequencies, is a consequence of probability theoretic considerations. Using this association matrix, a method is then suggested for selecting a small subset of the index terms as axes for a low-dimensional index term vector space. The method is intended to approximate a canonical factor analysis, but is much quicker to apply and easier to interpret. The position of an arbitrary term may be located in this reduced "image" space by reading off appropriate entries in the already- computed association matrix. The approximate method may, of course, be used in conjunction with

association matrices derived in ways other than that described in this paper. A procedure is then outlined for locating documents in this same image-space. Basically, this involves obtaining a description of the document consisting of a list of index terms with appropriate

weights or frequencies. This may be done by referring to the title, table of contents, selected portions

of the text, or what have you. Authors' names and cited authors and titles may also be incorporated

in deriving the position of a document in the image space. Simple linear calculations characterize

all the operations.

Then the procedure for locating an enquiry in the image space is presented. The form of the enquiry is extremely flexible, permitting the use of any number of index terms or authors' names,

with differential weighting. A quick method for retrieving "relevant" documents is proposed. The

method is basically a search for document images contained in a hypercube with the enquiry image

at its center. The proposed method of filing means that "relevant" documents may be identified immediately without any spurious scanning.

1. Introduction

1.1. Statement of the Problem

The elements of the problem are a collection of

documents, e.g., a library, and an enquiry. The

solution to the problem is a system which selects

(retrieves) that subset of the document collection

which contains the answer to the enquiry. Some

of the difficulties which present themselves immediately are as follows:

(1) Any verbalized enquiry is not usually more

than a good approximation to what one really wants

to know. Furthermore, the same verbalized en- quiry may have any number of connotations.

Hence, we will make this simplifying assumption —

an enquiry has a unique connotation, i.e., each

enquiry has only one correct answer;

(2) The obvious and trivial solution to the re- trieval problem is to scan the document collection

completely, selecting the subset which contains

the one correct answer. Presumably, this could

only be done by a human being who "knew" the content of the entire collection; in general, such a system is unavailable. We shall, therefore, assume

that the solution, the retrieval of the correct documents, can be achieved by a mechanical, objective,

and operational system;

(3) Inevitably, any system which is mechanical

can communicate only in a prescribed and pro- scribed form. Thus, we further assume that every

enquiry can be translated to a form which can be

communicated to the system. However, the system

to be proposed in these pages will be sufficiently

flexible so that this assumption will not prove to be very restrictive;

(4) Even though we have assumed that an enquiry is unambiguous and thus can have only one

correct answer, it is usually the case that the answer

is complex, with varying degrees of generality.

Therefore, the subset of documents which contains

the complete answer may be very large, wherein some documents may contribute very little to the answer. Thus, we assume that all the documents

can be differentiated with respect to their relevance

to a particular enquiry and, further, that this relevance can be measured.

These are four major difficulties and the four

corresponding basic simplifying assumptions we

are employing. Each assumption may introduce

into the system noise which may be difficult or impossible to assess. Though these assumptions

are almost always incorporated in a retrieval system, they are rarely articulated. The worth of

any mechanical retrieval system hinges crucially on the degree of validity of the foregoing assumptions.

1.2. Scope

This paper will concern itself primarily with a model for the mechanical selection of documents

most relevant to an enquiry. It is based chiefly on

the construction of a low-dimensional document

space and the development of a meaningful method

of locating a document in this space. The vector

representation of a document in this space will be

called the document image.

Retrieval is achieved by

(1) translating the enquiry into an enquiry image,

(2) entering this enquiry image in the space of

all document images, and

163
(3) selecting those document images which are nearest to the enquiry image according to the

defined criterion.

1.3. Elements of the Document Image

Since, by assumption, the idea of human scanning

of documents has been abandoned, it becomes

necessary to devise some means of mechanically

describing the information contained in a document.

This might be achieved by a more or less complex

statistical and/or syntactical analysis of the entire

document. Alternatively (because it is much

easier to do and may not result in too much loss) we will use only

(1) the document title, (2) the author's name or authors' names and

(3) the titles and authors' names of any documents cited by the given document or which cite the given document.

In a certain sense this model will therefore be

a combination of Salton's model for use of citations

[l] 1 and Baxendale's model for title analysis [2]. However, we will not be attempting any of Miss

Baxendale's semisyntactic analysis of titles. In addition, authors' names are included in the description of the document. Implicitly assumed then,

is that (1), (2), and (3) together in some way represent

the information content of a document. This basic

assumption is not totally unreasonable and effects the economy of not having to look at the contents of

2. Term

2.1. Preliminary Definition

The argument now hinges on the ability to find m characteristics by which index terms could be

described as m-dimensional vectors. Ideally,

if m = t= number of distinct index terms, then

a given term tj could have the unique representation

t, = 0, 0, . . . , 0, 1, 0, . . . , 0), where t, is a vector of t elements whose tth element is a 1. Then we find ourselves working with a f-dimensional space where t is impracticably (and often

spuriously) large. In practice, we will want m,

the dimension of the image space, to be much

smaller than t, the number of distinct index terms. As soon as m< t, the problem of an m-dimensional

vector representation for an index term becomes

nontrivial. This problem can be approached in the following manner. Suppose there was some way of finding

those m index terms (out of the t terms available)

which in some way were the m most "characteristic." Denote these m terms by t a , t$, . . . t^. Then for a suitably defined distance measure, A, on the space of all index terms, we could define

' Figures in brackets indicate the literatur references on at the end of the paper.

the documents. For specialized collections, e.g., journal articles in a single field, the assumption

may be especially well justified. For those who

feel somewhat uneasy about ignoring the body of the document, there is a straightforward extension

of the model which provides for a scanning of

the body material in whole or in part; this extension

appears in Appendix B to this paper. As a convenient and flexible way of summarizing

and combining the information contained in titles and authors' names, we will be constructing an "image space" of m dimensions. Every index

term, document title, and author, whether actual,

cited, or citing, will be transformable to a vector

of m elements called an "image." All the images

relating to a particular document will then be

brought together to form a composite vector — the

document image. How these images will be used

for retrieval will be outlined later. In general, the transformation of index terms,

etc., to vectors will be achieved by scoring them on

.each of m characteristics, the characteristics being

chosen in a way to provide maximum discrimination among different documents in the collection. These scores will be the elements of the image

vector. As a preview of what follows, it will turn

out that once the images of index terms are defined,

then the images of titles, authors, and documents

can be derived in a simple manner from these

index term images. Thus, a good part of this paper

is devoted to a meaningful construction of the vector images of the basic index terms. Images

the m-vector representation, tj, of an arbitrary

index term, tj, as

t / =(AJ «,AJ 0, . . . , AjM ),

where Aja , etc., represent the distances of the term

tj from each of the specially chosen "characteristic"

terms. In this way we compress the total index

space to an m-dimensional index image space, while this method for compression does seem

reasonable, the argument for the method will be

strengthened by the detailed development which

follows. We have in this way shifted the problem

to

(1) finding a suitable distance measure, A, on

the total index space, and

(2) finding some way of selecting the m most

characteristic index terms by using these suitably

defined distances.

2.2. A, the Term-Term Distance Measure

A number of term-term distance measures have

been proposed. Most of these are based on the

number of co-occurrences, Na b, of a pair of index

terms, t„ and tih i.e., the number of documents in which the two terms co-occur. All these proposed measures tacitly assume that frequency of co-occur164
rence in some way reflects the degree to which t a and t

b are related. We, too, shall incorporate this assumption, though in a somewhat different form.

The proposed distance measure between two

index terms is as simple as it is meaningful. Suppose we observe Na b co-occurrences of the terms

t a and t b - We might ask the following question:

Given the occurrence frequencies Na and N0 , what

is the probability of observing as many as Na b co-occurrences, assuming there is no association

between t a and t b ? That is, what is the significance

probability of the event 'WQ & co-occurrences?"

It is this significance probability which will be

taken to measure the distance between t a and tb. In general, the larger the value of Na b the smaller

will be its probability of occurring purely by chance,

i.e., its significance probability; and the smaller its significance probability the more likely it is that

t a and t b are not unassociated. Therefore, the significance probability of Na b does provide us with

a meaningful measure of the closeness of the

terms t a and tb- To get this probability we need to know the the- oretical distribution of Nab, conditional on Na , Nb, and d (the total number of documents in the

collection). It may be checked that this distri- bution is in fact the hypergeometric distribution

with parameters Na , Nb, and d. So the distance

between ta and tb, say AQ6, is just the significance

probability and is given by

min^ ^ / d -Na \ /(d\

A-—"ab

Fortunately, this rather horrendous-looking animal

is tabulated [3]. Thus, we may get the t X t A- matrix by substituting the quantities Aab for the

quantities Nab in the co-occurrence matrix. Since

the distances are probabilities, we have that Aa* is in the interval (0, 1).

2.3. The m Separators — Axes for the Space

of Images

The primary purpose of calculating the term- term distances was to construct the m-dimensional

index-term images. It was suggested that this might be done by selecting m index terms out of

the t available index terms in such a way as to be

most "characteristic." What was implied was a choice of those m terms which give rise to the most

variation in the matrix of distances. These specially chosen terms will from now on be called "separators," and they will be denoted by t a , to, . . . , tM

. The image of an arbitrary index term, t a , will then be the vector whose elements are the distances

of t a from t a , tp, . . . , t^, respectively, denoted by

ta =.(A«, Aaj8 ,

. . . , kav.).

If the m separators are well chosen, terms which

are closely related will have similar images while

terms which are essentially unrelated will be

"pulled apart" and will have widely different images.

The usual approach to a problem of this kind would

be to perform a factor analysis of the matrix of

term-term distances. We could then pick those m

factors which have the largest variances and use

these as separators. However, the factors would no

longer be single terms but would, in general, be

linear combinations of all t terms of the vocabulary.

The inherent difficulty of calculation and interpretation have led me not to consider factor analysis for

this problem. Instead, consider the following:

Denote

b^a Then AQ is the average distance of the terms of

the vocabulary from the term ta - If the individual

distances, &ab, differ considerably from their average

value, AQ , then it is reasonable to say that ta is a good discriminator (or that t a carries a lot of variation); that is, if

tq = 2) | A«ft — Aa | a

is large, then t a is a good discriminator. Thus

compute the quantity r« for each index term t a in the vocabulary. The m separators, t a , tp, . . . , tM will be those m index terms whose r-value is greatest.

The nature and amount of calculation involved

for this process of selection are outlined in appendix

A to this paper; it is certainly superior to factor

analysis in this respect. However, this method of

selecting separator variables is not, to my knowledge, discussed in the statistical literature. Therefore, I am not able to discuss its statistical

properties, but they should be investigated more

fully. Nevertheless, the process does have the

strong intuitive argument of the preceding para- graphs.

Nothing has been said so far about how one goes

about choosing m, the number of separators (the

dimension of the space of images). Unfortunately,

there does not seem to be any "internal" objective

way of doing this. The best that can be said now

is to choose m to be conveniently small. Clearly,

the smaller m becomes, the simpler and less sensitive the retrieval system becomes; experience in

this regard would certainly help. We can, however,

formulate the following rule for getting m: The

set of separators consists of those m index terms

which have a r-value greater than a threshold value,

To. Thus, the problem of choosing m is in this way

shifted to the problem of choosing t0

, which could

perhaps be more objectively chosen from a consideration of the distribution of the t's.

165
2.4. Recapitulation

Having found our set of separator index terms, the

question now is what shall we do with them? A

purpose of this study was to create an image for each document which was to be constructed in such a way that similar documents would have

similar images. (What use would be made of these

images will be taken up in greater detail further

on.) The image was to consist of a document's

score on each of m characteristics, i.e., the image

is a point (vector) in an m-dimensional space. These m characteristics were then taken to be a special subset of the vocabulary of index terms. These m index terms were called separators. Any

term, ta

, in the vocabulary could then be represented

by an m-vector, t„, whose components were the

distance of t„ from each of the separator index

terms, according to the metric, A. The separators were chosen in a way that gave them maximum

discriminating power according to a defined cri- terion. The metric, A, was also carefully chosen

so that it would have a natural probabilistic inter- pretation.

Now we are at the stage where we can construct

the images of each of the index terms in the vo- cabulary. This involves no further calculation —

merely the picking out of the appropriate entries

from the term-term distance matrix. It was re- marked earlier that title images, author images,

and document images would be a direct conse- quence of the index-term images (which we have

just calculated). The next part of this paper shows

how this is accomplished. The fourth part of this paper will treat of applications.

3. Scoring the Document

3.1. The Title Image

The scoring of any document on the m separators

may be conveniently divided into two parts:

(1) finding the title images, and

(2) finding the author images.

It turns out to be rather straightforward to create

the title image. First select all the index terms in the title — this means all words except those which,

by themselves, do not convey any substantive meaning, e.g., most quantifiers, prepositions, conjunctions, etc. This operation is performed quite easily

by human beings but could be mechanically per- formed by storing a vocabulary of the nonsubstantive words. (Here, this operation is assumed to have already been performed when the original

term-occurrence counts were made.) Suppose

the title, T, contains the y index terms t\, tz, . . ., ty whose corresponding m-dimensional image vectors

are ti, t-2, . . ., ty

. Then define the title image of

T as the m-vector, T, which is the weighted average

of the images of all the index terms which appear

in the title T, i.e.,

where X\j = l and y = number of terms in the title. The weight Xj is chosen to correspond to the

importance of term tj relative to the other index

terms in the title. There appear to be two ways

of choosing kj in an objective and mechanical manner:

(1) kj= 1/y for all j, that is, each term of the title

is given equal weight in the construction of the title image T. In this case we have simply

(2) \j = l/Njl%y=1 l/Nj, where Nj = total frequency

of term tj in the titles of the collection. Thus, the more rarely does a term occur, the greater is its weight in the construction of T. In this case, T={XtjINj }IXl/Nj

. The second method for assigning kj seems to have

stronger appeal since rarely occurring terms are given greater weight than commonly occurring

terms in the construction of the title image, while

the first method is a "no-information" type of

weighting. In collections which are so small that the quantities TV, are not especially reliable esti- mates of the relative frequencies of occurrence of

the different index terms, it may be just as well to use the simpler first method of weighting.

3.2. The Author Image

The construction of the author image is carried out in the same straightforward manner as the

construction of the title image. The author image

is built up by considering all the index terms he used

in the titles of all his documents which are in the

collection. In fact, it is natural to regard the author

image as some composite of all the title images of

his titles. The obvious and simplest composite is just the average, i.e., if an author, W, has p titles in the collection, Ti, T2, . . ., Tp , whose corre- sponding m-dimensional titles images are Ti,

T2, . . ., Tp , then the author image, W, is defined

as the m-vector

W=fjTj/p.

Thus, we have for each document a title image, T,

and an author image, W. (It is worth noting that

166
the elements of T and W are still contained in the

interval (0, 1).) The problem now is — how can T

and W be combined to produce a single image for the document? Again, we resort to an average,

but we must first decide on the relative importance

of W, the author image, with respect to T, the title image. Therefore, consider: If the author of the

given document, say D, has p documents in the

collection, then the given document represents

1/pth of the author image, on the hypothesis that

all his documents contribute equally to his author

image. Hence, a natural weighted average, D'"', of W and T isDto =l/pW+(l-l/p)T,

which will be called the author-and-title image.

Loosely speaking, the more documents an author

has in the collection, the more varied will their content be, the less important is the author's name

for the purposes of describing a particular document; this fact is incorporated in the expression

for Dtw. (Note that if p= 1, i.e., if the given document is the only one that the author has in the col- lection, then T = W= Dfu', as one would hope.)

The use of authors for retrieval is definitely no more than a conjecture and this, in itself, might

justify the light weighting. But note that

D'"=l/p(l/pXT/) + (l-l/p)T

\ 1 / = l/p2 X T, + (l-l/p+l/p2 )T.

Thus the title gets a weight of 1 — 1/p + l/p2 and

all the other p — 1 titles by the same author get a combined weight of 1/p— 1/p

2 (1/p2 each). Thus

if p = 3, the title gets weight 7/9 while all other

titles by the same author get combined weight of

2/9. If, in practice, it turns out that author "de- serves" more weight, then it might be worth an- other look.

3.3. Citations

The vector, Dtw

, is not quite the final document

image, for it has not taken into account the docu- ment's citations. To complete the picture, the

first step is to fist all the titles and authors of the

documents which

(1) are cited by the given document, and

(2) cite the given document.

The "cited" list is easy to compile and usually consists only of scanning the bibliography of the document. The "citing" list is impossible to compile

unless the collection is "closed." However, since

collections are rarely if ever actually closed, it is preferable not to incorporate this assumption.

Thus, the citing list is restricted to those documents

which cite the given document and which are in the

collection. Except for a brief note in the appendix, we will not be distinguishing between cited and

citing, so the two lists may be combined for each

document.

How do we use this list of citations? Suppose

that for the document D we have the set of q cited

documents and r citing documents, denoted

Di, Do, ., Dq+r- For each of these q + r documents, compute the corresponding m-dimensional

title-and-author images, Dtw (as defined above).

The average of these q + r title-and-author images

will be called the citation image, Dc

, for the document D, i.e., D<-=£ B'fl(r+q).

The next step is to combine this citation image Dc with the given document's own title-and-author

image D'"'. Now, it is often unfortunately true

that citations are not very closely related to the contents of the document. In fact, it seems that the more citations we have, the less closely are

they, on the average, related to the document in question. This last observation is now incorporated

as an assumption: the weight of the citation image

will now be taken to be inversely proportional to the number of citations. Thus, we finally have that the document image Dfor a document D is the vector found by taking the weighted average o/Dtw and Dc

, where Dc

, the citation image, is weighted inversely

to the number of citations, i.e., D= ,

1

, Dc +(l- -, ,

1

, ) Dto

- l + q + r \ l + q + r/

(Note, if there are not citations, i.e., q + r = 0, then

D = D'"'.) At long last we have arrived at an expression for the document image. Figure 1 provides

a summary of the process used to derive this ex- pression. And now we are in a position to construct an image for each document in the collection. En passant, we also defined these other images:

t, the basic index-term image

T, the title image

W, the author image

D'u', the title-and-author image

Dc

, the citation image.

In certain applications, these intermediate images

will be useful and interesting in themselves.

It might be noted that the document image D is a linear function of index-term image vectors t, where the elements of t are the A-distances of the

index term T from each of the separator index terms

ta , tp, . . ., t^. In fact, D may be written entirely

in terms of iVi, N2 ,

. . ., Nt and N12 , N1S , . . ., Nt-i,t, the frequencies of occurrences and co- occurrences of all the t index terms in the vocabulary—this is, of course, in accord with the basic

assumption made at the beginning of this paper.

167
Authors W

Titleand- Author

Images

Documents

D

Citations]

Titles

T

Author Title Images X— Images

W r

Term

Images

t

Term Occurrences

and Co-occurrences

Term- term

Distances

Separators

a' e •,t„

Document

Images D

Title Images

T

Titles Authors

T W

Author

Images W

Titleand- Author

Images

ntw

Citation

Images

DC

FIGURE 1. From documents to document images.

4. Application

4.1. The Enquiry Image

We now examine how the document images and

other images thus generated can be useful for retrieval. Any retrieval operation starts with an

enquiry. In most systems the enquiry must be in a closely specified form. One of the great advantages of this proposed system is its extreme flexibility with regard to the form of the enquiry, as

will now be shown.

The enquirer is given a preliminary form which

is divided into two sections.

Author-names section: In this section the en- quirer may write the names of any authors who

he believes have some relevance to his problem.

He may assign differential weights, stressing certain authors, if he wishes. He is not limited in the

number of names he may write down, and he may

may, if he wishes, leave this section blank. The

only restriction is that he should use only names of

authors who are represented in the collection. A

list of these authors would be available to the

enquirer. Text section: In this section the enquirer may

scribble down any "textual material" which he

feels may help in retrieving relevant documents.

By "textual material" we mean any titles, sentences,

phrases, single words, or what have you. The

restriction is that he should not use words which

are not one of the system's original terms or which

are not in the system's glossary of nonsubstantive

words (typically prepositions, quantifiers, conjunctions, etc.). Actually, this restriction and the

similar one for the author-names section may be

relaxed if it is assumed that ineligible names and

terms can be edited out of the enquiry. The en- quirer may assign differential weights to any of the

substantive words (index terms) he has written

down, either as individuals or in groups. He may

leave this section blank if he has not left the other

section blank.

This preliminary enquiry form in two sections

then goes to the interpreter (possibly mechanical),

who has before him the following:

(1) an alphabetic list of authors represented in the collection. Next to each name is a string of

m numbers (all between 0 and 1) representing the

author image;

(2) an alphabetic fist of all the index terms rep- resented in the titles of all the documents in the

collection. Next to each term in the list is a string

of m numbers (all between 0 and 1) representing

the index-term image;

(3) a form ENQ, which is reproduced in figure 2. The interpreter then looks up each of the authors

cited on the preliminary enquiry. He notes whether

the enquirer has assigned weights to the authors' names. If so, he multiplies the m numbers by the

stated weight and records them on the form ENQ,

repeating this for each specified by the enquirer.

168
Index Term

or Author Wt. 1 2 3 m— 1 m

Smitn oI l.OO 1.44 i.yo i fin 1.04 Jones 11 . 1 1 .ol i.UU . / 1 cat 5 3.60 3.95 4.75 2.00 3.15 house 2 1.44 1.50 2.00 1.20 1.66

TOTALS 10 7.11 7.70 9.73 4.70 7.16

Enquiry

Image .71 .77 .97 .47 .72

Figure 2. Form ENQ (with hypothetical numbers).

If no weights are indicated, then all weights are taken to be 1. The interpreter then goes to the

text section of the preliminary enquiry and crosses

out any words which are not on his list of index

terms. For the words which remain he enters the

m corresponding numbers, duly multiplied by any

weighting factor, on the form ENQ. Having done

this, he then totals each of the m columns on the

from and also totals the weights. Each column

total is then divided by the weight total. The re- sulting 77i numbers represent the weighted average

of the various image vectors, the weights having

been chosen subjectively by the enquirer. The

enquiry image is the vector represented by the

numbers on the last line of the form ENQ.

4.2. Measuring Resemblance

To effect retrieval, it is now necessary to compare

the enquiry image with the image of each document

in the collection. According to the hypotheses

and assumptions made at the very outset and else- where throughout this paper, those document

images which most resemble the enquiry are most

likely to represent the documents which contain

the information relevant to the enquiry.

There are a number of ways of defining the resemblance between two images. One way is to compute the correlation between them, this being the

usual method. The higher the correlation, the

greater we assume the resemblance to be. Thus, one could compute the correlation between the

enquiry image and each of the d document images.

These correlations can then be ranked. Then the

z documents giving the highest correlations with

the enquiry image would be picked as the solution

to the retrieval problem; alternatively, all documents having a correlation greater than p0 with the

enquiry would be picked. The value of z or p0 would be selected to yield the right blend of precision and accuracy as defined by Giuliano et al [4]. However, d, the number of documents in the

collection is usually large, and calculating d cor- relations for every enquiry could be undesirable.

Therefore, consider the following alternative method

for picking out resemblances. Suppose the enquiry

image vector is denoted by (u, u2 ,

• • ., vm) and a document vector by (ui, u2 , . . ., um). Then, for a preselected e<>, retrieve only those documents such

that

\vj — Uj| < eo for/=l,2, . . .,771.

This corresponds to retrieving all those documents

whose image points he within an m-dimensional

hypercube centered at the enquiry image point and

having side length equal to 2e 0 . Alternatively, the

enquirer may prefer that the system retrieve

exactly z documents. Then, the method goes as

follows: For each document compute the quantity C/max = maxltfj — m\. Then retrieve those z documents which have the lowest Umax scores. The

images of this set of documents are all the points

within a minimal hypercube centered at the enquiry

image point. For at least two reasons this simpleminded method is preferable to the use of correla- tions—first, because it is easier to interpret; second,

because it is far easier to do the calculations. Fur- thermore, it requires the scanning of only a small

fraction of the file of documents, whereas the cor- relation method requires a complete scan for each

enquiry. This last point holds only if we have the

following kind of document file — to each document

in the collection there is a "card" on which are

listed the m scores of its document image; these

cards are then filed in hierarchical order beginning

with the first score and proceeding through to the

TTith score. The effectiveness of this file ordering

in reducing the scan is detailed in appendix A.

4.3. Other Applications

We will only very briefly suggest some other

applications. For example, one may be interested

in knowing which authors are most closely asso- ciated with a specific problem; in this case one would work in the author-image space rather than

the document-image space, using procedures

identical to those described above. Now let us

give another example of the flexibility of the sys- tem. Some people may not trust the use of bibli- ographic citations in retrieval; in that case, one need only restrict himself to working with the title- and-author image space of the vectors D'"', instead

of the document image space of the vectors D, using

exactly the same methods as in section 4.2. above.

Further generalizations and modifications of the

model are outhned briefly in the appendices to this paper and will attest further to its flexibility.

4.4. Additions to the Collection

As with most systems, the system here proposed

suffers from the fact that the basic term document

matrix of occurrences and co-occurrences is altered

every time a new document enters the collection. For already large collections, each addition constitutes a very small portion of the collection and

169

772-957 0-66— 12
will do very little to upset the A measures already

established. It would then be safe to construct

the new document images and author images, etc., on the basis of the existing A matrix. From time

to time, however, updating of the A matrix would

be in order. For small collections the problem becomes somewhat more serious, and more frequent updating may

be necessary. In addition, there is the problem of new index terms being introduced through new

documents. This is a much thornier problem.

The most practical solution seems to be to equate

it to that term which is closest to it in meaning and

which is already in the vocabulary of the system

(to be done by human inspection). Whenever the

system is updated the whole matter could then be

set straight. The introduction of new authors

into the collection does not, of course, present any

problems, since author images are a direct conse- quence of title images which are, in turn, a direct

consequence of index-term images.

As a general remark it should be reiterated that mechanical or objective retrieval systems as applied

to small collections are bound to be unstable and

hence unreliable. It is only for large collections

that they have any hope of becoming useful or trust- worthy. The example which appears in the

appendix is for illustration purposes only and is not meant either to prove or to disprove the efficacy of the model. It is the misfortune of those working

in this field that small experiments, while often

difficult to execute, cannot tell us very much.

5. Appendix A. Required Computation

5.1. The Term-Term Distance Matrix

The first step is to list the documents with their associated index terms. (Both the documents and

index terms should be represented by numbers for convenience, and it is sometimes helpful if the

number ordering corresponds to an alphabetic

ordering.) Call this list L. The list should then

be inverted to give a second list L*. L* should dis- play each term in sequence along with the documents it indexes. From the two lists, L and L*,

it is a simple matter to get the symmetric co-occur- rence matrix by the tally method. This is the matrix

whose ab\h element is the number of documents in which the index terms ta and ti, both occur, i.e., the

matrix (N,,/,). The listing and tallying operations

may well be performed by a mechanical device, as indeed may every operation in the system. Each

entry in the co-occurrence matrix is then replaced

by its significance probability which may be read

out of the Owen and Lieberman tables [3] (e.g., if <f=100, A^„ = 10, No — 8, then the significance probability of N„b = 2 is 0.18). The resulting matrix is the term-term distance matrix A. For large collections the term frequencies, Na , will tend to be in a fixed proportion to the number

of documents d. It will, therefore, be possible to use the normal or Poisson approximations to the

hypergeometric probabilities which are also all tabulated. All probabilities should be rounded to a few (perhaps two) decimal places. In any event,

the total number of table lookups cannot exceed

\/2t(t— 1), where t is the total number of index

terms used.

5.2. Selecting the m Separator Terms

This involves calculating a quantity r

Q for each

of the index terms, where

A„

To get the A's we need to sum each of the t columns

of the matrix A. Then for each t we perform (t — 1) subtractions; and since there are t t's to calculate,

there are in all t{t— l) + £ = £

2 additions and subtractions to perform. We then choose the m largest

values of t and the corresponding index terms will be our separator terms. In practice, it will not be

necessary to compute t for all the index terms —

by inspection or some other mechanical criterion,

it will be obvious that most of the index terms will not even be contenders.

We may now take our (X / A-matrix and trim it down to an mX( matrix, say A,„, since the only

distances we will be considering are those to the

m separator terms.

5.3. Computation of Images

The t rows of A,„ are in fact the index-term images, t a , for each of the t terms in the system which we get free. Getting the higher-order images merely involves taking prescribed weighted averages of

the rows of AOT . The amount of actual work in- volved in getting the weighted averages depends

on such things as length of title, number of documents by the same author, and number of citations. On the average, to get a final document image would

require about yip + 1 + x + xp) + 2x additions and

4a + 6 multiplications, where

y= average number of index terms in a title, p — average number of titles by an author (in the collection), x = average number of citations per document.

Substituting typical values of x, y, p will show that the amount of computation cannot be very large

and grows increasingly slowly with d, the size of the document collection.

170
5.4. Matching the Enquiry Image

If \ = (vi, vz, . . ., Vm) is a document image and u = (ui, U2, . . ., Um) is the enquiry image, then the

suggested retrieval method is to compute max la,- — Vi\ for each document in the collection.

i Then retrieve all documents such that this quantity, called Umax , is less than e, where the enquirer

may choose e; or else, retrieve those z documents

with the smallest Umax scores, where the enquirer

may specify z. This seems to entail mXd subtractions, where d is the number of documents in the

collection. However, if we assume the hierarchical

arrangement of document "cards" as previously

described, then only a small fraction of this be- comes necessary. For example, suppose the document image elements were taken to the nearest

0.01 and suppose e was given to be 0.04. Then,

for a start, we need only look at the solid segment

of the file defined by the interval ^ = ^±0.04.

Thus, we can immediately eliminate 92 percent

of the file from the scanning operation. Similar

economies are affected when we pass to v2

, and

so on. Assuming uniformity, the fraction of the

file to be scanned will, on the average, be

m-l 2 (2e)fc = 2e(l-[2e]m- 1 )/(l-2e).

On the whole, the computations involved are all very elementary, and in quantity are quite reasonable. There is no offense made to simplicity.

6. Appendix B. Model Modifications

It is not our purpose here to develop any model

modifications but merely to suggest them. The

first thing that comes to mind would be to relax the restriction which confines us to the use of titles and authors. One might feel more secure if the

body or part of the body of the document were also taken into account. Within the framework of

images based on the tXm matrix, A,„, this could

be done in a very easy and natural way. Scan

the body of the document and list the frequencies

of the index terms which appear — suppose

h, f2, • • ., it appear with frequencies/1,/2, . . .,/a-, respectively. Look up the corresponding term

images ti, t2 , . . ., in the A m-matrix. Then we

can define the body image as

In a similar fashion, we may define and compute

first-paragraph images, summary images, chapterheading images, etc. The problem then arises as

to how much weight ought to be assigned to these

new creations relative to the existing title images,

author images, and citation images. We leave this problem with the hope that experiment and ex- perience may provide useful answers. We conclude with just one more suggestion.

We should also compute the images of those documents which are cited by documents in the collec- tion but which are not themselves in the collection. The image cards for these cited documents might be

filed separately. This file could then also be

searched in the usual way if the enquirer desires it.

7. References

[1] Salton, G., The use of citations as an aid to automatic content analysis, Information Storage and Retrieval, Rept. No. ISR-2, The Computation Laboratory of Harvard Univ.

(Sept. 1962).

[2] Baxendale, P., An empirical model for machine indexing, 3d

Inst, of Information Storage and Retrieval (Washington,

D.C., 1961).

[3] Lieberman, G. J., and D. B. Owens, Tables of the Hypergeometric Probability Distribution (Stanford Univ. Press,

Stanford, Calif., 1961).

[4] Giuliano, V. E., et al., Studies for the design of an English

command and control language system, A. D. Little Rept. ESD-TR-62-45 (June 1962).

[5] Doyle, L. B., Indexing and abstracting by association: Part

I, System Development Corp. Rept. SP 718/001/00 (Apr.

1962).

[6] Luhn, H. P., The automatic creation of literature abstracts, IBM J. Res. Develop. 2, No. 1 (1958).

[7] Maron, M. E., Automatic indexing: An experimental inquiry,

J. Assoc. Comp. Mach. 8, No. 3 (July 1961).

[8] Maron, M. E., and J. L. Kuhns, On relevance, probabilistic indexing, and information retrieval, J. Assoc. Comp. Mach.

7, No. 3 (July 1960).

171

Threaded Term Association Files

Mark Seidel

Datatrol Corporation

Silver Spring, Md. 20910

Since a term-association file, or file listing all terms bearing a given symmetric relationship to each

other, constitutes an inverted file of itself, only half of it need actually be stored. To find all the associates of a given term, one scans the contents of the profiles of all terms which precede the given

term, and finally pulls the entire profile of that term. Those associates which precede the term are obtained during the first phase: all others are in the profile of the term. Thus we have a half-length inverted file which is searched like a serial file until the desired entry is reached. More important, the file can be organized to be neither serial nor inverted, but to combine the advantages of both forms in a totally new way. The entries are arranged sequentially, as in a serial

file. The search information is the same as in an inverted file, but is distributed vertically as a set of threads through the sequential entries. Such a threaded file may be organized on magnetic tapes

without the sorting required for an inverted file, but combining the rapid directness of inverted-file search with the completeness of information found in a direct file.

A funny thing happened to me on my way through

school. A faculty advisor suggested to me that Kullback's work on information theory in statistics was interesting, but hopelessly impractical since

statisticians did not have access to unlimited computer time. This particular minor premise is probably true; but I am totally unable to accept a negative conclusion. And so my current preoccupation comes to be an efficient and economical

computer system capable of furnishing the statis- tician large bodies of experimental data at reasonable cost. As Hammond has mentioned in his paper for this Symposium, we have been maintaining a continuing

project with this goal. We wish to mechanize an

associative document retrieval system which is optimized with respect both to file maintenance

and to actual search time. In general, the philosophy of the system is based upon a large-scale

computer for generation of highly organized files. These files are complex but rapid to form and

maintain, and they may be used very rapidly by

even a small computer. We feel that one aspect

of such a file would be of some interest to this group.

e will be speaking of term profiles, and will show how the file size and search time may be

halved at a single stroke, in addition to any other

compression. While we use terms whose association is measured by Stiles' technique, our comments apply equally to any symmetric binary

relation within a finite vocabulary of terms. By way of background, let me first clarify our concept of a document file. Let us mean, by

document, a unique accession number and a set of terms from a finite descriptor vocabulary. We

will start with a matrix whose column headings are

the terms of this vocabulary; each document-entry

constitutes a new row in this matrix, with appropriate checks, or weights, or concept-coding defined across the relevant terms. Now no one would

actually use such a matrix on a realistic document

file, since it is extremely sparse: in NASA for instance, it is 99.8 percent empty. All the same,

the concept is a convenient reference; we will call

it the document matrix. This matrix and the two

files we will form from it are illustrated in table 1. When a matrix becomes very sparse, one thinks

of writing only the nonvoid rows or columns. We

call this form a file, as distinguished from the basic

matrix. It is necessary to repeat the name of the row or column at each recurrence, but this is easier

than filling in all the zeros for the voids of the matrix.

All of this is quite ordinary, except for our roundabout approach to it. If one writes out the rows

in order, with the column names (or terms) included, one has a simple document file; if the matrix is written in column order, with row names (or accession numbers) included, it is known as an inverted

term file. The point of all this introduction is that our term

profiles constitute a term-association matrix in which the row-headings and the column-headings

are the descriptor vocabulary; the row-entry for a term, consisting of all the associates of the term,

looks just like a document to the retrieval system;

and for a symmetric association, this file is its own

inverted term file. This is illustrated in table 2. Consider this term-association matrix: What do we need from it, and how shall we get what we

need? The matrix as such will be quite sparse, and we definitely want it encoded in file form; is there a distinction between the row and the column

representations? As we see at once from table 2, there is none, except in our view of the material.

Well, we are going to retrieve from the resulting

file, definitely; for generality, assume that we have

a set S of search terms, and we want to use our

collection of term profiles by finding and including

every other term which is associated to half the

members of S. By our analogy to the document

file, this means searching for all terms which "contain" half the members of S as associated terms,

written as pseudodescriptors. But with this particular file, there are two ways to go about this. These are illustrated in table 3.

173
The first of these is a straightforward document

search. Each entry is treated like a row of the

document matrix; its terms are compared with the

search set, and the entry is retained for output if

it contains half the members of S. The trouble

with this approach is that such implicit searching,

where almost every member of the entry must be

tested, can be rather time-consuming. Not necessarily prohibitive, of course; but a process based

on asking a question whose answer is almost al- ways "no" really ought to be held suspect.

The alternate method is to treat each entry like a column of the document matrix, or a member

of the inverted term file. The association matrix

is at least free of one cumbersome flaw, inasmuch

as no sorting is required — remember, the same set of entries may be treated either way. What one

does is to extract, explicitly this time, the entry

of each of the search terms. These entries are matched, and any term common to half of them is retained for output. Unfortunately, this method

also has its difficulties; chief of these is that the

matching process can necessitate simultaneous

manipulation of a great many terms. We were pleasantly surprised to discover that a hybrid approach is possible which has the advantages of both methods and the disadvantages of

neither. Consider the association matrix once

again. Since it is symmetric, it is completely de- termined by the triangle above the main diagonal.

The information we need will be found by tracing

the search terms down their respective columns to the main diagonal, and then across their rows. Of

course, that procedure is easier said about a matrix

than done on a computer file; but we will try to show that the doing is almost as easy as the telling.

Recall that we are eliminating the lower triangle of

the matrix, which contains all term-pairs whose sec- ond term is smaller than the first; and note that when a term's entry, or row, appears, this marks

the last appearance of that term anywhere within

this file. Another way to view this is to recognize that the

serial file of the upper triangle is the same as the inverted file of the lower triangle. All that we

are proposing is the simplification achieved by

recognizing it as such a dual file, using it as a serial representation until the diagonal is reached and

only then utilizing the immediate entry as an in- verted form.

To set the entire problem in practical perspective, we have in hand a set of 100,000 associated termpairs from 28,000 NASA documents, presently

arranged in full-matrix form. We intend to rear- range these in the half-matrix form we are discussing, with other compressions besides; the expected

savings will allow us to expand to 500,000 associates from which we hope to achieve a two-generation search expansion in 5 minutes of IBM 1401

time. Let us take search terms in hand and begin as

in table 4. The first entries in the file will be

examined in the manner we have termed document

searching. This is the direct but time-consuming

method; each entry so tested may be rejected or accepted on the spot, according as it contains less than or more than the required fraction of the search

terms. At some point we will reach the entry cor- responding to the first of the search terms; the term now shifts to what we call the inverted phase. It no longer participates in the direct search, but all

its righthand associates are retained as in the in- verse search. There is a major difference, however,

in the bulk which must be so retained. No associate in this inverse list need ever be kept beyond

the point where its own entry is encountered.

When we come upon that entry, we accept or reject

it on the basis of the number of active search terms

which it contains, plus the number of inverse search

terms on whose lists it appears, and then forget it completely since the remaining file contains no

information about it from here on. As the search

terms are passed, the number of inverse lists in- creases but their length decreases by deletion from

the top. When the entry of the last term is reached, we have only a few short inverted forms to finish. What has all this accomplished so far? We are working with only half the normal set of term profiles; we are spending half that time in an inverted

mode of search from which the twin barriers of

presorting and bulkiness have been eliminated; it remains only to deal with the complaints we have

voiced against the direct search. It is here that we

believe we have made the most novel contribution

in technique, for it is possible to thread the search

terms through the direct portion of this file. The concepts of lists, threaded lists, and multilists began to evolve in order to satisfy requirements

for dynamic storage allocation within random-access

storage media, and were gradually found to be ideal

to afford simultaneously some measure of content

addressability. We believe our application is among the first either to be intended solely for content-addressing, or to be used within a serial storage

medium.

To clarify once more: bear in mind that we are now speaking only of the direct phase of the total process we just described, the period when an entry

is being tested to see which, if any, of the search

terms it contains. What is needed at this point is to have some continuity down these search-term

columns which we are scanning during a particular

search, to permit us to ignore that large majority

which is of no interest to us. And, of course, threading does precisely this. To attempt to illustrate this threading within the

example we have been using would, unfortunately,

confuse rather than clarify. In application the

search actions and the auxiliary material for the

file and for interim manipulation will all be quite

small compared to the bulk of some hundreds of

thousands of term associations, and do result in appreciable savings which are not apparent in manageable illustrations. These savings result from

174
the fact that the thread of each term embodies the

inverted file of that term from the upper triangle.

The upper triangle is stored in serial form, and this

is also the inverted form of the lower triangle: the

dual search approach moves each term from one

phase to the other as the diagonal is reached.

What we now suggest is that the inverted form of the

upper triangle can be imbedded in the file as a set of lists which are threaded through these serial entries. All that is needed is to have each occurrence of a term carry with it the name of the next entry in which it is to be found. One maintains at the front of one's file the location of the first occurrence of each term in the vocabulary. At search time, one

picks up the first location for each of the search

terms. The smallest of these is the first entry in which one has any interest at all: during the time

until it arrives on tape, one obviously has almost

nothing to do except perhaps to be concerned with

a considerably larger search than one would other- wise have had time for. Even before it arrives, one

can observe within the computer whether enough

other terms are also waiting so that the impending

entry qualifies for acceptance. When it finally

comes, all that is needed is the directly accessible

association factor with each of the search terms it is known to contain, together with the next occur- rence of each of them. As the search progresses

and the search terms diminish in number, occur- rences get further apart and one has more time to deal with the growing number of inverse lists. All

told, we feel the dual-search threaded file has that certain reassuring harmony which bodes so well.

Table 1. Document matrix

Documents A B c D E F G H

X X X

2 X X

3 X X

4 X X

5 X X

6 X X

7 X X

8 X X X

9 X X

Document file Inverted term file

1 A, C, G, A 1,4,5

2 B. D B 2, 6

3 C, F C 1. 3, 8 4 A. H D 2, 7

5 A, E E 5,9

6 B. G F 3, 8

7 D. H G 1, 6, 9

8 A, C. F H 4, 7 9 E, G

TABLE 2. Symmetric term-association matrix

A B c D E F G H

A 0 X X X

B X 0 X

C 0 X X

D X c 0 X

E X 0 X

F X 0 X X

G X X X 0

H X X 0

"'Document" or serial form Inverted form is identical A B, E, G A B. E. G

B A, D B A, D

C D. H C D. H

D B, C. F D B. C. F

E A, G E A. G

F D, G. H F D. G, H

G A. E, f G A. E. F H C, F H C. F

Table 3a. "Document" or serial search for S = (A, D. H)

A B(A?D?) E(D?H?) G(H?) B A(A?*) D(D?*) Output B C D(A?D?*) H(H?*) Output C

D B(A?D?) C(D?) F(D?H?) E A(A?*) G(D?H?) F A(A?D?*) G(H?) H(H?*) Output F G A(A?*) E(D?H?) F(H?)

H C(A?D?) F(D?H?)

Table 3b. Inverted search for S = (A. D, H)

A(A?*) Keep A: B E G

B(D?)

C(D?)

D(D?*) Keep D: B C F Form AuD; B B C E F G

Output B

E(H?) Keep AuD: C E F G

F(H?)

G(H?)

H(H?*) Keep H; C F

Form AuDuH; C C E F F G

Output

Output

C

F

Left with AuDuH; E G

(X?): Is this search term X?

* : Yes, this is a search term.

175
Table 4a. Triangular term-matrix Table 4b. Dual search for S: (A, D, H)

Profile Associates

A B. E, G

B D

C D, H

D F E G

F G, H

r ronle Associates Act ion

A(A?*) BEG

Form P: r> t, d( u r

r orm P:

ft i t m• t vJ input v * T>. Iveep r

t>

r>D

R.

ri, V P

A,a nL*

L>(L> : ) rl(rl r ) f orm r: c c E G

Output C: D, H

Keep r

Cj (j U[u: ) r-r r orm P: E t Keep P: F G

F(H?) G(H?) H(H?*) Form P: F F G

Output F: D, H

Left with P: G

176
Statistical Vocabulary Construction

and Vocabulary Control with Optical Coincidence

Basil Doudnikoff and Arthur N. Conner, Jr. Jonker Business Machines, Inc. Washington, D.C. 20760

For several years vocabularies in mechanized documentation systems have been constructed

with the assistance of various statistical techniques. These procedures are generally accomplished

remotely through the use of computers.

It remains the job of the documentation specialist to analyze, correlate, and manipulate the data

thus provided, and periodically to ask for more data. The recent availability of an optical coincidence scanner (hole-counter) offers an entirely new type

of assistance in this area. This automatic desk-top device gives counts of holes in optical coincidence

cards within 10 seconds or less per count. And most importantly, the figures are not presupposed, but obtained as needed during linguistic analysis. A relatively new field is in the process of developing. This is the analysis and actual management

of current research efforts through evaluations of the descriptive vocabulary. In this area also, im- mediate counts are obtained through optical coincidence of any combinations of superimposed cards. Thereby, simultaneous combinations of conceptual processes and unlimited numerical manipulation and correlation are possible at the point of need.

The recognition of need, interest in, and all of us being here, at this Symposium, demonstrates

the need for new approaches to statistical vocabulary development.

Our title says we're going to talk about statistical vocabulary construction and vocabulary control

with optical coincidence — and we mean just

that. We'll give you a little background on how we got into this and why we feel there is a need in

this area. We'll review quickly the basic principles of optical coincidence — or peek-a-boo, as some of you may know of it — and, also, how this technique now operates as a counting and statistical tool. We'll try to look into the analyst's mind

as he constructs a vocabulary and indicate how this new statistical tool can help him. We'll talk briefly

about how the scanner can also be used as an

analytical tool for managing research. In one of the early, and still excellent, papers on the subject of statistical word association, Luhn

[1]

1 was concerned that: "For pictoral representation, the machine is at a disadvantage, at least at the present stage of the art. The best that can be

done is to instruct the machine to create a multidimensional array and to further instruct the machine to analyze all the many relationships contained

in this array. For a machine to do this it must have

an internal memory where it can store the representation and analyze it over and over again in accord- ance with a specific program." This limitation is, unfortunately, still all too true. The basic concepts to improve this situation

have been in the minds of the authors for over two

years. The application of these ideas, however, was held up by the lack of the required hardware.

About four months ago the optical coincidence

hole-count scanner became a reality.

At first, basic experimentation with this hard- ware was done by using hypothetical input. But

'Figures in brackets indicate the literature references on p. 180.

about the first of this year the contractual study

leading to the construction of a vocabulary was

started at the Army Research Office (ARO). For more experimentation in a real situation, actual

input from the ARO raw vocabulary has been used,

and the end product vocabulary has been, and will be, in part, accomplished through the techniques

described in this paper.

\ the relationship of \ j correlated index terms

is indicated by light dots ^ appearing in the super-

' imposed cards.

Figure 1.

177
Normal procedure in the construction, or more

accurately, selection, of the key words for the

vocabulary follows a routine somewhat as follows:

1. The documents are "freely indexed" without

a control vocabulary, or with only occasional ref- erence to one as a guide.

2. These index terms, or key words, are converted

to a machine language, usually punched cards.

The latter may, in turn, be converted to magnetic

tape or some other type of computer memory.

3. These terms are then processed by the

punched card equipment, or computer, into print- outs such as correlation tables, alphabetically se- quenced listings, and document sequenced listings.

4. These listings are then subjected to human

analysis for synonyms, near-synonyms, generic

relationships, ambiguity, redundancy, semantic

correctness, and the like. A committee of specialists from different disciplines may be brought

in for consultations and decisions. It is primarily

in this general area that we are concerned.

5. As a result of the findings of the analyst and

the committee, new lists and tables are requested.

6. Based on the decisions made and the judgments of the interrelationships of the key words,

or concepts, the vocabulary is thereby formalized

into printed form.

The new approach that we are presenting is based on using the relatively old principle of optical

coincidence and building on this concept an elec- tronic counter enabling it to read out valuable

statistical data at high speeds.

Although it is now used widely, many statisticians and documentalists are only casually familiar with

optical coincidence. In this relatively old concept,

each of the key words, or descriptors, has a card

dedicated to it. Document accession numbers are assigned X—Y coordinate positions on the cards.

When a hole is drilled in a specific position on a card, that document has a key word ascribed to that card. When several cards are stacked to- gether, coincident holes appear — and indicate those

documents that are described by each card thus

superimposed. Until recently, much of the effec- tiveness of the technique as a statistical tool was

lost because of the problem of visual reading — or

of "eyeballing" — of the coordinates of the holes. The recent availability of a device to count these

holes, when combined with ability to convert, or should I say "invert," punched cards into optical

coincidence cards, adds two new dimensions to

this technique, enabling its widespread use in

statistical vocabulary manipulation.

The input process into optical coincidence is analogous to punched-card input conversion to the computer. Just as punched cards go through

a converter to be put into a buffered memory or magnetic tapes of the computer, so the punched

cards go through a converter to be put in the memory

medium of the optical coincidence cards. The

mode of output, quite obviously, is radically dif- ferent. How is this so?

GAMMA + DOSIMETRY

a (2) ».

GAMMA RADIATION DOSIMETRY

(31) (95) (8)

\MMA + RADIATION RADIATION + DOSIMETR"

(13) (8)

GAMMA + RADIATION + DOSIMETRY

(2)

Figure 2.

The scanner is a device that electronically looks

at an optical coincidence card and counts the number of holes in it — puts numbers into a memory

unit, and through circuitry, optically displays the

summation. It works the same way with a stack

of cards — only here, the coincident holes are counted. This process takes only a few seconds

per card, or stack of cards.

Back to the computer. The computer re- quires considerable programming to facilitate desirable clustering and relationships of the key words.

This programming is usually done ahead of time by

EDP specialists lacking familiarity with the documentalist's problems and needs. Changes and

modifications of the analytical routine, if they are

to be meaningful to the analyst, again need to be

reprogrammed. In utilizing the optical coincidence

hole-count scanner, however, the analyst "programs," as the questions are posed.

The new methodology we are proposing is deeply

interwoven with the standard approach that was

mentioned earlier. This is somewhat of an intel- lectual switch, with more emphasis on people doing

the analysis. This process puts the "machine

room" at their fingertips. Determination of pro- cedures to be followed in this intellectual analysis

of the freely generated key words is very difficult to prescribe. The analyst does this partly by

intuition. Certainly he looks for relationships.

But he must browse, and think, and check back and

forth. Notwithstanding the freewheeling approach,

certain ground rules have been set up to maximize

efficiency of the analysis:

1. Process the input progressively, in a pre- designed series of gradual steps, rather than having

it immobilized for a one-time lengthy analysis. In

this manner the index is continuously available for retrieval operations in a form which is periodically

improved with every language-processing step

performed.

178
2. Maximum care should be exerted in the

syntactic parsing of word groups, following the consideration that every syntactic alteration carries

with it some alteration of semantics. Statistical data are needed before decisions are made with

regard to the ultimate parsing of word groups up

to the possible one-word level. Utilization of the scanner begins here. It facilitates the step-by-step

processing described.

3. A continuous feedback routine has been es- tablished between the system and the indexer.

This helps the indexers to improve their input lan- guage by showing them what machine form their

terms have assumed, and provides a continuous

updating of the vocabulary, thus eliminating the

defects of fixed thesauri and dictionaries, which are

in part obsolete at the time they go to press.

4. The vocabularies of the different laboratories

have been identified as to their origin. This per- mits the study of various input languages in the context provided by their origin, automatically

eliminating ambiguities which may arise at this stage, and permitting future comparison of homographs and other ambiguities due to the use of similar words in different contexts. At the same

time, this technique, with the hole-count scanner,

provides us with a tool for evaluating the kind of

work done in various locations.

DOCUMENT

LISTING

1140 1362

1363 1589

1627

1809

2431 3976 4885

THE TERM — CARD SCANNER

PRODUCES TWO TOOLS FOR VOCABULARY ANALYSES.

STATISTICAI DATA

TOTAL = 9

Figure 3.

Some of the possible manipulations here are the

pairing of key word to scientific field, of key word to responsible laboratory or organization, of key word

to key word synonym, of key word to key word

antonym. These will yield term clusters, pairing,

relationships, nonrelationships, and correlations.

Access time is on a demand basis. In addition, generic relationships are easily es- tablished and counts can be made at all levels. How deeply was the input made? What is the

percentage at different levels? These are the

questions asked by the analyst. His intuitive

logic is relied upon to determine which card is to be compared with which card.

Size of the card and ease of handling would seem

to impose certain constraints on the report population. In the system under study, the collection was

well within the 10,000-item limitation. But many

systems may not be so limited. We admit that such

a collection would be difficult to handle in its entirety. But in the event that the collection be large,

say 50,000, we believe a statistically sound random

sampling could be made with a high confidence

limit to enable this technique to be used.

Does the other parameter, one of the number of

candidate key words for the vocabulary, impose

prohibitive limitations on optical coincidence use and counting? Of course large numbers of candidate terms are a problem in any system. Wall [2] puts this into what we consider its proper perspective: "One is inclined to wonder whether all the

hundreds of thousands of words in the English

language must be included, and if so, one is appalled

by the multitude of the task. But, in fact, the

vocabulary of science is quite limited. Numerous

investigators have pointed out that the vocabulary

of any one field of technology is limited to approximately 5,000 terms, that the vocabulary of all technologies is limited to approximately 20,000

terms, and that the whole of human knowledge

could be expressed in less than 40,000 terms."

Qa

radiation a

= qa

radiation b

= q b

= amount of overlap between lab a and lab b.

Figure 4.

179
It should be remembered that this hole-count scanner is not intended to be used to count all of

the documents indexed by each and every key

word on a card by card count basis. Initial key

word counts will be done more easily and simply

by the tabulating machine while running the initial listing. Kurt Lewin [3] never hesitated to advise the

student: "Only ask the question in your research

that you can answer with the techniques you can

use. If you can't learn to ignore questions that you

are not prepared to answer definitely, you will never answer any." Indeed, only a small proportion of the key words is subjected to an in-depth

scrutiny and statistical comparison by the analyst.

Most current thinking in documentation is oriented to the static document and its retrieval. Another school of thought is being applied to statis- tically "managing" current work. For example,

how many projects are being worked on in a given area? What is the relative funding? Is there

any overlap of effort? How much, and specifically where is it? With the rapid advance of the

state of art of information retrieval in recent

years, it is not only possible, but mandatory, to resolve some of these problems.

Since the scanner makes it possible to analyze

vocabulary development, it is equally simple to interweave into this some studies of considerable

depth of the work done in different laboratories

and research groups within the organization or its contractors, plus the various subgroups within

them.

This new approach to statistical vocabulary

development provides the analyst, or the decisionmaking group, with a rather simple tool, which

when used in conjunction with his knowledge and

imagination lays the foundation to the information

system. Creative simplicity is one approach that we feel should not be overlooked in this age of

complexity.

References

[1] Luhn, H. P., A statistical approach to mechanized encoding and searching of literary information, IBM J. Res. and

Develop. 1, 313 (Oct. 1957).

[2] Wall, Eugene, A practical system for documenting building research, from Documentation of Building Science Literature, from the proceedings of a program conducted as part

of the 1959 Fall Conferences of the Building Research

Institute, Division of Engineering and Industrial Research,

NAS-NRC Publ. 791 (1960).

[3] Lewin, Kurt, Field Theory in Social Science: Selected Theoretical Papers, ed. D. C. Harper, p. 29 (1951).

180
A Computer-Processed Information-Recording and Association System

G. N. Arnovick

Planning Research Corporation

Los Angeles, Calif.

As a result of previous research studies in analyzing the problems of automatic data association in a man-machine information environment, a set of conditions is defined which represents a system logic concept for automatically processing input data for information content and relevance. The system

technique which is presented is the result of several separate research investigations and is defined

as a system concept which indicates a possible breakthrough in automatic information association. Automatic syntactical analysis and automatic reference to vocabulary lists may be used to construct

a formal operating statement given in equation form, by utilizing current methodologies of machine

language translation. Various levels of statistical association can be determined which represent a

logically manipulatable information unit. The association system logic which is presented can be con- ceived as a new and more efficient approach for a computer-processed information-recording and

association system.

1. Introduction

In the course of designing an informationprocessing system, a major problem becomes apparent, namely that of selectively identifying specific

information as it is related to information meaning

or coherence. The problem is further complex

when one considers the parameters of information

control that must process, correlate, or extrapolate

data elements in a rational manner. The tasks in- volved in information handling of syntax and semantic variables, and how they are identified and related

to a multiplex of stored items for comparison and

correlation purposes, are extremely difficult to process by a human analyst. The analysis and

processing of information as described above in- creases in magnitude when constraints such as

effective real-time inputs are part of the system, and

data buffering for prolonged off-line operations

cannot be tolerated due to loss of information

message content over a time continuum.

The information-processing logic and techniques

described in this paper are considered and defined

as an overall system concept in which system subtasks for automatic information association are computer processed. Significant research and systems

development in information association for (1) analysis and (2) machine organization have been

reported by G. Salton, V. Giuliano, R. Barnes,

H. P. Edmundson, L. B. Doyle, H. E. Stiles, and

others. (See references at end of paper.) These

findings and the technical methods suggested for information association are taken into account, with

the expectation that they can be effectively utilized within an information-processing environment such

as conceptually presented in this paper, and that the method or combination of methods to be selected

would depend on the application requirements.

To develop an optimum system configuration it is necessary to specify a man-machine informationprocessing environment, in which information re- cording and association are defined as the major

system task. Accordingly, a subsystem task framework is provided for automatic information recording and association based on the utilization of

machine language translation (MLT) methods for analyzing recorded information statements. The

methods for utilizing MLT employ functional

developments which are optimally suited to the

system solution. The technical approach and system design rationale for recording and associating information by

the utilization of MLT are dependent on suitable

functional solutions and special-purpose processing

equipment, and will be dependent on application

variations as they relate to (1) real versus non-real

time data handling; (2) file format and organization;

(3) semiautomatic or manual processing; (4) cost/

system tradeoffs for optimal utilization; (5) memory

size and type needed and available; (6) utilization of serial or parallel file processors; (7) random order

of data arrival; (8) priority interrupt; (9) on- or off- fine to a computer; (10) queuing and information

distribution. In summary, a method is described for data

analysis which considers information sets as an

operating group of formal statements as part of an

input message. The basic approach for a functional

system design is based on the use of MLT for analyzing recorded information statements. The

system utility is not expressly designed for library/

document system solutions as they are related to current automated library requirements. However,

the man-machine concepts utilized by the system

definition may be practical with further design

constraints for automatic document content

analysis, and on-line document browsing for the

library of the future, incorporating a man/console/

computer system suggested by Dr. D. Swanson,

at the Airlie Conference on Libraries and Automation, Warrenton, Va., 1963. The proposed

system concept is more applicable and suited to the

technical and decision-making requirements of

information control systems as applied to (1) management information systems; (2) control center management; (3) mission analysis and information

processing; (4) simulation.

181
2. System Concept

A basic requirement for an information storage

system is the ability to draw together all the relevant

pieces and bits of information in answer to interrogations which may be made at any hierarchical

level of relationships. Systems which have in the

past as well as currently, employed simple descriptors and low-level association between those factors allow for the use of relatively simple computer

processing. The result of such relatively simple

and limited operational capability placed a heavy

burden on the analyst, who has to determine the

relevancy of the retrieved information, much of

which is redundant, therefore reducing the rele- vancy of the retrieved data, as well as allowing for nonpertinent information flow. Previous work for very large automatic information-processing systems utilizing tree-structure techniques expressed as multiplets provided a

logically manipulatable information unit. However,

the system planning and design for such systems,

which theoretically provided complete automatic

information handling, was not able to process data

automatically as planned. This was due to the

inability of the subsystem to maintain automatically

logical consistency checks for input message completeness as verified by a stored item file for data

correlation. The item-compare subsystems ex- pressed as a function of word association pertinent

to incoming statements failed to provide message

reasonableness as defined by logical rules for semantic reliability. Thus the system design goals were not satisfactorily met; this appears to limit the possibilities of utilizing automatic input processing. Empirically, there is no doubt that fully auto- matic systems are inherently limited, and must

require human analysts to be an integral part of

the system performance functions. This manmachine interface is mainly centered on the need

for human analysts to be in complete control for input message encoding.

3. Technical

The system design rationale proposed for a computer-processed information-association and

recording model is specifically concerned with

several major system variables, which are as follows:

1. The system is semiautomatic by definition.

2. Humans (the analyst) are linked to the system.

3. The control element is a man-machine function.

4. The computer's role is defined as a servo- system for rapid processing slaved to the analyst.

5. The inferential technique for information

analysis (association and recording) utilizes machine

language translation as the major interface between

data control and computer processing.

6. The system is relatively dualistic (dependent

and nondependent on machine translation methods

relative to the time domain frequency for computerprocessed data), e.g., information content may be

processed in raw form independent of translation

requirements and at select time sequences, and in- formation processing is a control function dependent

on the logical algorithms of machine language

translation procedures.

The techniques of machine language translation

offer a means for automatically analyzing the syntactical structure of sentences. The semantic

content of a sentence is dependent both upon the

words used and upon their relative order of use. In

this instance, automatic syntactical analysis and

automatic reference to vocabulary lists (formally

equated to hierarchical code lists) which are gov- erned by a formal set of rules will be used to construct an operating set of formal statements,

expressed in the form shown in eq (1):

{I[t(S-0-A n )^P]} 1, . . . {I[t(S-0-A n )^P]}„->R

(1)

Approach

where:

{ } — operating level formal statements R = total stored intelligence item (gives loca- tion of storage and acts as a link between

statements included in a particular item

/ = field of interest (e.g., strategy, tactical,

intelligence, economics, etc.)

i = time of statement or origination of subject

or object (A n may modify)

S = subject taking the action

0 — object acted upon or co-subject of intransitive actions

A — action

P — product or result —> = leads to. A symbol before a bracket may modify the hier- archical structure of the code elements within that bracket, e.g., / modifies S, 0, A„, and P. t may

modify S, O, and An but is unlikely to modify P,

as this should be chosen to include time-stable

terminology. For example, 5 and O might contain names of countries or cities whose names may be

subject to change with time, t itself may express

either relative time, as dates, or absolute time rela- tionships such as elapsed time, velocity, rate, etc. An information set is defined as that group of

operating level formal statements derived from one message input to the system. This can be repre- sented by eq (2):

(Xu X2 , .... Xj)i . . . <XU X*, . . . XX R (2) where the A"s inside the parentheses stand for some

of the symbols defined above, where the items in- side the parentheses are numerically coded repre- sentations of the original statement information,

182
n is the total number of operating level formal

I statements in one information set, and R is the set

identifier. This information is compared with the /-file. A nonduplicate statement is stored as a hierarchical structure in the /-file. In the case of a duplicate

statement, only the set identifier, R, is stored. The number of /?'s stored serves to enforce the

validity of the corresponding statement. The hierarchical structure is now modified by

interchanging S and /; the above process is then

repeated, using the S-file. This process continues

until all six combinations of /, S, 0, A, n, and t have been exhausted. The last combination of items will be sorted in the f-file. These six files will enable rapid retrieval of information based on any one of the six categories.

The system concept expressed as a subtask of

file identification and flow of data sequence for input analysis and operations is shown in figure 1.

DATA

SOURCES EZU3

PRELIMINARY ANALYSIS & EVALUATION

OF INPUT DATA

HIGH VALUE

DATA, RAW

HIGH VALUE DATA IN MACHINE LANGUAGE

LOWER VALUE RAW DATA

CANNOT

RESOLVE WITHOUT HUMAN DECISION f CONVERT TO \

< 1 FORMAL OPERATING J \^ STATEMENTS J

ANALYST DATA

RESOLVE & CONVERT , TO FORMAL OPERAT- r*~

ING STATEMENTS

CONTRADICTION TO STORED DATA COMPARE TO

STORE AND

FORM ASSOCIATIONS WITH

EXISTING DATA

STORF MACHINE FUNCTIONS

(WITHIN DOTTED LINE)

FIGURE 1. Input information-processing flow.

Various index files of the formal statements will be derived from the combination of input data

and logical decisions applicable to those data by

the system or by the human analysts.

A file of logical statements will be created to serve

as a check upon the reasonableness of incoming

statements. For example, an input statement re- garding the movement of the troops of one nation

through the territory of another nation cannot be

considered as reasonable unless (a) these two

nations have some treaty or agreement regarding

such movements; (b) these two nations are at war

with each other; (c) one of these nations is in a

critical geographical location with respect to some

aggressor nation. Such statements may themselves

be derived from verified input data. Any contradictions to the stored logical rules of

reasonableness, any lack of completeness or other

inherent defects of the statement would be sensed

automatically, and cause the statement to be trans- mitted to the analyst for further investigation.

4. Application and System Extension

At this time, an analysis of the proposed associa- variables for weighting functions expressed as

tion and recording system concept suggests several probabilistic association events; (2) the utilization areas of possible applications. Some of the more of the proposed model in screening data redundancy

immediate applications concluded from the system for management information systems; (3) the

are (1) mathematical simulation of syntactical extrapolation of select associative terms related to

183
message identification; (4) the use of MLT models

for programming conversion suitable to input data

format; (5) utilization of MLT techniques as a manmachine system for information concept building;

(6) generation of a compiler for common message

translation which is computer independent as to type of equipment; (7) automatic thesaurus generation and development. These applications as expressed above are logically possible, and repre- sent a potential breakthrough for current problems

in information handling and manipulation. The

technical problems associated for such projected

functions are not easy, as it is obvious that the solu- tions required do not deal with simple data, but

rather with complex sets of data, expressed as infor- mation for human understanding.

Further study for the development and implementation of a computer-processed informationassociation and recording system is needed at this time. It is recommended that a study program be

initiated which would allow for the systematic

development of functional tasks that are logically

related to each other as a chronological step for each subevent in the total analysis effort. The

major analysis criteria are as follows: — Analyze various kinds of information to be used

for the system. — Determine various relevancy requirements and

techniques for total information match. — Study and analyze various methods for recording hierarchical relationships of data. — Analyze various methods of syntactical analysis

appropriate to the system. — Determine methods for establishing the equivalence of statements on the basis of syntactical

analysis and hierarchical relationships. — Ascertain the appropriate man-machine inter- face requirements. — Design information system model. — Describe the logic and computer program to simulate and test an information-recording and

association model. — Recommendation for methods of implementing

the system.

5. References

1. Arnovick, G. N.. A linear programming model for system

design, EDP Division, Radio Corporation of America,

Cherry Hill Laboratories, Cherry Hill, N.J. AATM-1

(Apr. 1960).

2. Arnovick, G. N.. Advanced techniques information search and retrieval systems. Data Systems Division, Radio

Corporation of America, Van Nuys, Calif. EM-61-583-6

(June 1962).

3. Arnovick, G. N.. Information Processing systems: system and subsystem characteristics. Space and Information Systems Division, North American Aviation, Inc., Downey,

Calif. SID-63-1362 (Jan. 1963).

4. Arnovick, G. N., New machine development for information processing system, presented at NATO Advanced Study

Institute on Automatic Document Analysis, Venice, Italy (July 1963).

5. Arnovick, G. N., J. A. Liles, and J. S. Wood, Information

storage and retrieval: Analysis of the state-of-the-art, presented at 1964 Spring Joint Computer Conf., Washington, D.C. (Apr. 21-23, 1964).

6. Bernier, C. L., Correlative indexes I. Alphabetical correlative indexes, Am. Documentation 7, 283-288 (1956).

7. Borko, Harold, The construction of an empirically based

mathematically derived classification system. System

Development Corporation. SP585 and FN-6164 (Oct. 26,

1961).

8. Doyle, L. B., Indexing and abstracting by association. Am.

Documentation 13, No. 4 (Oct. 1962).

9. Giuliano, V. E., Automatic message retrieval by associative techniques, Preprints of 1st Congr. on Information System

Sciences, The MITRE Corporation (Sept. 1962).

10. Giuliano, V. E., Analog networks for word association, BIONICS Conf., March 1963. To appear in IEEE Trans.

Mil. Elec.

11. Giuliano, V. E., and P. E. Jones, Linear associative information retrieval, Ch. 2 of Howerton and Weeks, Vistas in Information Handling 1 (Spartan Books, Washington, D.C,

1963).

12. Maron, M. E., Automatic indexing: an experimental inquiry,

J. Assoc. Comp. Mach. 8, No. 3 (July 1961).

13. Maron, M. E., and J. L. Kuhns, On relevance, probabilistic indexing and information retrieval, J. Assoc. Comp.

Mach. 7, No. 3 (July 1960).

14. QuiUan, Ross, A revised design for an understanding machine, Mechanical Translation (MT) 7, No. 1 (July 1962).

15. Stiles, H. E., The association factor in information retrieval,

J. Assoc. Comp. Mach. 8, No. 2 (Apr. 1961).

16. Vazsonyi, Andrew, Automated information systems in plan- ning, control and command, presented at 10th Intern. Meeting of Inst, of Management Sci. (TIMS), Tokyo,

Japan (Aug. 21-23, 1963).

17. Yngve, Victor H., A model and an hypothesis for language

structure, Proc. Am. Phil. Soc. 104, No. 5 (Oct. 1960).

184
Applications to Citation Indexing

Statistical Studies of Networks of Scientific Papers

Derek J. DeSolla Price

Yale University

New Haven, Conn.

Statistical analysis is made of the way in which papers are linked together by the citation of one paper by another. The distributions of numbers of references and of numbers of citations per paper

are estimated, and from this a general structure of the network is derived. Every paper once published

is cited on the average about once per year. The linking of papers is such, however, that an Immediacy

Effect tends to join new papers to relatively recent ones rather than the entire available body of litera- ture. Perhaps, half the literature is of the immediate type and the other half "immortal record." The nature of the research front is shown to correspond to a fabric of knitted strips, the width of each

strip being such that it corresponds to the work of a few hundred men at any one time. These form

natural parcels of subject matter.

187

Can Citation Indexing be Automated?

Eugene Garfield

Institute for Scientific Information

Philadelphia, Pa. 19106

The main characteristics of conventional language-oriented indexing systems are itemized and compared to the characteristics of citation indexes. The advantages and disadvantages are discussed

in relation to the capability of the computer automatically to simulate human critical processes reflected

in the act of citation. It is shown that a considerable standardization of document presentations will be necessary and probably not achievable for many years if we are to achieve automatic referencing.

On the other hand, many citations, now fortuitously or otherwise omitted, might be supplied by

computer analyses of text.

This paper considers whether, by man or machine, we can simulate the process of "documenting," the process by which authors provide

reference citations to pertinent and usually earlier documents. My paper does not concern the

manipulative or mechanical problems of auto- matically compiling or printing citation indexes.

The existence of the Science Citation Index is adequate testimony to the ability of the computer

rapidly to sort, edit, and print large-scale citation indexes [l]. 1 My paper also does not consider the problem of

automatically recognizing (reading) and/or extract- ing explicit citations appearing in published documents by use of character-recognition devices.

Programming such a device will require the resolution of fantastic syntactic problems even if the

machine has a universal multifont reading capability. For example, in the citation, "J. Chem. Soc. 1964, 1963," which number is the year and which

the page number? These are not trivial problems.

To handle the vagaries of bibliographic syntax we

"pre-edit" all documents before key-punching the

citation data needed for the Science Citation

Index. We also "post-edit" both by computer and

human editing procedures. Do not confuse the

"automatic" or "routine" nature of citation indexing with a syntactically intelligent automaton.

Our citation indexers do not require subject-matter

competence, but they do require considerable

bibliographic training. The diverse and un- standardized citation practices in the world's litera- ture make this necessary. In addition, there are

linguistic variations in names and publication

titles which must be handled. Our citation in- dexers essentially must be trained in descriptive

cataloging.

My paper does concern the ability of an artifi- cially intelligent machine to deal with, among other

things, the implicit reference citation as distin- guished from the explicit reference citation. Such

might be the case in a paper where the author, for one reason or another, has neglected to provide a pertinent bibliography. The editor of a scientific journal would ask such an automaton to supply all "pertinent" references, if for no other reason than

' Figures in brackets indicate the literature references at the end of the paper.

to make certain the research was original. Citations are generally used to provide "documentation"

or support for specific statements. However,

reference citations are also provided in papers for numerous reasons including, among others:

1. Paying homage to pioneers

2. Giving credit for related work (homage to peers)

3. Identifying methodology, equipment, etc.

4. Providing background reading

5. Correcting one's own work

6. Correcting the work of others

7. Criticizing previous work

8. Substantiating claims

9. Alerting to forthcoming work

10. Providing leads to poorly disseminated,

poorly indexed, or uncited work

11. Authenticating data and classes of fact —

physical constants, etc.

12. Identifying original publications in which an

idea or concept was discussed.

13. Identifying original publication or other work

describing an eponymic concept or term as, e.g., Hodgkin's disease, Pareto's Law, Friedel-Crafts

Reaction, etc.

14. Disclaiming work or ideas of others (negative

claims)

15. Disputing priority claims of others (negative

homage)

The problem of identifying all "pertinent" refer- ences, to support implicit citations, is a special case

of the general problem of automatic indexing.

It has previously been reported that machines can

index or abstract by use of key words in context taken from titles [2], by use of statistically signifi- cant sentences [3], kernels [4], etc. O'Connor

has recently reviewed these methods [5], as has

Artandi [6]. Associative methods have been

widely discussed by Stiles [7], Maron [8], Giuliano

[9], etc. All of these systems, however, are con- cerned with indexing by use of the text only.

Bibliographic citations are regarded as metalinguistic elements.

Recently, however, Salton [10] has discussed

the use of bibliographic citations as indicators of document content. Essentially he proposes to treat citations as descriptors, which may seem

strange to those who think in terms of conventional

indexing. Indexers do not ordinarily think of cita189
tions (addresses of cited documents) as descriptions of the citing document. However, that does

not alter the fact that they are [11]. Citations (document addresses) are brief repre- sentations of the documents they identify. As

one sacrifices compactness, such as is found in serial numbers for patents [12], and expands to

full titles and then to abstracts, one sees the gradual

enlargement of the document description toward

the complete text. In this transition from "cita- tion" to "document," redundancy is introduced as well as additional information content. Indeed,

a document and a citation approach equality as the depth of indexing decreases (from the full text)

and the length of the citation increases. This

corresponds to my earlier definition of the document

as the set of descriptors which describe it [13]. In an information retrieval system, information

content can be measured only on the basis of in- dexed information that is supplied in the indexing

process. By this definition a document is a unique

combination of descriptors not assigned to any

other document in the collection. In most the- saurus-based collections indexing is not sufficiently

deep to achieve such uniqueness. However, the

combination of conventional subject headings or descriptors with the bibliographic citations used as references increases our ability to describe documents uniquely and specifically. Indeed, those

who have studied citation indexes and so-called bibliographic coupling are well aware that only a small number of reference citations are needed to isolate uniquely a particular document in the collec- tion from all others [11]. That is why a search of

a citation index generally produces a highly selective and useful search result. In discussing citation indexing it is frequently

stated that weaknesses of the method include undercitation (the deliberate or unwitting failure to cite pertinent literature) and over-citation (the excessive

reference to presumably nonpertinent literature).

Under-citation is illustrated by the patent literature,

since there is an economic motivation to cloud rather

than clarify the information disclosed in a patent.

However, the patent examiner, otherwise motivated,

attempts to clarify the prior art by providing a list of "references cited" [14]. Suppose, however, the

patent examiner, or a journal editor, wishes to examine a document quite critically and asks that the "machine" provide all the pertinent documentation or prior art. This brings me once again to the main theme of my paper. To answer the question "Can citation indexing

be automated," as we have seen, obviously entails a discussion of the entire range of question-answering problems encountered in designing any information retrieval system. Consideration of the auto- matic procedure for supplying reference citations,

when they are missing, merely focuses attention on the complex indexing task performed by the

author when he does give pertinent reference cita- tions. Such considerations help us focus attention on the significant differences between a priori and

a posteriori indexing [15]. Since each person may

interpret the meaning or significance of words and

documents differently, the problem we are dealing

with inevitably involves the human ability to create

novelty, to invent, to discover, and to be critical. Are machines, or machinelike people, capable

of imitating or simulating the human process of

being critical? What are the peculiarly "human"

earmarks of certain sentences containing citations?

When do such sentences contain implicit citations that could be supplied by an intelligent machine

and when would this appear to be difficult or impossible?

Consider the following example: "Mr. X, an

impossible idiot, has recently published a paper on

gobbledegook. The conclusions reported in his paper are wrong as are the data on which the conclusions are based. The recommendations made

by Mr. X, on the basis of his conclusions, will be

a calamity for mankind."

In polite circles, this is called the critical review.

Obviously, "intelligent" machines are not yet ready

to generate such criticism. Or at least program- mers are not yet able to program machines to prepare such critiques. If they were, then the

paper by Mr. X would probably never have appeared

because the same artificial intelligence would have

been available to tell him that his data were wrong

before he published and why! (If he persisted in publishing, we probably would have identified a

quality common to humans, but invariably attrib- uted to machines — stupidity.)

The first sentence in the example illustrates the case for an implicit citation that our machine ought

to be able to provide. What could be more simple

than the kernel sentence "Mr. X has published,"

which one would hope could be the result of a transformational analysis [4] when such methods

are perfected. Such an analysis combined with a complete computer listing of the papers by Mr.

X is a good starting point. Since we know that this

is not sufficiently specific we must then expect of

the linguistic analysis "Mr. X has published on gobbledegook" and then we have reduced the computer search to the "simple" task of identifying the one paper out of the thousands by men named X

to those which concern gobbledegook. Alas, this simple task alone requires the resolution of all the

linguistic and semantic problems associated

with matching the word "gobbledegook" with the

possibly different words in the title of the implicitly cited paper or book. Indeed, there is no rea- son at all to assume the same word has occurred

either in the title or the text of the "cited" work.

If these problems were not sufficient, keep in mind

that the word "recently" is quite significant in the

example chosen because it stresses the possibility that Mr. X may have written extensively on gobbledegook and it is only one particular, or a few recent papers, that is the target for discussion.

Fortunately authors usually do provide, explicitly,

the citations needed to support such sentences. As a consequence the citation index, created by

190
human indexers, does correlate the cited work with

the critical statements which appear in the second

and third sentences of the example paragraph.

This feature of the citation index alone would have

justified its creation. However, it is interesting to speculate whether transformational or any other

automatic analysis of such a paragraph could pro- duce a useful additional "marker" which would de- scribe briefly the kind of relationship that exists between the citing and cited documents.

These "markers" would appear in the pubhshed

citation index along with the usual citation data. In the case of the paragraph above, for example,

"critique" or one of several other terse statements

like "Mr. X is wrong," "data spurious," "conclusions wrong," "calamity for mankind," etc., might

be appropriate. The "intelligent" machine would

examine a new document and generate a critical statement such as "rather poor paper." As we

have seen above, a less intelligent machine might

analyze the paragraph and conclude that a biblio- graphic citation to the work of Mr. X is missing and

needed. The machine might also conclude that the cited work was under "critical" discussion be- cause of certain syntactic or vocabulary characteristics associated with "critical." Presumably they

would be identified by transformational or other

sophisticated analyses not yet available. This

would be no mean accomplishment. Among other

nontrivial problems is the fact that the information

needed to assign the marker can be spread throughout, not in a single sentence of, the source paper.

O'Connor's studies on the term "toxicity" are quite pertinent to this problem because the problems have in common the need to discover methods

for assigning descriptions of documents which are subject to considerable variation [16]. What is toxic to one man may be euphoric to another!

To examine a document from the "citation"

point of view, to determine what reference citations could or should be provided which link the sentence,

phrase, or word in question to man's prior recorded

knowledge, is to say the least a formidable chal- lenge. The task is an excellent exercise for new

journal editors. To follow the "citation" method of appraising a paper is in essence to challenge

rigorously each statement in that paper. If an

author does not provide documentation for state- ments it does not mean that they are false. However, they should ideally be supported by a "refer- ence" to some prior document, conversation, etc.

It would appear that in the "ideally" documented

paper almost every sentence or phrase could be in- terpreted to require reference to the past. While

one can accept intuitively the notion that there are novel sentences that one can express in English,

novel concepts appear to be comparatively rare.

Most novel combinations of words, punctuation, etc. could be transformed into concepts that had ap- peared before. Indeed, patent examiners like to remind inventors of this when disclosing generic

concepts, alone or in combination, which anticipate

specific embodiments.

I recently did an experiment with a group of my

students at the University of Pennsylvania in which

I asked them to read a paper pubhshed in the

Journal of Chemical Documentation [13] which con- tained no bibliographic citations. The reason

this paper did not have a bibliography is simple.

Many pubhshed papers don't have bibliographies

for similar reasons. The paper was originally

presented at a meeting. The editor of the journal

asked for a copy, but it was pubhshed without

the bibliography which obviously was not needed in the oral presentation.

Each student was asked to supply the missing

bibliography for this paper. Twelve students were

involved in the experiment. One student assigned

12 references while another assigned 75. The

average was about 40. This is not surprising, as a considerable amount of hterature was reviewed in the paper. The bibliography could have been ex- panded to hundreds of items if the common German

practice were adopted of giving a complete hst of

papers every time a topic is mentioned. Thus, in a discussion of information theory where I felt one

citation was sufficient, someone else might have

cited numerous related works.

The comments above are intended to give you a

feeling for the problem we face in automating cita- tion indexing. It is a wide open area of research

and it will take us into every fundamental area of

textual analysis — something comparable to exegesis [17]. It is apparent that each author re- stricts his use of reference citations according to the importance he places on the statements in- volved. From our knowledge of quantitative cita- tion data, a doubhng or trebhng of the number of

citations in the average paper would not overload

the system from the user's viewpoint. The average

paper that was cited in 1961 was cited about 1.5 times [18]. To double the amount of citation would not even double this figure, because not the exact same set of papers would be cited. However,

even if we did significantly increase the average

number of references to a particular work, we would

then give consideration to a more specific approach

to citations. This is well illustrated in the citations

to books where one finds the hst of sources subdivided by the page cited. This only adds an additional dimension in the specificity of citation indexing. There is no reason why this same

principle cannot be extended to the paragraph,

sentence, or word. Indeed, this is exactly what

happens in exegesis.

191
References

[1] Garfield, E., and I. H. Sher, Science Citation Index, 2672 pp.

(Inst, for Scientific Information, Philadelphia, Pa., 1963).

# [2] Luhn, H. P., Keyword-in-context index for technical litera- ture (KWIC Index), ASDD Rept. RC-127 (IBM, Yorktown

Heights, N.Y., Aug. 31, 1959).

[3] Luhn, H. P., The automatic creation of literature abstracts, IBM J. Res. and Devel. 2, 159-165 (1958).

[4] Harris, Z. S., Linguistic transformation for information

retrieval, Proc. Intern. Conf. Sci. Inform. 1958, vol. 2, 937-950 (Natl. Acad. Sci., Washington, D.C., 1959).

[5] O'Connor, J., Mechanical indexing methods and their

testing, AD#409, 276, J. Assoc. Comp. Mach. 11, 437-

449 (1964).

[6] Artandi, J., A selective bibliographic survey of automatic

indexing methods, Special Libraries 54, 630-634 (1963).

[7] Stiles, H. E., The association factor in information retrieval,

J. Assoc. Comp. Mach. 8, 271-279 (1961).

[8] Maron, M. E., Automatic indexing: an experimental inquiry,

J. Assoc. Comp. Mach. 8, 404-417 (1961).

[9] Giuliano, V. E., Analog networks for word association, IEEE Trans. Mil. Elec. MIL-7, 221-234 (1963).

[10] Salton. G., Associative document retrieval techniques using

bibliographic information, J. Assoc. Comp. Mach. 10,

440-457 (1963).

[11] Garfield, E., The science citation index — new dimension

in indexing, Sci. 144, 649-654 (1964).' [12] Garfield, E., Forms for literature citations, Sci. 120,

1030-1040 (1954).

[13] Garfield, E., Information theory and other quantitative

factors in code design for document card systems,

J. Chem. Doc. 1, 70-75 (1961).

[14] Garfield, E., Breaking the subject index barrier — a citation index for chemical patents, J. Patent Office Soc. 39,

583-595 (1957).

[15] Garfield, E., Citation indexes — new paths to scientific knowledge, Chem. Bull. (Chicago) 43, No. 4, 11-12

(1956).

[16] O'Connor, J., Mechanical indexing studies of MSD, toxicity (DDC No. not yet assigned. Contact author for copies

c/o Inst, for Scientific Information).

[17] Garfield, E., Citation indexes to the Old Testament, Am.

Documentation Inst. (Nov. 1955).

[18] Garfield, E., Citation indexes in sociological and historical research, Am. Documentation 14, 289-291 (1963).

192
Some Statistical Properties of Citations

in the Literature of Physics 1

M. M. Kessler

The Libraries. Massachusetts Institute of Technology

Cambridge, Mass. The bibliographic sources in a number of physics journals are analyzed. The frequencies of

inter-citation between the journals, expressed as percentages, are arranged in a matrix. It is postulated

that the properties of this matrix may be used to define a functionally related family of journals.

The Technical Information Project of the M.I.T.

Libraries is engaged in the design of a working

model of a technical information system that will serve a local community of scientists on a test basis. The choice of an experimental body of literature became a crucial question in the design of the

system. It was recognized that the literature must be large enough to provide a realistic search

situation and yet it should not be too large for model

operation. The physics periodic literature was

chosen as the experimental corpus for the model

library. The choice of specific journals was based

on the associative statistics of the various journals,

the criterion of association was the frequency of

inter-journal references.

The design of the retrieval system, its components, and operations will be described in a forth- coming report. The present paper is concerned

with the statistics and association measures that give guidance to the choice of an experimental

literature. The statistics presented in this paper

are based on a study of the citations in 36 volumes

of the Physical Review (Vol. 77, 1950 to Vol. 112,

1958). These volumes contained 8521 articles that yielded 137,108 references to 805 sources. Spot studies were made on 18 other journals.

Except for minor editing to eliminate misprints,

duplications, and obvious errors, the given data are exactly as copied from the journals. Repetitions

due to lack of standardization in notation or ab- breviations were left unchanged. Such repetitions are common in references to the foreign literature,

particularly the Russian.

These data must not be interpreted as a defini- tive list of periodicals but rather as a sample of the

operational hterature of a large number of research

physicists who publish in the Physical Review and

other journals. As such it sheds light on the collective nature of the working hterature of physics and

provides significant guidance for the design of a science communication network. It is from this point of view that the data were of most interest to the author.

Table 1 is a summary of the statistical highlights

of the references in the Physical Review. Table 2

lists the titles in order of decreasing frequency of

citation. The first column in Table 2 (order number) locates the title along the frequency scale.

1 This work was sponsored by the National Science Foundation and in part by Proj- ect MAC, the experimental computer facility at M.I.T. which is sponsored by ARPA.

The second column (frequency) indicates the

number of times the title was referred to in the

36 volumes of the Physical Review. The last column is the title of the source as it appeared in the hterature. Table 2 does not list those titles that occur only four times or less. We draw three conclusions from the statistics of this list: A. There exists a definitive journal (Jo), in our

case the Physical Review, that occupies a unique

and dominant position as the most-referred-to

source.

B. The definitive journal plus a relatively small

number of additional titles account for the over- whelming majority of all the references. In our

case the Physical Review plus 55 titles out of a

total list of 805 titles account for 95 percent of the source material. The significant property that this class of journals shares with Jo is stability in time.

The same list of 55 journals (plus J0

) will account

for the majority of references year after year. C. The remaining 5 percent of the references is to a large and ever-growing list of rarely used

sources. Unlike the titles in Groups A and B, this

list has no stability in time; each new volume ex- amined yields some 15 to 20 new titles. This

phenomenon is illustrated in Table 3. The

total number of references to the periodic hterature

in the 36 volumes was 113,997. The titles that appeared in Vol. 77, the first volume examined,

account for 107,385 references. In other words,

the titles that appear in the first volume examined

are destined to carry 96 percent of the references in the subsequent 35 volumes. As we examine those

subsequent volumes, 78-96, it is clear that although

the list of new titles never ends, their contribution

to the total reference hterature is comparatively

small. The investigation was continued to journals other

than the Physical Review but related to it. Table

4 shows the distribution of citations between titles previously coded (i.e., those encountered in the

Physical Review study) and new titles. These

data are much hke those in Table 3, indicating

that these journals contribute to the hst of titles of Class C but share the same Class B journals.

An estabhshed, well-edited journal is not a static and isolated phenomenon. It is an active carrier of information within the community of scientific workers. Thus, a given journal relates to a family

193
of journals by referring to them and in turn serving

as a source for others. There is a two-way flow

of information between any two journals which is a measure of their correlation.

In our analysis, we shall use the following notation: Jm m =0, 1, 2, 3. ... k represents a list of journals. Jo is the definitive journal.

Jmn is the percentage of references in Jm to J„- We can construct a matrix that shows the flow of

information between the individual journals in the

list. Figure 1 is a schematic representation of

such a matrix.

A column such as Jzn {m = 3, n variable) repre- sents the distribution of references in J3 among a

list of n journals, Jn . A row such as Jm% (m variable, n = 3) represents the references of a list of journals,

Jm, to the specific journal, J3 . Jmm, the diagonal of

the matrix, represents in each case the references

of a journal to itself. Thus, Joo refers to the per- centage of references in the definitive journal to

itself.

FIGURE 1. Matrix representation of information flow between

journals.

(See text for meaning of J,„.)

Jo Jl Jz Js Jk

Jo Joo J\o J20 J30 JkO

Jl J01 Jn 72 1 J 31 Jkl

Jo. J02 Jn J22 y.32 Jk2

J, Joa Jn J23 Jm Jk3

A Jol! J\k J2k J3k Jkk

We shall define a family of journals and the posi- tion of each member relative to all others in the

family by means of a matrix such as in figure 1, using percentage of references for the Tmn's. Fig- ure 2 is an illustrative example of such a family.

The numbers in figure 2 are relative percentages

for illustration only and do not represent any

particular case. Referring to figure 2, we generalize

that a family matrix of journals may be generated

by a definitive journal. A journal matrix constitutes a family if it has a strong upper lefthand corner

(Joo), a strong diagonal, a strong upper row, and if each column adds up to about 50 percent. Formally we may characterize a family matrix by the

following:

a. Jmn = Jmo= 15 percent

b. Joo = 2Jmn = 30 percent

m=constant

c. ^ Jmn = 50 percent

n=0, 1,2 . . . . k

{m is any member of the family and n includes all the other members ending at Jk.) We can define several classes of journals within

the matrix (refer to fig. 2). Class 1. Jo the definitive journal, as previously

defined.

Class 2. Ji, J.i, Jz'- a group of journals that, in addition to being strongly coupled to Jo, are also strongly mutually coupled within themselves.

In this region Jmn = J„mClass 3. J4 , J5 : a group of journals that refer strongly to J0 and to Jt-3 but are not strongly re- ferred to by others. Jmn , however, is strong.

Class 4. All others, J6 -9 . These journals do

not satisfy the conditions for inclusion in this particular family. Within this last group we note three

phenomena depending on the magnitude of Jmm:

FIGURE 2. Illustrative example ofjournal family matrix \jm

Jn\ Jo J, 7= y3 74 75 Je y7 y8 79

Jo 30 15 15 15 15 15 0 0 0 0

Ji 5 15 5 5 5 5 0 0 0 0

J2 5 5 15 5 5 5 0 0 0 0

Jz 5 5 5 15 5 5 0 0 0 0

74 1 1 1 1 15 1 0 0 0 0

Js 1 1 1 1 1 15 0 0 0 0

Je 0 0 0 0 0 0 15 0 0 0

Ji 0 0 0 0 0 0 0 0 0 0

j* 0 0 0 0 0 0 0 0 30 15

Jo 0 0 0 0 0 0 0 0 5 15

a. ^66— 15 percent: although J6 does not fit into

this family, it may well fit into some other family.

b. 777 — 0: the expectation is low that J7 will fit into any family matrix.

c 7*88 = 30 percent: J% is very likely to act as Jo

for a new family and indeed is showing signs of

starting the family with Jg. Figure 3 is a family matrix of actual journals.

The main difference between it and the illustra- tion of figure 2 is that the boundaries between the

classes are gradual transitions rather than sharp

fines. This is of course to be expected in the case where definitions depend on statistical properties.

The regions are nevertheless recognizable and the

family structure clear. 194
Referring to figure 3, we note the strong diagonal

since we chose only journals of some character

and standing in the field. The family matrix is

: generated by the Physical Review. {Jo}- Zoo = 47

percent. A strong Jmo row extends from Jo to J15 where we have drawn the family fine. Ji to Jg

represent the Class 2 journals, namely, strong contributors and receptors of information within the

family. J$ to /13 are strong receptors but negligible

contributors. (Note, however, that Jmm is still

Figure 3. Reference matrix of a family ofjournals

Jo 7, 72 7s 74 75 7e Ji Js 79 7io 7n 7,2 JlS 7>4 7,5 7l6 7,7 7l8 7l9

REFERENi

TO

:es

FROM

»-

Phys.

Rev.

Proc.

Phys.

Soc.

Phys.

Rev.

Letters

J.

Appl.

Phys.

Sov.

Phys.-JETP

Physica Nuovo

Cimento

Zeit.

Physik

Progr.

Theor.

Phys.

Sov.

Phys.

—

Sol.

State

Can.

J.

Phys.

Czech

J.

Phys.

Phys.

Fluids

J.

Phys.

Soc.

Japan

Proc.

Roy.

Soc.

J.

Chem.

Phys.

Can.

J.

Chem.

J.

Chem.

Soc.

J.

Phys.

Chem.

J.

Am.

Chem.

Soc.

7o Phys. Rev. 47.2 34.1 28.4 14.5 18.5 15.8 25.0 19.7 29.8 12.8 15.1 12.3 8.7 15.4 6.9 12.8 1.3

Ji Proc. Phys. Soc. 2.0 9.4 1.2 2.4 4.3 1.0 2.5 1.1 1.2 2.0 2.0 3.7 2.9 h Phys. Rev. Letters 12.6 1.6 29.5 1.8 2.5 1.7 14.4 4.3 13.7 2.6 1.0 2.0

J3 J. Appl. Phys. 1.3 2.4 1.8 23.0 2.1 1.4 3.5 3.4 2.6 1.0 L Sov. Phys.-JETP 2.8 2.6 1.3 32.0 3.6 2.2 1.1 8.3 2.5 1.0

7s Physica 1.1 21.5 1.7 2.6 1.5 1.2 2.2 1.3

7e Nuovo Cimento 4.0 1.6 4.5 3.1 21.4 8.4 2.0 1.6

Jt Zeit. Physik 3.1 3.0 20.4 1.0 2.5 1.4

7s Progr. Theor. Phys. 1.5 3.7 25.7 h Sov. Phys. — Sol. State 23.8 1.8

J10 Can. J. Phys. 9.1

Ju Czech. J. Phys. 12.5

7,2 Phys. Fluids 19.5

713 J. Phys. Soc. Japan 1.0 16.2

7.4 Proc. Roy. Soc. 1.8 6.2 1.1 2.7 4.3 1.7 2.5 2.0 1.0 4.8 4.0 3.3 14.7 3.3 1.1 1.2 2.1

7l5

7l6

J. Chem. Phys. 3.9 1.1 2.1 5.0 2.1 3.0 13.1 5.4 6.5 33.4 8.4 1.1 7.9 5.4

Can. J. Chem. 12.3 1.3 1.1

7,7 J. Chem. Soc. 2.7 10.2 25.4 3.7 8.0

7l8 J. Phys. Chem. 1.3 1.4 3.0 12.8 2.4

7,9 J. Am. Chem. Soc. 2.0 3.8 6.3 17.6 19.4 22.2 39.2

strong.) Within the family each column, ^Jm?i,

n=i, 2, . . . adds up to about 50 percent. Journals outside

this family include 7*19 which shows signs of start- ing a new family extending up to7*i 4 . Two journals,

7i4 and /is belong to both families.

It is our hypothesis that the location of a journal

in a family matrix is a quantitative measure of the

probability that the journal will carry a specific

type of information.

195
TABLE 1. Statistical Summary of Citation Sources in Physical

Review

Material examined: Physical Review, Vol. 77, 1950 to Vol. 112, 1958 inclusive. Total number of articles: 8521

Total number of journal titles referred to: 805

Total number of references: 137, 108 of these

68,162 references were to the Physical Review.

11,695 were to private communications and unpublished works.

9,191 to books.

1,929 to reports and memoranda.

296 to theses. 4,252 to Reviews of Mod. Physics. 3,725 to Proc. Roy. Soc. (London).

7,072 to 3 titles each used 2000-2999 times. 12,957 to 9 titles each used 1000-1999 times. 12,377 to 43 titles each used 100-999 times. 1,642 to 25 titles each used 50-99 times. 1,107 to 32 titles each used 25-49 times. 1,304 to 79 titles each used 10-24 times. 595 to 88 titles each used 5-9 times. 523 to 519 titles each used 4 times or less.

Table 2. List of journal titles cited in Physical Review, Vol. 77-Vol. 112

(Arranged in order of decreasing frequency) Order FrelIX1 1Hill 1 rn f~if>r L/C ' quency Source Title

I 68,162 Physical Review

2 11,695 *Private Comm., Unpublished, To Be Published

3 9,191 *Books

d4" 4,252 Revs. Mod. Phys.

c: o 3,725 Proc. Roy. Soc. (London)

5 2,473 Z. Physik

; 2,459 Proc. Phys. Soc. A (London)

g 2,140 Phil. Mae.

g 1,929 * Reports, Technical Memos

10 1,831 Rev. Sci. Instr.

] I 1,796 Physica

12 1,724 J. Chem. Phys.

13 1,662 Bull. Am. Phy-s. Soc.

1 1 1,473 Nature

15 1,330 Nuovo Cimento

16 1,096 Helv. Phys. Acta.

17 1,023 Ann. Physik

1

1,022 Progr. of Theoret. Phys. (Japan)

1') 867 J. App. Phys.

20 755 Compt. Rend.

21 741 Kgl. Danske Vidensdab. Selskab. Mat-Fys

Med

22 586 Z Natur Forsch

23 567 Can. J. Phys. 24 539 J. Phys. et. Radium

25 518 Proc. Camb. Phil. Soc. 26 443 J. Phys. (USSR)

27 418 J. Exptl. Theoret. Phys. (USSR)

28 416 J. Am. Chem. Soc. 29 352 Nucleonics

30 336 Astrophys. J. 31 321 J. Opt. Soc. Am.

32 320 Physik Z

33 313 J. Phys. Soc. (Japan) 34 296 Arkiv Fysik 35 296 *Theses 36 249 Ann. Phys.

37 244 Nuclear Phys. 38 237 Proc. Nat. Acad. Sci. U.S. 39 223 Naturwiss

40 222 Bell System Tech. J.

II 209 Acta Cryst. 42 208 Proc. Inst. Radio Engrs.

1.1 202 Arkiv. Mat. Astron. Fysik

*Nonperiodic Literature.

Table 2. — Continued

Order Fre- Number quency Source Title

AA 1 Oft Trans. Roy. Soc. (London)

A^4-0 i1 ony\J Can. J. Research

AA4-U lOO soviet rnys-jHiir Al4 /

1 A/1 104 J. Research Nat. Bu. Stand.

4-0 1lOUfX\ Physik. Z. Sowjetunion

AQ 1iO^7I Repts. Prog, in Phys. ou^0 11DO^3 Science

Ol 140 Z. Physik. Chem.

^9 14-U Xrans. Faraday Soc. Oo^3 1loo33 Acta Metailurgica 04^A 11ZU90 J. Phys. Chem.

00 lift 1 lo J. Phys. and Chem. Solids OU 116 A m \ l-'niro hi. j . r iiys. D t 1 1

Proc. Indian Acad. Sci. JO^ft 1UO 1 Oft Proc. Phys. Math. Soc. Japan

COoy 1IU07/ Proc. Am. Acad. Arts and Sci.

1 07 1U / Ann. Rev. Nuclear Sci. aoOU iuo Leiden Comm.

a i yy Philips Research Repts. AOOZ OQyo Znur. Lksptl. 1 leoret. riz. Oo oo 7 A II All „ fL

A. Anorg. 11. Allgem. f.liem.

A/1 04 QAo4 J. Llectrocnem. ooc. OO RO Terrestrial Magnetism and Atm. Elec. uu 77 niin. iviain. 77 J. Franklin Inst. 67 76 Z. Krist. 68 74 Advances in Phys. oy Aft oo Proc. Acad. Sci. Amsterdam

Aft OO Discussions Faraday Soc. 70 AAOO Proc. Roy. Irish. Acad.

- 1 1 1 OO Trans. Am. Inst. Mining Met. Engrs.

79 U

J. Geophys. Research

73to AOOU Nachr. Akad. Wiss. Gottingen Math. Physik VI

1A U7 RCA Review

7^/ D C7O ( j. ivietais ^7O I Sci. Repts. Tohuku Univ.

lf\ i <J OO^3 Monthly Notices Roy. Astron. Soc. OO^3 J. Inorg. Nuc. Chem.

77 Ol Z. Electrochem.

51 Australian J. Phys.

O 1 v^iompL rvena. rvcaa. oci. un.oo

7ft ou Ricerca Sci. OU^0 Indian J. Phys.

7Q tu J. Sci. Instr. OU AO4Z lzvestia Akaa. iNauk. 5o5n oer. riz.

< . 1 A 1 41 Sci. Papers Inst. Phys. Chem. Research

(Tokyo)

ft9 OZ oy3G J. lecn. rnys. (U.o.o.rv.j oV3Q Z. Astrophys.

OQOO OO3ft J. Nuclear Energy

ft 1 OU36 J. Acoust. Soc. Am.

36 (

' q r, T \T it \~i V_. a1 1 . J . IV! dill. OO o4 J. Atmos. Terr. Phys.

3A04 Anal. Chem.

o43d. Proc. Roy. Acad. Sci. (Amsterdam)

RAOO oo33 Australian J. Sci. Research

87 OZ39 i3nt. j. Appi. rnys.

QQ< to 31o 1 Z. Tech. Phys.

31ol Nuclear Science Abstracts

ol31 Ann. Ann TV V \ r.*A Q^I in. i. Acaa. oci. 031

Appl. Sci. Research

ftQ oy oU30 J. Am. Ceram. Soc. 30oU Proc. Koninkl. Ned. Akad. Wetenschap

onyu zyOO Sci. Repts. Research Insts. Tohoku Univ. OOzy Geochim. et Coschim. Acta

Ql Zo Prog. Nuclear Phys. zo9ft \/uarr. /vppi. lviatn. 28 Acta. Phys. Polonica 28 Ergev. Exact. Naturw.

28 Wien. Ber. II A

92 27 Rec. Trav. Chim.

93 26 Proc. Am. Phil Soc. 26 Am. Mineralogist 26 J. Electronics

196
TABLE 2.— Continued

Order Fre- Xumber quency Source Title

Table 2. — Continued

Order Fre- Number quency Source Title

94

95

96

97

98

99

100

101 102

103

104

105

106

107

108

109

25 J. Chem. Soc. 25 Gen. Elec. Rev.

24 J. Phys. 24 Z. Metallkunde

24 Trans. Electrochem. Soc. 23 Ind. Eng. Chem.

23 Zhur. Tekh. Fiz. 23 Optik. 22 Proc. Am. Acad. Sci. 22 Acta Chem. Scand.

22 Nachr. Ges. Wiss. Gottingen 22 Kgl. Norske Videnskab. Selskabs. Skrifter 21 Anais. Acad. Brasil. Cienc.

21 Elec. Eng.

21 J. Inst. Metals

21 Acta Phys. Austriaca

21 Communs. Phys. Lab. Univ. Leiden

20 Verhandl. Deut. Physik. Ges. 20 Acta Physicochim. U.R.S.S. 20 Kgl. Fysiograf. Sallskap. Lund. Forh. 20 Commun. Pure and Appl. Math. 20 Trans. Am. Math. Soc. 20 Arch. Sci. Phys. et Natur. 20 Proc. London Math. Soc. 20 Can. J. Chem.

19 Arch. Elektrotech. 19 Sitzber. Akad. Wiss. Wien. Math.-Naturw.

Ki.

19 J. Math. Phys.

19 Math. Ann.

18 Atti. Accad. Natl. Lincei 17 Physics

17 Phys. Today

17 Acta Phys. Acad. Sci. Hung.

17 Cahiers Phys.

17 J. Chim. Phys.

17 Proc. Inst. Elec. Engrs. Ill 17 Acta Mat.

16 Philips Tech. Rev.

16 Proc. Roy. Soc. (Edinburgh)

16 Proc. Natl. Inst. Sci. India 16 Ann. Chim. Phys.

16 Chem. Revs.

16 Ann. lost. Henri Poincare

15 Busseiron Kenkyu

15 Observatory

15 Ann. Geophys.

15 Wireless Engr.

15 Sitzber. Preuss. Akad. Wiss., Physik-Math Kl.

15 Phil. Trans. Roy. Soc. (London)

14 Electronics

14 Phil Mag. Suppl. I 14 Am. J. Roentgenal Radium Therapy

14 Communs. Kamerlingh Onnes Lab. Univ. Leiden

13 Astrophys. Norv.

13 TeUus

13 Z. Angew. Phys.

13 Nova Acta Reg. Soc. Sci. Ups.

12 Pubis. Astron. Soc. Pacific 12 Bull. Soc. Franc. Mineral

12 Mem. Soc. Roy. Sci. Liege

12 Rept. Ionus. Research Japan

12 Quart. J. Math

12 Nuovo Cimento Suppl.

12 Current Sci. 12 Bureau Standards J. Research

12 Duke Math. .]. 11 Bull. Astron. Netherlands

10 Trans. Am. Soc. Metals.

10 Technol. Repts. Osaka Univ.

10 Ned. Tijdschr. Natuurk.

10 Ann. Rev. Phys. Chem.

10 Rend. Reale Accad. Nazi. Lincei

10 Comm. Leiden.

10 Radiology

10 Atti. Congr. Intern. Fis. Como

10 Brit. J. App. Phys. Supplement

10 Acta Phys. Hung.

10 Preuss. Akad. Wiss. Berlin. Ber. 10 Ann de Physique

10 Atominaia Energya

10 Soviet Physic Doklady

110 9 Ann. Astrophys.

9 Rept. Inst. Sci. Tech. Univ. Tokyo

9 J. Aeronaut. Sci. 9 Cent. Bras. Besq. Fis. (Notas de Fisica) 9 Advances in Electronics

9 Trans. Roy. Soc. Can. Ill 9 Nuclear Instr. 111 8 Trans. Am. Geophys. Union

8 J. Math, and Phys.

8 Kgl. Norske Videnskab. Selskav. Forh. 8 Trans. Am. Inst. Elec. Engrs.

8 J. Geomag. and Geoelec. 8 Metal Progr. 8 Am. J. Math.

8 Verhandel. Koninkl. Akad. Wetenschap

Amsterdam Afdeel Natuurk.

8 Czechoslov. J. Phys.

8 Brit. J. Radial. 8 Appl. Spectroscopy

8 J. Iron and Steel Inst. 8 Sorysiron Kinkyu

8 Phys. Chem. Solids 8 Nuclear Sci. and Eng.

8 Phys. Fluids 112 7 Chem. Weekblad

7 Arch. Math. Naturvidenskab.

7 American Scientist

7 J. Sci. Research Inst. (Tokyo)

7 J. Sci. Hiroshima Univ.

7 Bull. Inst. Nuclear Sci. Belgrade

7 Ber. Deut. Chem. Ges.

7 Skrilter Norse Videnskaps-Akad. Oslo I MatNatur. Kl.

7 Trans. Am. Soc. Mech. Engrs.

7 Sylvania Technologist

7 J. Washington Acad. Sci.

7 Rev. Mex Trs

7 Trans. Am. Inst. Mec. Engrs.

7 Ann. Radioelec Compagn Gen de T.S.F.

113 6' Bull. Akad. Sci. URSS 1 6 Actualities Sci. et Ind. 6 Naturw. Anz. Ungar. Akad. Wiss. 6 Zhur. Fiz. Khim.

6 J. Phys. and Colloid Chem.

6 Amer. Math. Mon.

6 Proc. Leed Phil. Lit. Soc. Sci. Sect. 6 Arkiv. Kemi. Mineral. Geol. 6 Experientia

6 Progr. Metal Phys.

6 J. Proc. Roy. Soc. (N.S. Wales)

6 Encykl. D. Math. Wiss.

6 Am. J. Sci. 6 Uspekhi Fiz. Nauk.

6 Elec. Comm.

6 Bull. Am. Math. Soc.

6 J. Colloid Sci. 6 Geofus Publ

6 Soviet J. Atomic Energy

6 IBM J. Research and Development

114 5 Proc. Intern. Conf. Refrig. 5 Bull. Soc. Chim.

5 Z Hochfrequenz

5 Akad. Wiss. Wien.

5 Festschr. Akad. Wiss. Gottinger MathPhysik Kl

5 Kolloid-Z. 197
Table 2. — Continued

Order Fre- Number quency Source Title

5 Z. Angew. Math. U. Mech.

5 Abhandl. Braunschweig. Wiss. Gen.

:> J. Ind. Eng. Chem.

5 Akad. Nauk. S.S.S.R. Ceram. Age.

5 Svensk. Kern. Tidskr.

5 Kgl. Svenska. Vetenskapsakad. Handl.

5 Ilium Engr.

5 Ann. Univ. Grenoble

5 Wiss. Veroffentl. Siemens-Werke

5 Bull. Soc. Roy. Sci. Liege

5 Ann. Math. Stat. 5 Carnegie Inst. Wash. Publ. 5 Physik Bl.

5 Radiation Research

5 Memoirs and Proceedings of the Manchester

Literary and philosophical Soc.

5 Wied. Ann. J. 5 Chinese J. Phys.

5 Astron. J. 5 Phil. Trans.

5 Fortschr. Physik

5 J. Rational Mech. and Anal.

5 Rocqniki Chem.

5 Univ. I. Bergen Arbak. Naturvidenskap. Rekke

5 Soviet Phys-Tech. Phys.

Table 4. Incremental growth of the list of cited journals as new journals are examined

(This table illustrates the stability of the most cited journals in the physics literature outside the Physical Review.)

Total number Number of Source journal of citations citations to new* titles

Phys. Rev 1120 10 Phys. Rev. Letters 1004 8

Proc. Phys. Soc. 1000 27

Z. Physik 1000 23 Physica 379 19 JETP 1011 18

Jn. Phys. Soc. Japan 1250 57 Can. J. Phys. 996 43

Prog. Theor. Phys. 1016 10 Czech. J. Phys. 476 16 Nuovo Cimento 996 8

Rev. Sci. Instr. 839 32

Jn. Appl. Phys. 1002 26

Phys. Fluids 956 32

Sov. Phys. Sol. State 1000 44

Philosophical Mag. 1000 34

*Citations of titles not encountered in Phys. Rev. Vol. 77-112.

Table 3. Incremental growth oj the list of cited journals as new

issues are examined

iThis table shows that a relatively small number of sources account for most of the references found in the Physical Review.)

Phys. Rev. Number of Number of Number of volume new titles times cited times cited in cited in this vol. Vol. 77-112

77 (1950) oCO 1517 107,385 78 40 57 1,025 79 29 42 605

80 27 35 249

81 (1951) 18 26 662

82 21 28 163 83 .30 49 987

84 19 19 126

85 (1952) 12 13 81 86 9 12 47

87 12 18 150 88 28 38 340

89 (1953) 13 14 57 90 20 21 72

91 24 29 137 92 17 20 183

93 (1954) 18 23 57 94 21 28 138 95 14 15 32

96 10 15 50

198
Evaluation Methodology, and

Criticisms
Isi
An Evaluation Program for Associative Indexing 1

Gerard Salton

Harvard University

Cambridge, Mass. 02138

Statistical association techniques have been widely used in information retrieval to relate items of information such as documents or words occurring in documents. The desired relationships between

the given items are normally determined by means of a variety of different criteria, including in particular the co-occurrence of words in documents, the similarity in bibliographic citations, and the

identity of authorship. Associative techniques are particularly useful as a means for adding to the index terms attached

to a given document, a number of new, related terms. Such associated terms then effectively broaden

the scope of the original terms in such a way as to increase the number of relevant documents retriev- able in response to a specific search request. Word associations can therefore be used in an adaptive

retrieval system in which requests for information are successively altered until a satisfactory response

is obtained. One of the difficulties which beset associative systems is the problem of evaluating the effective- ness of the procedure. Specifically, it is not clear whether an improvement in retrieval is actually obtained by using term and document associations, or whether equally effective results might not be generated with a small thesaurus, or synonym dictionary, used to normalize the vocabulary.

An adaptive information retrieval system is presented which can be operated with or without a synonym dictionary, with or without term and document associations, and with or without a hierarchical subject arrangement. By processing the same search requests under a variety of different modes it is possible to compare the relative effectiveness of the various automatic methods without large-scale human effort. The retrieval system is described in detail, and test results obtained by processing a sample document collection on the 7090 computer are exhibited.

1. Introduction

Within the last few years the design of automatic

information systems has become increasingly complex, and so have the techniques which are used to analyze and manipulate the information. As more

and more different types of systems are proposed

and generated, the evaluation of these systems

becomes of increasing urgency. Unfortunately, no real guidelines are available which could be used

in the design of evaluation procedures, and most

of the methods actually proposed are based on ad hoc rules which stress theoretically desirable

features, and do not concern themselves with practical questions. As a result, much of the proposed

methodology cannot, in fact, be implemented

reasonably in a test situation. In the present report, an evaluation program is outlined which is believed to be both useful and

practical. No attempt is made to treat all aspects

of a retrieval system; the program confines itself, instead, to the evaluation of retrieval techniques,

including methods for analyzing document and in- formation content, and methods for the comparison

of stored information with search requests. Specifically excluded from the testing process are operational criteria such as cost, access time, re- sponse time, and so on, since these factors are not

of immediate interest in experimental automatic

information systems.

us study was supported by the National Science Foundation under grant GN-82.

Furthermore, in order to circumvent the diffi- culties which arise from the dual, and probably

incompatible, requirements of demanding, on the one hand, an absolute standard against which the

performance of each retrieval system is to be compared, and of insisting, on the other, that the user himself be the ultimate judge in deciding what part

of the retrieved information is to be relevant to any

given request, the evaluation procedures described

here are based on relative measures of system effec- tiveness. In particular, an attempt is made to rank the various retrieval procedures as a function

of their excellence in performing certain desired

tasks without, however, specifying how far removed

each performance is from some optimum standard.

Such a relative evaluation process cannot then be

used to design an ideal system, but will make it possible to choose from among a set of available

procedures the one which may be expected to render

the best performance in a given situation. Moreover, the use of a relative standard of excel- lence makes it unnecessary manually to produce

an index of relevance for each document with

respect to each question, and permits instead a

largely automatic testing procedure. This in turn

implies that the tests can be performed on relatively

larger collections of stored information than is possible in a purely manual operation, thus insur- ing a reaso'^ible statistical base for the test results. In addition, since the cooperation of large numbers

of persons over long periods of time is no longer

H)\

772-957 0-66— 14
needed, one of the basic weaknesses built into conventional testing systems — namely the variability

of the environment — is now removed.

The principal criteria used in the design of the

testing procedure are outlined in the next section;

the system itself is briefly described in section 3; and some of the many possible testing routines are

listed in the concluding section. 2

2. Evaluation Criteria

A number of diverse systems for the identification

of stored information have come into general use

within the last several years. The first and most

widely known is the key word system in which certain terms, manually chosen or automatically ex- tracted from the body of documents, are used for purposes of information identification. These

terms are normally assumed to be independent in the sense that they do not exhibit relations among

each other, and may be chosen from a controlled

vocabulary, or else may be completely free. In a key word system, the information relevant to a given search request is identified by comparing,

respectively, the term sets representing stored

information with the term sets representing infor- mation requests.

In order to eliminate the variations resulting from

an uncontrolled vocabulary, and to supply some of

the more obvious inclusion and generic relations

between terms, a synonym dictionary, or thesaurus,

is often introduced. Key words, chosen as before, are then looked up in the dictionary and replaced

by the corresponding thesaurus heads before being

used as information identifiers. Within the the- saurus, the items may be hierarchically arranged

in such a way that terms appearing "high up" in the hierarchy (near the roots of the corresponding

abstract tree structure) are general terms which

are generically related to the more specific terms

listed under them on a lower level. Such an arrangement makes it possible to use the thesaurus

for a variety of term expansion procedures, as will be seen. Additional relations between key words may also be taken into account by using for purposes of document identification clusters or phrases, consisting

of subsets of terms with specified relations between

them (instead of individual key words alone). Such

phrases may again be chosen manually or else may

be generated automatically by a variety of statistical,

syntactic, or semantic techniques. The relations

which obtain between the individual terms within

a cluster may be purely formal ones, such as co- occurrence of words within the sentences of a document, or within the documents of a collection, or

2 Some recent works dealing with the design of testing and evaluation systems for information retrieval are included in the reference list [1, 2, 3, 4, 5, 6, 7]. (Figures in brackets indicate the literature references on p. 210.)

3 The precision ratio of a search is that fraction of the retrieved documents which is in fact relevant to the user's request; the recall ratio, on the other hand, is that fraction of all the relevant documents in a collection which is in fact retrieved [7].

4 It is an unfortunate fact that recall and precision ratios cannot, in general, both be improved simultaneously, because as recall increases through retrieval of additional relevant material, more irrelevant matter will also be produced, thus decreasing precision; similarly, as precision improves through decrease in the amount of irrelevant material, recall may deteriorate because some of the newly missing material may origi- nally have been relevant [5, 7].

else they may be described in very specific terms,

such as cause-effect or whole-part relations; in the

latter case, extensive syntactic and contextual

analyses may be needed to identify them. Relevant

information in such a system is retrieved by more or

less complicated phrase-matching procedures.

In addition to information extracted from the text of documents, or supplied by auxiliary dictionaries

and tables and by various analytical procedures, it is often convenient to use a number of related sources for purposes of information analysis. Thus

it is possible, under certain circumstances, to uti- lize contextual criteria such as the date of a publication, the name of the author, the references

cited in the bibliography of each document, and

other related indicators.

In a typical retrieval situation, the user is first given some indication of the parameters within

which the system operates, and is then free to formulate any acceptable search request. In response to each request, the system then furnishes a certain set of items which is considered relevant

to the respective requests. The user may now find himself in one of three situations:

(a) the information retrieved is in general satis- factory, and there is no need to rephase the request;

(b) the information retrieved is not satisfactory

because too much irrelevant material is included

(the precision ratio 3 of the search is too low);

(c) the information retrieved is not satisfactory

because too little relevant material is included

(the recall ratio 3 of the search is too low). In the last two situations the user will want to rephrase his search request in an attempt to obtain

a more nearly satisfactory answer. Specifically,

to improve the precision ratio it is necessary to narrow the scope of the terms used to specify the

search request, and to tighten the criteria used to match the stored information with the requests for information. Contrariwise, to improve the recall

ratio the search specifications must be broadened,

and the matching criteria between the respective

sets of terms relaxed. 4 In a practical, useful retrieval system, the follow- ing types of operations are then seen to be of

primary concern:

(a) the construction of matching procedures

which would make it possible to produce successively more and more relevant, or less and less irrelevant, material in answer to a given search

request;

(b) the generation of term expansion and contraction methods which could alter the coverage of

the original terms used to specify a search request

202
by addition, deletion, or modification of terms, in such a way as to produce response alterations in the desired direction;

(c) the assembly of a variety of methods of the

kind described under (a) and (b) into a unified,

flexible retrieval system.

The discussion at the beginning of the present

section indicates that a considerable number of

different methods have already been proposed for

! the automatic identification of search requests and

stored information. Adaptive matching techniques

which can be used to compare items under more or

less stringent conditions have also been generated

[8]. The difficulty which arises in the actual implementation of retrieval systems is that very little is known about the precise effect of each of the many

possible steps which may be taken in a given situ- ation. For example, which of many possible correlation coefficients should be used to measure the

similarity between sets of key words? Given a

specific correlation coefficient, what cutoff point

should be chosen to distinguish relevant from ir- relevant information? How much more (or less)

information is retrieved by replacing each original

key word by a more general (or a more specific)

one? Is it better to use a synonym dictionary or a

statistical association method for the expansion of

index terms? And so on.

In the next section, a retrieval system called

SMART is described which is believed to be useful

in answering questions of this type. The SMART

system makes it possible to process data in dozens

of different modes by calling into play different methods for the determination of information content, different criteria for matching items of stored

information, and different ways of specifying the

information requests. This system may be used

for the evaluation of retrieval techniques by proc- essing the same search requests and the same document collection several times and effecting each

time a slight change in the processing conditions.

To evaluate the effect of a certain processing tech- nique it is then sufficient to concentrate on the

differences in output produced by two search operations in which the given technique is used in one

case but not in the other. This is further described

in section 4 of this study.

3. The SMART Retrieval System [9]

A simplified flowchart of the complete system is shown in figure 1. The system is seen to consist

of a sequence of largely optional, text-processing

routines, including dictionary lookup processes,

statistical correlations, and syntactic matching

procedures. Documents consisting of English

texts, as well as search requests, are submitted to

Incoming Text or Search Request

Dictionary lookup to obtain

syntactic and semantic labels

I Expansion of semantic labels 1

I through search in concept hierarchy

•"Computation of sentence significance ' and automatic sentence extraction

] Syntactic analysis of significant

| sentences and structural matching

| with criterion phrases

[Expansion of semantic labels ~

^through statistical term correlations | jComparison of search request with

I document identifications and possible

document correlations

/

/

^optional steps

'compulsory steps

Figure L Simplified SMART system.

the same process and a complete run consists of a sequence of text manipulations including input

operations of new texts, and matching operations

between certain specified texts (the search requests)

and all other texts. The system is designed around a monitor called

CHIEF, which can in turn call on many different subroutines. The monitor accepts input instructions to specify the type of operation to be per- formed, and control data to choose the subroutines

which are to be called. At the present time, four

basic input operations are available and about 35

different processing options. The processing

options fall into seven basic categories: general

processing methods, alphabetic dictionary pro- cedures, operations using the semantic concept

hierarchy, statistical correlation options using co- occurrence of terms within sentences, syntactic

prodecures using a phrase dictionary and structural

matching methods, statistical term correlations

using co-occurrences within documents, and document-matching procedures.

Four basic dictionaries or tables are used by the

system: an alphabetic-stem dictionary designed to supply each word stem with a number of syntactic

and semantic codes, an alphabetic-suffix table to obtain syntactic codes for word suffixes, a numeric

concept hierarchy to represent various relations

between semantic categories, and a criterion-phrase

dictionary to aid in the syntactic processing.

3.1. The Alphabetic Dictionary Programs

The input texts are first segmented by identifying

the individual words of the texts and noting the

203
sentence number and text code for each word. The

individual words are then looked up in an alphabetical dictionary to supply each word found with

both syntactic and semantic codes. The alphabetic

dictionary consists actually of two parts: a stem

dictionary and a suffix dictionary, and both parts are stored in list form. An attempt is made by a dual left-to-right and right-to-left letter-by-letter

scanning procedure to find a match between each

input word and the respective entries in the stem

and suffix dictionaries. When a match is actually

found, the semantic concept codes and the syntax

codes included in the dictionary are used to replace

the alphabetic characters which specify the input

word.

The importance of the dictionary lookup proce- dure is threefold: first, it reduces the dependence

of the various procedures on the vocabulary of the original texts by assigning the same concept

numbers to a variety of synonymous expressions;

second, it permits the remainder of the process to be carried out with standardized numeric codes

instead of with variable alphabetic information;

third, a replacement of the original words by con- cept codes tends to broaden the coverage of each

term and therefore affects the retrieval action, as

will be seen.

For purposes of comparison and evaluation, it may in some circumstances be desirable to operate

with the original input words. Provision is therefore made to substitute for the alphabetic stem dic- tionary a simulated vacuous dictionary. This

dictionary includes no entries initially, but is con- structed during the "lookup" operation by entering

in the dictionary every occurrence of a new word

found in the input text, together with a fictitious "concept" code. Each new word type is thus

assigned a different concept code, so that a one-to- one correspondence exists in the simulated dic- tionary between dictionary entries and concept

codes. When the simulated dictionary is used,

the statistical correlation programs, while still technically operating on numeric concept numbers,

are in fact then associating the original alphabetic

text entries. An excerpt of a text, including both real concept

numbers as well as simulated dummy numbers,

is shown in figure 2. It is seen that the actual

concepts are assigned to a variety of different words,

whereas the simulated numbers are repeated only

if the corresponding word is repeated also. Highfrequency function words are not assigned any

concept numbers.

A 113 V. H. Yngve, "A Framework for Syntactic Translation, 1 Mechanical Translation, 4, pp. 59-65 (December 1957).

1 APPROACHES TO MECHANICAL TRANSLATION BASED ON ADEQUATE

(230)45 (119)46 (98)5 (15, _16)47 (64)48

("STRUCTURAL DESCRIPTIONS OF THE LANGUAGES INVOLVED AND ON AN

| (107)49 (57)50 (102)51 (73)52

| ^

• ADEQUATE STATEMENT OF EQUIVALENCE ARE DISCUSSED.

(64)48 (181, 57, 32, 245)53 (94)54 (230)55

I t * I

2 TRANSLATION IS CONCEIVED OF AS A THREE-STEP PROCESS:

(98)5 (230)56 (NF)57 ( 147)58

^ — — — — — ^ RECOGNITION . . . TRANSFER . . . (123)59 (22,42)64

(a) : concept numbers

a : dummy concepts

» FIGURE 2. Excerpt of typical abstract.

3.2. Processing of the Concept Hierarchy

Whereas the lookup in the alphabetical dictionary,

real or fictitious, is compulsory since the numeric

concept codes must be obtained in one way or another, all operations involving the concept hier- archy are entirely optional. If no hierarchy is available, these operations can be skipped. The

concept hierarchy is a treelike arrangement of

numeric concept numbers as illustrated in the

simplified excerpt of figure 3. Each node in figure 3 represents a concept number, and the hori- zontal dashes next to the nodes symbolize the text words which are replaced by the corresponding

concept numbers during the dictionary lookup.

Associated with a given concept appearing in the

201
o =

FIGURE 3. Hierarchical concept dictionary with cross references.

hierarchy are more specific concepts which appear

on a lower level in the hierarchy, more general

concepts which appear on a higher level, and cross- referenced concepts which appear on the same

level. Thus, when a concept number is obtained

as a result of the lookup operation in the alphabetical

dictionary, it is possible to enter the hierarchy in order to obtain a number of related concepts or, alternatively, more general or more specific ones. The hierarchy is stored in the computer as a

multiply-chained list, and list processing operations

are used to obtain the "parent" of a given node on

the next higher level, the "brothers" on the same

level, the "heirs" on the next lower level, and the

cross references. Each concept may be said

to "include" other concepts located on lower levels, or to "be included" in concepts situated on higher

levels: no such inclusion relation is implied,

however, for the cross references. In the SMART

system, search requests as well as document specifications may be broadened by moving upward in the hierarchy or restricted by moving downward,

and related concepts are picked up through the

cross-reference fists.

3.3. Statistical Concept Associations

The text-segmentation and alphabetical-dictionary

lookup programs furnish for each sentence a list of all the included concept numbers. An inverse

sort followed by a simple counting procedure can

then be used to obtain for each concept a list of the

corresponding sentences, as well as the frequency

of occurrence in each sentence. This in turn per- mits the construction for each document of a concept- sentence incidence matrix in which the ijth element

is set equal to n if sentence j contains concept i ex- actly n times. A typical concept-sentence inci- dence matrix is shown in figure 4. In the same manner, it is possible to take the

sets of concepts attached to each document within

a complete document collection and to form a single

concept-document matrix. The ijth element in such a matrix is set equal to 1 if and only if concept

Ti is assigned to document Dj. A typical conceptdocument matrix is shown in figure 5.

—-^Documents

Concepts

^ D

2 Dr

T

l

l cj---- --- c

1r r

Z

l

2 Si

1

1

1

1

cj~~ -- - c

2r

T

s

1

1

1 =; c'---- ... c

'sr

-—^Sentences

Concepts —_^

S

l s2 sm

T 1

1 Cl1 1 Cm

T 2 4 i~" c im

k---- - - - c' nm

FIGURE 4. Concept-sentence incidence matrix for a given document. C = n«—* Sentence S, contains term T, exactly n times.

FIGURE 5. Concept-document matrix for a given document

collection. C

l = 1 *—* Term T, has been assigned to document Dj (otherwise C^' = 0).

To obtain a measure of similarity between a pair of concepts, it is necessary to compute a correlation

coefficient between the two corresponding rows of

the concept-sentence incidence matrix or of the

concept-document matrix. If correlation coeffic- ients are computed for all concept-pairs, a conceptconcept correlation or similarity matrix is obtained

in which the ijth element denotes the strength of

association between concept i and concept /, based

either upon the number of co-occurrences of two

concepts within the sentences of a given document,

or within the documents of a given collection.

205
Concept-correlation options are included in the

SMART system for two principal reasons. First,

it may be desirable to replace a given set of old concepts by a new concept formed of a cluster of

highly correlating original ones. Second, it may

be useful to add to an original concept new ones which correlate significantly with the original. The

clustering procedure is carried out by starting with

a single term, and then adding a new term whose

correlation coefficient with the old one is larger

than a given threshold. To the pair thus formed,

a third term is added whose correlation with both

of the others is significantly high, and so on. Three

types of output may be obtained to represent similarities between terms: the "term correlations"

exhibit all correlation coefficients for a given term;

the "term relations" include only those related

terms which have significant correlation coefficients

with a given term; finally, the "term clusters" in- clude terms which have significant correlations

with all other terms in the cluster.

It may be noted that the generation of new con- cepts formed from sets of old ones is similar in

effect to the concept expansion obtained by means of

the concept hierarchy. The two methods may then

be compared by performing first the one and then

the other and checking results. Options are

available to skip the concept-correlation process

if desired.

3.4. Syntactic Processing

A syntactic-analysis program may be a useful

part of an information-retrieval system since it permits a further refinement of the matching criteria between information requests and document identifications. Specifically, the document sentences and search requests may be analyzed syntactically,

and individual concepts or terms may be clustered

only if the syntactic relationships between con- cepts are identical. Similarly, a phrase or cluster

included in a search request can then be made to match the corresponding phrases included in the

document identifications only if the syntactic

relations also match.

A syntactic analysis program is included in the

SMART system which can transform each sentence processed into dependency tree form. Tree- matching procedures are then used to compare

sentences and sentence parts [8, 9, 10]. Specifically, a dictionary of so-called "criterion phrases"

or "criterion trees" is used. Each entry in this dictionary consists of a set of concept numbers

corresponding to a phrase in ordinary written texts. Typical phrases might be "information retrieval,"

"computer design," "syntactic analysis of phrases,"

and so on. Also included in the criterion-phrase

dictionary are the semantic concept numbers and

the syntactic codes corresponding to the terms

included in each phrase, as well as a specification

of the syntactic connection pattern between the

concepts. A typical criterion phrase is shown in figure 6, including also the syntactic indicators and

semantic concept numbers attached to the nodes

of the phrase.

If the "criterion tree" option is chosen, each of

the previously syntactically analyzed sentences is compared against all entries in the criterion-phrase

dictionary, and those phrases are identified which

match a given part of a sentence. To match, not only must the semantic and syntactic labels compare

properly, but the syntactic connection pattern

must also be the same. Thus, a phrase such as "information retrieval," where the concept "infor- mation" is syntactically dependent on "retrieval,"

would not match the sentence, "Because the text contains secret information retrieval is vital,"

but would match the sentences, "The retrieval of information is necessary," or "He discusses infor- mation and document retrieval." A tree which

matches the criterion phrase of figure 6 is shown in figure 7. A comparison of figures 6 and 7 shows

that nodes (a) and © of figure 6 match nodes (J) and @ of figure 7, respectively, and that the paths

between the nodes are properly preserved.

©

LABELS

(S, O, C)

THESAURUS

CATEGORY 014: INFORMATION,'

DOCUMENT (S), FACT (S), DATUM, DATA,

ETC.

THESAURUS

CATEGORY 017: RETRIEVAL, \ PROCESSING,

ORGANIZATION,

SEARCH, ETC. , (014,023) (017)

FIGURE 6. Typical criterion phrase.

206
©

©

SYNTACTIC

LABELS

SEMANTIC " LABELS THE RETRIEVAL INFORMATION

FIGURE 7. Tree structure which matches the criterion phrase offigure 6.

At the end of the matching process, the criterion

routine furnishes for each document a count of the

number of matches obtained between each criterion phrase and the sentences included in that document.

The concept numbers identifying the criterion

phrases which match sufficiently often can then be

added to the concept lists of the corresponding documents, thus resulting in an expansion of the concept

vectors similar to the expansion previously obtained

through the hierarchy and the statistical correlations. By using the option "no syntactic processing,"

the complete syntactic analysis and the criterion

phrase processing can be eliminated.

3.5. Document Associations and Request

Processing

The programs described for the generation of

concept correlations can be used unchanged to obtain document similarities by performing column

instead of row correlations of the concept-document

matrix. Specifically, one of the documents,

newly introduced or previously included in the

collection, may now take the place of a search re- quest. This special request vector can of course be subjected to the same procedures as the other

documents, including lookup in the alphabetic

dictionary, expansion through the concept hier- archy, and so on. By correlating the request vector with all other documents in the collection, a "rele- vance coefficient" is obtained for each document,

and documents with sufficiently high coefficients can be considered to answer the request. Moreover, given a set of documents obtained in response

to some request, new documents may be added by

using the document-document similarity matrix,

including the correlation coefficients between all pairs of documents, to form document clusters. The clustering techniques are the same as those

used before for concept clusters, and these clusters can be used as an entity in the generation of answers

to search requests.5

4. Test Procedures

The system described in the preceding section can be used to generate document identifications

by a variety of methods. In particular, starting

with a simple term-document matrix of the type

shown in figure 5, it is possible to generate an ex- panded matrix as shown in figure 8, including new

terms derived by hierarchical expansion, syntactic

processing, and statistical associations. The problem is then to find a way for constructing in each

case the most effective possible matrix and the

most useful matching procedure for the comparison

of the matrix columns.

The following general methods are available for

this purpose:

(a) a variety of correlation measures may be used

to compare the similarity between the information

identifications and search requests;

5 Procedures for the generation of term and document associations have been de- scribed in the Uterature and are not repeated here in detail [11, 12 |. Extensions of

the term association, to include bibliographic information, have also been proposed [13].

(b) a variety of coefficient thresholds may be

chosen for each correlation coefficient, so as to increase or decrease the amount of retrieved infor- mation in each case;

(c) the matching procedures may be altered

(without change in the search specification) by

using, for example, binary-term document matrices

instead of numeric ones, or by disregarding various

kinds of relations between terms;

(d) the search specifications themselves may be

modified, for example, by addition or deletion of

terms, or by replacement of original terms by new

ones.

It is seen that each of these four principal proc- essing alterations can be brought into play independently of the other three. Not much can be

said concerning the choice of a useful correlation measure; it is in fact conceivable that, for practi- cal purposes, this step may be of little importance.

In any case, experimentation may indicate that

207
Documents Original Citation and Search

Documents Author Re lations 1 Request

Concepts

' D

1 D ...

2 Dm

TD , m+

D "

1 . . . t

1 R

' T

l ^ 1 1 C

t

| R

1

Original 2 c2

. . . c

2

1 t

1 R

Te rms

• T

n

•cj

1-

n 1 R

n

T

n+1

n+l C

l n+Z °2 ' ' ' _n+2 1

t n+l R

n+i

Hi-srarchy

Expansion ! T

*

I P

!

Syntactic p+1

C

l r

P+l

• c

p+1

| t R

P+1 K

Phrases ' T

'r

9

! ; 4

T , r+1 c

r+1 C

l c

r+1

t R

r + 1

Statistical Associations* T

s 4 cj • • ' c

s

"

t r*

FIGURE 8. Expanded concept-document incidence matrix.

some coefficients are more satisfactory than others;

in particular, everything else being equal, it is most

efficient to use that coefficient which minimizes the

amount of computation to be performed.

One of the simplest ways to increase or decrease

the amount of information produced in response to a given search request is to alter the threshold of the coefficient of correlation used in the matching

process. Clearly, the lower the threshold, the more

information is produced. A change in the cutoff point will not, however, be effective if different kinds of responses are expected, but will affect mainly the number of answers.

Alterations in the matching process itself are most useful in the dictionary lookup operations.

For example, word endings could be disregarded

in the alphabetic-dictionary lookup; alternatively,

syntactic codes might be deleted as a matching

criterion in the tree-matching process. In general,

the fewer the number of restrictions affecting a lookup process, the larger the number of matches

between arguments and stored information.

The most powerful process available for altering

the kind (rather than merely the amount) of infor- mation produced in answer to a search request is

to change the search specification itself. The many

methods by which this can be done are summarized

in figure 9. In general, addition of new terms to a given search specification may be expected to yield a more narrowly defined document set, thus increasing precision; on the other hand, deletion of terms

may have the reverse effect, thus increasing recall. Replacement of old terms by new ones may have one , or the other effect, depending on whether the new . terms have a more restricted definition than the , original, or a broader one. Thus the use of clusters of terms, or syntactic phrases, instead of individual

terms alone should refine the definition, as indicated

in figure 9. Clearly, each of these possible devices may be

expected to have a different effect upon the eventual i outcome of a search, in the sense that recall and

precision are affected in different ways. In order

to be able to design a useful system, it is then neces- I sary to obtain a measure of the effect of each indi- vidual processing step alone. This can be done by n keeping the main system invariant and making

one judicious processing change at a time. If the differences in output are then evaluated, a measure should be obtainable of the usefulness of

208
Type of Process Method of Alteration of Specification

Probable Effect Improve s Recall Improve s Precision

Dictionary

Lookup

(1) Each input word is replaced by one

or more terms (or term numbers)

Hierarchical

Processing

(2) Each term is replaced by its "parent" on the next higher level

in the hierarchy

(3) Each term is replaced by its "sons" on the next lower level in the hierarchy

(4) To each term are added its "brothers" on the same level in the hierarchy, and its first-order cross references

Statistical Correlation

Methods

(5) To each term are added all other terms from within the same signifi- cant term cluster

(6) Each term is replaced by the term

cluster of which it is a part

Syntactic

Matching

(7) Each term is replaced by the criterion phrases in which it is contained

(8) To each list of terms are added the criterion phrases which match the original input

V

Simple Addition

and Deletion

(9) To each list of terms are added a

set of new terms

(10) From each list of terms are deleted

a set of specified terms

FlGUBE 9. Alterations of search specification or of document identifications.

the given step in relation to the usefulness of the

possible alternative steps. A continuing type of

process can then be envisaged, as illustrated in

figure 10, in which a sequence of processing altera- tions is executed until such time as the right kind

and amount of information are produced.

The weakest link in this procedure is the manual

evaluation of output differences produced by two

given search procedures. This cannot, unfortunately, be done automatically, since it is necessary

to determine to what extent the information added

by a given processing modification is in fact relevant,

and the information deleted is in fact marginal. No

method exists for eliminating this step entirely; by

adjusting the system in such a way that only small

amounts of output are produced (so that output

differences are . also small) the difficulty of this manual evaluation process can, however, be minimized.

It is hoped that tests now under way will lead to the construction of preferred sequences of processing steps. This in turn may lead to the determination of specific processing options which may be

particularly useful for certain kinds of subject

matter. Eventually, it may be possible to suggest

to the user at each step a set of alternative moves

to reach a given goal most efficiently.

209
Document Collection and Search Requests

Choose a reasonable correlation measure to compare

document identifications with search requests

Choose a reasonable threshold

to distinguish relevant from

nonrelevant information

Vsati

Process search requests against

documents using a standard mode V

the retrieved information

sfactory ?

YES

YES

EXIT

Can the situation be improved

by merely changing the amount

of retrieved information? NO

Choose one of the available methods for narrowing the search

specification (to increase

precision) or broadening the search specification (to increase

recall) and reprocess

FIGURE 10. Repeated processing procedure.

5. References

[1] Goldwyn, A. J., The place of indexing in the design of infor- mation system tests, Automation and Scientific Communication, ADI Annual Meeting (1963).

[2] The Metallurgical Searching Service — An Evaluation, NASNRC Publ. 1148 (1964; includes bibliography on testing and evaluation).

[3] Swets, J. A., Information retrieval systems, Sci. 141

(July 19, 1963; discusses measures of effectiveness of systems).

[4] Mooers, C. N., The intensive sample test, Report to Rome

Air Development Center ZTB-132, Zator Company

(Aug. 1959).

[5] Cleverdon, C. W., and J. Mills, The testing of index language

devices, ASLIB Proc. 15, No. 4 (Apr. 1963).

[6] Borko, H., Evaluating the effectiveness of information

retrieval systems, Rept. SP-909/000/00, System Develop- ment Corporation (Aug. 2, 1962).

[7] Centralization and documentation, Final Rept. to NSF No.

C-64469, A. D. Little, Inc. (July 1963).

[8] Salton, G., and E. H. Sussenguth, Jr., Flexible information

retrieval systems using structure-matching procedures,

Proc. Spring Joint Computer Conf. (Washington, D.C.,

1964).

[9] Salton, G., A flexible automatic system for the organization,

storage, and retrieval of language data (SMART), Rept. ISR-5 to NSF, Harvard Computation Laboratory (Jan. 1964).

[10] Salton, G., Manipulation of trees in information retrieval, Commun. Assoc. Comp. Mach. 5, No. 2 (Feb. 1962).

[11] Doyle, L. B., Indexing and abstracting by association, Am.

Documentation 13, No. 4 (Oct. 1962).

[12] Stiles, H. E., The association factor in information retrieval,

J. Assoc. Comp. Mach. 8, No. 2 (Apr. 1961).

[13] Salton, G., Associative document retrieval techniques using

bibliographic information, J. Assoc. Comp. Mach. 10,

No. 4 (Oct. 1963).

210
The Unevaluation of Automatic Indexing

and Classification

Terry R. Savage

Documentation, Inc. Bethesda, Md. 20014

The published papers reporting statistical methods of automatic indexing and classification have

invariably described and used a method of evaluation which is practically ineffective and theoretically unsound. The method seems to presuppose that a document "contains" something like a "meaning," that humans somehow find such meanings, and that an automatic system is to be judged on the basis of its relative agreement with what humans report finding. The method is ineffective mainly due to the lack of inter-subject agreement among humans. It is unsound because agreement with human reports is irrelevant to the question of evaluation. Such

agreement provides neither necessary nor sufficient conditions for making any judgments about per- formance. Realistic performance measures as well as methods to obtain them are described in detail. The paper critically examines the work of Luhn, Baxendale, Maron, Swanson, and Borko.

211

Automatic Indexing Using Cited Titles

Mary Elizabeth Stevens and

Genevie H. Urban

National Bureau of Standards

Washington, D.C. 20234

A brief account is given of an automatic indexing method which uses significant words in titles and cited titles for the assignment of descriptors to new items. Assignments are based on statistics of co-occurrence of significant words with descriptors assigned by human indexers to a "teaching sample" of items representative of the collection. Problems of evaluation arise in terms of changes

in indexing vocabulary and questions of inter-indexer consistency.

During recent months some small-scale experiments in automatic indexing have been conducted

by personnel of the Information Technology Divi- sion (formerly Data Processing Systems Division),

National Bureau of Standards. The experimental

method, which is called SADSACT (Self-Assigned

Descriptors from Self And Cited 71tles), involves two distinct procedures.

The first procedure is applied to a substantial

representative sample of items (e.g., papers, books,

reports) for which human indexers have already

assigned descriptors; this sample is called the

"teaching sample." The procedure develops

statistics of co-occurrence of substantive words

in titles and abstracts and the previously assigned

descriptors. The result of processing the "teaching sample" is a master vocabulary list with fre- quencies of association for each word and each

of those descriptors occurring in 3 percent of more

of the sample items with which that word had co- occurred. A list is also maintained of any other

descriptors occurring in the sample, but without

word association data. These are treated as "candidate" descriptors and may be assigned to new items if and only if a word identical with the name of such a descriptor occurs in the new

item. The second procedure is the automatic assignment of descriptors to new items. The titles of new items and the titles of bibbographic references

cited in these items are keystroked on a tape typewriter, converted to punched cards, and fed to the

computer. This input material is run against the master vocabulary list to derive for each input word

that matches a vocabulary word a "descriptorselection score" (based upon various weighting

formulas) for each of the descriptors previously

associated with that word. If a word occurs that coincides with the "name" of one of the "candidate

descriptors" retained in the list of those occurring

in less than 3 percent of the teaching sample items,

a selection score is also developed for the candidate

descriptor. When all words from the title and

cited titles of a new item have been processed,

the descriptor-selection scores are summed and at

1 Figures in brackets indicate the literature references on p. 215.

an appropriate "cutting" level those descriptors

having the highest scores are assigned to the new

item.

The SADSACT method differs from other auto- matic assignment indexing techniques in several

respects. A relatively smaller amount of textual input material is required both in setting the system

up and in the indexing of new items. Neither extensive human tailoring of word-descriptor association

lists nor extensive matrix manipulation by machine

is required. The SADSACT method is an ad hoc

statistical association technique in which the same

word may be associated, whether appropriately or inappropriately, with a number of different descriptors. By taking cited titles as sources of input

clues, clues are picked up that are not limited to the terminology of the author alone. Word co- occurrence patterns and redundancy then tend to depress the effects of inappropriate word-descriptor associations, to enhance the significant associations, and to increase the likelihood of successful

indexing of items which have an uninformative title.

Results of SADSACT experiments to date have

been based on two "teaching samples" taken from

the collection of the Research Information Center

and Advisory Service on Information Processing

(RICASIP). These samples have consisted of

approximately 100 items each, with about 70 per- cent overlap of items, and involve such subject

fields as computer technology, information selection and retrieval research, mathematical logic,

pattern recognition, and operations research.

These items had previously been indexed by DDC

(Defense Documentation Center, then ASTIA)

indexers in 1960. Results obtained on rerunning

these "source" items have been reported elsewhere

[1,2].»

New items that have been tested have also been

drawn from similarly indexed documents in .the same subject fields. To date, approximately 100

tests have been run on 59 different items. The

lists of descriptors assigned by machine have been

compared with those previously assigned by DDC

to determine the "hit" accuracy, that is, the per- centage of DDC-assigned descriptors that are also assigned by machine. The overall average hit accuracy for these tests is only 40.1 percent,

213
considering all descriptors assigned to these

items by DDC.

However, in spite of the use of test items drawn

from the same time period as the teaching sample

in order to maximize the consistency of indexing

and descriptor vocabulary, 19.1 percent of the

descriptors assigned by DDC were not available to the machine. When corrected for this factor,

the average hit accuracy was 48.2 percent. Applying a further correction factor for the case of the

candidate descriptors which were available to the

machine if and only if their names occurred in the

input material, the accuracy in terms of those

descriptors fully available to the machine rose to 58.1 percent.

A second approach to the evaluation of the results was to ask several representative users of the

RICASIP collection to analyze test items and in- dependently to assign descriptors from the list of

descriptors available to the machine. The extent

to which the descriptors assigned by machine were

judged to be relevant to the item by these users was then checked. Results for 25 items are shown

in figure 1, which gives the percent agreements

No. of Indexers No. of Items No. of Descriptors

12 6 3- 1

2 i 50.0 66.7 100.0 100.0

3 4 47.9 58.3 66.7 75.0

4 10 40.0 55.6 53.3 60.0

5 10 54.2 68 3 80.0 90.0

OVERALL 47.4 61.6 67.9 76.0

Figure 1. Average agreement with machine assignments.

between indexers and machine, averaged over

the items indexed where agreement of the indexers

with the machine is the percentage of descriptors

assigned by machine to that item which one or more

of the indexers also assigns. The proper definition of average agreement over a number of indexers

presents an area for further investigation.

TITLE: "construction of convolution codes

BY SUB-OPTIMIZATION

"

DESCRIPTOR NAME

Coding (A,B,C,D,E) 6820

Theory (A,B) 4837

Errors (B,C,D,E) 4816

Data Transmission (B, D,E ) 4633

Electronic Circuits 4370

Information Theory (C,D) 4326

Communication Systems (B,E) 4030

Synthesis 3502

Communications Theory ( D, E ) 3375

Figure 2. Typical result.

In general, the fewer the descriptors assigned,

the better was the overall agreement, ranging from

47.4 percent in the case where the machine had

assigned twelve descriptors to each item up to 76

percent in the case where the machine had assigned

only one. In particular, for ten items which were

independently analyzed by five indexers, the

chances that one or more would also select the

machine's first choice (highest scoring) descriptor

averaged 90 percent.

Figure 2 shows, in part, a typical result of the

SADSACT assignments to test items. The numeric

data shown are the computed selection scores. The upper case alphabetic characters in parentheses following the descriptor names indicate

which of five human indexers independently

selected the same descriptor as being relevant to the item. Two important aspects of the evaluation

problem are evident here. First is the problem

of inter-indexer consistency, or lack of it. Closely

related is the chance that a descriptor judged by

one indexer-user to be appropriate will be "missed"

by another indexer. This in turn means that in retrieval operations, for example, if user D of Figure

2 requested items on "coding," "errors," and either

"information theory" or "communications theory,"

then the item shown, which he would consider

specifically relevant to his query, would have been

missed if it had been indexed by either A or B. Figure 3 shows the percentage "misses" for items

A

A B C D Ave. M DDC

43.6 50.0 37.2 43.6 43.6 62.8

B 45.0 45.0 35.0 41.7 48.9 61.3

C 46.6 39.8 39.8 42.1 41.

63.0

D 37.9 34.2 44.3 38.8 43.0 58 2

Ave. 43.2 39 2 46.4 37.3 41.6 44.2 63 8

Figure 3.

indexed by four typical users by comparison with

machine assignments (column "M"). It can thus

be seen that the chance of disagreement with the

machine's assignments are not significantly greater

than the chances of an individual's disagreement

with the assignments made by any other indexeruser, at least for these test items. Finally, figure

4 shows agreement-disagreement by one or more of

the indexers with the machine indexing for a sample

of the specific descriptors of particular interest

assigned to 25 of the SADSACT items tested to date, e.g., for eight items to which the machine had

assigned the descriptor "coding," one or more

of the indexers independently assigned that descrip214
\ ANY \PERSON

MACHINE\\

s §

=

Is

DATA

STORAGE s

|1 INFO

THtORV §

COMMUNICATION

THEORY 3 1

Mine 1 i

1 — i EiiOtS

g§C li II I

1 1 ErrsESIS 1 —

e«u« srsrats 1 esn .

1 —

1— i

i i i 1 1 1

data 1 LAICUACE 1 III ! 1 i

D»T» PMC. 1 t MM i I in PERANUM am 1 i KACK.TMIS. 1 1 II I

i ii AITOHATIC ; 1 ii

1« : 0. THEORY 1 11 i PWIAIlim till i

::.«.< --;:=• 1

1 1

CIRCUITS i

tor to six of those items, whereas for the other two

items, the descriptors "theory" and/or "errors" were assigned by one or more of the indexers but were not assigned by the machine in these two

cases. The results therefore appear to compare favorably

with those of other automatic indexing techniques

that require, generally, more input text, more

machine processing, or more human intervention

[3, 4, 5, 6, 7, 8]. They also compare rather favorably

with respect to the levels of human inter-indexer

consistency that can typically be expected [9, 10,

11, 12]. A further implication of these preliminary

tests is that titles and cited titles do appear to give as good subject content indications as do titles and abstracts.

Figure 4.

References

[i]

[2]

[3]

[4]

[5]

[6]

Stevens, M. E., Preliminary results of a small-scale experi- ment in automatic indexing, to be published in Proceedings, NATO Advanced Study Institute on Automatic Document Analysis, Venice, 1963. Stevens, M. E., and G. H. Urban, Training a computer to assign descriptors to documents: experiments in auto- matic indexing, to appear in Proceedings 1964 Spring

Joint Computer Conference, American Federation of Information Processing Societies. Borko, H., The construction of an empirically based mathematically derived classification system, in proceedings

1962 Spring Joint Computer Conference, American Federation of Information Processing Societies, 1962, pp. 279-289.

Borko, H., and M. D. Bernick, Automatic document classification, J. Assoc. Computing Machinery, 10, No. 2, 151-162 (Apr. 1963). Borko, H., and M. D. Bernick, Automatic document classification. Part II — Additional experiments, Technical Memorandum TM-771/001/00, System Development

Corp., Santa Monica, Calif., 18 Oct. 1963, 33p. Maron, M. E., Automatic indexing: An experimental inquiry,

in Machine Indexing, Progress and Problems, Washington,

D.C., American Univ., 1961, pp. 236-265. Swanson, D. R., Automatic indexing and classification.

paper presented at the NATO Advanced Study Insti- tute on AutomaticDocument Analysis^ Venice, 1963.

[8] Williams, J. H., Jr., A discriminant method for automatically classifying documents, International Business Machines Corp. in Proceedings of the Fall Joint Computer

Conference, 1963, American Federation of Information Processing Societies, Conference Proceedings V. 24. Baltimore, Spartan Books, 1963, pp. 161-166, 7 refs.

[9] Jacoby, J., and V. Slamecka, Indexer consistency under

minimal conditions, Rept. No. RADC TR 62-426; Documentation, Inc., Bethesda, Md., Nov. 1962, 1 V. Contract AF 30(602)2616, AD-288 087.

[10] Painter, A. F., An analysis of duplication and consistency

of subject indexing involved in report handling at the

Office of Technical Services, U.S. Department of Commerce, Rutgers State Univ., Ph. D. Dissertation, 1963, 135 p. Washington, D.C., Office of Technical Services, PB181501.

[11] Slamecka, V., Indexing aids, Final report, TDR-62-579,

Documentation Inc., Bethesda, Md., Jan. 1963, 33p.

[12] Slamecka, V., and J. Jacoby, Effect of indexing aids on the

reliability of indexers, Final technical note RADC TDR

63-116, Documentation, Inc., Bethesda, Md., June 1963,

1 V.

215
i5
Results of Classifying Documents with

Multiple Discriminant Functions

J. H. Williams

International Business Machines Corporation

Bethesda, Md. 20014

An important, but frequently underemphasized, step in the classification process is the selection of attributes. In classification problems of mutually exclusive assignment, a set of attributes is se- lected to represent the category. For information retrieval applications the assumption of mutually

exclusive categories may not hold. Therefore, the problem of the selection of measurable attributes to represent the categories becomes more acute. Discriminant analysis appears to offer a solution not only to the selection-of-attributes problem,

but also to the document relevance problem. In the selection phase it provides a method of selecting a se» of attributes whose ratio of among-category variance to within-category variance is largest. In the actual classification process, a distance measure can then be employed to determine the degree of relevance of a given document with respect to each category. Since a category can be defined by a

set of documents having the desired category attributes, the measure also enables one to determine

the degree of overlap among the categories — a valuable check on the soundness and manageability of the classification structure.

Classification experiments have been conducted on 794 solid state abstracts. Classification accuracies up to 90 percent were achieved using the discriminant procedures.

1. Introduction

This report describes a continuing effort, within

IBM, devoted to developing and testing statistical techniques to aid in the content analysis of documents. Techniques currently exist for the extraction of key terms and phrases, as long as a definition

of the desired terms is given. However, there re- mains an important class of documents for which

no techniques have as yet been developed. These

documents contain concepts whose meanings are

not expressed directly by proper nouns, key terms,

or specific sentences, but by the total pattern of

words throughout the whole document. The typical

solution for this class of documents offers a rele- vance value relating the document to each concept

represented in it. In the computation of a relevance value, the

problem of word dependence becomes apparent

in this latter class of documents. It arises from

the common assumption that the occurrences of

words are independent of each other. Two aspects

of the dependency problem should be mentioned

here: (1) words are indeed dependent on each other

for some class of documents; (2) the dependency

relationship may change from context to context. The assumption of independence of words in a document is usually made as a matter of mathematical convenience. Without the assumption,

many of the subsequent mathematical relations

could not be expressed. With it, many of the conclusions should be accepted with extreme caution.

The importance of this independence assumption can be observed as progress is made from a coordi- nate indexing system to a subject classification system. In coordinate indexing systems, key

terms are selected because their meanings are thought to be independent of the context. If their

meanings were unique, and therefore independent

of the context, then they would be ideal indicators

of subject content. However, experience with

these systems has revealed examples of the two

aspects of the dependency problem. The first aspect can be illustrated by the computer literature,

where the words "compiler" and "Fortran" are not independent of each other. However, if the

degree of the relationship of these two words were known, an adjustment could be made. The second

aspect of the problem can be illustrated by a word

whose meaning changes with the context, such as "pitch" in baseball, music, or aerodynamics.

As a result, the need arises to determine and measure the relationships of words to each other and to the context in which they occur. The purpose of the present study is to test the

applicability of discriminant analysis, a multivariate

statistical technique, which appears to represent

the intuitive concepts of dependency of words for coordinate indexing as well as for subject classifi- cation systems.

2. Previous Experiments

An earlier series of experiments was conducted

to test the feasibility of automatically classifying

documents by means of a statistical technique.

The data base employed was a set of 400 abstracts

1 Figures in brackets indicate the literature references on p. 224.

from the computer field. Classification accuracy

for the independent test set ranged from 60 to 90

percent when compared with professional indexers.

The empirical classification equation used in these

experiments is described in reference [l]. 1 To ensure that the classification technique was

772-957 0-66— 15 217
not biased by the data base from which it was derived, another series of experiments was performed

on a subset of 2700 solid state abstracts. After

achieving a reasonable degree of accuracy on this subset of abstracts, attention was turned to the

analysis of the classification parameters. Isolating

parameters and determining the conditions under

which they assume their optimum values was the

point of interest here. Some of these parameters

considered were the number of categories in the

structure; the number of documents in each cate- gory; the number of words in each document; the

number of discriminating words to be retained for the classification phase; and the representativeness

of documents. In the next series of experiments,

NUMBER of Ref. Docs.

CATEGORY ,

91

80 100

93

80 100

94

80 100

95

80 100

Type of Document: REFERENCE

INDEPENDENT TEST

.70 .68

.43 .70

.68 .56

.65 .29

.86 .88

.29 .49

.98 1.0

.43 .46

FIGURE 1. Percentage of correct classifications as the number

of reference documents changes.

-2 0 +2 RELEVANCE VALUE - STANDARD UNITS

FIGURE 2. Distribution of document relevance values for

category 95.

.6 -i

-2 0 +2 >4 RELEVANCE VALUES - STANDARD UNITS

.8 -i /

B. 80 DOCUMENTS /

.7 - ' /

/ /

RELEVANCE VALUE - STANDARD UNITS

Figure 3. Distribution of document relevance values for category 91.

observations on the effects of changes , in these

parameters on the overall performance of the System were made. In one of the experiments, the

number of reference documents in each category was decreased from 100 to 80. The results shown

in figure 1 indicate that a more detailed analysis

of this parameter is required. The number of

documents required may change from category to category. Figure 1 shows that category 95 achieved

98 percent sucess with only 80 reference documents,

whereas category 93 achieved 68 percent success. The effects of a change in the number of reference

documents cannot be analyzed independently of

the other classification parameters. The effect of

the number of documents on classification accuracy

as well as the inter-effect of representativeness of

documents can also be observed from the same

figure. It cannot simply be assumed that if 20 more documents are added the classification ac- curacy will improve. A check on the representativeness of the documents being added is required.

When documents that are not as representative

are added, a decrease in accuracy can result, as shown by category 93.

Figures 2 and 3 show how the distribution of

relevance values can be used to measure the rep- resentativeness of the documents. In figure 2, the solid line shows the distribution about the mean relevance value of documents known to belong

to category 95. Ideally, the dashed fines should

218
be low around the mean relevance of zero and be- come higher with increasing distance from the mean. Lack of representativeness in a document can

be caused in two ways. (1) A document may contain one and only one concept at that level, but it may be shorter or longer than the average. The

word frequencies then would be atypical with re- spect to the category and thus cause an increase

in the within-category variance. (2) A document

could contain more than one concept at the same

level. Such a document could contain words from

several categories and would cause a decrease in the among-category variance.

A preliminary experiment in which 10 documents

of each type were removed from each category has

borne out this hypothesis. Figure 2B shows that

after removal only 10 percent of the other category

documents were near the mean of category 95,

and 50 percent were more than four standard deviations away. Figure 3 shows additional information

concerning the degree of similarity of two categories. The dashed line close to the solid line is the distribution of documents belonging to cate- gory 93. When two distributions are close to each

other, it can be interpreted that they belong to the same population rather than two distinct populations. Even after removal of the 20 less representative documents from each category, the lines are closer than expected. Thus the categories

probably represent a related subject.

As a result of these experiments, it became ap- parent that a more analytical technique would be

required to classify documents, and also to analyze misclassifications. A metric that is not biased

by the parameter of the data from which it was de- rived seems to be needed in measuring relevance

and the effects of the parameters. Mahalanobis'

D2 is a metric that appears to satisfy these conditions. Therefore, the objective of our latest experiment was to test the effectiveness of multiple dis- criminant functions' and Mahalanobis' D2 for classi- fying documents. The steps in the classification procedure will be illustrated in section 3 by the detailed description of the latest experiment.

3. Classification Procedure

A user starts with a set of documents and decides on a group of subjects of interest to him. He then

partitions this set into subsets of documents belonging to the various subject categories. These

documents will be called reference documents and

are used to compute mean frequencies and vari- ances of each word type. In this experiment the

solid state categories as defined by the Cambridge

Communications Corporation (CCC) were used.

The reference set consisted of 320 documents.

CCC had previously classified 80 of these documents into each of four categories, as shown in figure 4. In this experiment classification was

performed only at one level. Topics included

in each of the categories are shown in figure 4,

to indicate the level of difficulty presented by this data base.

Applications

of SS Devices

SS Metallurgy

and SS Devices Chemistry Physics

of Solids

Communications Conductive Devices Computers Photo Devices Power Magnetic Devices

Instrumentation

Crystal Structure Crystal Physics

Crystal Growth Electrical Properties Magnetic Properties

Optical Properties Thermal Properties

Crystal Surfaces Environmental

Effects

FIGURE 4. Experimental solid-state structure.

Since CCC can be considered the user, the defi- nition and structure of categories are determined

by their outline of solid state categories. In an

operational situation, the method provides an

opportunity for the user to improve the initial defi- nition of the categories after a preliminary computer run. The degree of improvement is entirely

under the direction of the user. He is given several control statistics which tell him the amount

of dispersion in each category, the amount of over- lap of each category with every other category, and

the discriminating power of the variables. He

can add, remove, or redefine categories to suit the

specificity of his particular needs. These statistics are based on the sample of documents that he assigns to each category. Thus the user is not obligated to define each subject category with

merely a word label. He is free to supply any documents which contain his concept of that subject.

Various users of an identical set of documents can

thus derive their own structure of subjects from

their individual points of view.

At the next step, the reference documents are

input to the word counting program. The program

computes, for each word type in a category, the mean frequency as well as the variance. The

pooled within-category variance, the among-category variance, and an F ratio (described below)

are computed. At this point there is an F value

for every word type that occurred in a document.

Previous experiments indicate that all word types

do not need to be retained for the classification equation. But what criterion can be used to select the words to be retained? This is a question which

has frequently been underemphasized in the clas219
sification process. Ideally the criterion should be

similar to the one used by indexers and classifiers. Therefore, we have used a statistical criterion which

appears to quantify the intuitive criterion that has

been used.

The intuitive criterion is one in which words that represent a category should occur in nearly every

document of that category and should not occur

in documents belonging to another category. If they do occur in documents of another category, near the same frequency, ambiguity exists, and the

word will not be a good predictor. Two easily

obtained statistics can represent this criterion. The consistency with which a word occurs in each

document in a category can be measured by the

pooled within-category variance, W. The deviation of the frequency of occurrence of a word in documents belonging to different categories can

be measured by the among-category variance, A. The ideal predictor should occur regularly in all the documents of a category; therefore its W should

be low. It should not occur with the same fre- quency in documents of the other categories; therefore its A should be high. It was noted that, by

forming the ratio F = A/W, the value of F quantifies the qualitative criterion because it is high for excellent predictors and low for poor ones. This

F ratio is similar to the multivariate maximizing

condition of discriminant analysis. Figure 5 lists the 48 most discriminating words selected in this experiment relative to the above F ratio. Only the frequencies of these 48 words are used

in the actual computation of the discriminant function. The object of this computation is to find the

optimum linear combination of weighting coefficients for these words. Each of the 48 words has

a set of weighting coefficients which represents its discriminating ability with respect to each of the

various categories. Since these coefficients are

affected by the definition of the categories, words

will have a different set of weights depending on

the context.

Classification can now be achieved by comparing

the observed frequency of each of the 48 word types

to their corresponding mean frequencies in each

category. When the comparison is performed by

the classification equations, each word type is

Category

91-Applic. of 93-SS 94-SS Metallurgy 95-SS

SS Devices Devices and Chemistry Physics CIBCUI .94 .25 .00 .03 COUNTE .33 .01 .01 .01 DESIGN .25 .08 .05 .01 DETECT .50 .04 .05 .03 NOISE .22 .05 .00 .05 OUTPUT .73 .14 .00 .01 POWER .63 .21 .10 .19 PULSE .68 .06 .00 .05 REGULA .24 .01 .03 .00 STABIL .16 .05 .04 .00 SWITCH .60 .29 .00 .03 TRANSI .98 .90 .04 .28 CONSTR .08 .09 .03 .01 CURREN .78 .85 .06 .56 DEVICE .21 .45 .04 .00 FERRIT .06 .25 .06 .01 FREQUE .28 .83 .05 .54 HIGH .34 .53 .21 .31 JUNCTI .10 .79 .10 .03 MADE .23 .29 .14 .19 MAGNET .43 .76 .03 .61 P .06 .34 .13 .23 TUNNEL .00 .05 .00 .01 VOLTAG .56 .71 .00 .38 CRUCIB .00 .00 .25 .00 CRYSTA .14 .20 2.28 .63 DISLOC .00 .00 .68 .03 FURNAC .00 .00 .13 .00 GROWTH .00 .00 1.18 .04 ION .06 .00 .15 .15 MICRON .01 .00 .14 .09 OXIDE .00 .00 .16 .11 SEED .00 .00 .16 .00 SINGLE .16 .09 .59 .29 TEMPER .25 .24 .74 1.58

V ArOK .00 .00 .28 .00 AND 3.06 3.28 3.34 4.54 DEPEND .05 .13 .20 .63

.98 EFFECT .16 .25 .14 ELECTR .34 .43 .31 1.96 FERROE .00 .00 .00 .24 FIELD .06 .69 .05 1.25 IMPURI .00 .09 .30 .24 INTERA .00 .03 .03 .24 OXYGEN .00 .00 .09 .20 PHONON .01 .00 .20 PIEZOR .00 .00 .00 .16 TRANSV .00 .04 .01 .24

Figure 5. Mean frequencies of discriminating words.,

weighted by its discriminant coefficient, its own

variance, and its covariance with other word types.

Thus frequency is not the sole criterion for classification. Compensation for its discriminating ability in context and for its dependence on other words,

is included. A relevance value is computed for each document with respect to each category.

All relevance values can be retained for retrieval

purposes, or an additional step of assignment to one or more categories can be made.

4. Linear Discriminant Functions

Suppose there are c categories with pj documents

in the jth category (/=1, 2 . . ., c). For each

document find the n values representing meas- urements on the n variates xi, X2, ., xn . One

problem of interest here is to classify a document

into the appropriate categories on the basis of the

set of n values when it is known that the document

belongs to at least one of the categories. The first aspect is concerned with whether these n variates can distinguish among c categories. If so, then the

distance between separating pairs of categories and

the assignment of an individual document to one

or more of the c categories can be considered.

The linear discriminant function is one of the tools available for this process. The linear discriminant function is a function of n variables measured on each category such that

this linear combination provides the best discrimination between categories. Specifically, the best

discrimination is effected by maximizing the ratio of the among-category sum-of-squares of this function to its within-categories sum-of-squares. As

220
will be noted later, appropriate generalizations of

this discriminant criterion have been made in the case of several groups of categories.

Since the concern is with discrimination among

categories, one of the first tests of interest deals

with the problem of separation. That is, are the

category means (centroids) distinct? Under the

assumption of equality of category variances, one

test of the degree of confidence with which it can

be assumed that the centroids are indeed distinct

is given by the Wilks' statistic:

\W+A\ (1)

The symbol A is the ratio of two determinants

where W is an n by n matrix whose elements are

the pooled within-category sums-of-squares and

sums-of-cross products. A is an n by n matrix

whose elements are the among-category sums-ofsquares and sums-of-cross products. Values of

the A matrix which are correspondingly larger

than values of the W matrix result in an increasingly

smaller ratio with increasing confidence in rejecting the hypothesis of equality of category means. Now, if the centroids are distinct as measured by

the A criteria, the questions of distance between

categories and assignments of individual documents may be analyzed next. For a single word type, a possible method of

classification would involve comparing the measure- ment of that type in the new document against

the corresponding category sample mean, and as- signing the item to the category for which the mean

is closest to the measurement.

For the multivariate case (i.e., the case in which

there are n 3: 2 variates) one of the simplest trans- formations would be a linear combination of the n variates resulting in a single quantity. Consider

for example, the linear combination

X = C1X1 + C2X2 +

where X is the value resulting from the linear coml bination, Xi, X2, xn measurements, and

Ci, C2, ., Cn are a set of coefficients chosen in such a way that the best discrimination is effected. That is, the set of coefficients which should be

chosen is of the type which satisfies the discrimi- nant criterion stated above.

It has been shown (see, e.g., Bryan [2]) that the

condition for maximizing the ratio of the amonge category sum-of-squares to the pooled withincategory sums-of-squares is satisfied by solving

• the determinantal equation,

IW-'A-XI^O, (2) where I is the identity matrix, W and A are as defined previously, and X is any one of the n eigenvalues to be determined. The eigenvector cor- responding to A provides the set of coefficients

j for a discriminant function which transforms the

n individual measurements into a single value or discriminant score. This discriminant score is then the basis for assigning an incoming document

to one of the categories. In dealing with the problem of discriminating

among several categories, more than one dimension is considered, since there is no reason to as- sume that the centroids are collinear. It follows

that by taking only one linear combination, in effect a linear ordering of the categories is made. Further, a linear ordering cannot exhaust all the

information in the data relevant to group separation.

It has been shown (see, e.g., Bryan [2]) that the

linear combinations corresponding to the pre- viously discussed eigenvectors have the following

property: the first linear combination, corresponding to the largest eigenvalue \i, maximizes the

discriminant criterion in the sense that one is dis- criminating between two categories; the second

linear combination, corresponding to the second

largest eigenvalue A.2, maximizes the ratio of the

residual among-category sums-of-squares to the

residual within-category sums-of-squares after the

effect of the first has been removed, and so forth. Furthermore, the number of solutions of the de- terminantal equation such that k ^ 0 is at most

equal to the smaller of the two numbers c— 1 and n. These solutions are the multiple discriminant functions (MDFs) and exhaust the total discriminative

power of the variables relevant to category separation. The MDFs are a powerful tool in that they pre- serve the information given by the variables relevant

to group separation and yet allow one to classify

in an m-dimensional space, where m = min (c— l,n).

The eigenvectors of the MDF can be used to form a transformation matrix V, where

Vn V21

v =

(n, m)

(3)

The vector of means for each category, the dispersion matrix for each category, and the vector of

observations for an incoming document are each

appropriately transformed to a reduced discriminant

space having only m dimensions.

The classification question is now posed in the

reduced space. How far does an observation he

from the centroid of each category? Mahalanobis'

D2 (see, e.g., [7]) can again be used to measure this distance, using values derived for the reduced

space by the transformations indicated above.

An incoming document will then be assigned to the

category for which its Mahalanobis' D2 value is 221
smallest. The number of dimensions has thus been

reduced considerably and at the same time the

MDFs have preserved, in this reduced space, the

effect of the most discriminating variables.

The D2 value in the reduced space is also used

to represent the relevance value of an individual

document. For the distributional properties of

Mahalanobis' D2

, see reference [7]. Upon making

the necessary assumptions (which need, of course,

to be tested further), most of the necessary computer

programs for the procedure described above can

be found in reference [3].

5. Interpretation of Discriminant Functions

The separability of the solid state categories can

be observed in either the original 48-dimensional

variable space or in the reduced three-dimensional

space. Figure 6 shows the centroids of the four

categories in the reduced three-dimensional dis- criminant space. Figure 7 shows that category 93

has a larger percentage of overlap than any other

category. In addition to these visual checks, a

statistical check can be made with Wines' A test.

91(-0.95, 0.54, -0.73)

93(-0.19, 0.58. 0.47)

95(1.05, 1.37. -0.47)

(0.76, -0.68, -0.39)

FIGURE 6. Category centroids in three-dimensional discriminant

space.

\\ ' >RDS AXES

i II III

Ul - .20 • - 14 COUNTE -.25 -.03 - .15 DESIGN - .30 .06 -.18 DETECT -.28 .04 - .30 NOISE -.13 .10 - . 14 OUTPUT - .12 "?? POWER - .10 - .01 "o8 PULSE -.17 .08 -.16 REGULA -.20 -.10 -.21 OTA TJT T - 11 owl 1 L.H 1Q

~ (n - 14

1 1 'L> A MCI -.05 .06 nfi 09

OUINo Lti - .01 12 20 CURREN - .07 .02 Dhj vice. -.13 -.03 24 FERRIT - .1

- . 11 .33

r nt^Ut -.02 .06 HIGH -.07 -.02 ' JUNCTI - .10 -.08 .27 MADE .03 .19 .07 MAGNET -.10 .02 .05 P .13 .05 .09 TUNNEL .03 .06 .35 VOLTAG .05 .15 .04 CRUCIB .09 -.18 -.06 CRYSTA .06 -.18 0.0 DISLOC .11 -.26 - .05 FURNAC .21 -.31 .02 GROWTH .13 -.25 - .07 ION .07 -.01 -.18 MICRON .18 -.05 - .16 OXIDE .21 0.0 -.09 SEED -.10 .22 -.01 SINGLE .06 .04 -.09 TEMPER .11 .09 -.07 VAPOR .15 -.39 -.08 AND .02 .04 -.02 DEPEND .25 .18 -.13 EFFECT .18 .23 -.09 ELECTR .10 .16 -.07 FERROE .07 .29 -.15 FIELD .10 .16 .04 IMPURI .10 -.11 -.04 INTERA .15 .08 -.12 OXYGEN .07 .17 -.03 PHONON .06 .12 .06 PIEZOR .07 .27 -.04 TRANSV .18 0.0 -.06

Figure 8. Normalized coefficients of words in discriminant

space.

CATEGORY

93 94

FIGURE 7. Overlap of categories.

Analysis of the coefficients of the discriminant

functions shown in figure 8 indicates how the

separation of categories is achieved. The first 24 words generally have negative coefficients,

and the last 24 generally have positive coefficients. This means that the first discriminant function

divided the space into two parts. If discrimination

between only the two pairs of categories 91 and 93

or 94 and 95 were desired, it could be achieved along

this axis. In the second discriminant function

the coefficients of words for categories 91, 93, and

95 are generally positive and for category 94 are

negative; therefore it appears to provide a decision

boundary between categories 94 and 95. In the

222
third discriminant function the coefficients of words

for categories 91, 94 and 95 are generally negative

and for category 93 are positive: therefore it ap- pears to provide the decision boundary between

categories 91 and 93. The relation of the decision

boundaries to each category can also be observed

from the coordinates of each centroid as shown in figure 6. A few examples will show how discriminant functions are used to transform a 48-dimensional space

to a three-dimensional space. Since the coeffi- cients in figure 8 are normalized, the square of

these values is the percentage of discrimination

contributed by each word. Thus, the word AND

contributes less than one percent on each of the

axes, whereas OXIDE accounts for four percent on the first axis. The direction of the effect of

each word can be observed in the three-dimensional

reduced space by letting its value in each discrimi- nant function equal one and the value of all other

words equal zero. Three different types of words

to be discussed are: (1) CRUCIB — occurs in one

and only one category: (2) OXIDE — occurs in two

categories: (3) AND — occurs in all four categories.

CRUCIB (0.0, 0.0, 0.25, 0.0) is a word which has a

significant difference between its means and which

occurs in only one category. Its discriminant

coefficients (0.09, — 0.18, — 0.06) as shown in figure 8

he near the centroid of category 94 as expected.

OXIDE (0.0, 0.0, 0.16, 0.11) has a significant dif- ference between pairs of means, but not within the

pairs. In some techniques this word would not be retained as a predictor. However, in the dis- criminant technique, utilization of this information can be easily seen through its discriminant coeffi- cients (0.21, 0.0, —0.09). The positive value on

Axis I indicates that it is in either 94 or 95, whereas

the zero value on Axis II indicates that OXIDE

has little discrimination power between 94 and 95. AND (3.06, 3.28, 3.34, 4.54) has an insignificant

difference between all its means and, as expected,

its discriminant coefficients (0.02, 0.04, —0.02)

are very low on all axes. Thus, analysis of the dis- criminant procedures indicates that the results do have a meaningful interpretation. Significant

words will have high discriminant coefficients,

whereas insignificant words occurring in the inter- section of all categories will he near the origin.

6. Results and

The classification procedure just outlined in section 3 was used to classify both the 320 reference

documents and 474 independent test documents.

The percentages of correct classifications shown in figure 9 are based on all documents input to the

INDEPENDENT TEST DOCUMENTS

CATEGORY

TOTAL NUMBER OF DOCUMENTS

\. A

D N. 91 93 94 95

91 63.95 30.23 3.49 2.33 86

93 18.75 64.58 6.25 10.42 48

94 1.21 10.3 80.61 7.88 165

95 4.67 21.7 30.21 43.42 175

REFERENCE DOCUMENTS

CATEGORY TOTAL NUMBER OF

N. A DOCUMENTS

D

91 93 94 95

91 87.5 11.25 1.25 0.0 80

93 11.25 75.00 10.0 3.75 80

94 0.0 5.00 90.0 5.0 80

95 1.25 6.25 8.75 83.75 80

LEGEND: A: Actual

D: Desired

FIGURE 9. Percentage of correct classifications.

Potential Use

system. Even though some may not contain any

of the discriminating words (i.e., the small set of only 48 types), their results are included in the

percentages. Therefore these results were

achieved by using only 48 out of the 3155 total word

types in all 320 reference documents. Only 80

documents were used to represent each category. In the selection of reference documents for cate- gory 95, the longest documents were intentionally

placed in the reference set and the shortest in the

test set. The results for the test set of category

95 indicate that compensation for variation in document length must be considered. These two are the most obvious parameters to change in order to increase classification accuracy. Another important parameter is the range of document length.

The procedure described in this paper was uti- lized in order to assist in content analysis, that is, in determining what subject or subjects are cov- ered by a particular document. The unique feature

of this statistical approach is that it provides for an

analysis of a set of documents from many divergent

points of view. For example, if three user groups, who are interested in the political, electronic, and

military aspects of a situation, all receive the same

set of documents, how can they be indexed or classified to serve the different needs of each user? The present technique permits a matching of in- coming documents against statistically derived

profiles which are specifically oriented towards

the user's point of view. These profiles could

be derived for each group and to any level of detail specified. They could be determined independently of the other users' needs, or combined at a higher and more general level.

223
Since the technique is based on an analysis of

variance of word-type frequencies, the definition of these word-types can be changed to suit specific

requirements. A word can be defined as a string

of n characters, so that foreign language documents

as a separate group can be processed without

translation. The technique is also general enough

to handle various intervals of text. The textual

interval to be classified could be either a whole

document, an abstract, a section, a paragraph, a sentence, or a set of key words.

The system output is not limited to subject classification because relevance values are computed

and retained for each document with respect to every category. The output for each document

could also include each of the discriminating words

that actually occurred in the document, at each

level of the structure. Furthermore, the following

retrieval aids could be made available: (a) association factors at every level (either for each subject separately or for all subjects within that group),

(b) lists of the most discriminating words for every

category, ranked in descending sequence of their discrimination ability.

With these aids, retrieval could be accomplished

either by subject heading, descriptors, associated

words, or by a narrative query. For retrieval by

subject heading, the user would request all documents in the desired category having a relevance

value higher than some specified threshold. Retrieval by narrative query would be entirely analo- gous to the matching of an incoming document

against all available categories. The output in

this case would indicate which categories are most

relevant to the request, and these categories could

then be searched in descending sequence.

It appears that the system would be capable of detecting changes in disciplines or relationships

of subjects. Each group of categories should

contain one which will be "general" or "all other."

Periodically the distribution of relevance values

for all documents processed in the preceding period will be compared with the distributions previously established for each category. Detection of the fact that words from two disciplines are now

being used interchangeably can be made easily by

noticing that the measured overlap between two

categories is becoming greater. Detection of the

arrival of new words and concepts can be achieved

either when the dispersion of a category increases

or when a new word moves up on the ranked dis- criminating word fist. Consistent increases in rank can be detected very early, for example a change

from rank 1000 to 900. When a change in the structure is required, documents can easily be re- classified since the permanent machinable form of

the document is condensed at one point to a single

record of word-frequency pairs. When a change

occurs in a group only the documents having a sig- nificant relevance value with respect to the cate- gories of that group are reclassified and the appropriate files updated.

Interpretation of textual subject matter may vary

widely depending on a user's background, current

interest, and other factors. For effective classi- fication and retrieval, it is essential that some means

be provided which will allow a variable "point of

view" in information processing. It is believed

that the discriminant procedures described here

are not only responsive to this operational requirement, but also furnish valuable analytical tools for use in content analysis.

7. References

[1] Williams, J. H., A discriminant method for automatically

classifying documents, Proc. Fall Joint Computer Conf. 24, 161-166 (1963).

[2] Bryan, J. G., The generalized discriminant function: mathematical foundations and computational routine, Harvard Ed. Rev. 21, 90-95 (1951).

[3] Rao, C. R., Advanced Statistical Methods in Biometric Research (New York, N.Y., John Wiley & Sons, 1952).

[4] Cooley, W. W., and P. R. Lohnes, Multivariate Procedures

for the Behavioral Sciences (New York, N.Y., John Wiley

& Sons, 1962).

[5] Borko, H., and M. Bernick, Automatic document classification, J. Assoc. Comp. Mach. 10, No. 2, 151-162 (1963).

[6] Edmundson, H. P., and R. E. Wyllys, Automatic abstracting and indexing— survey and recommendations, Commun.

Assoc. Comp. Mach. 4, No. 5, 226-234 (1961).

[7] Maron, M. E., Automatic indexing: and experimental in- quiry, J. Assoc. Comp. Mach. 8, No. 3, 404-417 (1961).

[8] Posten, H. O., Bibliography on Classification Discrimination, Generalized Distance and Related Topics, RC—743, IBM, Yorktown, N.Y. (1962).

[9] Tatsuoka, M. M., and D. V. Tiedman, Discriminant analysis, Rev. Ed. Res. 24, No. 5, 402-416 (n.d.). [10] Williams, J. H., Statistical Analysis and Classification of Documents, IRAD Task No. 0274, IBM, FSD, Rockville, Md. (1963).

224
Rank Order Patterns of Common Words as Discriminators 01 Subject

Content in Scientific and Technical Prose

Everett M. Wallace

System Development Corporation

Santa Monica, Calif. 90406

There is a style of language characteristic of different subject areas which is particularly notice- able in scientific and technical writing. It is not only the unique vocabulary of a subject field which

sets it apart from others, but also the different habits of writers in using the most common words. An experiment was devised to test whether these differences could be used for subject discrimination

in addition to identification of unique vocabulary, particularly to determine whether or not author

variation in style is sufficiently great to override the variation from field to field. Fifty IRE abstracts in the field of electronic computers and fifty Psychological Abstracts were

matched, one abstract at a time, one word type at a time, against two lists of words ranked in descending order of frequency as they occurred within two different sets of 300 psychological and computer

abstracts. All fully inflected forms of all function and content words were included in the rankings.

Using the first 50 ranks only of the two lists, 93 percent of the abstracts were successfully discrimi- nated. For the first 75 and 100 ranks, the success rates were 96 percent and 97 percent, respectively.

1. Introduction

There is little reason to be satisfied with current

information system designs for either dissemination

or retrieval. The use of condensed representations in the form of class categories or index terms

has limitations. Systems using such devices appear, inherently, to produce a great deal of "noise,"

as can be seen in the recent work on relevance/

recall ratios. Whole text or "natural language"

processing approaches appear to offer the greatest

promise of improvement in retrieval systems. The

designers of prose processing schemes, however,

have encountered serious difficulties in building

systems which are both practical and economical.

A major problem in working with natural language

is the range of variation in linguistic behavior.

The wide range of variation has been an obstacle

to successful predictive generalization, whether

applied to mechanical or human information storage

and retrieval. One reason for the current diffi- culties is that we do not have a sufficiently precise

knowledge of the stochastic parameters of lan- guage, particularly as it is used in different subjects and contexts. A second reason is that efforts directed at statistical techniques of linguistic

analysis have concentrated upon the relatively

infrequent verbal constructs.

It has been a common practice in building lan- guage-processing programs to reduce the number

of different entities which must be handled by ex- cluding the most common articles, prepositions,

conjunctions, and auxiliary verb forms, and by combining ^fleeted forms of common roots. Such

procedures do result in the loss of a certain amount

' Figures in brackets indicate the literature references on p. 228.

of information. Through reading the reports of

G. Yule [l], 1 G. Herdan [2], and F. Mosteller and

D. Wallace [3] in establishing the authorship of

disputed works, I was led to consider ways in which this lost information could be recovered and

used to supplement established methods. G. K.

Zipf [4] had already shown one way of using rank

order distributions of words. Others have indicated

that there is a considerable range of variation in the way individual authors use the most commonly

occurring words in a language in different contexts. There is a style of language characteristic of

different subject areas which is particularly noticeable in scientific and technical writing. It

is not only the unique vocabulary of a subject field which sets it apart from others, but also the different habits of writers in different fields in using common

prepositions, nouns, and verbs. This is most

clearly illustrated in mathematical writing, in which

symbology is embedded in a highly stylized form of prose, sufficiently unlike ordinary language to be

considered a distinct dialect. The growth of

"dialects" in this sense is common to all subjects

in varying degrees. The question is whether these

behavioral differences are sufficiently distinctive

to provide a basis for subject discrimination in addition to the identification of unique vocabulary.

One of the first considerations in estimating

whether a practical discriminator could be built was whether or not author variation in style is sufficiently great to override the variation from

field to field. An experiment was devised to test

this proposition and to gather evidence for identifi- cation of statistical parameters and techniques

useful for subject discrimination.

225
2. The Experiment

An experimental corpus was selected consisting

of 350 Psychological Abstracts and 350 IRE abstracts from the Transactions of the Professional

Group on Electronic Computers (PGEC). The

abstracts were available at System Development

Corporation in machine-readable form.2 This

corpus was considered to provide an adequate re- flection of author variation, in that the abstracts

had largely been written by different persons,

including authors of the papers abstracted.

Three hundred psychological abstracts and 300

PGEC abstracts were taken from the corpus for establishment of population "profiles" of the two

subject areas. The profiles consisted of two lists of the most frequent 100 words ranked in descending order of occurrence within the two sets of 300

abstracts. A System Development Corporation

computer program called FEAT was used to provide

the counts and listings. The appendix presents

a consolidated alphabetic list of the words in the

two profiles, together with their rank numbers.

Where occurrence frequencies of two or more

words were equal, a word-length criterion was

applied such that the shorter word was given the

higher rank. This was based on the assumption

that, in general, short words are more prevalent

than long. When word length as well as frequency were equal, the words were ranked in alphabetic

order. A version of the FEAT program was used to count

and list the words in each of the 100 abstracts re- maining in the experimental corpus of 700. Each

abstract was matched, one word type at a time,

against the two profiles of 100 rank-ordered words.

The words in each abstract occurring in one or both of the two profiles were recorded, together

with their rank numbers.

PSYCHOLOGICAL ABSTRACT * 1 - 54 word type

in Abstract Psych. Profile PGEC Profile 50R 75R 100R 50k 75R 100R

a 6 3 and 3 4

be 17 13

but 63

by 1 J 14

first 74

have 56 69

information 40

in 4 7

is 7 5

of 2 2 on 13 16 the 1 1 to 5 6 were 18

with 9 17

No . words in conimon 12 15 15 1? 13 13

Rank no . sum 99 128 Figure 1. Psychological abstract No. 1—54

word types.

2 The abstracts were drawn from the experimental sets used originally by Borko for automatic classification and by Maron for automatic indexing.

IRE PGEC ABSTRACT * 1 - 15 vord types

Word in Abstract Psych Profile PGEC Profile 50R 75R 1 00R 50R 75R 100R

are 8 9 automatic 80

be 17 1

cons idered 85

data - - 80 37 may 50 91 of 2 2

or 21 27

that 12 19 no. words in common 6 6 7 6 6 9

Rank no . sum 110 1 10 107 107 Figure 2. IRE PGEC abstract No. 1-15

word types.

The purpose of this procedure was to segregate

the abstracts into two files — psychological and

PGEC abstracts, respectively. After considering

a number of decision rules, the following criteria were adopted:

1. An abstract belongs to psychology if the number of words in common with the psychology profile

is greater than the number in common with the

PGEC profile, and conversely.

2. If the number of words in common in the

abstract and the two profiles were equal, the sum

of the rank numbers of those words on the two

fists would be determined, and the abstract assigned

to the profile with the smaller sum. If the sums

were equal, no decision would be made.

Figures 1 and 2 illustrate the data recorded and

the results of matching two abstracts against the

first 50, 75, and the full 100 ranks of the two profiles. In both cases the number of words in the

abstracts contained in the first 50 ranks of the two

profiles is the same. Summing the rank numbers

permits both abstracts to be correctly discriminated by the rule given.

The following table summarizes the results of | matching the psychological and PGEC abstracts

against the first 50, 75, and 100 ranks of the profiles, j

Number correctly discriminated for 50 Ranks 75 Ranks 100 Kanks

50 Psychological abstracts 43 46 47

50 IRE PGEC abstracts 50 50 50

Success ratio 93% 96% 97%

All of the abstracts which were cast into the

"wrong" category by this procedure were psychological abstracts. Examination of the abstracts

contributing to the profiles suggests several reasons

for this. The PGEC abstracts represent a more

specialized subject matter than those from Psychological Abstracts. In general, the PGEC abstracts

contain fewer word types used more frequently.

Consequently the counts contributing to the PGEC

profile are higher than those of psychology.

226
Rank Rank Rank Rank Rank Rank PGEC Word Psych PGEC Word Psych PGEC Word

6 3 56 69 have 43 70 other 16 12 4 7 in 58 59 presented

3 4 and 91 64 into 52 88 problems

8 9 are 7 5 is 45 67 some

11 20 as 23 24 if 71 48 such 39 43 af 49 90 its 82 18 system

17 13 be 50 91 may 29 49 than 14 14 by 44 22 method 12 19 that 100 23 72 63 methods 1 1 the

s: 37 data 30 66 more 33 44 these 34 21 discussed 90 53 new 19 25 this 84 54 94 50 number 46 62 time

10 8 for 2 2 of 5 6 to 96 68 function 13 16 on 36 61 two

69 94 general 65 41 one 20 11 which

64 58 has 21 27 or 9 17 with Mean difference in Rank - 17.4

FIGURE 3. Rank numbers of the 48 words in common in the first 100 ranks ofpsychological and IRE PGEC abstract profiles.

In examining the results it was found that, at the 100 rank level, 88 percent of the successfully

discriminated abstracts were dependent on the 52

words that are unique to each profile, with 9 percent

successfully decided through summing the rank

numbers. It was considered useful to investigate

the discrimination to be obtained by the rank sum

criterion alone, using only words common to the

profiles.

There are 48 words in common on the profiles

in the first 100 ranks. Figure 3 lists the words in common and their ranks. The mean difference of

rank for these words is 17.4, with the lower ranks

tending to larger differences than the higher ranks.

As can be seen from the figure, function words

predominate. The following table shows the results of matching the 100 abstracts against the fist of

48 words common to the profiles and applying the

rank sum criterion:

Correct Incorrect 50 Psychological abstracts 36 14 50 IRE PGEC abstracts 42 8 Percentage 78% 22%

3. Conclusions

The results of this experiment indicate that author

variation in style imposes no serious obstacle to using patterns of common words as discriminators.

Considering the length of the profiles, the small

size of the sample contributing to the profiles, and

the hmited number of word types contained in individual abstracts, the success ratios are sur- prisingly high. It is uncertain, however, to what

degree the results are biased by editorial conventions and style.

The results also tend to support the idea that there is much useful information to be found in the

high-frequency area of word occurrence, and that frequency alone can provide a basis for subject

discrimination of widely different fields, particularly

when all word type occurrences of fully inflected

forms are taken into account. Further work is required to establish the precision which may be

expected of such a technique, especially if ap- plied to fields more closely related than psychology

and computers.

4. Potential Applications

A system designed to make use of common word

patterns through a technique similar to that de- scribed in this paper would include a short table intended to combine the functions of an exclusion

fist with identification of broad subject areas. Such a quick initial segregation would reduce the

search time required for matching against the particular vocabulary of those areas. Figure 4 illus- trates the contrast between using a large dictionary

with the familiar features of exclusion lists, root

stripping, and an extended search of a long table,

and the approach suggested here. The initial

segregation would lead directly to a relatively short

specialized dictionary or to a mismatch monitor.

The thesaurus devices necessary to a large dic- tionary could be simplified, and the range of ambiguity inherent to terms used in many different

fields would be narrowed. It is quite feasible to use specialized tables now, provided the texts are segregated by subject prior to input. This ap- proach, however, looks forward to the application

of optical readers for the transformation of printed

text to machine readable form in systems that do

not require the intervention of a human mind for prior subject classification.

227
TEXT

Exc lusions & Root

Stripping

ProcessedText

Specialized

Dictionary

Subject A

Spec ial ized Dictionary Subject B

Specialized

Dictionary Subject C

Mis-match

Monitor

Processed Text

Figure 4. Schematic flow contrasting a conventional technique with suggested

approach using common word patterns.

5. References

[1] Yule, G. V., A Statistical Study of Vocabulary (Cambridge

Univ. Press, 1944).

[2] Herdan, G., Type-Token Mathematics ('S-GravenHage, Mouton & Co., 1960).

[3] Mosteller, F., and D. L. Wallace, Inference in an authorship

problem, J. Am. Statist. Assoc. 58, No. 302 (June 1963).

[4] Zipf, G. K., Human Behavior and the Principle of Least

Effort (Addison-Wesley, 1949). Borko, H., The construction of an empirically based mathematically derived classification system, Proc. Spring Joint Computer Conf. 21, 279-289 (1962).

6. Appendix. The Profiles

The 300 Psychological Abstracts used to build the rank- ordered profiles for this experiment contained a total of 22,175 word occurrences of 4,587 word types. The 300 IRE PGEC

abstracts contained 23,200 word occurrences of 3,678 word

types. The mean number of word occurrences per abstract was 77.3 for PGEC versus 73.9 for Psychology. When broken

into subsets, both samples exhibited a broad internal range of variation for the expectation that a given word would appear at

a given rank, with the broader range appearing in the Psychological Abstract set. The following table presents a consolidated alphabetic list of words occurring in the first 100 ranks of the IRE PGEC and

Psychological Abstract Profiles, together with their rank numbers. Dots (....) are used instead of a rank number to indicate

that the word does not occur in the first 100 ranks of one or other

of the profiles.

228
Rank number

Psych. PGEC

Word type

63

14 10092

54

0740

10064

05

24

90

71

93

52

3033

may

means memory

mental method

methods more

network new

no

not number

ofonone

only

operation

operations

orother

out output

part perception performance

personality

possible presented problem

problems program programming

psychological psychology

reinforcement

relationship required research response

results

set shown

social solution some

storage study such

switching system

systems

technique

techniques

test than

that the

their theory

these

this time

totwo under

use used

using

various

visual was were when

which

with

229
Clumping Techniques and Associative Retrieval 1

A. G. Dale and N. Dale

The University of Texas

Austin, Tex.

Experimental work applying clump theory to the problem of defining word associations useful

for document retrieval is described. A clump-finding computer program developed by the authors has been successfully used to clump key words in a document-key word data set previously used by

H. Borko of System Development Corporation and M. Maron of RAND Corporation for classification experiments described in the literature. The main features of the program, which permits several

analytical options at execution time, are described. An analysis is made of word associations implicit in a collection of GR-clumps found under a given term-term connection definition. Clump intersections define small subsets of terms that possess

identical properties of contextual distribution and the structure of the subsets forms an associative network useful for retrieval. An algorithm for associative retrieval is suggested. Information on the membership of key words

in GR-clumps can be used to define the context of a retrieval request and to provide a rapid parti- tioning of the document set into relevant and nonrelevant subsets. Clump associations can then be used to order the prospectively relevant documents for output.

1. Introduction

This paper summarizes experimental work

applying clump theory [1, 2, 3, 4] 2 to the problem

of defining word associations in a context where

documents are described by key terms, and of

implementing a retrieval process within an associative network produced by key-term clumps.

For reasons discussed by R. M. Needham [3], who

has been responsible for much of the existing work

on clump theory, experimentation has been largely

confined to work with GR-clumps.3

2. Key-Term Clumping: Data and Software

The clumping experiments were made with a data set supplied by H. Borko of System Development Corporation [5]. The data characterize the use of 90 key terms in 260 documents in a classifi- cation array in which the elements are 1 or 0, depending on whether or not a key term is used in a given document.4 Several connection definitions have been used

in experiments to date, two of which have proved

most useful with these data. Let l(n, m) be the

number of l's in the intersection of rows n and m

in the classification array (i.e., the number of co- occurrences of the nth and /nth terms in the set of

documents), and l(n) be the number of l's in row n

(i.e., the total number of occurrences of the nth

term in the set of documents):

1 Work described in this paper was supported in part by the National Science Foundation under Institutional Grant GU-483 at The University of Texas.

2 Figures in brackets indicate the literature references at end of paper.

3 The definition of a GR-clump is as follows:

U: a finite set of elements, between pairs of which there is a symmetrical rela- tion attaching a real number to each pair, called the connection of the pair. cU, s^: The connection of a pair of elements x and 5. _S: a sub_set of U (s,, s

2

, . . ., s

n

) S*: V-S

CU, S): Sdx, s)VseS _

CU, 5*): 2dx, s*)Vs*tS* AU,S): C(x,S)-C(*,S*)

Hence the bias (bix, S)) of an element x to a subset S is the excess (positive or negative)

of the total connections of x to the members of S over the total connections of x to the members of S*.

GR-clump S: {x | b{x, S) 3 0 and b(y, S) < 0V yeS*}

A subset S of (/_is a GR-clump if all members of S have a positive or zero bias to S and

all members of S* have a negative bias to S, given the convention that rtx, x) =0.

1 The documents are 260 abstracts published in the March and June issues of the 1959 IRE Transactions on Electronic Computers; the topics cover computing hard- ware and computer applications.

Connection def. 1: l(n, m)

Connection def. 2: l(n, m)

Vl(n) • l(m)

FORTRAN programs have been written to compute the appropriate connection matrices and to implement an algorithm for finding GR-clumps in the connection space. Since the clumping pro- cedure works iteratively from an initial partitioning

of the universe, and since a prohibitive number of

possible initial partitions exists, the practicability

of the procedure depends upon heuristics governing

the selection of initial partitions. For clumping

in sparse matrices characteristic of the type of

data used in the experiments, initial partitions

defined by what we have termed the pivot variable

method provide useful starting points. For each

variable a set S consisting of that variable and all other terms with which it has a nonzero connection

is defined, so that in a system of n terms, n initial partitions are considered. The clumping algorithm is essentially as described by Needham [4], following the initial partitioning operation.

Since the size, z, of a GR-clump is typically large —

n/3 < z < n in an n-element universe — several

methods for defining smaller clumps within GRclumps have been tried. For some purposes it may be desirable to work with clumps possessing

strong internal connections, and many GR-clumps

contain fringe elements with small positive bias. Two promising methods found to yield useful

smaller clumps within GR-clumps are as follows:

230
Method 1:

1. Remove elements with minimum bias (min(6)).

2. Recompute the bias of each remaining ele- ment over u.

3. Repeat 1 and 2 until all remaining elements

have a bias to the reduced set greater than min(6).

4. The reduced set is a clump with threshold

min(6).

5. Repeat 1-4.

6. The process ends when the set collapses, i.e., when no set containing elements with bias greater

than min(6) can be found.

Method 2: This uses the same procedures as method 1, except that biases are computed only over the set consisting of the elements of the previous clump

found.

The two methods produce quite different minimal

clumps. For example, consider an element x of

a GR-^lump, S, with a large number of connections

over U. Its bias to S is likely to be small despite

its large number of connections, and it would be

transferred from S early in the method 1 procedure.

However, since the sum of its connections to S

may be large, its bias to reduced clumps in method

2 would also be large, and it would therefore probably be retained in the reduction process. The clump-finding program used in the experiments is executed under three major options per- mitting: (1) location of GR-clumps, (2) location of

GR-clumps and method 1 reduction, and (3) location

of GR-clumps and method 2 reduction. The pro- gram works in core (32K) with connection matrices

of up to 100 variables, and with up to 100 pivot

variable initial partitions on one run. Reprogramming to handle significantly larger connection

matrices is planned. The programs are being

implemented on a Control Data 1604 (FORTRAN

compile-and-go system), with the following average

execution times for finding and reducing one GRclump (or reaching a dead-end) in a 90 X 90 con- nection matrix:

Fixed point

Option 1: 10.8 sec. Option 2: 20.4 sec. Option 3: 30.0 sec. Floating point

Option 1: 5.6 sec. Option 2: 56.3 sec. Option 3: 43.8 sec. 3. Key-Term Clumping: Results

Table 1 summarizes the output of the clumpfinding prodecures outlined above, showing the

number and mean size (number of elements) of

clumps found.

The network implicit in the GR-clump structure,

using the second connection definition, is shown

in figure 1. The relationships for the definition-2 clump structure are shown for illustrative purposes

since the association structure is simpler than that

Table 1.

Connection

GR-clumps

Reduced —

Method 1 Reduced —Method 2

definition No. Mean No. Mean No. Mean

found size found size found size

12

198 52

49

13

7 4449

8

3

1958

No. of clumps

containing

term subsets General Context: Applications General Context; Hardware

7 j©

6

| (2?

5 Q ) « 7)

4 Q o) 02c' Qo)

3

> c ) >

2

FIGURE 1. Strong term associations implicit in GR-clump structure, connection definition 3.

(See table 2 for contents of numbered subsets.)

231
implicit in the definition-1 set of clumps and is more easily diagrammed. Preliminary investigation suggests, however, that the more complex

association structure given by definition-1 clumps

provides better retrieval outputs. The circled

numbers identify subsets of terms appearing in identical clumps; the number of clumps in which

the subset appears is indicated on the left of the

diagram. The contents of the numbered subsets

are identified in table 2. The connecting lines in the network indicate inclusion relations. For

example, two of the three clumps in which subset

No. 3 (mechanical, translation) appears form an

intersection in which subset No. 1 (complexity,

language, Uncol) uniquely appears. These connections specify the strongest association paths in the

network. An interesting contextual partition of

the entire set of index terms is evident; one subnetwork deals largely with hardware topics, the

second with applications, with relatively weak

connections between the two. The retrieval model described below uses the

contextual distributional properties of terms as a basis for associative retrieval.

Table 2. Key to numbered term subsets in figure 1

Subset number

9

hi 1112

1314

15

16

17

111 19

21

2122

2324

25

26

2728

29

30

31

complexity, language, Uncol

arithmetic, expressions mechanical, translation bound, definition, parity chess, mechanisms, process, program, programming, programs

pseudo-random, random

square

average, differential, division, equation, equations, multipbcation, solution,

traffic character, delays, Monte Carlo, shuttle, stage, unit numbers

abacus, boolean, functions, matrix

diffusion, error characters, office section simulation

analog, control, function, generator, plane

code, conversion, elements

adder, carry, network, networks, scientific, synthesis communications, register, decoder, shift, wire

circuit, circuits, counter, logic, pulse, transistor, transistors storage switching

fields element barium

file, information, library, magnetic, processing, tape memory

transmission

printed, recording

side coding, compressions, film, speech

4. Retrieval Model

The retrieval model will be described informally.

Given a collection of m documents described by

n index terms, and k clumps of terms, the initial data arrays are

1. A clump-key term binary matrix, T, with

elements Ty = l or 0 depending on whether or not

the yth term is a member of the ith clump.

2. A document-key term binary matrix, C,

with elements Cy=l or 0 depending on whether

or not the yth term is in the ith document.

A secondary data array D=CTT can be formed,

such that Dij = the number of terms in the ith document contained in the yth clump.

Considering an input request as a binary vector q of dimension n, with <7i=l if the ith term is in- cluded in the request and 0 otherwise, a simple

retrieval model would be

e = DTq (1) where e is an output vector of dimension m, and

€i is the relevancy weight of the ith document with

respect to the input request.

It is evident, however, that this model has several

defects. In particular:

1. It is desirable to partition the set of m documents so that only relevant documents are con- sidered for output. A possible definition of relevancy is to require that an output document

possess a clump list that encloses the clump list of the request (i.e., that the union set of clumps

associated with the key terms of a document enclose

the union set associated with the key terms of a

request). This condition proves to be overrestrictive, since it can lead to the exclusion of

documents that possess some key terms included

in a request. Consequently, we define a relevant

document to be one that either (a) contains key

words included in the request, or (b) possesses a i clump list that encloses the clump list of the request.

Only such documents will be considered for output.

2. It is desirable to normalize the weights, e,, in the output vector, since in the simple model

these weights are directly proportional to the

number of key terms in a document.

3. Other things being equal, a relevant document

with an extensive clump list should have a lower

relevancy weight than a document with a shorter

, clump fist. |. 4. Other things being equal, a relevant document

with a larger number of key terms matching key

terms contained in the request should have a higher

relevancy weight than one with a lower number of matches.

A model satisfying these conditions is: e=[D's GV]+m (2) where

D' is a submatrix of D of dimension rX k, and r = the

number of documents satisfying the relevancy

criteria.

s = Tq defined above.

G is a diagonal matrix of dimension r X r, such that

G,i is the ratio of the number of relevant clumps

attached to document i (i.e., the number of clumps

that match the clump fist of the request) to its ! total clump list.

232
V is a diagonal matrix of dimension rXr, with Vu

the reciprocal of the number of key terms in document i. m is a row vector of length r, such that mi = k

xi ly,

where k is a constant, Xi is the number of request

terms contained in the key term list of the ith document, and y is the number of key terms contained in the input request.

Thus, [D's~\ is a row vector, the elements of which

are the crude relevancy scores for r relevant docu- ments; [D'sG] is a row vector in which the document scores have been modified to reflect what

5. Retrieval

An algorithm simulating retrieval model (2) de- scribed above has been programmed. Retrieval

requests were executed in each of the associative

networks implicit in the clump structures found

under the two different definitions. The principal purposes of the retrieval experiments were

1. To examine the efficiency of the model in partitioning the document collection into relevant

and nonrelevant subsets.

2. To compare retrieval output from the two as- sociative networks.

3. To examine the validity of the relevance

weighting scheme.

5.1. Partitioning Efficiency

In evaluating the suitability of the retrieval model

for use with large document collections, its effi- ciency in initially partitioning the set of documents

to identify a prospectively relevant subset is important. Efficient partitioning will reduce search

time and computation time associated with the cal- culation of relevance weights.

In 19 test retrieval requests, the mean number of documents retrieved per request was approximately

84.5 from the clump structure of connection definition 1 (19 clumps), and 110.8 from the clump

structure of connection definition 2 (8 clumps).

The standard errors are approximately 5.8 and 7.1 respectively.

These data suggest that the mean number of

documents that would be retrieved per request

using clump structure 1, over a large number of requests, would be in the range of about 73 to 96

documents, and using clump structure 2 in the range 97 to 125 documents (at the 95 percent confidence level). Thus, from clump structure 1, we

would expect, on the average, an initial partitioning

of the set of documents to be of the order of 28

percent to 37 percent of the collection; from clump

i structure 2 the retrieval algorithm would produce

an average initial partition in the range 37 percent

to 48 percent of the collection. These figures illustrate that the partitioning effi- ciency of the model is directly related to the number

might be termed the "contextual dispersion" of the document key terms; and [D's GV\ is a row

vector of normalized relevancy scores. The values

in m give added weight to documents for key word

matches. The exponential weighting scheme has

the desirable property of increasing the relative weight of rrii in the model for larger values of y and

%i\ this is intuitively satisfactory since requests

containing a large number of key terms are likely

to require more specific outputs than general

requests using fewer (and probably broader) terms. The value of the parameter k may be modified to adjust the relative weight of m in the model.

Experiments

of key word clumps available to it in a given col- lection of documents with a given set of key words.

It can be shown that the expected number of clumps

to be found in some set S' will probably be greater

than in some set S, if S' D S, since the possible number of clumps is greater in S' . Thus, for a document collection of a given size, it is probable that partitioning efficiency would improve if the set of

descriptive key terms were increased.

It should be recognized that the efficiency of the

retrieval algorithm, as measured by the number of

documents returned as a result of a search, is a function of a number of variables, including (a) the

frequency of use of key terms in the documents

and (b) the distributional characteristics of terms

in the key term clump structure, in addition to the

number of key term clumps. The properties of

this function are being investigated. In general, however, if it is assumed that the

initial partitioning ratio is improved by the use of

larger key term sets (producing more key term

clumps), then the model appears to be adaptable

for retrieval in large collections, provided a suit- ably large set of key terms is used for clumping

and a suitably large number of clumps are identified. Further experimentation is planned to permit

estimates of initial partitioning ratios attainable in larger collections.

5.2. Comparison of Retrieval Outputs

As noted above, the output fists from clump struc- ture 2 tend to be larger than from clump structure

1. Considering the set of retrieval requests as samples with re =19 in each structure and testing

for a significant difference between the mean length

of output fists generated, the null hypothesis is rejected at the 0.01 significance level (t = 2.958,

exceeding the critical value of approximately 2.72 with 36 deg of freedom). Thus, there is a signifi- cant difference between the mean lengths of the

output fists, outputs from structure 1 being significantly shorter. The relevancy ordering of documents within re- trieval outputs was also compared. Output lists

772-957 0-66— 16 233
from three of the 19 retrieval requests were ran- domly selected, and relevancy weights computed

for retrieved documents by the system were nor- malized. For each request, documents retrieved

from structure 1 were located in the corresponding

structure 2 outputs, to produce paired observations of normalized relevance weights. If a document from structure 1 did not appear on the

corresponding structure 2 output list, the second

member of the pair was assigned a zero value. This

procedure provided 260 observations of paired

relevance weights. Linear correlation of the

variables yielded a correlation coefficient of 0.3448, a rather low value, but nevertheless significant

at the 0.01 significance level. Two conclusions are permissible:

(a) Significantly shorter output lists are generated

from structure 1. (b) Significant correlations exist between the

relevancy orderings generated by the retrieval algorithm in clump structures using different con- nection definitions defining term associations.

The second point is of interest since it indicates

that different nearness definitions can produce

comparable relevancy orderings (or, alternatively,

the association structure generated by one near- ness definition will resemble, at least grossly, the

associations produced by an alternative definition).

A practical consequence of the two conclusions

noted above is that it may be desirable to work

with a connection definition that yields the most

clumps, rather than making an a priori selection

of a particular definition as a basis for clumping.

5.3. Validity of the Retrieval Model

We have not, at this stage, undertaken any rig- orous validation of the retrieval model, or of the

relevancy weighting scheme. However, informal

validation of the following type has been undertaken:

(a) Four individuals with general familiarity of

the subject fields covered by the set of 260 documents were given four randomly selected retrieval requests and asked to independently prepare fists of documents relevant to the requests by scanning

the 260 abstracts and identifying documents on a three-valued relevancy scale ranging from most

relevant (1) to possibly relevant (3).

(b) The manually prepared lists for a given re- quest were consolidated and a sublist of documents

most relevant to the request was prepared. This

sublist comprised documents rated with a value

of 1 by at least two of the four individuals, or rated

with a value of 1 by one individual and rated 2 by

at least two others.

(c) Comparisons of manual and automatic re- trievals are given in table 3. Request 1 asked for documents dealing with

language translation. Request 2 asked for documents dealing with circuitry in analog computers.

Request 3 was for documents on simulation. Re- quest 4 called for documents dealing with pro- gramming languages.

TABLE 3. Comparison of manual and automatic retrievals

Number

Total number of documents retrieved Number of most rele- vant docu- ments in up- per fourth of output tostsNumber of most rele- vant docu- ments in reNumber of most rele- vant docu- ments not

Request number

of most

relevant documentsAutomatic mainder of output Hsts retrieved

identified Manual Structure Structure Structure Structure

1 2 1 2 1 2 1 2

1 10 19 104 105 7 9 2 1 1 0

2 14 63 89 100 10 10 4 4 0 0

3 15 43 119 94 8 11 6 1 1 3

4 12 32 115 181 8 11 3 1 1 0

Using the rule outlined in (b) above, 10 documents

most relevant to the first request were identified from a union set of 19 documents retrieved by the four investigators. The retrieval algorithm pro- duced ordered lists of 104 documents using struc- ture 1, and 105 documents using structure 2. In the upper fourth of the output list from structure 1, 7 of the 10 most relevant documents were located,

and in the upper fourth of the structure 2 output

list, 9 of the 10 most relevant documents. The

algorithm failed to retrieve one of the most rele- vant documents using structure 1, but retrieved all the relevant documents using structure 2. The table indicates the generally satisfictory

performance of the retrieval model and confirms

the reasonableness of the definition of relevance

used.

It also again suggests that the choice of nearness

definition as a basis for clumping may not be criti- cal to retrieval performance.

In some respects the output from the model is even better than the data suggest. For example,

in executing the retrieval request for documents

dealing with the use of computers for simulation,

the algorithm produced towards the top of its output

lists a number of documents covering Monte Carlo

processes and the generation and use of random

and pseudo-random numbers. Keference to these

documents in response to a general request for information on simulation is quite reasonable, and

is an interesting indication of the associative capabilities of the system.

234
6. Summary

The experiments described above were designed

to yield information on the utility of a document

retrieval model working with term associations

implicit in a system of key term clumps, and the

potential performance of such a retrieval model in large collections. The results are suggestive rather

than conclusive, but justify further empirical work

with larger collections than the one used. The

data in table 3 also suggest that efficient retrieval

in large collections might utilize user feedback,

based on scrutiny of initial system output. Thus,

if it is the case that the system's denotations will generally coincide with those of a given user, one

retrieval strategy would be to output the upper part

of the response list generated in response to the

initial request, and take the user's specifications

of most relevant items in this subset as a basis for a reordering of the remaining documents.

7. References

[1] A. F. Parker-Rhodes and R. M. Needham, The Theory of Clumps (Cambridge Language Research Unit, Cambridge,

England, 1960).

[2] R. M. Needham. The Theory of Clumps II (Cambridge

Language Research Unit, Cambridge, England, 1961).

[3] Research on Information Retrieval Classification and Grouping, 1957-61 (Cambridge Language Research Unit, Cambridge, England, 1961).

[4] R. M. Needham, A Method for using computers in information

classification (Presented at IFIPC, Munich, 1962, mimeo).

[5] See H. Borko and M. Bernick, Automatic document classification, J. Assoc. Computing Machinery 10, 151-162

(1963).

235

Statistical Association Methods for Simultaneous

Searching of Multiple Document Collections

William Hammond*

Datatrol Corporation

Silver Spring, Md. 20910

A technique is described for using statistical association methods for machine retrieval from a

large collection of documents when individual elements of the collection have been indexed by different agencies employing different indexing vocabularies. The objective is to develop a mechanized approach lor providing the kind of Government-wide

clearinghouse information retrieval service described in the "Crawford Report" [l] 1

; or, in the words

of the report, "to undertake and coordinate, on demand, appropriate simultaneous searches and service multiple collections." The approach envisions superimposing a common subsumption scheme onto the indexing data

of the different agencies; this would inject a significant degree of commonality, and would provide the base, or framework, for deriving equivalent retrieval terms by computer. In actual practice, each

agency would tag each report it enters into its system with the common terminology of the scheme. The association profiles of these common terms would serve as points of departure for mechanized

searching.

Experimentation in this approach with NASA and DDC indexing data is discussed. Examples

of term association profiles generated during the experimentation are included.

To condition myself for this program, I turned to my favorite reference work: How to Lie with Statistics \2]. (It is really how to catch a liar, rather

than be one.) This book makes reference to the

work of Sir Francis Galton, who once said of statistics: "I have a great subject to write upon, but

feel keenly my literary incapacity to make it easily

intelligible without sacrificing accuracy and thor- oughness."— Some of us recognize the same literary

incapacity a century later. For this reason we

welcome the opportunity to discuss our work at such a forum as this in advance of publication.

Our unique contribution in this field — if indeed

our contribution is unique — is in the area of computer software and in our application of statistical associative techniques to operating systems — and

in particular, our current experimentation with

these techniques to achieve compatibility among

the large Federal technical information systems.

We are currently working with the NASA and DDC

files. Our presentations to this Symposium, mine and

that of Mark Seidel, are somewhat in the form of progress reports. My paper deals with our efforts

to achieve compatibility among different information systems — that is, compatibility of the nature

required for integrated announcement and retrieval

of Government research reports. Seidel deals with some aspects of the computer software that we

have developed for the manipulation of the files of large information systems in the course of our

investigations. In June of 1963, we were asked to undertake a study of the various approaches to the common

vocabulary problem of the large Federal technical

1 Figures in brackets indicate the literature references at end of paper. *Now with ARLES Cooperation. McLean, Va. 22101.

information agencies. This was one of the many

problem areas that had to be resolved for the suc- cessful operation of an integrated clearinghouse

service. To provide us with expert consultation on the objectives and operations of the various

Government agencies involved, an Inter-Agency

Vocabulary Study Group was formed under the

Operating Committee of COSATI (Committee on

Scientific and Technical Information, Federal

Council for Science and Technology). This group

of consultants was composed of senior personnel

from the information facilities of the Department

of Defense, Department of Commerce, Atomic

Energy Commission, Department of Health, Education, and Welfare, Department of Agriculture, National Aeronautics and Space Administration, and

the National Science Foundation. The study was

accomplished under a National Science Foundation contract, and under the monitorship of the

Head, Office of Science Information Service, of the Foundation [3]. We concluded that if the decentralized facilities retain their current mission orientation, a com- mon indexing vocabulary would be essentially a composite of the working vocabularies that the

operating agencies currently employ. Assuming

such a composite vocabulary were in use, we stiU could not formulate reliable search patterns for multicollection retrieval solely on the basis of the

prescriptive indexing data of any "common the- saurus" of this nature.

It is true that where the interests of the different agencies coincide or overlap, their indexing of a common subject it recognizably similar, at least

to those familiar with the subject matter. However,

where the interests of the different agencies do not

coincide, their indexing of common subject matter

is dissimilar even if they have common indexing

terms available.

237
Table 1 was compiled from current indexing of the two major information facilities. It is a sampling of the extreme variations in use of a common

set of indexing terms by NASA and DDC to index an identical set of 966 research reports. From a review ol the data in this figure, you can readily

appreciate the difficulty in selecting corresponding

search terms for the two systems solely on the basis

of identical terms appearing in a listing of their indexing vocabularies.

Table 1. Sampling of variations in DDC-NASA usage of the common terms for indexing an identical set of reports

NASA Term DDC NASA Term

use use use use

10 15 Ablation 1 19 Maps

30 60 Absorption 99 67 Measurement

4 20 Acceleration 1 12 Microscopes

1

45 Air 1 10 Navigation charts

lo Airborne 2 25 Numbers

13 18 Aluminum 15 36 Optics

1 7 Automation 12 28 Oscillation Brightness 8 14 Oxidation

o 18 Calibration 1 7 Pilots 13 19 Combustion 1 5 Planets

g 22 Configuration 45 108 Pressure

5 17 Connection 33 57 Propagation

7 17 Cooling 7 16 Protons

12 7 Copper 1 12 Pumps

4 20 Deceleration 25 17 Reliability 4 14 Deflection 19 37 Resonance

43 21 Deformation 1 3 Sapphires 50 104 Density 1 14 Skin

5 17 Diffraction 2 8 Sky

13 70 Distribution 7 15 Spheres

13 29 Earth 7 14 Spin

35 26 Elasticity 31 52 Stability

1 35 Emissivity 1 44 Steel 30 99 Energy 10 60 Stresses

15 30 Excitation 8 18 Sun

22 50 Functions 27 10 Table

17 8 Glass 3 22 Telescopes

8 16 Graphite 75 190 Temperature

11 91 Heat 104 41 Theory

4 29 1 leating 3 20 Tracking

36 25 Instrumentation 29 12 Turbulence

23 49 Ionization 28 87 Velocity 26 56 Ions 1 5 Venus

3 7 Learning 30 50 Vibration

13 38 1 nading 12 18 Viscosity 6 8 Visibility

We seek to achieve a degree of compatibility

that will permit a clearinghouse operation to accept

the original abstracting and indexing of the different federal agencies (at this point in time we are con- cerned with AEC, NASA, DDC, and OTS) and auto- matically integrate these different data into an- nouncement publications to meet the varied

interests of the national scientific community. The

clearinghouse should also be capable of providing

effective retrieval of report literature on the basis

of original indexing.

One of the significant conclusions resulting from

our study for COSATI was that a common subscription scheme, superimposed on the indexing data

of the different agencies by a human intermediary,

would inject a significant degree of commonality

for integrated announcement — and at the same time

would provide a context or framework of "common

generic denominators" for identifying equivalent access paths for searching the multiple collections. For the approach to compatibility that we are

investigating, we have compiled a list of broad

subject headings that subsume the entire subject

coverage of the Federal scientific and technical

report literature. These broad subject headings —

or generic denominators — as we have developed

them in our initial effort actually comprise a basic

common vocabulary of some 225 terms. Although

our experience to date is far from conclusive, the

indications are that the current list may be too small. Perhaps our final hst will be closer to 300 terms. Much will depend on the consistency — recognizably

consistent patterns — that indexers can maintain

with an acceptable degree of reliability.

It is proposed that each participating agency

require its indexers to assign one or more of these

broad subject headings to each document processed

into its system. In this manner, the subject indexer

would be adding the set of common generic denominators that we just referred to, providing points

of departure for generating context sets or term profiles of statistically associated terms. These

term profiles of the generic denominators, as you

will see later, suggest the equivalent access paths

for retrieval. We have had many obstacles to overcome in establishing the validity of our concept. Not the

least was to design a computer system that would

permit economical manipulation of the data for experimentation. We can now generate the statistically associative data and produce the term

profiles for either the NASA or DDC system in about two hours on an IBM 7090. We can update

the system in a fraction of that time.

For our present experimental corpus we have

generated the individual term profiles for all 12,000

terms in the NASA machine vocabulary and the

7,000 terms in the DDC thesaurus.

Although the NASA subject indexing vocabulary

has not been structured into the subsumption

scheme of a thesaurus, our generic denominators

accommodate the NASA indexing patterns more

readily than they do the DDC indexing patterns.

We can organize the existing NASA indexing data

into our own scheme with a modest computer

effort. The existing DDC indexing data will require

a good deal of human effort. We have printed out the corresponding term

profiles in the DDC and NASA systems for several

of our generic denominators. We are now investi- gating the use of these corresponding profiles

from the two systems for selecting the initial search

terms for each system. From this point on the

search, including associative expansion to formulate

the final hst of search terms, continues independently in each system.

Since the profiles of the generic denominators in fact reflect the "state of each collection" for the

given subject, this approach appears to be most

promising. We have been able to examine only

those subject areas where the indexing data of both

systems are already in consonance with our scheme

of generic denominators. Some examples are shown in the appendix. Individual profiles in the

two systems are shown for NAVIGATION, GUIDANCE, THERMODYNAMICS, and HEAT TRANS238
FER. We have also shown the terms listed in the

DDC Thesaurus of Descriptors under the group

THERMODYNAMICS and under the group NAVIGATION and GUIDANCE.

At the present time we are using the Stiles Association Factor [4] as a threshold for selecting the

associated terms in the profiles. We also plan to use the statistical associative concept as one of the

elements in ordering the output of the computer

search. We have currently suspended our experimental

work on multisystem searching while we are implementing the full associative search capability for

the NASA collection which by now has grown to 60,000 reports and is increasing by almost 5,000

reports a month. This will provide an ample test bed for future experimentation.

We feel that it is important to keep in mind that our discussions concern retrieval of report literature,

not retrieval of data or generation of information

from data stored in a machine system.

Our current emphasis on retrieval of report

literature is based on the belief that we are going

to have to five for some time to come with the status quo in the indexing and abstracting of the large

Federal technical information agencies. Additionally, our actions must be tempered by the vast "information in being" represented by several

million reports in the various agency collections. Mechanized information retrieval — that is, re- trieval of report literature as it is practiced today —

is at best a "gray" affair. It involves the interplay of many models of human endeavor throughout

the information transfer chain — from the recorder

to the information handler to the ultimate user. The objective of retrieval under the current modus

operandi is to satisfy the needs of the user without

requiring him to review an undue amount of nonessential bibliographic data to select pertinent

reports. In any given instance, it is unlikely that the information handler will know how well-informed

the user may be, and what is nonessential. One

realistic compromise that we are striving to attain through statistical associative techniques is to provide a high recall ratio and to fist a probable

order of relevance for the reports cited. When we consider the human indexing model —

as yet not clearly defined — together with information

retrieval practices of the operating agencies, it is difficult to provide a firm measure of effective- ness of any approach to retrieval, particularly multicollection retrieval. There are many elements,

however, that are measurable. We can evaluate

parallel operations on the basis of time and cost

factors, and the usefulness of the output. Another

important factor is the optimum use of the human

resources that are available to perform the intel- lectual tasks required to support the system. These

factors, together with the vast "information in being" that we referred to earlier, were the basis

for our initial experimentation with statistical associative properties of indexing data. Our cur- rent efforts are motivated by the positive results of our experimentation over the past two years.

References

[1] Scientific and Technological Communication in Government. AD 299 545 (Apr. 1962).

[2] Huff. D.. How to Lie with Statistics (W. W. Norton and Co., New York, 1954).

3] Hammond, W., and S. Rosenborg, Common approaches for

Government scientific and technical information systems, Tech. Tept. IR-10 (AD-430,000) (Datatrol Corporation,

Silver Spring, Md., Dec. 1963).

[4] Stiles, H. E., The association factor in information retrieval,

J. Assoc. Comp. Mach. 8. 271 (1961).

239

Total usage frequency of parent term

Total usage frequency of associated term

Total co-occurrence with parent term

Association factor ( X 100) TERM PROFILES

NASA

442 GUIDANCE

13 7 529 Aboard

57 16 550 Abort

130 30 594 Apollo*Project 60 16 544 Autopilot 1101 70 513 Computer

2059 141 599 Control*/Noun/

824 71 560 Controls, *Control*Systems

126 21 518 Gyroscope

285 52 623 Inertia 410 47 556 Landing*/Noun/

489 54 565 Launch*/Noun/

119 25 564 Maneuver

61 14 513 Matching

49 33 720 Midcourse

640 55 533 Missile*/Noun/ 340 40 542 Mission 356 132 798 Navigation 933 80 572 Orbit*/Noun/

22 8 502 Pershing*Missile

61 14 513 Platform 838 63 528 Propulsion*/Noun/

800 55 500 Reentry

163 35 602 Rendezvous

8 6 546 Sextant

52 21 619 Space*Navigation

979 107 635 Spacecraft*/Noun/

15 7 514 Spacecraft*Navigation

2681 139 552 System

302 34 520 Target

51 14 533 Telecommunications

86 22 573 Terminal

30 10 518 Tracker

545 50 532 Tracking*/Noun/

761 115 683 Trajectory*/Noun/

DDC

403 GUIDANCE

250 28 628 Astronautics

136 21 632 Automatic Pilots 207 16 527 Booster Motors

14 9 689 Celestial Guidance

125 18 608 Command & Control Systems

762 34 541 Communication Systems

670 32 543 Control 1564 120 735 Control Systems

69 14 618 Doppler Navigation

1188 58 607 Errors 345 29 599 Flight Paths

32 11 647 Guided Missile Computers

285 31 635 Guided Missile Trajectories 2476 156 740 Guided Missiles 242 23 589 Gyroscopes

42 16 698 Homing Devices

115 39 779 Inertial Guidance

80 12 569 Inertial Navigation 44 7 515 Interception 242 25 607 Landings

81 11 549 Launching Sites 12 7 650 Light Homing

228 28 638 Lunar Probes

172 26 652 Manned

348 21 527 Moon

298 36 662 Navigation

79 15 618 Navigation Computers

861 85 728 Orbital Trajectories

Appendix

DDC —Continued

137 17 586 Planets 277 23 574 Propulsion

12 6 616 Radar Homing

523 26 526 Reentry Vehicles 59 22 729 Rendezvous Spacecraft

18 5 534 Retro Rockets

1782 67 589 Satellites (Artificial) 800 74 707 Space Flight 170 56 813 Space Navigation 303 27 598 Space Probes

901 82 715 Spacecraft 99 10 506 Stabilization Systems

23 8 612 Star Trackers

1025 67 657 Surface to Surface 4 4 637 Terminal Guidance

TERM PROFILES

NASA

356 NAVIGATION

oy n ft Ooy

lip* zu Air*Traffic

d.9 Lo Ooo Airspace

130 30 618 Apollo *Project

21 7 500 Avoidance

15 7 536 Circumlunar

665 47 520 Communication

18 9 572 Compass

1101 73 557 Computer

16 8 559 Doppler*Navigation 959 58 519 Flight*/Noun/ 442 132 798 Guidance*/Noun/

10 7 579 Gyrocompass

126 23 564 Gyroscope

285 34 554 Inertia 119 17 503 Maneuver

49 17 603 Midcourse

340 31 511 Mission

933 54 505 Orbit*/Noun/

61 12 503 Platform

83 57 800 Proportion 838 63 559 Propulsion*/Noun/

163 21 513 Rendezvous

8 5 527 Self-Contained 8 6 568 Sextant

52 27 694 Space*Navigation

979 64 541 Spacecraft*/Noun/

15 7 536 Spacecraft*Navigation

2681 112 530 System

30 11 562 Tracker

545 45 536 Tracking*/Noun/

761 48 505 Trajectory*/Noun/

DDC

298 NAVIGATION

203 18 588 Air Traffic Control Systems

1156 34 526 Airborne

218 19 592 Airplane Landings

57 11 618 All-Weather Aviation 136 17 619 Automatic Pilots 53 14 677 Beacon Lights 25 5 531 Bombing

50 10 611 Buoys

50 7 533 Celestial Navigation 39 12 676 Compasses

7 3 543 Course Indicators 204 13 517 Direction Finding 643 33 591 Display Systems

69 9 555 Doppler Navigation

241
DDC—Continued DDC THESAURUS

Continued

179 17 590 Flight Instruments 345 19 541 Flight Paths 970 28 503 Flight Testing 36 7 568 Fog Signals 48 6 503 Glide Path Systems

57 17 710 Ground Controlled Approach Radar

9 3 517 Ground Position Indicators 403 36 662 Guidance

242 16 543 Gyroscopes

27 12 713 Hyperbolic Navigation 80 14 634 Inertial Navigation 44 8 576 Instrument Flight 81 19 697 Instrument Landings

164 55 844 Lighthouses

22 6 585 Loran

11 5 615 Loran Equipment

10 3 506 Low Altitude 8 3 529 Navigation Charts

79 20 710 Navigation Computers

69 14 649 Navigational Lights 247 21 600 Position Finding

161 15 574 Radar Beacons

9 3 517 Radar Bombing

795 32 559 Radar Equipment

116 31 762 Radar Navigation 58 8 547 Radar Reflectors 65 10 584 Radio Beacons

458 32 623 Radio Equipment

135 34 765 Radio Navigation

187 13 527 Shipborne

199 24 652 Ships

170 16 582 Space Navigation 788 63 707 Symposia

9 4 585 Terrain Avoidance

337 17 519 Transport Planes

DDC THESAURUS

GROUP 106 NAVIGATION AND GUIDANCE

ALL-INERTIAL GUIDANCE

AUTOMATIC NAVIGATORS

AUTOMATIC PILOTS

AZIMUTH

CELESTIAL GUIDANCE

CELESTIAL NAVIGATION

CIRCULAR ERROR PROBABILITY

CONTROL SIMULATORS

DEPTH FINDING

DEPTH INDICATORS

DIRECTION FINDING

DIRECTION FINDING SIGNALS

DOPPLER NAVIGATION

GLIDE PATH SYSTEMS

GUIDANCE

HEAT HOMING

HOMING DEVICES

HYPERBOLIC NAVIGATION

IMPACT PREDICTORS

INERTIAL GUIDANCE

INERTIAL NAVIGATION

INJECTION GUIDANCE

LIGHT HOMING

LORAN

LORAN EQUIPMENT

MAGNETIC GUIDANCE

MAGNETIC NAVIGATION

NAVIGATION

PRESET GUIDANCE

PROPORTIONAL NAVIGATION

RADAR HOMING

RADAR NAVIGATION

RADIO HOMING

RADIO NAVIGATION

RENDEZVOUS GUIDANCE

SHORAN

SPACE NAVIGATION

STABILIZED PLATFORMS

STAR TRACKERS

STELLAR MAP MATCHING

TELEVISION GUIDANCE

TERMINAL GUIDANCE SYSTEMS

TERRAIN AVOIDANCE

VIDEO MAP MATCHING

WIRE GUIDANCE

TERM PROFILES

DDC

1426 THERMODYNAMICS

525 49 507 Aerodynamic Heating

722 77 573 Air

145 24 511 Beryllium Compounds

49 15 534 Boiling 421 50 544 Boron Compounds

146 39 619 Calorimeters

120 44 667 Chemical Equilibrium 1598 145 615 Chemical Reactions 1007 127 647 Combustion

114 30 590 Combustion Chamber Gases

260 79 705 Dissociation 1046 92 563 Energy

196 111 808 Enthalpy

182 107 808 Entropy

131 44 657 Equations of State

71 21 566 Eutectics 343 39 512 Exhaust Gases

10 6 505 Film Boiling 329 40 524 Flames

476 49 522 Fluid Mechanics

1208 139 644 Gas Flow

618 58 526 Gas Ionization 1673 246 735 Gases

366 55 585 Heat

123 65 746 Heat of Formation

18 11 576 Heat of Fusion 48 23 629 Heat of Reaction

9 6 516 Heat of Solution 24 15 612 Heat of Sublimation

1935 281 747 Heat Transfer

1820 169 634 High Temperature Research

1037 100 586 Hydrogen

451 47 519 Hypersonic Characteristics 437 55 561 Hypersonic Flow

45 14 528 Hypersonic Nozzles

16 9 545 Irreversible Processes

169 34 571 Liquid Metals 514 59 556 Liquids 292 35 508 Lithium Compounds

276 33 502 Mass Spectroscopy 340 48 563 Mixtures

25 13 577 Nucleate Boiling 1964 128 549 Oxides

1158 89 539 Oxygen

688 83 598 Phase Studies 1839 117 535 Physical Properties 3041 152 515 Pressure 49 14 519 Propellant Properties 474 83 646 Reaction Kinetics 201 34 550 Recombination Reactions 874 74 535 Refractory Materials 155 34 582 Rocket Propellants 1242 87 521 Skock Waves

596 53 508 Solid Rocket Propellants 850 89 586 Solids 170 27 518 Solubility 249 106 773 Specific Heat

130 26 543 Specific Impulse

242
DDC — Continued

5195 235 540 Temperature

5051 217 521 Theory

450 86 660 Thermal Conductivity

134 29 563 Thermal Diffusion 259 116 787 Thermochemistry

322 67 645 Transport Properties 145 51 677 Vapor Pressure 236 51 621 Vaporization 379 55 580 Vapors

222 40 575 Zirconium Compounds

DDC —Continued

NASA

498 THERMODYNAMICS

OO 18 515 Calorimetry

607 53 514 Combustion

268 43 574 Dissociation 24 12 569 Effusion 198 57 672 Enthalpy

52 21 606 Entrance

148 67 738 Entropy

81 22 566 Envelope

488 91 670 Equilibrium

41 22 641 Free*Energy

1811 132 583 Gas*/Noun/

1214 106 587 Heat*/Noun/

65 24 610 Heat*Capacity

18 9 537 Heat*Content

1036 80 538 High*Temperature

1015 93 580 Property 326 39 526 Specific 2834 155 552 Temperature*/Noun/

70 16 513 Vapor*Pressure, *Tension

119 27 567 Vaporization

DDC THESAURUS

GROUP 157 THERMODYNAMICS

EQUATIONS OF STATE

ENTROPY

ENTHALPY

HEAT

HEAT OF ACTIVATION

HEAT OF FORMATION

HEAT OF REACTION

HEAT OF SOLUTION

HEAT OF SUBLIMATION

HEAT TRANSFER

JOULE-THOMSON EFFECT

SPECD7IC HEAT

THERMODYNAMICS

TERM PROFILES

DDC

1935 HEAT TRANSFER

264 92 702 Ablation 1578 119 511 Aerodynamic Characteristics 519 57 502 Aerodynamic Configurations 525 226 817 Aerodynamic Heating

722 77 528 Air

541 103 640 Atmosphere Entry

291 79 658 Blunt Bodies

331 54 554 Bodies of Revolution 49 26 620 Boiling 783 209 754 Boundary Layer

1007 90 514 Combustion

253 51 576 Compressible Flow

360 60 568 Conical Bodies

215 119 780 Convection

21 13 563 Cook-off

128 64 706 Coolants

591 196 773 Cooling 900 115 597 Cylindrical Bodies

196 56 629 Enthalpy

10 9 563 Film Boiling 27 15 567 Film Cooling

20 10 511 Flat Plate Models

975 142 637 Fluid Flow

476 79 595 Fluid Mechanics

208 34 506 Fluids 66 562 Friction 1208 244 736 Gas Flow

1673 178 613 Gases

366 55 544 Heat

226 118 773 Heat Exchangers

502 66 544 Heating

62 17 500 Hemispherical Shells 1820 132 514 High Temperature Research

451 109 676 Hypersonic Characteristics 437 126 712 Hypersonic Flow

217 43 556 Hypersonic Wind Tunnels

300 68 620 Hypervelocity Vehicles 429 169 778 Laminar Boundary Layer

26 13 540 Liquid Cooled

169 47 607 Liquid Metals 514 65 537 Liquids 168 31 513 Mach Number

7383 369 534 Mathematical Analysis 167 39 567 Nose Cones

25 18 615 Nucleate Boiling 214 43 558 Pipes 3041 222 570 Pressure

31 15 552 Radiators 28 12 514 Reactor Coolants 523 85 600 Reentry Vehicles 179 38 552 Reynolds Number

414 61 552 Rocket Motor Nozzles 950 83 502 Rocket Motors

428 53 513 Shock Tubes

NASA

1100 HEAT TRANSFER

237 ou 550 rVULalHjll 175 58 598 Aerodynamic*Heating

109 53 635 Boiling 666 201 714 Boundary*Layer

261 52 518 Conduction

252 120 716 Convection 450 109 622 Coobng*/Noun/

198 53 561 Enthalpy

304 61 535 Flatness, *Flat 2537 350 659 Flow*/Noun/

615 86 513 Fluid*/Noun/

19 12 509 Free*Convection

1811 170 505 Gas*/Noun/

1214 334 754 Heat*/Noun/

98 41 591 Heat*Flux

127 116 786 Heat*Test

481 99 589 Heating, *Heated

691 138 619 Hypersonics

392 127 675 Laminar

626 85 507 Layer

86 42 612 Mass Transfer

799 98 503 Nozzle*/Noun/

22 17 570 Nucleate

23 17 565 Nusselt*Number

651 89 513 Plate 594 83 509 Point*/Noun/ 64 34 600 Prandtl*Number

43 26 587 Radiative 267 67 576 Reynolds*Number

240 48 510 Skin

295 107 672 Stagnation 2834 261 547 Temperature*/Noun/

140 62 640 Temperature *Distribution 34 17 520 Temperature*Profile , 1255 154 550 Thermal*/See*Also*Thermo-, *Heat/

187 40 501 Thermocouple

677 309 808 Transfer*/Noun/

362 82 583 Turbulent 448 70 511 Viscosity 419 82 562 Wall*/Noun/

50 34 628 WalPTemperature

243

Studies on the Reliability and Validity of

Factor-Analytically Derived Classification 1 Categories

Harold Borko

System Development Corporation

Santa Monica, Calif. 90406

A series of experiments has been conducted in order to determine whether a factor-analytically derived classification system is reliable and valid. In a previous experiment, 10 classification cate- gories were derived by factor analyzing 618 abstracts of psychological reports. Two new samples

of psychological abstracts, numbering 659 and 338 respectively, were factor analyzed. The three independently derived classification schedules were compared and found to be quite similar. It was

concluded that factor-analytically derived classification categories are reliable in that the factors remain essentially stable from sample to sample. The categories are also valid in that they are descriptive of the main divisions of the psychological literature.

1. Introduction and Purpose

One aspect of documentation research is con- cerned with deriving a mathematical theory of

classification that will provide a basis for dividing a

collection of documents into major subject categories. A number of mathematical techniques for deriving classification systems have been suggested.

These include factor analysis [l],2 clump theory

[2, 3, 4], latent-structure analysis [5], and discrimination analysis [6]. At the System Development Corporation, with support from the National Science

Foundation, we are continuing to investigate the

application of factor analysis to the problems of

document classification with the aim of determining

whether a factor-analytically derived classification system is (a) reliable — in the sense that successive samples

from a given data base will yield the same factors,

and

(b) valid —in the sense of being descriptive of

the content of the documents.

2. Determining Reliability

A classification schedule is said to be reliable if the categories, which were derived on the basis of one sample of documents, are equally descriptive

of other samples taken from the same population.

One of the claims made for mathematically derived

classification systems is that the categories so derived are descriptive of the documents used in the

analysis. However, if the categories prove to be

so unique that they describe only the one document

set and no other, they would be of little value. In order to determine the stability, or reliabihty, of

factor-analytically derived classification categories, a series of experiments was conducted using three

different samples of documents selected from the

psychological literature.

3. Results of Previous Study

In the 1961 experiment by Borko [1], 618 abstracts of psychological reports were selected from the

publication Psychological Abstracts, vol. 32, number

'This document was produced in connection with a research project cosponsored l lgro

by SDC's independent research program and a grant from the National Science Founda- ?

1 Figures in brackets indicate the literature references at end of paper. These abstracts Were keypunched, analyzed by

245
means of the FEAT program [7], and 90 highfrequency clue words, called "tag terms", were

selected. The 90 words and the 618 abstracts were

arranged in the form of a data matrix and correlation coefficients based upon the co-occurrence of the

words were computed. The resultant 90 X 90

correlation matrix was factor analyzed [8], and the

10 factors extracted were interpreted as classifica- tion categories. A report of this study has been

published previously.

4. Selection of Sample

To establish the proposition that a factor-analytically derived classification system is reliable and

does not vary from sample to sample, it is necessary

to repeat the factor analysis using a new collection

of abstracts. Approximately 1,000 abstracts of

psychological reports were selected from Psychological Abstracts, vol. 35, number 1, 1961. Abstracts vary in length and in style. To insure that the sample would be relatively uniform and the

selection unbiased, only abstracts between one and

two inches in length were included in the study.

This reduced the number from 1,430 abstracts contained in that issue to 997. Next, the collection was divided into two groups by selecting approximately every third abstract. The first group, consisting of 659 abstracts, was labeled the experiment

group; the second, consisting of 338 abstracts, was

called the validation group. An independent factor analysis was performed on each group, thus providing an additional check on the reliability of the

resulting factors.

5. Selection of Tag Terms 3

All 997 abstracts were keypunched for computer

processing by means of the FEAT program, which

prepared a fisting, by frequency of occurrence, of

all words appearing in the text. Function words

and other common words were excluded. One

hundred and fifty tag terms were chosen by the

investigators from this fist of frequently occurring

words. Appropriate words with the same root were combined manually. In the previous study,

90 tag terms were used, but since then the capacity

of the factor-analysis program has been expanded,

and it is now able to handle a larger matrix. The

150 tag terms are listed in table 1. The words

marked by an asterisk are also on the list of 90

words used in the previous study. Only 16 words

from this original list do not appear on the present

list of 150 terms.

6. Data Matrix, Document-Term

Having selected the terms, it was necessary to determine which documents (i.e., abstracts)

contained each of these words. This information was recorded in the form of a matrix; the columns

show the 150 terms, and the rows indicate the docu- ments. Each document is an abstract selected for

3 The writer perfers to use "tag term" rather than key words or index terms to describe the automatic assignment of labels to documents. The words assigned are

tags by which a document can be identified and compared with other documents. The tag terms do not necessarily describe the basic contents of the document nor are they true index terms; they are, to repeat, simply tags.

analysis in this study. A small portion of this matrix is illustrated in table 2. Two such matrices were prepared, one for the 659 documents in the

experimental group and the other for the 338 documents in the validation group. A computer

program prepared the document-term matrix in a form suitable for input to the factor-analysis pro- gram. Since the data consisted of 150 terms, two

80-column cards were produced for each of the docu- ments. Every term was assigned a unique column

on the cards, and the number of times a word

occurred in the document was punched in the proper

column.

246
Table 1. Tag terms.

*1. ability 39. employed *77. mental *115. science

2. academic 40. error 78. monkeys 116. sensitivity

*3. achievement *41. experiment 79. motivation 117. sensory

4. action 42. eye 80. motor 118. situation

*5. activity *43. factor *81. nature *119. social

6. adaptation 44. failure 82. negative 120. sound

7. adjustment *45. family 83. nervous *121. speech

8. administered 46. feebng 84. noise 122. statistically

9. adults *47. field *85. normal *123. status

*10. analysis 48. fond *86. organization 124. stimulation

11. animals *49. frequency *87. patient *125. stimulus

*12. anxiety 50. frontal 88. people 126. stress

*13. attitude *51. function *89. perception *127. structure

14. auditory 52. grade *90. performance *128. student

15. average *53. group *91. personal 129. subjective

*16. behavior 54. hand *92. personality 130. support

17. baby 55. health *93. personnel *131. system

*18. boys 56. hearing 94. physical 132. task

*19. brain 57. hospital 95. population *133. teacher

*20. case 58. hypnosis 96. probability *134. technique

*21. child 59. hypothesis *97. problem 135. temporal

*22. clinical 60. image *98. procedure *136. test *23. college 61. independent *99. program *137. theory

24. color *62. information *100. psychiatric *138. therapy

25. communication *63. intelligence *101. psychological 139. threshold

*26. community 64. intensity 102. questionnaire 140. tone

*27. concept 65. interaction 103. rat *141. training

28. conditioning 66. interest 104. rate *142. treatment

*29. correlation 67. I.Q. 105. reaction 143. trials 30. cortex *68. knowledge *106. reading 144. validity *31. data 69. language 107. reflex 145. value

32. delinquency *70. learing *108. reinforcement 146. verbal

33. dependent *71. level *109. research *147. visual

*34. development *72. Ufe *110. response 148. vocational

35. discrimination *73. light 111. retarded 149. women

36. dogs 74. male *112. role 150. words

*37. (^HiifJitinn CUULdLlUU to. man *1 1 Q

1 lo. scale

*38. emotion 76. medical *114. school Table 2. A portion of the data (document-term) matrix.

Doc. #

Behavior

Experiment

Group

Learning

Response

Stimulus

Test

313 0 0 4 1 0 0 2 323 1 0 2 1 1 1 1 334 0 0 0 4 0 2 2 347 0 1 0 6 1 1 1 349 0 2 3 0 0 0 1 364 0 1 3 3 0 0 1 382 2 1 0 1 2 2 1 383 2 0 0 2 0 0 0 385 1 0 0 0 3 0 (I 438 0 0 4 3 0 0 3

7. Correlation

The data matrix indicates the number of times

each term occurred in the various documents.

Based upon this information, the degree of association among terms can be computed as a function

of their occurrence in the same set of documents.

A measure of this association is the correlation coefficient, the formula for which is shown in table 3.

* Items marked by an asterisk were also on the list of 90 words used in the previous study (see ref. [5]).

trix, Term-Term

The solution to this formula results in a decimal

number ranging from + 1.000 to - 1.000. + 1.000

indicates a perfect correlation, namely, that every

time word X occurs, word Y is sure to appear in the same document. A zero correlation means that there is no predictable relationship in the co- occurrence of these words in documents. A negative correlation means that if word X occurs then

word Y is not likely to occur in the same document.
The actual correlations were calculated on a computer and printed in the form of a 150 X 150

matrix. Over 11,000 correlation coefficients were

computed. A portion of this matrix is illustrated in table 4. The number in each cell is the correlation

coefficient. Here we can see that behavior has a

slight positive correlation with experiment and

learning, an essentially zero correlation with group

and response, and a negative correlation with

stimulus and test.

TABLE 3. Computation of correlation coefficient. NZXY-(2X) (2F)

rxv Vt/YXY2 - (XX)2

] [NXY* - (2F)2

] N— Number of documents

: Terms being correlated

TABLE 4. A portion of the correlation (term-term) matrix.

Behavior

Experiment

Group

Learning

Response

Stimulus

Test

16. Behavior .2702 .0359 .0096 .0454 .0112 - .0353 - .0818

41. Experiment .0359 .1930 .1190 .0424 .1594 .0432 .0297

53. Group .0096 .1190 .2835 .0746 .0070 -.0161 .0981

70. Learning .0454 .0424 .0746 .2958 .0549 .0714 .0524

110. Response .01 12 .1594 .0070 .0549 .2489 .2265 - .0220

125. Stimulus -.0i5i .0432 -.0161 .0714 .2265 .3334 - .0422

136. Test - .0818 .0297 .0981 .0524 - .0220 - .0422 .3141

8. Factor Analysis

By means of factor analysis, the information

contained in the 150 X 150 correlation matrix is compressed into a smaller matrix with fewer

columns. Obviously, as a result of this compression, some information contained in the original

matrix is lost. Information must always be lost as we go from the specific to the general — as we go

from specific data about collies, terriers, and poodles

to the single concept "dogs" — or more appropriately

as we go from a series of papers dealing with the causes and treatment for hysteria and schizophrenia

to the single classification category labeled "etiology

and treatment of mental disorders." Factor

analysis is a mathematical technique designed to reduce the matrix to a small number of eigenvectors

accounting for a large proportion of the total variance. There is always some questions as to when enough factors have been extracted. In this case, in order to maintain consistency with the pre- vious study, 10 factors were extracted and rotated

orthogonally before interpretation. One factor was bipolar and so was interpreted as representing

two classification categories.

Two factor analyses were computed — one using

the 659 documents in the experimental group and

the second using the 338 documents in the valida- tion group. These, plus the 1961 study, provide

three derived classification schedules for psychological literature.

9. Comparison

In interpreting the stability of the factor-derived

classification categories, the three sets of factors

will now be compared. All three are based upon

different samples of documents as recorded in Psychological Abstracts, 1958 and 1961. Further- more, in the earlier experiment only 90 tag terms were used, as compared with 150 in the current

study. Nevertheless, it is hypothesized that the

factors will be relatively stable from sample to sample and regardless of difference in the tag terms

used for analysis. Is this the case?

Let us examine in detail the factors from each

study that are labeled "academic achievement."

For convenience, the words with significant load- ings on each of these factors are listed side by side

in table 5. In the 1961 study, the words with the highest

loadings on this factor are girls, and boys. While

boys was used as a tag term in the present study,

girls was not. However, the word with the highest

loading for both the other groups is student. This

carries substantially the same meaning as girls and

boys. School and achievement appeared with high

loadings on all three sets of factors.

248
Table 5. Words with significant loadings on academic achievement factor.

Study

Current study

Experimental

group

Validation group

girls student student boys achievement achievement

school test college achievement school ability reading grade school

college grade administered test independent average program academic

knowledge motivation

correlation science medical

scale

Reading was a legitimate word, but it did not appear in the current study; however, the other

words on the two current lists are clearly related to "academic achievement."

Based upon this analysis, we conclude that all three studies contain a factor which could be properly labeled "academic achievement." In other

words, this factor is stable and reliable. As a second example, let us examine the factors dealing with "physiological psychology" (table 6). These are not nearly as similar as was "academic

achievement," and the interpretation had to be

stretched on a Procrustean bed to achieve some

degree of commonality. The three lists in table 6 have very few words in common, and yet there is a unifying theme dealing with the structure and

function of the central nervous system. The words

cerebral, cortex, frontal, temporal are all related to

Table 6. Words with significant loadings on central nervous system factor.

1961 study

Current study

Experimental

group

Validation group

emotional animals perception

development activity color cerebral frontal communication

child (children) cortex field theory field structure

life behavior analysis nature nervous temporal

factors) conditioning

the brain. Research in this area has many facets. Some studies are concerned with the development

of the cerebral cortex in children and its psychological concomitants. Extirpation experiments on

animals are designed to study behavior as a means

of determining localized brain activity. In the case

of humans with structural brain damage, one is concerned with functional loss, such as perception

and communication, and the possibilities of conditioning and retraining. Consequently, in spite of

the fact that the words are different, all three factors

refer to a single broad category of research papers

and so are given a common interpretation.

Finally, let us examine the factor named "etiology

and treatment of mental disorders" (table 7). Clearly the words in the two groups of the current study are quite similar. There is also considerable

agreement with the 1961 study; however, the 1961

study had an additional factor called "therapy —

case studies," which did not appear as a separate

factor in the current analysis. A possible reason

is that the older data contained significantly more

reports of therapy cases than did the more recent sample of literature. At any rate, the net effect

is that two factors under the general heading of

"clinical and abnormal psychology" were compressed into one. Nevertheless, it is reasonable to conclude that this factor configuration is relatively

stable. Let us now take a more global view of all three

factor-analytic studies and compare them for similarity (table 8). Under the major heading of

"educational psychology," we see a factor in each

analysis labeled "academic achievement." Similarly each analysis has a factor dealing with

"physiological psychology" and the slight dif- ferences among these factors were discussed.

Next, under "clinical and abnormal psychology," we note that the two original factors on this topic were compressed into one. In "experimental

psychology" the opposite situation occurred.

The 1961 study was based upon a relatively limited

literature in this area — an accident of sampling —

and as a result only one factor emerged. In the

present study — again as a vagary of sampling —

there was a large amount of experiment literature and five separate and distinct factors were derived.

This change reflects the heavier concentration of

experimental papers in the more recent psychological literature. At the same time, we lost the special category of "clinical case studies"

and combined this group of documents with the more general class of "clinical and abnormal

psychology." Two factors in the 1961 analysis

did not appear at all in the present study. These

are Factor 4, "studies of college students," which was known to be a poorly defined factor, and Factor

8, "general psychology." This latter factor probably deserves a place in the classification system.

The documents which could reasonably be classified under "general psychology" were probably divided

into the various experimental categories.

249

772-957 0-66— 17
Table 7. Words with significant loadings on

etiology and treatment of mental disorders

factors.

Current study

117U1 Of » 1 ctnrJis siuuy

Experimental Validation group group

treatment patient patient

psychiatric hospital hospital

clinical therapy treatment

ps y ciiui irci dp y treatment psychiatric medical community

schizophrenia group techniques

ii ici ti^j y mental attitude group(s) psychiatric therapy

psychoanalysis program population

counseling community emotion women

personal

case(s) therapy

level

The obtained results help reveal both the

strengths and weaknesses of the factor-analysis

technique for deriving classification categories.

The factors which emerge from the analysis are

closely related to the data used in the study. To

the extent that the data base is an adequate sample

of the total document collection, the factor-derived

categories will represent the entire collection. To the extent that the sample is only partially representative, the factors will be only partially representative of the total collection, but adequately rep- resentative of the sample on which they are based.

The reasonableness, or validity, of the factor- analytically derived classification categories can

be determined by comparing the derived classi- fication schedule with the classification system

used by the American Psychological Association

(APA). As is to be expected, the factor-analytically derived categories are fewer in number and more general in character. Many fine distinctions are lost as, for example, the distinction between

"human experimental psychology" and "animal

psychology." Nevertheless most of the major

Table 8. Comparison offactor names.

Factors derived from current experiment

Factor name

Experi- mental

group,

factor #

Validation group,

factor #

Factor number and name

Experimental psychology

Conditioning 2 1 Learning and reinforcement 8A 2 Feelings, emotion, and motivation 5 10A 2. Perception and learning Vision and the special senses 9 5 Speech and hearing 10 8

Physiological psychology

Central nervous system 6 9 9. Developmental psychology

Social psychology

Community resources 8B 6 3. Community organization

Clinical and abnormal psychology

Etiology and treatment of mental

disorders 4 4 6. Clinical psychology and

therapy

10. Therapy — case studies

Educational psychology

Academic achievement 1 3 1. Academic achievement

Interest and ability testing 3 10B 5. School guidance and

counseling

Special problems 7 7 7. Educational measurement

4. Studies of college students

8. General psychology

Factors derived from

1961 experiment

250
headings do appear, as do some of the important

subdivisions. It is thus reasonable to conclude

that the factor-analysis technique has uncovered

the most important dimensions, or trends, in published psychological research literature. On the basis of the above analyses, it is concluded

that factor-analytically derived classification categories, based upon representative samples of the

total document collection, are reasonably reliable and descriptive. However, because of the diffi- culty of obtaining a truly representative sample of

a document collection, more than one factor analysis

should be made to attain a stable constellation of

factors. By repeating the analysis every year or

so and adding the new accumulations to the data

base, changes in the character of the collection can be identified quickly and automatically, and a

revised classification schedule created. Obviously,

a change in classification categories without a concomitant reclassification of all the documents

in the collection would be worse than useless. The documents will all have to be reclassified, and

while this is normally a chore, it can be accomplished automatically by using a factor-score prediction equation. In actual practice, the physical

documents will be stored by accession number, and

the reclassification will consist of a new set of

properly arranged file cards, which will be printed

as an output of the computer processing routines.

Used in this manner, factor-analytically derived

classification categories provide the flexibility and

responsiveness to change that are needed in scientific documentation and provide a basis for an auto- mated document storage and retrieval system.

9. References

[1] Borko, H., The construction of an empirically based mathematically derived classification system, Proc. Spring Joint Computer Conf. 21, 279-289 (1962).

[2] Needham, R. M., The theory of clumps II, Cambridge Lan- guage Research Unit, M. L. 139 (Cambridge, England,

1961).

[3] Needham, R. M., Research on information retrieval classi- fication and grouping 1957-61, Cambridge Language

Research Unit, M.L. 149 (Cambridge England, 1961).

[4] Parker-Rhodes, A. F., Contributions to the theory of clumps,

Cambridge Language Research Unit, M. L. 138 (Cambridge, England, 1961).

[5] Baker, F. B., Information retrieval based upon latent class

analysis, J. Assoc. Comp. Mach. 9, No. 4, 512-521 (1962).

[6] Williams, J. H., Jr., A discrimination method for automatically

classifying documents, Proc. Fall Joint Computer Conf. 24, 161-167 (1963).

[7] Olney, J. C, FEAT, an inventory program for information

retrieval, FN-4018 (System Development Corporation, Santa Monica, Calif., 1960).

[8] Harman, H. H., Modern Factor Analysis (Univ. of Chicago

Press, Chicago, 111. 1960).

251

10. Appendix I. Factors Derived in the 1961 Experiment

1. Academic Achievement 6. Clinical Psychology and Psychotherapy

Tag-Terms

girls

boys

school

achievement

reading

Loadings

.74

.73 .30

.20

.18

Experimental Psychology-Perception and

Learning

Tag-Terms

perception(ual)

learning

experimental

theory

evidence

visual

field

Loadings

.46

.36 .29

.25 .24

.23

.21

3. Social Psychology and Community Organization

Tag-Terms Loadings

organization .67 community .54 structure .38 workers .22

field .15 analysis .15 social .11 role .10 job .10

4. Studies of College Students

Tag-Terms Loadings

student(s) -71 college -70 group(s) .17 mental -16

factor(s) -15 teacher -14 intelligence -11 personality -10

5. School Guidance and Counseling

Tag-Terms Loadings

program .42 education(al) .36 child(children) .33 parents .29 guidance .29 teachers .28 intelligence .27 school(s) .25 counseling .20

Tag-Terms

treatment

psychiatric

clinical psychotherapy

case(s)

schizophrenia

theory

group(s)

psychoanalysis

counseling

7. Educational Measurement

Loadings

.44

.35

.32 .22

.16

.16

.16

.12

.12

.11

Tag-Terms Loadings

achievement .46

ability .36 correlation .35 scale .32 group(s) .22 reading .30 intelligence .20

test(s) .20 school(s) .19

General Psychology — Psychology As A Science

Tag-Terms Loadings

social .42 research .32 science .31 psychological .25 status .24

Developmental Psychology

Tag-Terms Loadings

emotional .32 development .32 cerebral .23 child(children) .22 theory .19

life .18 nature .18

factor(s) .18

10. Theory: Case Studies

Tag-Terms

personal

case(s)

therapy

level

Loadings

.56

.55

.42

.21

253
11. Appendix II. Factors Derived In the 1964 Experiment for Experimental Group and Validation Group

Experimental Group

1. Educational Psychology:

Academic Achievement

Tag-Terms

student

achievement

test school

grade

college

administered

independent

program

knowledge

correlation

medical

scale

Experimental Group

2. Experimental Psychology:

Conditioning

Tag-Terms

conditioning

reflex stimulus

academic

stimulation

visual

auditory

action

dogs

motor

sound

reaction

college

threshold nervous

Validation Group

3. Educational Psychology:

Academic Achievement

Tag-Terms

student

achievement

college

ability

school

grade

test average

academic

motivation

science

Validation Group

1. Experimental Psychology:

Conditioning

Loadings Tag-Terms

Loadings

.57 nervous to .51 reflex

.48 ability .47 conditioning

.44 dogs

.34 cortex

.32 system

.32 motor

.29 stimulus

.25 auditory

.24 failure .24

.23

.73

.63 .63

.62 .36

.32

.31 .29

.25

.21

Experimental Group

Loadings 3. Educational Psychology:

'75 Interest and Ability Testing

.43

.37

.36

.31 .28

.28

.26

.26

.25 .24

.23

.21 .20

Tag-Terms Loadings

physical .69 women .64 interest .63 achievement .58 teacher .39 grade .32

ability .29 motor .27

Loadings

.63

.61 .56

.40

.36

.33

.31 .29

.27 .26

.25

Experimental Group

Clinical and Abnormal Psychology Etiology and

Treatment of Mental Disorders

Tag-Terms Loadings

patient .56 hospital .43 therapy .32 treatment .32 medical .30 group .27 mental .27 psychiatric .25 program .20 community .20 254
Experimental Group

Experimental Psychology: Feelings, Emotion,

and Motivation

Tag-Terms

emotion

feeling

science nature

psychological

motivation

personality

Validation Group

Loadings

.67

.67

.47

.45

.32 .28 .27

10B. Educational Psychology:

Interest and Ability Testing

Tag-Terms

scale

physical

behavior

intelligence

child

test

Loadings

.35

.25

.25

.23

.22 .20

Validation Group

Clinical and Abnormal Psychology Etiology

and Treatment of Mental Disorders

Tag-Terms

patient

hospital treatment

psychiatric

community

techniques

attitude

therapy

population

emotion

women

Validation Group

Loadings

.64 .50

.47

.45

.36

.35 .34

.37

.27

.26

.23

10A. Experimental Psychology:

Feeling, Emotion, and Motivation

Tag-Terms Loadings

frontal .31 performance .29 training .27 concept .27 emotion .24 problem .22 research .20

Experimental Group

6. Physiological Psychology:

Central Nervous System

Tag-Terms

animals

activity frontal cortex

field behavior nervous

Experimental Group

7. Educational Psychology:

Special Problems

Tag-Terms

retarded

mental

child

LQ.

academic

achievement

behavior

boys

normal

Validation Group

9. Physiological Psychology:

Central Nervous System

Tag-Terms

perception

color

communication

field structure

analysis

temporal

conditioning

Validation Group

7. Educational Psychology:

Special Problems

Tag-Terms

normal

LQ.

intelligence

child

dependent

trials learning

boys

task

negative

verbal

test motor

Loadings

.60

.58 .58

.35

.33 .30

.29

Loadings

.52 .50 .44

.41 .30

.22

.22

.21 .20

Loadings

.77 .65

.42 .34

.34

.25

.21 .20

Loadings

.57 .49

.44

.39

.38

.33

.32 .29

.28

.24

.23

.22 .20 255
Experimental Group

8A. Experimental Psychology:

Learning and Reinforcement

Tag-Terms Loadings

learning .34 response .33 reinforcement .27 performance .26 rate .24 verbal .23

rat .23 discrimination .22 experiment .22 stimulus .21 task .21 group .20 function .20

Validation Group

6. Social Psychology:

Community Resources

Tag-Terms

health

development

child

education

physical

community

research

social

mental

personality

program

concept

emotion

frontal psychological

Loadings

.66 .54

.41 .37 .36

.35

.32

.31 .29

.28

.25

.23

.22

.21

.21

Experimental Group

8B. Social Psychology:

Community Resources

Tag-Terms

health

community

mental

social

psychological

Validation Group

Loadings

.27

.25 .24

.23

.22

Experimental Group

Experimental Psychology:

Vision and the Special Senses

Tag-Terms Loadings

image .60 baby .43 negative .32

field .28 procedure .26 visual .23

light .20 temporal .20

test .20

2. Experimental Psychology:

Learning and Reinforcement

Tag-Terms Loadings

animals .53 rate .52 response .47 group .42 sensory .41 rat .35

trials .35

light .31 reinforcement .30 conditioning .29 experiment .25 fond .23

Experimental Group

10. Experimental Psychology

Speech and Hearing

Tag-Terms

words

language

hearing

speech

structure threshold tone

Loadings

.34

.31 .28

.24

.24

.21

.21 256
Validation Group Validation Group

5. Experimental Psychology:

Vision and the Special Senses

Tag-Terms

light

sensory

stimulation

function

intensity

visual

rat baby

auditory

brain

eye

animals

cortex

frontal retarded

8. Experimental Psychology:

Speech and Hearing

Loadings Tag-Terms Loadings

.58 employed .54 .54 noise 4.0 .51 frequency .41 .42 stress .41 .39 population .40

.38 words .39

.36 speech .37

.35 emotion .35

.27 concept .27

.27 system .24

.22 response .23

.21 .21

.21 .20

257
»

I

I

I

f

I

I

i

e
Postscript:

A Personal Reaction to Reading the Conference Manuscripts

Vincent E. Giuliano

It was with great regret that I was unable to attend

the conference because of sudden illness. Nonetheless, in my capacity as a member of the committee backing the Symposium, I have had an opportunity to read over the manuscrips carefully. In reading the manuscripts I felt an absence of remarks

of an evaluative nature. I have been informed

that there was a great deal of lively discussion during

the conference, although it was unfortunately impossible to include this material in this volume.

This postscript represents a personal comment

based on the written record of the Symposium, since

the absence of commentary might otherwise make it difficult for readers not familiar with the field to piece together a coherent perspective.

The discussions in this book revolve around one

central theme, but the theme is approached from a

variety of viewpoints which are often conflicting in emphasis, objectives, and methodology. The

main questions which surround the theme are whether the work is of fundamental or transitory

significance, whether the techniques will actually

prove out in large-scale operational practice, and,

in general, what the future for research in this area

will hold. To repeat some remarks conveyed in the Intro- duction, my overall impression is that the work

rests on quite solid fundamentals, but that it remains

in a very preliminary stage of development and

further clarification of objectives is essential. There are excellent theoretical foundations drawn

from the fields of statistics, mathematical psychology, and a tradition of empiricist philosophy.

In many instances, the techniques and methodologies used have been previously applied to a number

of closely related problems in other fields besides

documentation, and are known to be effective. An

ability to produce potentially useful results has been

demonstrated in several problem areas, including

document retrieval, automatic classification, and

handling of citations. The methodologies are mostly based on use of very simple counting techniques, with relatively few major questions of workability yet to be resolved. In contrast with some of

the other research approaches to problems of

machine-aided documentation, such as those based

of complex types of logical or grammatical analysis,

many of those discussed in this volume seem to offer a real prospect of producing useful results in the

foreseeable future. Passing now to what remains to be done, there are

at least three areas in which more must be learned

about the statistical association techniques; one

area has to do with what the techniques themselves

consist of, another has to do with their usefulness,

and the third has to do with the very goals and

objectives of the work itself.

First, it soon becomes evident to the reader that

at least a dozen somewhat different procedures

and formulas for association are suggested in the

book. One suspects that each has its own possible

merits and disadvantages, but the fine between the

profound and the trivial often appears blurred.

One thing which is badly needed is a better understanding of the boundary conditions under which

the various techniques are applicable and the ex- pected gains to be achieved through using one or the other of them. This advance would primarily

be one in theory, not in abstract statistical theory

but in a problem-oriented branch of statistical theory.

Secondly, it is clear that carefully controlled

experiments to evaluate the efficacy and usefulness

of the statistical association techniques have not

yet been undertaken except in a few isolated in- stances. It is not surprising that this is so, for before one attempts to undertake a careful evaluation, one first of all wants to convince oneself that there is something worth evaluating. Nonetheless,

it is my feeling that the time is now ripe to conduct

carefully controlled experiments of an evaluative

nature, for example, experiments which are designed

to measure when and how much a statistical tech- nique for document retrieval yields improvements

over conventional coordinate-type retrieval systems.

Similar experiments are required for the other

applications. Such experimental work has, to some degree, been undertaken by several investi- gators using relatively small document collections. This work has been and continues to be useful,

but extension of evaluation experiments to document collections of realistic size is an essential next step: many problems of system performance

are known to be dependent on collection size. My third main point is to open to question the

perspective implicitly adopted in much of the exist- ing work in our area — that the techniques are to be

mainly useful for completely automatic rather than

merely machine-aided document retrieval, abstracting, etc. Personally, I am far from convinced that completely automatic document retrieval (i.e., without use of either an expert who knows the re- trieval system or of external user-machine feedback)

is ever going to be a really useful activity except

perhaps in certain highly specialized subject areas. Most of the machine searching systems that are now

in existence are man-machine systems; they are

likely to remain man-machine systems even if the

standards of machine performance can be improved.

As yet, however, there has been only modest investi- gation of using the associative techniques within

such a more general man-machine framework.

Also, a wide variety of alternative techniques for 259
scientific communication have been proposed and

discussed in the literature, including document

dissemination based on citations or based on re- searcher interest profiles, etc. It is my suspicion

that the system configuration for the next generation of automated documentation systems will not be merely an extension of a term-indexed coordinate

retrieval system, but be something quite different; thus consideration of overall directions must pre- cede the detailed planning of future research.

Finally, I would also like to remark briefly on equipment limitations. In the paper by Baker, a discussion is given on the limitations of existing

digital computers; the impression may be left that

it is impossible to deal with collections of more than

300 index terms with existing machines. I do not

feel that the limitation is this bad; there are numerous shortcut techniques for dealing with sparse

matrices. Both Spiegel and Stiles have dealt with

collections of more terms than these, and at Arthur

D. Little, Inc., we are currently experimenting with

association of over 1,500 index terms and over

100,000 documents using an IBM 7094 computer.

Nonetheless, the economics of manipulating very

large matrices of index terms leaves something to be desired. This has proved to be one of the constraints upon evaluating the proposed procedures on a reasonably large scale and may well be a ban

to implementation of the statistical association

methodology even if it is shown to provide improved

performance. These considerations continue to suggest, in my opinion, that it would pay to look

further into the area of large capacity, inexpensive

permanent memory devices which would handle

associative processing in a special-purpose manner.

For example the fact that certain forms of associative processing can be carried out directly by means

of simple passive analog network devices could

radically change the economics of reducing the

techniques to practice. The development of either

soft-ware schemes or processing devices which

affect the economics of associative processing by

making simpler the handling of relative large system

matrices thus merits our continued interest and

attention.

260
THE NATIONAL BUREAU OF STANDARDS

The National Bureau of Standards is a principal focal point in the Federal Government for assuring maximum application of the physical and engineering sciences to the advancement of technology in industry and commerce. Its responsibilities include development and maintenance

of the national standards of measurement, and the provisions of means for making measurements

consistent with those standards; determination of physical constants and properties of materials;

development of methods for testing materials, mechanisms, and structures, and making such tests as may be necessary, particularly for government agencies; cooperation in the establishment of

standard practices for incorporation in codes and specifications; advisory service to government

agencies on scientific and technical problems; invention and development of devices to serve special needs of the Government; assistance to industry, business, and consumers in the development

and acceptance of commercial standards and simplified trade practice recommendations; administration of programs in cooperation with United States business groups and standards organizations

for the development of international standards of practice; and maintenance of a clearinghouse

for the collection and dissemination of scientific, technical, and engineering information. The

scope of the Bureau's activities is suggested in the following listing of its three Institutes and their organizational units.

Institute for Basic Standards. Applied Mathematics. Electricity. Metrology. Mechanics.

Heat. Atomic Physics. Physical Chemistry. Laboratory Astrophysics.* Radiation Physics.

Radio Standards Laboratory:* Radio Standards Physics & Radio Standards Engineering. Office

of Standard Reference Data.

Institute for Materials Research. Analytical Chemistry. Polymers. Metallurgy. Inorganic

Materials. Reactor Radiations. Cryogenics.* Materials Evaluation Laboratory. Office of Standard Reference Materials.

Institute for Applied Technology. Building Research. Information Technology. Perform- ance Test Development. Electronic Instrumentation. Textile and Apparel Technology Center. Technical Analysis. Office of Weights and Measures. Office of Engineering Standards. Office

of Invention and Innovation. Office of Technical Resources. Clearinghouse for Federal Scientific

and Technical Information.**

*Located at Boulder, Colorado, 80301. "Located at 5285 Port Royal Road, Springfield, Virginia, 22171.

U.S. GOVERNMENT PRINTING OFFICE : 1966 0 —772-957

261

I