—V 

~~~~~ 

— 

~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

~~~~ 

V -- V V_ 1.U~~~

LEVEL

Technical Paper 319 LAD ____

THE PERCEIVED FAVORABLENESS OF

SELECTED SCALE ANCHORS AND RESPONSE

ALTERNATIVES

Josephine L. Matthews, Calvin E. Wright , and Kenneth L. Yudowitch

Operations Research Associates

V and

James Geddie and R. L. Palmer U.S. Army Research Institute for the Behavioral and Social Sciences

FORT HOOD FIELD UNIT

C~3

• !

~~~~~~

19T8

~~~~

___ 

B

U. S. Army

Research Institute for the Behavioral and Social Sciences

August 1978

Approved for public re lease . fis tru h u t r o n u n limi ted .
~~—~~~~-— : ~~~~~~~~~~~~~ - -

F4

U. S. ARMY RESEARCH INSTIT UTE

FOR TH E BEHAVIORAL AND SOCIAL SC IE NC ES

A Field Operating Agency under the Jurisdiction of the

Deputy Chie f of Staf,f for Personnel

V 

WILLIAM L. HAUSER

JOSEPH ZEII)NER Colonel, US Army

Technical Director (Designate) Commander

Research accomplished under contract

to the Department of the Army

Operations Research Associates

NOTICES

DISTRIBUTION Primary distribution of thit report has been made by ARI. Please address correspOndence

concerning distribution of reports to: U. S. Army Research Institute for the Behavioral and Social Sciences . ATTN. PERI l’, 5001 Eisenho~

wr Avenue , Alexandria , Virginia 22333

FINAL DISPOSITION: This report may be destroyed when it i no long er needed. Please di, not return it to the U. S. Army Research Institute for she Behavioral and Social Science,

~~~ The findings in this report are not to be construed as an official Department of the Army position . unless so designated by other authorized documents.

--

— -

~~
Unclassified 

SECU"ITV CLASSIP'ICATION OP' THIS ~AGE (Wit., DMo P:ntored) 

REPORT DOCUMENTATION PAGE READ INSTRUCTIONS 

BEFORE COMPLETING FORM 

[1. "E"'O"T NUMBE" r· GOVT ACCESSION NO J. RECIPIENT'S CATALOG NUMBER 

Tecpnical Paper 319 /_ 

4. TITLE (.,d ....,Uio) S. TYPE OF REPORT 6 PERIOD COVERED 

THE PERCEIVED FAVORABLENESS OF SELECTED SCALE 

ANCHORS AND RESPONSE ALTERNATIVES <:-::- --

•• PERP'O"MING O"G. REPORT NUMBER 

-- 7 . AUTHO"fo) •• CONTRACT OR GRANT NUMBER(e) 

Josephine L. Matthews, Calvin E. Wright, Kenneth L. 

Yudowitch (Operations Research Assoc.), D ' AHC19- 74-C- 003 

James c. Geddie, and R. L. Palmer 

t . ~EIU'O-ING O"GANIZATION NAME ANO ADDttESS 10. PROGRAM ELEMENT. PROJECT. TASK Operations Research Ass~iates, Palo Alto, CA 94303 AREA 6 WORK UNIT NUMBERS 

and U.S. Army Research Institute for the Behavioral 2Q763743A775 and Social Sciences (PERI-OH), 5001 Eisenh9w~r Ave . Alexandria, VA 22333 I I. CONTttOLLING OP'IfiCE NAME AND ADD"ESS 12. REPORT D.OTE _../"' 

TRADOC Combined Arms Test Activity Aug ust 1 978 

Fort Hood, TX 76544 13. NUMBER OF PAGES 

2 4 

. 14. MONITO"ING AGENCY NAME a ADD"ESS(II dllla,..t ,_ Controlll ... Olllco) IS. SECURITY CLASS. (of tllle -II) 

u.s. Army Research Institute for the Behavioral Unclassified and Social Sciences, 5001 Eisenhower Avenue, 

Alexandria, VA 22333 I So. ~~~~:~~I:ICATION DOWNGRADING 

-- 111. DtST"'BUTtOM STATEMENT (ol lftlo lt..,) 

Approved for public release; distributior unlimited. 

17. DIIT"IBUTION STATEMENT (ol tlto •ottacr .,leNd In IJioelr 20, If dllla,..t ,_ ltoport) 

--

11. SU~~LEIIENTA"V NOTES 

-- . 

II. ICEV WO"DS (C_II_., ,..woo oldo If noeo .. _, ...t ldonllly ty aloelr ,_ .. ,, 

Questionnaire construction 

Test development 

Response alternatives 

~ 

Scale anchors 

Response scales 

10. A81T"ACT (C_,..., ,. ... .,. If-~ -.1 l ..... llfr., aloelr _._) 

The purpose of the study was to obtain objective estimates of the extent to 

which respondents perceive selected questionnaire response alternatives as 

general indicators of favor or disfavor toward attitudinal objects. Li.,ts of· 

many descriptive terms denoting various degrees of acceptability (e.g., moderat~ll acce~table), adequacy (e.g., slightll inadeguate) , and relative 

goodness (e.g., much worse) were presented to 51 u.s. Army offi~s and 

enlisted personnel who rated each term on an 11-point scale of favorablenes~· and indicated whether or not the meaning of the term seemed ambiguous. The ~ -- DD I ~=-71 1473 EDtTIOM M ' NOV .. IS GeiOLETE Unclc..ssified 
J 1 ~~~~oi fi~~d

SCCURITY CLASSIFICATION OF THIS FAGE(WPIw Data Entered)

\ 20.

mean , standard deviation , and range of responses were computed for each

‘term and presented in tabular form. Those terms with smaller standard

deviations are more strongly advocated for use in questionnaire items than

those with larger standard deviations. A technique for constructing scaled

response alternative sets is demonstrated . The term borderline is recommended

as a replacement for neutral in many applications , because of an ambiguity

of meaning that may often accompany use of the latter term. ~

C3

~~

S 

Ijrs c].as si fied

SICUR1Ty CLASSI~~ CA~~ ON OF TH~~ PAGS(Whafl Data E,uar.d)
— 

. 

-,IuI

~~

~~~~~~

.

T hnIcatIaper~T9

THE PERCEIVED FAVORABLENESS OF

SELECTED SCALE ANCHORS AND RESPONSE

ALTERNATIVES

Josephine L.’Matthews, Calvin E.ANrigh~, 

I

and Kenneth L. Yudowitch ,

. —‘

Operations Rea.~~$t

~~~~~~ esf

and

i James/Geddie 

~~ 

R. L.jPalmer U.S. Army Research Institute for tWe’

~eha~ioraI and Social Sciences

ARI FIELD UNIT AT FORT HOOD, TEXAS

Submitted as compiete and Approved By: technically accurate, by:

George M. Gividen A.H. Birnbaum , Acting Director

Field Unit Chief ORGANIZATIONS & SYSTEMS

RESEARCH LABORATORY

Joseph Zeidner

TECHNICAL DIRECTOR (DESIGNATE)

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5011 Eisenhowe r Avenue , Alexandria , Virginia 22333

Office , Deputy Chief of Staff for Personnei

Department of the Army

f// ) Augu~~~978 , ,

:

Army Pro~ cULw~ber Human Performance in FieItAss~ssment

20763743A775 / CQe’,tTBOI DAHC 19.74.C.0032

‘ /‘ —‘ - —~~~~~“ - - ..

Appr~~~d for public release; distribution unlimited.

--.---~~~~~--. --- -—~~~~ 

‘

~~~~~~~— ~~~~~~~~~~~~~ ..
— V V_~

_ _.___,__

J

ARI Research Reports and Technical Papers are intended for sponsors of

R&D tasks and other research and military agencies. Any findings ready for

implementation at the time of publication are presented in the latter part of

the Brief . Upon completion of a major phase of the task , formal recommendations for official action normally are conveyed to appropriate military

agencies by briefing or Disposition Form.

/
-- V - -V. ..- c-— — ~ IlII

~~~

FOREW ORD

An important and continuing concern within the Army Research Institute for the Behaviora l and Social Sciences (ARI ) has been the influence

of factors that may enhance or degrade the usefulness of responses to

questionnaire and interview items . Examples of such factors are the number of items in the questionnaire, the numbe r of response alternatives in

itens , the presence of words in item stems that may bias the direction or

intensity of responses, the kind of modifiers used in response alternatives, the forcing or not forcing of the respondent to state a preference ,

the serial order of items and response alternatives, and the amount of

time required to complete the questionnaire.

This report represents a collaborative ARI effort designed to provide objective data concerning the denotative meanings and inherent scale

positions of common descriptors used as response alternatives and scale

anchors in questionnaire construction. The statistical analyses were con—

ducted by Operations Research Associates, Palo Alto, Calif., in connection with a contract that subsumed their contribution to the present report (Contract No. DAH C 19—74—C—0032). The larger project was directed

at producing an instructional manual for constructing and administering

questionnaires (MU Special ~~~~~ ,

~~~17—iLand a technical report of

the results of a ”e ri~ i~i~~ irvey of the literature on factors involved

in questionnaire construction and administration. Data collection and

manuscript preparation of this report were accomplished by the ARI Field

Unit, Fort Hood , Tex.

This report supports the objectives of FY 1974 RDTE Project

2Q763743A775 , Huma n Performance in Field Assessment.

4L~~ 

L

~~~

~ J~~ EPH Z NER

echnical Director (Designate )

hia-A . . . ~~~~~~~~ . . .. 

~~~~~~

,, , . . . .

~~~~~~~~~~~~~~~~~~~~~ 
- —.-,—-—.— ..—~ -----V.-’. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - — ---.-.- .-,-----,-V.~~~ —-——

~~ 

—.III~~

THE PERCEIVED FAVORABLENESS OF SELECTED SCALE ANCHORS AND RESPONSE

ALTERNATIVES

BRIEF

Requirement:

In the interest of improving methods of questionnaire construction,

the principal purpose of this study was to produce objective norms regarding the extent to which respondents perceive various descriptive phrases

(denoting degrees of acceptability, adequacy, or goodness) as indicators

of favorable or unfavorable attitude .

Procedure:

Lists containing many descriptors denoting degrees of acceptability,

adequacy , and goodness were distributed to a random selection of 51 Army

officers and enlisted personnel who rated each descriptor on a bipolar

scale of favorableness and indicated for each whether it seemed ambiguous.

Findings :

The majority of the tested descriptors were considered unambiguous by

a majority of the raters. In this connection , a specific finding of interest was that the adverb modifiers pretty and rather were contained in more

than half the adjective phrases considered ambiguous by the raters, which

indicated the advisability of exercising caution in using these terms in

questionnaire response alternatives and scale anchors.

The mean, standard deviation, and range of the raters ’ responses for

each descriptor were computed and are presented in tabular form for use in

questionnaire construction. Descriptors with large standard deviations

are noted and considered generally less desirable than the others but are

not necessarily to be avoided in every application.

A limited amount of data relevant to intrasubject consistency indicated that the ratings are reliable.

Examples of sets of scaled response descriptors with desirable scale

properties are presented in connection with a description of possible meth—

ods for choosing response descriptors for questionnaire items.

Regarding scale midpoints , it was noted that the term neutral, although possessing desirable statistical characteristics , may normally be

unsuitable as a midpoint descriptor because of an inherent amb iguity of

LA ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~ . _ _

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -
.— -.,

~

..-‘-.-,. ~

0

~~~~~ — .—~ — V.

meaning. The term borderline seems appropriate as a replacement for neutral in many cont e xt s.

Utilization of Findings:

The objective contents of this report have been integrated into the

Army Research Institute Questionnaire Construction Manua l, which elaborates procedures for selecting response alternatives and scale anchors

(and other topics). The manual is used within both the Army and the civilian community. This report also complements existing literature pertaining to the characteristics of descriptive terms for use in questionnaire item construction.

— , _ - _ V.- _— —_- -- . .- . 4 _ .-. . .----.- . -- — , ..--- --.--
—~~~ - . V.-V~~~~~~~~~ —- ~~~~- -— ~~~~~~~~~ ----—-- .. -

~~~

THE PERCEIVED FAVORA BLENESS OF SELECTED SCALE ANCHORS AND RESP ONSE

ALTERNATIVES

CONTENTS

Page

INTRODUCTI ON 1

PROCEDURES 2

Pilot Test 2

Main Test 3

RESULTS 5

Sample Characteristics 5

Normative Attributes of the Candi date Descriptors 5

Intrasubject Consistency 14

E xample Sets of Minimally Overlapping Response

Descriptors 14

DISCUSSION AND CON CLUSIONS 17

RE FE RENCES 21

DISTRIBUTION 23

LIST OF TABLES

Table 1. Candidate descriptors whose meanings were unclear

to two or more respondents 6

2. Results pertaining to “acceptability” descriptors . . 7

3. Results pertaining to “a dequacy ” descriptors 9

4. Results pertaining to “re lative goodness”

descriptors 11

5. Candidate descriptors with standard deviations

greater than l .25 13

6. Results pertaining to intrasub ject response

consistency 15

hI’- A

~ _______________

- --V —V. V. — — -,. . ..
. V.— 

- -

~~~~~~~~~~ 

—-. -.-

~~~~~~

---

~~~

-

~

,-.- -V.

~ —V.——-, ~~~~~~~~~~~ - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ .

WI

Page

Table 7. Example set of minimally overlapping descriptors

denoting degrees of “acceptability” 16

8. Another example set of minimally overlapping

descriptors denoting degrees of “acceptability ” . . . . 16

9. Example set of minima lly overlapping descriptors

denoting degrees of “adequacy ” 18

10. Example set of rn.&nirnally overlapping descriptors

denoting degrees of “re lative goodness” 18 

~~~~~~~~~~ . -- -- - .

~~~~~~~~~~~ 
~~~~~~~~~~~

THE PERCEIVED FAVORA BLENESS OF SELECTED SCALE ANCHORS

AND RESPONSE ALTERNATIVES

INTRODUCTION

The construction of multiple—choice rating—scale questionnaires frequently involves selecting adjectives , adverbs, or descriptive phrases

for use as response alternatives or scale anchors. The ultimate value

of the questionnaire often rests to a great degree on such properties of

these descriptors as clarity of meaning, evaluative intensity, and the

extent to which the denotations of the descriptors do not overlap. It is

desirable, therefore, that response scales be constructed with such factors in mind and in accordance with the objective properties of the candidate descriptors.

Objective information about the characteristics of candidate descri ptors is available from several studies reported in the psychological li terature (e.g., Altemeyer, 1970; Cliff , 1959 ; Dodd & Gerberick , 1960;

Gividen , 1973; Jones & Thurstone , 1955 ; Myers & Warner , 1968; Mosic-~~,

1941; and U.S. Army Test and Evaluation Command , 1973). Much of the relevant data in the literature (including the objective contents of this

report ) (Matthews, Wright, & Yudowi.tch, 1975) has been abstracted or

cited in a comprehensive literature survey p :epared for the Army Research

Institute (ARI) by Dyer, Matthews, Stulac, Wright , and Yudowitch (1976)

and integrated into the ARI Questionnaire Construction Manual (Dyer,

Matthews, Wright, Yudowitch , & Nystrom , 1976).

The primary purpose of this MU study was to complement previously

published studies dealing with the characteristics of response alternatives and scale anchors. More specifically, it was desirable for ARI to

obtain norms , based on military respondents , regarding the perceived characteristics of descriptors that denote degrees of acceptability, adequa cy ,

and relative goodness. The following are examples of these three types of

descriptors.

Acceptability Adequacy Goodness

descriptors descriptors descriptors

Wholly acceptable Extremely inadequate Best of all

Sli ght ly acceptable Somewhat inadequate Alike

Highly unacceptable Completely inadequate Absolutely worst

Of primary concern in this study was the extent to which such descriptors are perceived by respondents as signifying favorable or unfavorable (positive or negative ) attitude. Thus, the primary procedural objective of the study was to determine for each of a sizable selection of the

three types of candidate descriptors a scale va lue on a bipolar ( positive

to negative ) scale of favorableness.

V. ~~~~~~ ~~~~~~~~~~~~~~~~~~~~~ .

~V _V._ V.V ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
~~~~ 

A secondary purpose associated with this study was to demonstrate the

construction of response alternative sets from normative data.

PROCEDURES

Pilot Test

An original pool of 220 descriptors representing the three aforementioned categories was prepared by Operations Research Associates (ORA).

Indivi dual items were garnered from the literature or created especially

for this study. The intent was to generate a list for each category that

would contain most of the descriptors of that type that researchers might

use in the construction of questionnaires. The 220 items contained 62 acceptability descriptors , 62 adequacy descriptors , and 96 relative goodness descriptors.

The three original lists were pilot tested by ARI , using 10 Army officers as respondents. Each respondent was asked to rate each of the 220

descriptors on an 11—point favorableness scale ranging from 1 to 1 1 , with

6 as the neutral point. The respondents read the following instructions

prior to rating.

Given on the following pages are three lists of phrases. To

the right of each phrase is an 11—point scale ranging from 1

to 11 on which 1 means Most Unfavorable , 6 means Neither Unfavorable nor Favorable, and 11 means Most Favorable. The

distance between each number should be regarded as equal.

Indicate by circling one number for each phrase the degree

of favorableness expressed by that phrase. Cross out any

phrase the meaning of which is unclear to you.

Following the rating procedure , comments regarding the items and rating

procedure were solicited from the respondents.

The data from the rating procedure were analyzed by calculating the

standard deviation of the 10 (or fewer) responses for each of the 220 de—

scriptors. In addition, the number of times each descriptor was crossed

out (rejected as unclear or unratable) was recorded.

Subsequent to the analysis, the three lists were edited by eliminating each descriptor whose standard deviation was 1.00 or greater or that

was rejected as ambiguous by two or more of the respondents.

The comments of the respondents indicated that the third list of

descriptors—- the relative goodness items—-contained two general types of

items, some expressing degrees of likeness , or equality, others expressing degrees of relative goodness. Furthermore, the respondents reported

having experienced difficulty in attempting to rate the former type on

the favorableness scale. Consequently, the list was further edited to

eliminate items of the likeness type that caused respondents confusion.

2

__ - - - V ~~~~~~~~. . -
V. ~—_ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~ V. V.

The editing yielded 48 acceptability, 49 adequacy, and 41 relative

goodness descriptors.

The comments of the respondents also indi cated that the favorableness

scale would proba b~.y be more compatible with the required rating task if

the sca le va lues range d from —5 (most unfavorable ) to +5 ( most favorable),

with 0 as neutral, instead of from 1 to 11.

Main Test

Test Lists. Two of the descriptors in the acceptability list (mildly

acceptable and high~ y unacceptable) were included twice, creating a 50—

item list. One descriptor in the adequacy list (exceptionally inadequate)

was included twice, a lso creating a 50—item list. No repetitions were incorporated into the relative goodness list, so the final test version contained only the 41 items that resulted from the pilot—test revisions. In

addition to the repetitions within lists, there were certain repetitions

across lists: The descriptors neutral and borderline occurred in all

three lists; the descriptor marginal occurred in the acceptability and

relative goodness lists but was inadvertently left of f the adequa cy lists.

These repetitions within and across lists permitted a crude assessment of

intrasubject response consistency.

Each list was prepared so that the descriptors were presented in a

haphazard serial order.

Administration of Test Lists. Questionnaires containing the three

lists were distributed to 50 Ar my officers and 50 Ar my enlisted men assigned to the Modern Army Selected Systems Test , Eva luation, and Review

(MASSTER), Fort Hoo d, Tex. These subjects were randomly sampled from

the MASSTER telephone directory. A cover letter accompanying the questionnaire included the following statement.

A significant portion of the data collected in MASSTER tests

is subjective and jud gmental in nature. Collection of this

type of data frequently requires construction of questionnaires. In order to generate a list of response alternatives

which have been evaluated in terms of their meanings to respondents, your cooperation is needed in eva luating the enclosed phrase lists. Please return the completed phrase list

to the ARI Field Unit . .

The questionnaire itself contained the same instructions as those used in

the pilot test (see above), except that the favorableness scale was described as ranging from —5 to +5, with a midpoint of 0.

Completed questionnaires were returned by 92 of the 100 subjects.

Unfortunately, however , 60 of the returned questionnaires had been corn—

pleted under an apparent misunderstanding about the intent of the instruc—

tions. The respondents involved obvi ously had not rated the candi date

V.
_____________ - 

.

~~~~~~~~~~~~~

descriptors on the extent to which they were perceived as signifying f a—

vorable or unf avorable attitudes. Rather, the descriptors appeared to

have been assessed on the basis of the respondents’ personal preferences

for them. Therefore , questionnaires were redistributed to those respondents and to the few respondents who had not returned questionnaires from

the first distribotion.

The cover letter accompanying the second questionnaire contained statements intended to clear up the misun derstanding. They read as follows.

As part of its support to MASSTER, ARI and a contractor are

investigating various factors which inf luence obtaining jud gments and subjective evaluations of equipment , procedures and

concepts by participants/observers in MASSTER tests.

Attached . . . are three lists of phrases which were sent . earlier . . . . The same lists of phrases are being sent out

again with additional guidance because there was apparently a

misunderstanding by many respondents concerning the task involved. Many of the phrase lists were returned with the

phrases rated according to whether the respon dent approved

or disapproved of the phrase as a descriptor——while what was

desired was their judgment of how positive/favorable or negative/unfavorable each rating phrase would be if it were given

as a rating of some equipment, procedure or concept. Please

process the attached lists of rating phrases in accordance

with the guidance below and return to the ARI Field Unit . .

Guidance. List A consists of phrases used to express different degrees of adequa cy or inadequa cy . List B consists

of phrases used to express different degrees of acceptability

or unacceptability. List C consists of phrases used to compare an item with a standard in terms of better or worse.

Oppooite each rating phrase are eleven numbers , —5 to +5 ,

which constitute a sca le for you to use to tell us how negative/unfavorable or positive/favorable each rating phrase would

be fo r you if you used it to rate some equipment , procedure or

concept. A rating of —5 stands for the most negative or unfavorable rating possible, and +5 stands for the most positive

or favorable rating possible, with less extreme degrees of

favorableness or unfavorableness represented by the numbers

in between. A rating of 0 means that the phrase is neither

negative/unfavorable nor positive/favorable. For each rating

phrase , circle the n umber which describes how negative/urifavor—

able or positive/favorable the rating phrase is when applied

to you. For example, if you saw in a list the phrase highly

satisfactory and you considered it to be a very positive/favorable descriptor, you would record your response by circling a

#4 or +5 as shown be low .

Hi ghly satisfactory —5 ~“4 3 -2 —1 0 +1 +2 +3 

~

J+s

L_ Ad
-.

On the other hand, if you saw the phrase highly unsatsifactory,

you might give it a -4 or -5 as shown below since it is an

unfavorable/negative descriptor.

Hi ghly unsatisfactory —5~~~~~—3 —2 —1 0 +1 +2 +3 +4 +5

If you saw the phrase neither satisfactory or unsatisfactory,

you might consider it to be a neutral descriptor and give it a

rating of 0 on the 11—point scale.

Note that the dimension on which the phrases are to be rated

is their meaning——not your personal approval or disapprova l

of the phrases as descriptors.

If you have any questions concerning these rating phrase lists

or how to process them please call Dr. [name ] at [phone number].

RESULTS

Sample Characteristics

The number of completed questionnaires obtained from the second distribution was 22. Two of these were eliminated from the analysis because

more than half the items were crossed out by the respondents. Another

was eliminated because more than 10% of the responses were judged unrealistic (e.g., completely adequate was rated —5 , and extremely inadequate

was rated +5). Thus the second distribution yielded 19 valid questionnaires. These, combined with the 32 valid questionnaires from the first

distribution, produced a total of 51 usable questionnaires.

Specific items on some of these questionnaires were eliminated from

the analyses because the response was considered very unreasonable, more

than one scale va lue had been circled by the respondent , or the item had

been crossed out as ambiguous by the respon dent. Table 1 shows the candidate descriptors that were crossed out by two or more respondents. Fourteen of the 141 descriptors were crossed out, but never by more than 4 of

the 51 respondents. It appears, then , that most of the descriptors were

sufficient ly unambiguous to most of the respondents. It should also be

noted that 6 of the crossed—out descriptors contained the adverb modifier

pretty or rather.

Normative Attributes of the Candidate Descriptors

Tables 2 , 3, and 4 , respectively, portray the acceptability, adequacy,

and relative goodness lists and the results of the primary analyses. In

each table the first four columns of figures describe the distribution of

responses for each candidate descriptor. Consider, as an example, the

first row of Table 2: the mean 4.73 indicates that, on the avera ge, the

subjects felt that wholly acceptable denoted a high ly positive attitude or

degree of favorableness (recall that the favorableness scale extended from 

-,—~~~~~~~ -.,-— -- -
~~~~~~~~~~~~~~~~~~ ,—=,-----— - .

-—- _ _ ~~~ ,,. _ç—-.--- - - —V

Table 1

CANDIDATE DESCRIPTORS WHOSE MEANINGS WERE

UNCLEAR TO TWO OR MORE RESPONDENTS

Descriptor No. of respondents

.4

“Acceptability ” descriptors :

Pretty acceptable 2

Reasonably unacceptable 2

“Adequacy ” descriptors :

Considerably inadequate 3

Exceptionally adequate 2

Mildly adequate 2

Pretty adequate 2

Pretty inadequate 2

Rather adequate 2

Rather inadequate 2

Somewhat adequate 2

“Relative goodness ” descriptors:

Rather better 2

Descriptors for scale midpoints :

Borderline 4

Margina l 3

Neutra l 2

Note. Descriptors within each category are listed alphabetically.

~ 

~~
— V .— V . ~~~~~~~~~~~~~~~ -— --- - -—-—- - ,- ~~~~~~~~ - .

-~~~~~~- -- — V.-, - -

Table 2

RESULTS PE RTAININ G TO “ACCEPTABILITY ” DESCRIPTO RS

Descriptor Mean SD Range No. of

M m. Max. subjects

Wholly acceptable 4.73 .56 3 5 51

Completely acceptable 4.69 .61 3 5 51

Fully acceptable 4.41 .87 2 5 51

Extremely acceptable 4.39 .72 3 5 51

Most acceptable 4.16 .92 2 5 51

Very very acceptable 4.16 .83 2 5 51

Highly acceptable - 4.04 .63 3 5 50

Quite acceptable 3.22 .96 1 5 51

Largely acceptable 3.14 .99 1 5 51

Acceptable 2.39 1.46 0 5 51

Reasonably acceptable 2.29 .72 1 4 51

Moderately acceptable 2.28 .72 1 3 50

Pretty acceptable 2.00 1.13 -3 4 49

Rather acceptable 1.94 .82 0 4 49

Fairly acceptable 1.84 .92 0 4 50

Mildly acceptable 1.80 .95 -1 4 51

Mildly acceptable 1.69 .70 -1 4 51

Somewhat acceptab le 1.46 1.24 -2 3 48

Barely acceptable 1.08 .52 -l 3 51

Slightly acceptable 1.04 .52 -l 2 51

Sort of acceptable .94 .65 -l 2 50

Borderline .00 .20 -l 1 50

Neutral .00 .00 0 0 51

Marginal - .12 .52 -2 1 50

Barely unacceptable -1.10 .30 -2 -l 50

Slightly unacceptable -1.26 .59 -4 -l 51

Somewhat unacceptable -1.77 .67 -3 -l 51

Rather unacceptable -2.02 .84 -4 0 50

Fa irly unacceptable -2.16 .88 -5 -l 50

Moderately unacceptable -2 .34 .68 -3 -1 50

Pretty unacceptable -2.41 .66 -4 -l 51

Reasonably unacceptable -2.44 .75 -4 -1 50

Unacceptable -2.67 1.38 -5 0 51

Substantially unacceptabl e -3.24 .90 -5 -l 51

(Continued)

7
— 

~~ ~~~~~~VTh - ~

;-_—~~ -- —r~

——-- ‘_V ~

__~

_ V —- ~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~ “ . ,v’~

rr — —

7

Table 2 (cont)

Descriptor Mean SD Range No. of - M m . Max. subjects

Quite unacceptable -3.39 1.07 -5 0 49

Largely unacceptabl e -3.39 .82 -5 -l 51

Considerably unacceptable -3.44 .78 -5 -2 50

Notably unacceptable -3.50 1.04 -5 -1 50

Decidedly unacceptable -3.84 1.02 -5 -l 49

Highly unacceptablea -4.22 .58 -5 -3 50

Highly unacceptableb -4.29 .54 -5 -3 51

Most unacceptable -4.42 .72 -5 -2 50

Very very unacceptabl e -4.49 .50 -5 -4 51

Exceptionally unacceptabl e -4.54 .61 -5 -3 50

Extremely unacceptable -4.69 .46 -5 -4 51

Completely unacceptable -4.90 .36 -5 -3 50

Entirely unacceptable -4.90 .36 -5 -3 50

Wholly unacceptable -4.92 .27 -5 -4 51

Absolutely unacceptable -4.92 .33 -5 -3 51

Totally unacceptable -4.94 .24 -5 -4 51

aFirst usage in test list.

bSecond usage in test list.

8

__ V. ~~~~~~~~~ , , .
- -

Table 3

RESULTS PERTA I NIN G TO “ADEQUACY ” DE SCRI PTORS

Descriptor Mean SD Range No. of

M m . Max. subjects

Totally adequate 4.62 .85 2 5 50

Absolutely adequate 4.54 .92 1 5 50

Completely adequate 4.49 .83 2 5 51

Extremely adequate 4.41 .72 3 5 51

Exceptionally adequate 4.38 .87 2 5 50

Entirely adequate 4.34 .86 2 5 50

Wholly adequate 4.31 1.04 1 5 51

Fully adequate 4.29 .91 2 5 51

Very very adequate 4.06 .88 1 5 48

Perfectly adequate 3.92 1.03 2 5 51

Highly adequate 3.84 .61 2 5 51

Most adequate 3.84 .98 2 5 51

Very adequate 3.42 .85 1 5 50

Decidedly adequate 3.14 1.54 -4 5 50

Considerably adequate 3.02 .87 1 5 51

Quite adequate 2.98 .98 1 5 49

Largely adequate 2.86 .99 -l 4 51

Substantially adequate 2.61 1.03 -l 5 51

Reasonably adequate 2.41 .77 1 4 51

Pretty adequate 2.31 .86 0 5 49

Rather adequate 1.76 .89 0 4 49

Mildly adequate 1.57 .67 1 3 49

Somewhat adequate 1.32 .79 -2 3 49

Slightly adequate 1.20 .57 -l 3 50

Barely adequate .63 .93 -4 2 51

Neutral .00 .00 0 0 50

Border li ne - .02 .32 -l 1 50

Barely inadequate -1.16 .64 -4 0 51

Mildly inadequate -1.35 .62 -3 0 51

Slightly inadequate -1.38 .77 -4 1 50

Somewhat inadequate -1 .88 .73 -3 0 51

Rather inadequate -2.10 .97 -4 0 49

Moderately inadequate -2.16 1.02 -4 2 51

Fairly inadequate -2.22 .80 -4 1 51

Pretty inadequate -2.35 .96 -4 1 49

Considerably inadequate -3.60 .68 -5 -2 45

(Continued )

-- - -V.-— ~~~~~~~~~
—V.- - V-V___~__VV •V.-VV._ ~~~~~~~~~~~~~~~ 

Table 3 (cont)

Descriptor Mean SD Range No. of

M m . Max. subjects

Very inadequate -3.74 .78 -5 -2 49

Decidedly inadequate -3.78 .94 -5 -1 50

Most inadequate -3.98 1.55 5 4 49

Highly inadequate -4.20 .74 -5 -3 51

Very very inadequate -4.46 .54 -5 -3 50

Exceptionally mnadequa teb -4.56 .64 -5 -3 50

Extremely inadequate -4.60 .53 -5 -3 51

Fully inadequate -4.67 .68 -5 -2 51

Exceptionaly inadequatea -4.68 .51 -5 -3 50

Wholly i nadequate -4.78 .50 -5 -3 51

Entirely inadequate -4.79 .64 -5 -2 48

Completely inadequate -4.80 .53 -5 -3 50

Absolutely inadequate -4.88 .43 -5 -3 50

Totally inadequate -4.90 .41 -5 -3 50

5First usage in test list.

bsecond usage in test list.

10 

---,—~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - V .—-- _~~~~~~~~~ V ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
—— ~

V.

~~ V.V.V.•-V_ V_ -V~ 

_

~V __V._

~~~ - ‘“?‘~~~W9 V.7! ”Z~~

?r

~ - ~~~~~~~~~~~~~~~~~ ~~, .. . .

-,

~ V.~~V.V.V.7V.~~• .’ V. V V ~•

~ V.V. _ V.•V.V.V.V.-V~~

V.

’

V V._’

Table 4

RESULTS PERTAINING TO “RELATIVE GOODNESS” DESCRIPTORS

Descriptor Mean SD Range No. of

M m . Max. subjects

Best of all 4.90 .51 2 5 48

Absolutely best 4.84 .46 3 5 51

Truly best 4.60 .72 3 5 50

Undoubtedly best 4.57 .82 2 5 51

Decidedly best 4.37 .84 2 5 51

Best 4.22 1.46 -3 5 51

Absolutely better 4.06 .99 2 5 50

Extremely better 3.92 .88 2 5 51

Substantially better 3.70 .92 2 5 50

Decidedly better 3.41 .93 1 5 51

Conspicuously better 3.06 .80 2 5 51

Moderately better 2.26 .74 1 4 51

Somewhat better 1.84 .80 1 4 51

Rather better 1.82 .72 1 4 49

Slightly better 1.16 .78 -1 4 51

Barely better .96 .66 -1 3 51

Absolutely alike .59 1.62 -l 5 51

Al i ke .22 .85 0 5 51

The same .16 .80 0 5 51

Neutral .00 .00 0 0 50

Borderline - .06 .31 -l 1 49

Marginal - .18 .92 -4 2 49

Barely worse -1.04 .82 -3 4 51

Slightly worse -1.22 .50 -3 -l 51

Somewhat worse -2.08 .86 -4 -l 51

Moderately worse -2.22 .94 -4 2 50

Noticeably worse -2.53 1.04 -4 2 51

Worse -2.67 1.42 -5 -1 51

Notably worse -3.02 1.04 -5 -1 51

Largely worse -3.22 1.11 -5 3 51

Considerably worse -3.28 - 1.21 -5 4 51

Conspicuously worse -3.28 .89 -5 -l 51

Much worse -3.29 .81 -5 -l 49

Substantially worse -3.46 .90 -5 -2 50

(Continued )

11

- - -- -- --- ---~~~~~~~~~~~~~~~~~~~~~~~~~ . ~~~~~~~~~ --
Table 4 (cont) 

Descriptor Mean SD Range No. of 

M1n. Max. subjects 

Decidedly worse -3.76 .91 -5 -2 50 

Very much worse -3.94 .75 -5 -2 51 

Absolutely worse -4.43 .82 -5 -2 51 

Decidedly worst -4.43 .75 -5 -2 51 

Undoubtedly worst -4.51 .87 -5 -2 51 

Absolutely worst -4.69 1.29 -5 4 51 

Worst of all -4.78 1.30 -5 4 49 

12 
- -- ‘~~~~~~~ ‘~ —- -

~~~ -~~ ,

~~~~ r ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -~~~~- ‘ - - - ‘V — -

.

-5 to +5); the standard deviation (SD) .56 indicates that differences among

the subjects ’ responses for this item were, on the average, a little more

than one—half scale unit; the range columns give the smallest and the largest scale value given to this item by any subject. The last column in each

table shows the number of subjects whose responses were analyzed for each

can didate descriptor.

Each of the means given in Tables 2, 3 , and 4 is, in effect, a compromise among different estimates of many respondents. The standard deviations, on the other hand, measure the degree of consensus among the

respondents. Thus, the standard deviations permit examination of the

“ clarity” of the descriptors in a manner different from that used to derive Table 1. Table 1 shows the descriptors that were considered ambiguous by the respondents. In contrast, the standard deviations in Tables 2 ,

3, and 4 show whether the respondents agreed on the scale values on those

descriptors that they considered unambiguous. Descriptors with large

standard deviations possess favorableness values that vary considerably

from subject to subject and, therefore, would often be less desirable as

point anchors for rating scales.

Table 5 lists all candidate descriptors with standard deviations

greater than 1.25 (an arbitrary cutoff). Note that for the most part, the

terms listed seem to be characterized by extremeness or generality.

Table 5

CANDIDATE DESCRIPTORS WITH STANDARD DEVIATIONS GREATER THAN 1.25

Descriptor SD

Acceptable 1.46

Unacceptable 1.38

Most inadequate ‘ 1.55

Decidedly adequate 1.54

Absolutely alike 1.62

Best 1.46

Worse 1.42 V

Worst of all 1.30

Absolutely worst 1.29

Note. The descriptors within subgroups are listed in order of

decre~ Thg SD.

_____ 

--- - - V. V.~~~~~~~~~~~~~ 1V. V~~~~~~ ___ _
Intrasubject Consistency

The results pertaining to descriptors that were repeated either within or across test lists have been gathered together in Table 6. The table

shows that the two means for each of the three descriptors repeated within

lists were quite similar, considering the 11—point range of the favorable—

ness scale. The same holds true for descriptors repeated across lists.

The standard deviations reveal a simi lar pattern. The variabilities associated with repetitions of a given descriptor appear quite similar—-the

largest discrepancy is .40 , between the two repetitions of marginal. Thus

the sub jects’ responses seemed to exhibit a satisfactory degree of consistency with respect to the few items repeated within or across lists. Because of the limited data ava ilable regarding intrasubject consistency,

further analyses were not con ducted.

Example Sets of Minimally Overlapping Response Descriptors

The construction of good questionnaire items of the type un der consideration requires that the denotations of response alternatives or scale

anchors located at different points on the response sca le do n~ t overlap

to an extent that mi ght cause respondents to attribute the same n~~aning to

any two different points on the scale. Hence it is often desirable to

choose descriptors that occupy narrow bands along the response dimension .

In this section, several examples of lists of minimally overlapping de— V

scriptors are presented.

Nonoverlapping or minimally overlapping descriptors can be chosen by

different methods and with different criteria that depend upon the purpos e

of the questionnaire, the theoretical orientation of the investigator, the

statistical methods employe d, etc. Two illustrative procedures are described here in connection with Tables 7 and 8, which present two differently scaled lists of descriptors denoting degrees of acceptabi lity.

Consider Table 7. This table was constructed on the basis of the information presented in Table 2 , using the following procedure. First, no

candidate from Table 2 was considered if its standard deviation was 1.00

or greater. This arbitrary restriction limited the candidates to those

occupying relatively narrow bands on the favorableriess dimension along

which they were eva luated. Secon d , from the remaining candi dates, the one

having the largest mean (i.e., wholly acceptable ) was chosen for the positive extreme of the response dimension. The rest of the descriptors were

then selected in descending orde r so that the interva l between each pair

of adjacent means was greater than zero but as small as possible without

being less than the larger standard deviation of the pair. This rule

guaranteed that the descriptors woul’l be at least one standard deviation

apart in all cases.

Table 8 portrays another list constructed from the candidate descriptors in Table 2. In this case, however, the mi dpoint (i.e., neutral) was

chosen as the starting place , and the selection process proceede d in both

L
V - -

_ — ___

Table 6

RESULTS PERTAINING TO INTRASUBJECT RESPONSE CONSISTENCY

Within list Across list

Descriptor 1st 2nd “Acceptability ” “Adequacy ” “Goodness ” usage usage

Mean responses

Mildly

acceptable 1.80 1.69 -- -- --

Highly

unacceptable -4.22 -4.29 -- -- --

Exceptionally

inadequate -4.68 -4.56 -- -- --

V Neutral -- -- .00 .00 .00

• Borderline -- -- .00 - .02 - .06

Marginal -- -— - .12 -- - .18

Standard deviations

Mildly

acceptable .95 .70 -- -- -- Highly

unacceptable .58 .54 -- -- --

Exceptionally

inadequate .51 .64 -- -- --

Neutral -- -- .00 .00 .00

Borderl i ne -- -- .20 .32 .31

Marginal -- -- .52 -- .92

15

___-- -
V. -—- - V — ~~~~~~~ ~~ 

•— -—-— ——V. —7-V-V--V., ~

__VV_ ’V’

__ ~

_ ,—_---_ ,-~

._ _ _,-,__.~~~~~._ ~~~~_ .______ .-~ .. 

. 

‘-,luI~~~

Table 7

EXAMPLE SET OF MINIMALLY OVERLAPPING DESCRIPTORS

DENOTING DEGREES OF “ACCEPTABILITY ”

Descriptor Mean SD

Wholly acceptable 4.73 .56

Highly acceptable 4.04 .63

Reasonably acceptable 2.29 .72

Barely acceptable 1.08 .52

Neutral .00 .00

Barely unacceptable -1.00 .30

Somewhat unacceptable -1.77 .67

Substantially unacceptable -3.24 .90

Highly unacceptable -4.22 .58

Completely unacceptable -4.90 .36

Note. This table is based on Table 2.

Table 8

ANOTH ER EXAMPLE SET OF MINIMA LLY OVERLAPPIN G DE SCR I PTO RS

DENO TING DEG REE S OF “ACCE PTABILI TY”

Descriptor Mean SD

Very very acceptable 4.16 .83

Largely acceptable 3.14 .99

Mildly acceptable 1.69 .70

Sort of acceptable .94 .65

Neutral .00 .00

Barely unacceptable -1.10 .30

Rather unacceptable -2.02 .84

Substantiall y unacceptable -3.24 .90

Highly unacceptabl e -4.29 .54

Completely unacceptable -4.90 .36

L 

Note. This table is based on Table 2.

16

—- -— - V. —--—--—-- —-- - V.-V--- - - - --- - ~~~~~V.V.~~~~~ V • -- - --—V.—
V. V ~~~~~~~~~~~~~~~ - --V- Vr7 . , —-V-V.~~~~~~ , V -V.-.- V

directions from the middle. The fact that the lower halve s of Tables 7

and 8 are identical results from the fortuitou s fact that neutral occurred as the mi dpoint in Table 7.

Tables 9 an d 10 , constructed in the same manner as Table 7 , portray

sets of descriptors denoting, respectively, degrees of adequacy and degrees

of relative goodness. The norms in these tables were obtained from Tables

3 and 4 , respectively.

DISCUSSI ON 1’,ND CONCLUSIONS

The return rate of 92% on the first administration of the test lists

was reduced to 54% after the readininistration. Other attrition reduced the

final number of usable questionnaires to 51% of the ori ginal total. The

precise effect of this loss on the normative data presented in this report

is, of course , unknown. Some amount of reduction in the variabi lity of responses to each item would be expected; but the item means may have been

largely unaffected, since no systematic relevant bias due to the attrition

appears obviously noteworthy .

The number of descriptors that were considered ambiguous was quite

small, as was the number of respondents who considered any descriptor ambiguous. Therefore, the descriptors listed in Table 1 are not necessarily

unsuitable for use as scale anchors or response alternatives. Neve rtheless, it may be wise to exercise discretion in using the modifiers pretty

and rather, which occurred in more than half the nonmidpoint descriptors

in the table. Three of the descriptors in Table 1——mildly adequate, considerably inadequate, and neutral—-were used in the example sets given in

Tables 7 , 8, and 9. However, those three tables (along with Table 10)

were intended to be not so much examples of universally ideal lists of descriptors as illustrations of possible methods for constructing good questionnaire items.

As rega rds descriptors for scale midpoints, the term neutral appears,

at first glance, to be ideal: both its mean and standard deviation were

zero on all three test lists. However, in a study described by Gividen

( 1973), subjects (Army test officers ) displayed some confusion about the

precise meaning of the term. Some interpreted it as denoting no opinion

rather than as an opinion halfway between scale extremes. Furthermore, a

majority of the subjects held the opinion that because of the ambiguity in

the term it should not be used in questionnaires. Thus, although the mean

and standard deviations observed in the present study favor the use of

neutral as a scale midpoint, in practical situations its use may sometimes

make the interpretation of obtained data difficult or impossible. Consequently, neutral should be employed only after prior consideration indicates that confusion would not exist within the particular context.

According to Gividen ( 1973), the term borderline does not suffer the

potential disadvantage of ambiguity associated with neutral. In the present study, borderline was reported as confusing by 4 out of the 51 subjects;

-- 

17 

_ - ——
Table 9

EXAMPLE SET OF M INIMALLY OVE RLA PPING DE SCR I PTO RS

D ENOTIN G DEGREES OF “ADEQ UACY ”

Descriptor Mean SD

Totally adequate 4.62 .85

Very adequate 3.42 .85

Reasonably adequate 2.41 .77

Mildly adequate 1.57 .67

Barely adequate .63 .93

Barely inadequate -1.16 .64

Somewhat inadequate -1.88 .73

Considerably inadequate -3.60 .68

Very very inadequate -4.46 .54 •

Note . This table is based on Table 3.

Table 10

EXA MPL E SET OF MINIMA LLY OVERLAP PIN G DE SCRI PTORS

DENOTING DEGREES OF “RELATIVE GOODNESS”

Descriptor Mean SD

Best of all 4.90 .51

Extremely better 3.92 .88

Moderately better 2.26 .74

Slightly better 1.16 .78

Alike .22 .85

Barely worse -1.04 .82

Somewhat worse -2.08 .86

Conspicuously worse -3.28 .89

Absolutely worse -4.43 .82

Note . This table is based on Table 4.

18

V.V

~

V.
~ VV. V V~V -~~~ ~~~~~~~~~~~~~~~~~~~~~~~~ V .VV ~~ -~~~~~~~~

neutral by 2. Regardless, it does not seem likely that borderline would

be used by subjects to indicate no opinion. This conc lusion, coupled with

the fact that the means and standard deviations observed in the present

study for borderline were close to zero, suggests that the term is generally appropriate as a midpoint descriptor. In contrast, neutra l may be

normally inappropriate.

An examination of Tables 2 , 3 , and 4 revealed that the descriptors

with the greatest amount of item variability tended to be characterized

by extremeness or genera lity (see Table 5). Why descriptors such as best,

absolutely worst, and the like should be interpreted so differently by different respondents is unclear. It is interesting to speculate that there

is considerable variability among subjects in the tendency to assi gn ex—

V treme va lues. As to terms like acceptable and unacceptable, it may be

that such unmodified common terms of moderate degree possess relatively

little specificity of meaning. The variabi lity of the terms in Table 5

would perhaps shrink to some extent if the terms were located among other

terms in actual questionnaire items.

The two methods for selecting descriptors, which were illustrated in

Tables 7 , 8 , 9 , and 10 , are easi ly subjected to reasonable variations.

One , for example , would be to begin by choosing from the candidate poo l

the two descriptors that have the hi ghest and lowest means and the descriptor with the mean closest to zero. Low—variability intermediate descriptors would then be selected from the rest of the pool in a manner similar

to that used for Tables 7 , 8 , 9 , and 10.

Note that the scale va lues (means ) obtained in this study should not

be used as scale values in the analysis of questionnaire items. Rather,

they should be used only as aids in selecting response alternatives or

scale anchors for questionnaire items.

Note also that the descriptor sets presented in Tables 7, 8, 9, and

10 were derived using criteria related only to scale position and item

variability . There may be other important considerations related to the

selection of response alternatives or scale anchors for any given real

questionnaire . Many of these potential considerations are discussed in

the ARI Questionnaire Construction Manual (Dyer et al. , 1976). That manual also presents sample lists of descriptors selected with regard to

other criteria.

19

- V -V
REFERENCES

Altemeyer, R. A. Adverbs and Intervals: A Study of Likert Scales.

Proceedings of the 78th Annual Convention of the American Psychological Association , 1970 , 5, 397—398 .

Cliff, N. Adverbs as Multipliers. Psychological Review, 1959, 66, 27—44.

Dodd, S. C., & Gerberi ck, T. R. Word Scales for Degrees of Opinion.

Language and Speech, 1960, 3, 18-31.

Dyer, R. F., Matthews, J. J., Stulac, J. F., Wright, C. E., & Yudowitch , K.

Questionnaire Construction Literature Survey. ARI Special Publication P—77—2 , July 1976. (NTIS No. AD A043 012)

Dyer, R. F., Matthews, J. J., Wri ght, C. E., Yudowitch , K. L., & Nystrom,

C. 0. ~uestionnaire Construction Manual. ARI Special Publication

P—77— 1 , July 1976. (NTIS No. AD A037 815)

Gividen, G. M . Order of Merit: Descriptive Phrases for Questionnaires.

Unpublished report, available from the ARI Field Unit at Fort Hood,

Tex., February 1973.

Jones, L. V., & Thurstone, L. L. The Psychophysics of Semantics: An

Experimenta l Investigation. Journal of Applied Psychology, 1955 , 39 ,

31—36 .

Matthews , J. J., Wright, C. E., & Yudowitch , K. L. Analyses of the Resuits of the Administration of Three Sets of Descriptive Adjective

Phrases. Palo Alto, Calif.: Operations Research Associates, 1975.

Unpublished report prepared for the ARt Field Unit at Fort Hood,

Tex., under Contract DAJ-IC 19-74-C-0032.

Mosier, C. I. A Psychometri c Study of Meaning. Journal of Social Psychol2~y

~’ 1941 , 13 , 123—140.

Myers, J. H., & Warner , W . G. Semantic Properties of Selected Evaluation

Ad jectives. Journal of Marketing Research, 1968, 5, 409—412.

U.S. Army Test and Evaluation Command. Development of a Guide and Checklist for Hwnan Factors Evaluation of Army Equipment and Systems.

U.S. Army Test and Evaluation Command (TECOM), 1973. (Contract

DAAD 05—7 3—C—0388)

- ~~~~~~~~~~~~~~~~~~~

__ -- -V — V
-

~~~~~~

DISIRIBUTION

ARI Distribution List

4 OASD (M&RA) 2 HQUSACDEC , Ft Ord, ATTN: Library

2 HQDA (DAM I-CSZ) I HQUSACDEC . FtOrd , ATTN: ATEC—EX—E—Hum Factors

I HQDA )DAPE-PBR 2 USAEEC, Ft Benjamin Harrison , AT’TN: Library

1 HODA (DAMA-AR) 1 USAPACDC , Ft Benjamin Harrison , ATTN: ATCP—H R

1 HODA (DAPE-HRE-PO ) 1 USA Comm—Elect Sch, Ft Mo~

imouth, ATTN: ATSN—EA

1 HODA (SGRD-ID) 1 USAEC, Ft Monmouth, ATTN: AMSEL,

~CTV.HDP

1 HODA (DAMI-DOT-C) 1 USAEC, Ft Monmouth, ATrN: AMSEL—PA—P

1 HODA (DAPC-PMZ-A) 1 USAEC, Ft Monmouth, ATTN: AMSE LV. Sl_~CB

1 HQDA (DACH-PPZ-A) 1 USAEC , Ft Monmouth, ATTN: C, Fact Des Br

1 HODA (DAPE-HRE) 1 USA Materials Sys Anal Agcy, Aberdeen , ATTN: AMXSY—P

1 HQDA (DAPF-MPO-C ) 1 Edgewood Arsenal . Aberdeen , ATTN: SAREA—BL-- H

1 HQDA IDAPE-OW) 1 USA Ord Ctr & Sch, Aberdeen , ATTN: AT SLV. TEMV. C

1 HODA IDAPE-HRLI 2 USA Hum Engr Lab, Aberdeen , ATTN: Library/Dir

1 HODA )DAPE-CPS I 1 USA Combat Arms Tng Bd Ft Benning, ATTN: Ad Superv~or

1 HODA (DAFD-MFA I 1 USA Infantry Hum Rsch Unit , Ft Benning, ATTN: Chief

1 HODA (DARD-ARS-P I 1 USA Infantry Ed , Ft Bennirig, ATTN STEBC—TE—T

1 HODA (DAPC-PAS-A I 1 USASMA , Ft Bliss , AUN: ATSS—LRC

1 HODA (DUSA-ORl 1 USA Air Def Sch, Ft Bliss . ATTN: ATSA—CTD- ME

1 HODA (DAMO-RQR) 1 USA A,r Dci Sch, Ft BI,ss , ATTN: Tech L,b

1 HQDA IDASG) 1 USA Air Dci Bd, Ft Bliss , ATTN: FILES

1 HODA (DA1O-P ll 1 USA Air Def Ed, Ft Bliss . ATTN: STEBD—PO

1 Chief . Consult Div (DA-OTSG) , Adelphi, MD 1 USA Cmd & General Stf College , Ft Leavenworth , ATTN: Lib

I M,l Asst. Hum Res, 000R&E , OAD IE&LS 1 USA Cmd & General Sti College , Ft Leavenworth , ATTN: ATSW—SE—L

1 HO USARAL , APO Seattle , ATTN: ARAGP-R 1 USA Cmd & General Sd College, Ft Leavenworth , ATTN: Ed Advisor

1 HO F,rst Army, ATTN: AF KA-O I-Tl I USA Combined Arms Cmbt 0ev Act . Ft Leavenworth , ATTN: DepCdr

2 HO F,fth Army, Ft Sam Houston 1 USA Combined Arms Cmbt Des Act , Ft Leavenworth , ATTN: CCS

1 Dir, Army Stf Studies Ofc , ATTN: OAVCSA (DSP) 1 USA Combined Arms Cmbt Des Act , Ft Leavenworth, ATTN: ATCPSSA

I Ole Ch,et of Sti, Studies Of c 1 USA Combined Arms Cmbt 0ev Act , Ft Leavenworth, ATTh : ATCACO—E

1 DCSPER , ATTN: CPSIOCP 1 USA Combined Arms Cmbt Dee Act . Ft Leavenworth , ATTN: ATCACC—C l

1 The Army Lib, Pentagon , ATTN: RSB Ch,ef 1 USAECOM , Night Vision Lab, Ft Belvoir , ATTN: AMSEL—NV—S D

I The Army Lib, Pentagon, ATTN: ANRAL 3 USA Computer Sys Cmd, Ft Belvoir , ATTN: Tech Library

I Ole. Asst Sect of the Army (R&Dl I USAMERDC. Ft Belvoir , ATTN: STSFB—DQ

1 Tech Support Ofc . OiCS I USA Eng Sch, Ft Belvoir , ATTN: Library

I USASA , Arlington, ATTN: IARD-T 1 USA Topographic Lab, Ft Belvoir , ATTN: ETL—TD—S

I USA Rsch Ofc , Durham, ATTN: Life Sciences Dir 1 USA Topographic Lab, Ft Belvoir, ATTN : STINFO Center

2 USARIEM , Natick , ATTN: SGRD-UE-CA 1 USA Topographic Lab, Ft Belvoir . ATTN: ETL—G SL

1 USATTC, Ft Clayton, ATTN: STETC-MO-A 1 USA Intelligence Ctr & 5th, Ft Huachuca, ATTN: CTD—MS

1 USAIMA, Ft Bragg, ATTN: ATSU-CTD-OM 1 USA Intelligence Ctr & Sch, Ft Huachuca, AT1~

N: ATS—CTD—MS

1 USAIMA, Ft Bragg, ATTN: Marquat Lib 1 USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: ATS I—TE

1 US WAC Ctr & Sch, Ft McClellan, ATTN: Lib 1 USA l”~~

lligence Ctr & Sch, Ft Huachuca, ATTN: ATS I—TEX—GS

1 US WAC Ctr & Sch, Ft McClellan, ATTN: Tng Dir I USA intelligence Ctr & Sch, Ft Huachuca, ATTN: ATSI—CTS-—OR

I USA Quartermaste r Sch, Ft Lee, ATTN: ATSM-TE 1 USA Intelligence Ctr & Sch, Ft Huachuca, AITN: ATSI—CTD—OT

I Intelligence Material Des Ofc , EWL , Ft Ho)abird 1 USA Intelligence Ctr & 5th, Ft Huachuca . ATTN: ATSI—CTD—cS

1 USA SE S,gnal Sch . Ft Gordon, ATTN: A1SO-EA 1 USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: DAS/SRD

1 USA Chaplain Ctr & Sch. Ft Hamilton , ATTN: ATSC-TE-RD 1 USA Intelligence Cu & Sch , Ft Huachuca , ATTN: ATSI—TEM

1 USATSCH , Ft Eusti s , ATTN: Educ Advisor I USA Intelligence Ctr & Sch. Ft Huachuca, ATTN: Library

1 USA War College. Carlisle Barracks . A’T”T’N: Lib 1 CDR , HQ Ft Huachuca, ATTN: Tech Ref Div

2 WRA IR , Neuropsychiatry Div 2 CDR , USA Electronic Prvg Grd , ATTN: STEEP—MT—S

I DLI, SDA, Monterey I CDR , Project MASSTER , AT’TN: Tech Info Center

1 USA Concept Anal Agcy, Bethesda, ATTN: MOCA.WGC 1 Hq MASSTER . USATRADOC . LNO

1 USA Concept Anal Agcy, Bethesda , ATTN : MOCA.MR I Research Institute . HO MASSTER . Ft Hood

1 USA Concept Anal Agcy, Bethesda , ATTN: MOCA.JF I USA Recruiting Cmd, Ft Sherdian , ATTN: USARCPM—P

1 USA Attic Test Cu, APO Seattle, ATTN: STEAC.MO.ASt. 1 Senior Army Ads., USAFAGOD/TAC , Elgin AF Au. FId No. 9

1 USA Artic Test Ctr , APO Seattle , ATTN: AMSTE.PL-T5 1 HO USARPAC , DCSPER . APO SF 96558. ATTN: GPPE—SE

1 USA Armament Cmd, Redstone Arsenal , ATTN: ATSK.TEM 1 Stimson Lib. Academy oi Health Sciences , Ft Sam Houston

1 USA Armament Cmd, Rock Island, ATTN: AMSAR .TDC 1 Marine Corps Inst., ATTN: Dean—MCI

I FAA-NAFEC , Atlantic City , ATTN: Library 1 HOUSMC , Commandant , ATTN: Code MTMT 51

1 FAA-NAFEC , At la,’t ic City , ATTN: Hum En~ Br 1 HOUSMC, Commandant . ATTN: Code MPI—2 0

I FAA Aeronautical Ctr , Oklahoma City, ATTN: AAC -44D 2 USCG Academy, New London , ATTN: Adm,ssjon

2 USA Fid Arty Sch , Ft Sill , ATTN: Library 2 USCG Academy, New London, A’fl’N: L,brary

1 USA Armor Sch. Ft Knox , ATTN : Library 1 USCG Training Ctr . NY , ATTN: CO

1 USA Armor Sch , Ft Knox , ATTN: ATSB -Dl.E I USCG Trainin g CIr. NY , ATTN: EJuc Svc Ofc ‘ -

I USA Armor Sch. Ft Knot , , ATTh: AT SB-oT.’rP I USCG, Psyc ho l Ret Br . DC, ATTN: GP 1/62

1 USA Armor Sch , Ft Knox , ATTN: ATSB-CD .AD 1 HO Mid—Range Br , MC Des , Ouantico , ATTN: P&S D,v

23

- - V- V ~~~~~~ 

V.

-

PRICZD1N~ P~~~ IM..A)~ j

V.V ~~~~~~~~~~~~~~~~~ - V.
_

I US Marine Corps Livision Ofc , AMC. Alesrondria. ATTN: AMCGS—F I Def & Civil Inst of Enviro Medicine. Canada

1 USATRADOC . Ft Monroe. ATT N: ATRO--ED 1 AIR CRESS, Kensington, ATIN: Info Sys 8r

6 USATRADOC, Ft Monroe , ATTN: ATPR—A D 1 Militaarpsykologiik Tjsns~

te, Copehag.n

1 USATRADOC. Ft Monroe, ATTN: ATTS—EA I Military Attache, French Embassy, ATTN: Doc Sec

1 USA Forces Cmd. Ft McPherson, ATTN: Library 1 Medecin Chef, C.E.R .P.A.—Arsena l . Toulon /Nav al France

2 USA Aviation Test Ed, Ft Pucker , ATTN: STEBG—PO I Prtn Scientific QfI, A~~ l Hu ts Eri

~ ~

Iscfi Due, Miniasr’y

1 USA Agcy for Aviation Safety, Ft Pucker , ATTN: Library of Defense . New Delhi

I USA Agey for Aviation Safety. Ft Rucker, ATTN: Edue Advisor I Pert Rsch Ofc Library. AKA , Israel Defense Forces

1 USA Aviation Sch, Ft Rucker , ATTN: P0 Drawer a 1 Minister is van Defens ie , DOOP/KL Aid Sociaal

1 HQUSA Aviation Sys Cmd . St Louis , ATTN: AM SAV—ZDR Psych ologisc he Zaken . The Hague, Netherlands

2 USA Av iatIon Sys Tt,tt Act ,, Edwards AFB, ATTN: SAVTE—T

1 USA Air Def Sch. Ft Bliss , ATTN: ATSA TEM

I USA Air Mobili ty Rsc h & Dee Lab , Moffett Fld . ATTN~ SA VOL— AS

1 USA Aviation Sch, Ret, Tng Mgt. Ft Rocker . ATTN: ATST—T--RTM

1 USA Aviation Sch. CO. Ft Rucker, ATTN: ATST—D—A

1 HO. DARCOM , Alexandria , ATTN: AMXCD—TL

1 HO , DARCOM , Alexandria , ATTN: CDR

I US Military Academy, West Point, ATTN: Serials Unit

1 US Military Academy, West Point, A’ITN: Ofc of Milt Ldrshp

I US Military Acade my. West Point, ATTN: MAOR

1 USA Standardization Op. UK , FPO NY , ATTN: MASE—GC

1 Olc of Naval Rsch , Arlington , ATTN: Code 452

3 Ofc of Naval Rsch , Arlington, ATTN : Code 458

1 Ofc of Naval Rsch. Arlington , ATTN: Code 450

I Ofc of Naval Rsch , Ailinglon , ATTN: Code 441

1 Naval Aerospc Med Res Lab , Pensacola , ATTN: Acous Sch Div

1 Naval Aerospc Med Ret Lab, Pensacola , ATTN: Code L51

I Naval Aerospc Med Ret Lab , Pensacola , ATTN: Code L5

1 Chief of NavPers , ATTN: Pert-OR

1 NAVAIRSTA , Norfolk , ATT N: Safety Ctr

1 Nay Oceanographic . DC. ATTN: Code 6251, Charts & Tech

1 Center of N~,al Anal, ATIN: Doe Ctr

I FlavAirSysCom , ATTN: A IR—53 13C

1 Nay BuMed, ATIN: 713

1 NavHelicopterSubSqua 2, FPO SF 96601

1 AFHRL IFT) William AFB

I AFI-IRL (Ill Lowry AFE

1 AFHRL (AS) WPAF B . OH

2 AFHRL IDOJZ) Brooks AFB

1 AFHRL (DOJN) Lackland AFB

1 HQUSAF (INVSD(

1 HOUSAF (DPXXA)

1 AFVTG (PD) Randolph AFB

3 AMRL (HE) WPAFB. OH

2 AF Inst of Tech . WPAFB , OH, A1TN: ENE/SL

1 ATC (XPTO) Randolp h AFE

1 JSAF AeroMmi Lib , Brooks AFE ISUL—4). ATTN: DOC SEC

1 AFOSR (N LI, Arlington

1 AF Log Cmd, McClellan AF8. ATTN: ALC/DPCRB

1 Air Force Academy, CO. ATTN: Dept of Eel Sen

5 NavPers & Dee Ctr, San Diego

2 Navy Med Neuropsychiatric Rsch Unit, San Diego

1 Nay Electronic Lab, San Diego , ATTN: lIes Lab

1 Nay TrngCeri, San Diego, ATTN : Code 9000—Lib

1 Nav PostGra Sch , Monterey, ATTN: Code 55Aa

1 Nav PoatG raSch , Monterey, ATTN: Code 2124

I NavlrngEquipCtr, Oi-lando . ATIN: Tech Lib

1 US Dept of Labor, DC, ATTN: Manpower Admin

1 US Dept of Justice , DC, ATTN : Drug Enforce Admin

1 Nat But of Standards , DC. ATTN: Computer Info Section

1 Nat Clearing House for MR—Info , Rrackville

1 Denver Federal Ctr , Lakewood , A’TTN: ELM

12 Defense Documentation Center

4 Dir Psych , Army Hg, Russell Of cs. Canberra

I Scientific Advsr, Mil Ed, Army Hg, Russell 01cc, Canberra

I Mil and Air Attache, Austrian Embassy

I Centre dv Recherche Des Fai~

teurs , Humaine de Ia Defense

Nationale , Brusaela

2 Canadian Joint Staff Washington

1 C/Air Staff , Royal Canadian AF . ATTN : Pets Std Anal Br

3 Chief , Canadian Del Rsch Staff . ATIN: C/CRDS(W)

4 Briti sh Del Staff , British Embassy , Washington

24