Technical Report 437 LEYEL( USEFULNESS OF READABILITY FORMULAS FOR

0ACHIEVING ARMY READABILITY OBJECTIVES:

RESEARCH AND STATE-OF-THE-ART APPLIED

00 TO THE ARMY'S PROBLEM

0

Richard P. Kern

TRAINING TECHNICAL AREA

4

U. S.Army

Research Institute for the Behavioral and Social Sciences

C9 January 1980 .-.

L L_... Approved for public release; distribution unlimited.

80 7 10 044
U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating Agency under the Jurisdiction of the

Deputy Chief of Staff for Personnel

FRANKLIN AL HART

JOSEPH ZEIDNER Colonel, US Army

Techniical Director Commander

NOTICE

DISTRIBUTION! Primnary distribution of this report has been made by ARI. Plow address correspondenice

oncerning distribtition of reports to: U. S. Army Research Institute for the Behavioral and Social Sciences.

ATTN: PERf-TP, 5001 Einenhomsr Avenue, Alexandria, Virginia 22333.

FINA flSP~h~i~~d:This report may be destroyed swhoe it is no longer needed. Pleas do not return it to

th U.S. Army Research Institute for the Behavioral and Social Sciences.

bL" The findings in this report are not to be construed nas n official Departiment of the Army Position.

unlss so designated by other authorized doeumients.
Unclassified

SECU RITY CL ASSIFICATION O F THIS P AGE (When D ate Entered) R E ADIN ST R U C TIO NS

REPORT DOCUMENTATION PAGE BEFORE COIPLETING FORM

REPORT NUMBER 2. GOVT ACCESSION No. 3 RECIPIENTS CATALOG NUMBER

Technical Report 437 / -

4. TITLE (and Subtltlo) 5. TYPE OF REPORT & PERIOD COVERED

Usefulness of Readability Formulas for Achieving

Army Readability Objectives: Research id Stateof-the-Art Applied to the Army's Problem 7 6. PERFORMING ORG. REPORT-NUMBER

7. AUTHOR(e) 8. CONTRACT OR GRAN't NUMBER(&)

Richard P. Kern

9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. PROGRAM ELEMENT. PROJECT, TASK

US Army Research Institute for the Behavioral AREA 6 WORK UNIT NUMBERS

and Social Sciences, 5001 Eisenhower Avenue 2Q162722A777

Alexandria, VA 22333

It. CONTROLLING OFFICE NAME AND ADDRESS 12. REPORT DATE

US Army Administration Center January 1980 /

Ft Benjamin Harrison, Ind. 46216 IS. NUMBER OF PAGES

36

14. MONITORING AGENCY NAME & ADDRESS(If different from Controlling Office) IS. SECURITY CLASS. (of this report)

Unclassified

15.. DECL ASSI FI CATION/ DOWNGRADING SCHEDULE

IS. DISTRIBUTION STATEMENT (of this Report)

Approved for public release, distribution unlimited.

17. DISTRIBUTION STATEMENT (of the abetract entered In Block 20, It different from Report)

IS. SUPPLEMENTARY NOTES

It. KEY WORDS (Continue on reverse aid* it neceeary and Identify by block nowber)

Readability Standards

Readability Formulas

Reading Comprehension

Reading Skill Levels

Comprehension of Text

20. ABSTRACT (Contfinue on reverse elde II noceeeary and identify by block n.mber)

This report was prepared to assist the Army in evaluating the usefulness of

readability formulas in identifying material that will be comprehended by

readers at a given reading skill level.

Existing readability methods and research findings were reviewed against

three major criterion questions. These questions and conclusions reached

for each are as follows: _>

DD 1JAN73 1473 EDITION Ol INOV S IS OBSOLETE Unclassified

i SECURITY CLASSIFICATION OF THIS PAGE (Whon Date Entered)
Unclassified

SECURITY CLASSIFICATION OF THIS PAGEM(IM DO&. bMf.0

(1 Question: How effectively do formulas predict comprehension at

targeted reading grade levels?

Conclusion: Formulas cannot match material to reader.

.0

(2) Question: Does rewriting to lower readability scores increase comprehension?

Conclusion: Rewriting to lower the formula reading grade level score

does not increase comprehension.

(3) Question: How will formula score requirements affect production of

written materials?

Conclusion: Requiring that text be written to satisfy a targeted

reading grade level score focuses attention on meeting the score requirement

rather than on organizing the material to meet the readers' information needs.

An approach is outlined for developing a broader based method and criteria to

by-pass the serious deficiencies of present day readability formulas.

Aoo8ssio'rl For

NTIS 6a1A&1

TAc 

Unawounc ed

j u st i f i c a t i o n _ _ _ _ - -- -

By_ -

Av'..jL

ii Unclassified

sCUMITY CLASSIFICATION Of THIS PAOE(When Date fttegmd)
'-- ,USEFULNESS OF READABILITY jORMULAS FOR

ACHIEVING ARMY 1EADABILITY O6JECTIVES:

RESEARCH AND JTATE-OF-THE-ART APPLIED

. ... TO THE ARMY'S PROBLEM.

,-.rc (JarPKern7

Joyce L. Shields, Team Chief

Submitted by:

Milton S. Katz, Chief

TRAINING TECHNICAL AREA

Approved by:

E. Ralph Dusek, Director

PERSONNEL AND TRAINING

RESEARCH LABORATORY

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhower Avenue, Alexandria, Virginia 22333

Office, Deputy Chief of Staff for Personnel

Department of the Army

S =22277 rIndividual Training

Apmuad for Il rejlons: dlnrib lie unillfknid.

iii

J4
ARI Research Reports and Technical Reports are intended for sponsors of

R&D tasks and for other research and military agencies. Any findings ready

for implementation at the time of publication are presented in the last part

of the Brief. Upon completion of a major phase of the task, formal recommendations for official action normally are conveyed to appropriate military

agencies by briefing or Disposition Form.

iv
FOREWORD

The Training Technical Area of the Army Research Institute for the

Behavioral and Social Sciences (ARI) conducts a program of research in

support of the systems engineering of training. A major part of this

research is to develop the fundamental data and technology necessary to

put into the field integrated systems for improving individual job

performance. Such systems include Skill Qualification Testing (SQT),

job performance aids, training courses in schools and in the field,

performance criteria, and management and feedback systems.

This r ort was prepared in response to a request for Technical

Advisory Service from Director of Evaluation, US Army Administration

Center, Fort Benjamin Harrison, Indiana. This request was for technical

assistance in evaluating and identifying methods that will assist the

Army in achieving objectives for improved readability of Army publications.

This review focuses on a specific methodological-problem basic to

job aid research currently being conducted under Army Project 2QI62722A777,

Training Technology.

EI E

Technical Director

v
USEFULNESS OF READABILITY FORMULAS FOR ACHIEVING ARMY READABILITY

OBJECTIVES: Research and State-of-the-Art Applied to the Army's Problem

BRIEF

Requirement:

Evaluate existing methods for determining readability of written

materials, and determine the most effective method for achieving Army

readability objectives.

Procedure:

Existing readability methods were reviewed against three major

criterion questions presented below with conclusions. This review was

based on the existing research literature. Data were obtained from

research reports on the development and revision of a number of readability

formulas. These data were used to examine the extent of agreement

between readers' comprehension test scores and the reading difficulty

level score obtained by applying readability formulas to the same passages.

In addition, research was reviewed on the effects of rewriting text to

improve ease of comprehension.

Findings:

Question: How effectively do formulas predict comprehension at

targeted reading grade levels?

Conclusion: Formulas cannot match material to reader

Current readability formulas provide unreliable predictions of

reading skill levels at which text can be comprehended and are therefore

unsuitable for meeting Army readability objectives.

Example: Text comprehended by readers with 7th to 8th grade reading

skills may yield formula readability scores ranging from the 3rd to llth

grade levels. Text understood by readers with 9th to 10th grade reading

skills may yield formula scores ranging from the 6th to the 12th reading

grade levels.

Current readability formulas appear most reliable when used to make

a gross screening distinction between material that is above or below

about the 10th grade difficulty level. Soldiers who read below the 10th

grade skill level will generally not be able to readily comprehend text

that scores above the 10th grade difficulty level.

vii ffZCFZG=JG PAW BLANdK N OT F1LJRD
Question: Does rewriting to lower readability scores increase

comprehension?

Conclusion: Rewriting to lower the formula's reading grade level

score does not increase comprehension.

Research reviewed indicates that rewriting text to reduce its reading

difficulty level (formula score) by using shorter words and sentences to

"say the same thing" does not result in cdrresponding increases in

readers' comprehension.

Question: How will formula score requirements affect production of

written materials?

Conclusion: Requiring that text be written to satisfy a targeted

reading grade level score focuses attention on meeting

the score requirement rather than on organizing the

material to meet the readers' information needs.

Criteria used to identify writing that meets readability standards

should be criteria that writers and editors can be encouraged to use as

a guide to detect problems and improve their text. Present day readability formulas cannot provide this guidance. By shifting attention

from content features to formula scores, readability standards ray even

promote production of writing that is harder to understand.

Utilization:

This report presents an approach for developing a broad-based method

for establishing readability standards and for evaluating proposed

publications against these standards. This method will address the

following objectives readability formulas cannot address:

(1) Provide a method and criteria writers and editors can use to

develop and improve their material.

(2) Identify specific kinds of problems present when material does

not meet standards.

(3) Apply to illustrations and all other parts of a publication.

(Readability formulas can only be applied to passages of written text).

viii
USEFULNESS OF READABILITY FORMULAS FOR ACHIEVING ARMY READABILITY

OBJECTIVES: RESEARCH AND STATE-OF-THE-ART APPLIED TO THE ARMY'S PROBLEM

CONTENTS

Page

BACKGROUND ............. ........................... 1

OBJECTIVES OF THIS PAPER .......... .................... 1

THE ARMY'S PROBLEM ........... ....................... 1

WHAT DOES A READABILITY FORMULA PREDICT? ...... ............ 2

USEFULNESS OF READABILITY FORMULAS FOR ACHIEVING ARMY OBJECTIVES 2

How Effectively Do Formulas Predict Comprehension At

Targeted Reading Grade Levels? ....... ............... 2

Does Rewriting to Lower Readability Scores Result In

Increased Comprehension? ......... .................. 9

How Will Formula Score Requirements Affect Production

of Written Materials? ........ .................... ... 12

A MORE EFFECTIVE APPROACH TO ARMY OBJECTIVES ..... .......... 14

REFERENCES ........... ........................... ... 21

APPENDIX ............................ 23

DISTRIBUTION ........... .......................... ... 35

ix
TABLES

Page

Table 1. Comparison of Scaled Reading Grade Level (RGL)

Values with Recalculated Formula Scores for the

Eighteen Passages used by Kincaid, et. al., 1975. . . . 4

Table 2. Comparison of Scaled Reading Grade Level (RGL)

Values with Formula Scores for the Twelve Passages

used in the Development of the FORCAST Formula by

Caylor, et. al., 1973 ..... .................... 5

Table 3. Comparison of Scaled Reading Grade Level (RGL)

Values with Range of Formula Values Obtained on

Passages used in Development of the KincaidFlesch and FORCAST Formulas ..... ................ 7

Table 4. Range of Formula Values Obtained by Applying KincaidFlesch and FORCAST Formulas to Passages used in

development of Formulas and New Passages .... ........ 8

Table 5. Average Comprehension Scores (Percent Correct)

Obtained by Reading Skill Levels at each Reading

Difficulty Level Version of Two Passages

(Kniffin, et. al., 1979) ...... ................ .10

FIGURES

Figure 1. Major Headings From: In-Process Review Checklist,

Part I - Presentation Quality, (Price, 1975) ...... .15

Figure 2. Major Headings From: In-Process Review Checklist,

Part II - Job Performance Adequacy, (Price, 1975). . 16

Figure 3. Major Headings From: Checklist For Evaluation of

Technical Manuals, (Ross, 1959) ....... ......... .17

x
USEFULNESS OF READABILITY FORMULAS FOR ACHIEVING ARMY READABILITY

OBJECTIVES: RESEARCH AND STATE-OF-THE-ART APPLIED TO THE ARMY'S PROBLEM

BACKGROUND

Department of the Army has undertaken a long-range program to improve

the Army publication system. A major objective of this program, improving

the readability of Army publications, is addressed by Department of the

Army Circular 310-9 (15 December 1978). This circular prescribes use of

readability formulas as the major method for determining when readability

objectives have been met for a proposed publication. Commanding General,

US Army Administration Center (ADMINCEN), and The Adjutant General are

tasked in this circular to review existing methods for measuring readability

and to develop a single method for use in preparing Army publications.

The present paper was prepared in response to the Administration Center's

request for technical assistance in formulating their recommendations.

OBJECTIVES OF THIS PAPER

1. Evaluate the usefulness of readability formulas and other methods

for achieving Army readability objectives.

2. Determine the method which most effectively assures achievement

of Army readability objectives.

THE ARMY'S PROBLEM

The Army's objective is to produce publications which all of its

reader population can use effectively. To achieve this objective publications

must satisfy two interrelated conditions. They must: (a) clearly

address the job and training needs of the target audience, and (b) be

written in a style that is easily comprehended by soldiers in the target

audience. The Army has made considerable progress in applying Instructional

Systems Development (ISD) as a tool to identify content oriented to the

job and training needs of a target audience. Present need is for a

method to use in monitoring and improving the production of written

materials to assure that readers in the target audience will be able to

comprehend them. Of particular concern are the below average to average

reade-rs. This concern is reflected in the reading difficulty level

standards specified in the recent Department of the Army Circular 310-9

(Improving the Readability of Army Publications). For example, these

standards set a target reading difficulty level of the 7th grade for

training and job publications prepared for soldiers in pay grades E-1

through E-4. Data recently reported by Mathews, Valentine and Sellman

suggest that as many as 30 percent of the soldiers entering the Army

score below the 7th Reading Grade Level (RGL) on a Gates-MacGinitie 1i
Reading Test (Mathews et al., 1978). These data suggest good reason for

the Army's concern over providing job and training publications that can

be used effectively by the below average reader. The question is, can

readability formulas provide a sound basis for achieving the Army's

readability objectives? Are there alternative approaches that will be

more effective?

WHAT DOES A READABILITY FORMULA PREDICT?

Readability formulas provide a method for rank ordering difficulty level of textual materials in relation to average comprehension test

performance of students at successive school grade levels. These rank

order scores are usually expressed as reading grade levels (RGL).

School grade levels provide a means of indexing general growth in

language skills displayed by the average student over the course of

their exposure to school and their culture. A great deal of their

appeal rests on the assumption that if a relatively difficult passage

can be pegged, say, at the 12th school grade level, then the user knows

how "high" that is compared to one pegged to some lower grade level,

say, the 5th grade level. To accomplish this "pegging," readability

formulas are developed to predict the school grade level at which students,

if given a comprehension test on the passage in question, will achieve

an average test score equal to some pre-set criterion, for example, 70

percent correct. For instance, a 10th grade level readability score

represents a prediction that approximately 50 percent of the 10th grade

students will be able to read the passage and pass a comprehension test

given immediately after the reading. If this prediction could be made

with complete accuracy we would expect less than 50 percent of the 9th

grade students and more than 50 percent of the llth grade students to

pass the comprehension test. Obviously these predictions cannot be made

with complete accuracy. So the question is how much error can be expected

in these predictions? What are the implications of the amount of error

and sources of error for the way the Army proposes to set standards and

use these formulas?

USEFULNESS OF READABILITY FORMULAS FOR ACHIEVING ARMY OBJECTIVES

Usefulness of readability formulas for achieving Army objectives

hinges on three major considerations: (a) How effectively do formulas

predict comprehension at targeted reading grade levels? (b) Does

rewriting to lower readability scores result in increased comprehension?

And, (c) How will formula score requirements affect production of written

materials?

How Effectively Do Formulas Predict Comprehension At Targeted Reading

Skill Levels?

Data contained in two recent studies permit a look at how well readability formulas predict the grade level criteria used in the

development of the formulas (Caylor et al., 1973; Kincaid et al., 1975).

2
i°

Both of these studies were concerned with development of readability

formulas for military personnel using military job and training materials.

Caylor et al. developed the FORCAST formula using Army recruits and

passages from Army job and training publications. Kincaid et al.

recalculated six readability formulas using Navy enlisted personnel with

service of six months or less and passages selected from Navy job and

training material.

Comprehension Criteria Expressed As Grade Levels. Caylor and Kincaid

used the same method for establishing the grade level-comprehension

criteria to be predicted. This method consisted of first administering

a standardized reading test to the people in the sample to identify

their reading grade level placement. 1he experimental passages were

then given in the form of cloze tests. Grade level comprehension

criteria were then determined by identify'.ng for each passage the lowest

reading grade level in which 50 percent of the men reading at that grade

level achieved a cloze score at or above the 35 percent correct criterion

level.' The resulting criterion reading grade levels, called scaled

Reading Grade Levels (RGL), are shown in Table 1 for the 18 passages

used by Kincaid and in Table 2 for the 12 passages used in developing

the FORCAST. The formulas were then developed, using multiple correlational

techniques, to predict the scaled RGL value for each passage. The

FORCAST formula uses number of one-syllable words in a 150 word passage

as its predictor. The recalculated Flesch formula uses average number

of syllables per word and average number of words per sentence to predict

the scaled RGL.

1Different standardized reading tests were used in the two studies. Caylor

et al. used the United States Armed Forces Institute (USAFI) Achievement

Tests III, Abbreviated Edition, Form A (a special printing of the Metropolitan

Achievement Test, Advanced Battery). Reading Grade Level scores on this test

can range from 1.3 to 12.9. Kincaid et al. used two forms of the GatesMacGinitie Reading Test. Form E (grade levels 7-9) was given to personnel

with GCT scores below 55. Form F (grade levels 10-12) was given to those with

GCT scores of 55 and above.

2Cloze tests systematically omit words which testees are asked to fill in as

a measure of comprehension. Caylor et al. used five cloze tests for each of

their 12 passages. Version I deleted every fifth word counting from the first

word in the passage. Version 2 deleted every fifth word counting from the 4

second word in the passage and so forth. Kincaid et al. apparently used

only one version for each passage.

3Based on earlier research 35 percent correct on a cloze test is assumed to

be equivalent to 70 percent correct on a multiple choice comprehension test.

3
TABLE 1

Comparison of Scaled Reading Grade Level (RGL) Values with Recalculated

Formula Scores for the Eighteen Passages used by Kincaid et al., 1975.

Recalculated Formulas

Passage Scaled Kincaid- FORCAST ARI FOG Gunning

Number RGL Flesch (Rev.) (Rev.) Count (Rev.) FOG (Rev.)

15 5.5 8.4 7.9 7.7 10.1 9.2

17 5.8 9.3 11.8 10.2 9.9 11.0

1 6.9 9.7 8.0 8.5 10.9 7.7

5 7.0 7.1 9.7 8.6 7.5 6.9

9 7.8 8.0 8.1 8.0 8.9 5.9

10 8.9 11.7 10.5 11.9 12.9 12.0

13 8.9 10.0 9.0 11.5 11.0 8.5

11 9.0 8.1 10.6 7.0 6.5 7.3

18 9.8 6.6 8.0 10.6 6.7 7.4

14 10.8 12.5 12.1 10.6 12.2 12.0

4 10.9 8.2 9.5 7.9 7.3 7.5

16 11.8 13.8 13.9 12.7 12.5 15.4

7 12.0 11.7 11.8 11.3 11.6 10.6

3 16.0 12.7 13.0 12.4 11.4 12.7

6 16.0 12.3 13.8 11.3 11.2 10.8

2 16.1 16.7 15.6 18.3 14.4 12.1

12 16.1 11.8 8.7 12.6 13.5 9.7

8 16.3 14.7 13.5 14.7 16.4 14.9

Note: Scaled RGL and Recalculated formula values for the Flesch, Automated

Readability Index (ARI) and the FOG count are from Table 2, Kincaid et al.,

1975. Values for the Recalculated Gunning FOG Index and revised FORCAST

were calculated by John Brand, US Army Administration Center (USAADMINCEN)

and the author based on data provided by Kincaid.

4
TABLE 2

Comparison of Scaled Reading Grade Level (RGL) Values with Formula

Scores for the Twelve Passages used in the Development of the FORCAST

Formula by Caylor et al., 1973.

Formula RGL Values

Passage Scaled FORCAST Dale- F-J-P

Number RGL (Original) Chall Flescha

21 6 8.6 7-8 7

12 7 7.8 7-8 6

11 7 7.6 5-6 5

13 7.3 9.4 7-8 8-9

23 9.1 10.1 11-12 13-16

22 9.6 10.7 9-10 10-12

15 11.4 12.1 13-15 13-16

16 11.8 13.2 16+ 16+

26 12.0 12.2 16+ 16+

25 17.0 13.2 16+ 16+

24 12.1 11.3 13-15 13-16

14 13.0 10.9 13-15 13-16

Note: All values were taken from Caylor et al., 1973.

aF-J-P Flesch is the Farr-Jenkins-Paterson revision of the Flesch

Reading Ease formula.

5

....... ..........
Comparison of Predicted RGL With Scaled RGL Criteria. Tables 1 and

2 compare the scaled RGL values for the passages in each study with the

RGL score obtained by applying the formula to the passage. The purpose

of these comparisons is to illustrate the fact that present readability

formulas do not predict comprehensibility of material for specific

reading grade skill levels. They simply do not possess that precision.

In fact, inspection of the comparisons shown in Table 1 for the KincaidFlesch suggests that the precision of this formula is limited to predicting

whether comprehensibility of the material (Scaled RGL) is above or below

about the tenth RGL.

Range Of Formula Scores For Scaled RGL Intervals. Lack of precision

in these predictions is illustrated by comparing the range of scores

obtained from a given formula with the scaled RGL scores for the same

passages. The scaled RGL criteria and formula scores for the passages

are shown in Table 3 for the Kincaid-Flesch. Suppose that the Army

wanted material that could be readily understood by soldiers reading at

the 5th to 6th RGL. The first row in Table 3 shows that the passages

that met grade level comprehension criteria at the 5th to 6th RGL could

have Kincaid-Flesch scores ranging from the 8th to the 9th grade level.

Likewise soldiers whose reading skills were at the 7th to 8th grade

level could receive material ranging in Kincaid-Flesch scores from the

7th to the llth RGL. The range of Kincaid-Flesch scores for passages

that required reading skills at the 9th to 10th grade level is even

broader, ranging from the 6th to the 12th grade readability score. The

right hand column of Table 3 shows corresponding data used in developing

the FORCAST formula. It should be pointed out that the lack of precision

in predicting specific reading grade skill levels just illustrated is

not specific to the Kincaid-Flesch. A similar lack of precision is

characteristic of all of the formulas in Tables 1 and 2.

Range Of Formula Scores When Applied To New Passages. What happens

when formulas are lapplied to new passages not used in developing the

formula? Tables piresented above have illustrated the lack of precision

in predicting grad level comprehension criteria for passages used in

developing the forulas. These are the conditions when the formula

should give its most accurate prediction. We would expect more error in

prediction when the formula is applied to new passages. The author

calculated the FORCAST scores for the 18 passages Kincaid et al. used in

their study. Kincaid-Flesch scores were also calculated for the 12

passages used in the development of the FORCAST formula. Table 4

compares the results obtained when each formula was applied to its own

developmental passages with the results obtained when each formula is

applied to the other formula's developmental passages. The last column

in Table 4 for each formula represents an estimate, based on these

available data, of the range in formula scores we might expect for

material meeting comprehension criteria at each of the scaled grade

level intervals. These data should severely shake the confidence of

those advocating production standards that will match RGL of materials

with RGL of readers.

6
TABLE 3

Comparison of Scaled Reading Grade Level (RGL) Values with Range of

Formula Values Obtained on Passages used in Development of the KincaidFlesch and FORCAST Formulas.

Scaled RGL Kincaid-Flesch FORCAST

Interval Kincaid Passages FORCAST Passages

5-6 8-9 8

7-8 7-11 7-9

9-10 6-12 10

11-12 11-13 11-13

13+ 11-16 10

7
TABLE 4

Range of Formula Values Obtained by Applying Kincaid-Flesch and FORCAST

Formulas to Passages used in Development of Formulas and New Passages.

Kincaid-Flesch Formula

Scaled RGL Kincaid New (FORCAST) Composite

Interval Passages Passages Estimate of Range

5-6 8-9 5 5-9

7-8 7-11 3-6 3-11

9-10 6-12 7-12 6-12

11-12 11-13 14-21 11-21

13+ 11-16 11 11-16

FORCAST Formula

Scaled RGL FORCAST New (Kincaid) Composite

Interval Passages Passages Estimate of Range

5-6 8 9-11 8-11

7-8 7-9 9-10 7-10

9-10 10 9-11 9-11

11-12 11-13 11-12 11-13

13+ 10 10-13 10-13

8
Conclusion: Formulas Cannot Match Material To Reader. Readability

formulas do not provide reliable predictions of reading skill levels

(RGL) at which material can be adequately comprehended. This conclusion

is not restricted to the Kincaid-Flesch and the FORCAST. All of the

readability formulas shown in Tables 1 and 2 have high rank order agreement

with each other.

Does Rewriting To Lower Readability Scores Result In Increased Comprehension?

The wide range of error possible in using a readability score to

predict whether or not material will be appropriate for soldiers at the

lower reading skill levels was illustrated above in Tables 1-4. These

tables demonstrate quite clearly that present day readability formulas

can not be used to match difficulty level of the material to reading

skill levels of below average readers.

People concerned with producing understandable reading material for

below average adult readers advance a second argument for use of readability

standards. This argument is that lowering the readability levels of the

material will result in higher levels of comprehension by the below

average reader. This apparently is the assumption on which readability

standards in Department of the Army Circular 310-9 are based.

Klare's Review of Research Literature. Klare reviewed 36 experimental

studies performed to study the effects of lowering or raising the readability

score level of passages on comprehension of the material (Klare, 1976).

The general approach in this type of research is to rewrite a given

passage to make it easier (or harder) than the original passage without

changing content meaning. "Easier" or "harder" is determined by applying

a readability formula. The easier and harder versions of the same

passage are then given to different groups of readers and comprehension

scores are obtained and compared. Klare concluded that differences in

comprehension were obtained in 19 of the 36 studies, no differences were

found in 11 studies, and the remaining 6 studies produced "mixed" results.

The importance of Klare's review to the present concern is that it

suggests that present day optimism that simply reducing reading difficulty

level of material will increase the reader's comprehension stands on

shaky ground. Recent research examining this same question has been

carried out by the Navy and by the Air Force.

Recent Air Force Research On Lowering Readability Level Of Text.

The Air Force studied basic trainees at two reading test skill levels,

8th and 10th Reading Grade Level (Kniffin et al., 1979). The purpose of

this research was to measure the effects upon reading comprehension of

three sizes of literacy gap, zero gap (readability score numerically

equal to tested RGL), -2 gap (readability score 2 RGL higher than tested

RGL), and -4 gap (readability score 4 RGL higher than tested RGL). Four

versions of two different passages selected from Air Force training

9
TABLE 5

Average Comprehension Scores (Percent Correct) Obtained by Reading

Skill Levels at each Reading Difficulty Level Version of Two Passages

(Kniffin et al, 1979).

Reading Difficulty Level of Material

Reading Grade Level of Airmen 8th GL 10th GL 12th GL 14th GL

10th RGL Passage 1 X 67 64 62

Passage 2 x 81 78 79

8th RGL Passage 1 58 61 56 X

Passage 2 82 76 73 X

10
materials were prepared. The 10th reading grade level subjects received

versions of the two passages written at the 10th, 12th, and 14th grade

readability level (Kincaid-Fleach formula). The 8th reading grade level

subjects received versions written at the 8th, 10th and 12th grade

readability levels. The same 52-item multiple-choice comprehension test

for each passage was used with all subjects. The authors conclude that

the literacy gap produced a statistically significant but small effect

upon comprehension scores.

Table 5 presents the average comprehension scores (percent correct)

obtained by each group in the Air Force study. These scores illustrate

again the inadequacy of readability formulas for predicting grade level

comprehension criteria. Note that readers at the 8th grade level do not

reich the predicted 70 percent comprehension score for passage 1 on the

version written at the 8th grade difficulty level. In addition they V

appear to do almost as well on the 12th grade version of passage 1 as

they did on the 8th grade version. Also note the differences in comprehension

scores for passages 1 and 2 at the same reader skill level when the two

passages each received the same readability score. Passage 2 is consistently

easier.

Recent Navy Research On Lowering Readability Of Text. A report on

the Navy's research is not yet available. However, major features of

the research plan and main conclusions were given in an oral presentation

by Duffy (Duffy, 1979). Navy research studied the effects of careful

rewriting "to formula" on comprehension at different reading skill

levels.

Navy researchers took the eight passages used in the Nelson-Denny

Reading Test and, treating them as a set, prepared three different

versions of the set. One version was prepared by changing only vocabulary.

Shorter word synonyms taken from standard 4th grade vocabulary lists

were substituted for approximately 25 percent of the passage words.

This resulted in lowering the average Kincaid-Flesch score from 11.5 to

10.1 Reading Grade Level (RGL). A second version was prepared by only

shortening sentences. This resulted in an average Kincaid-Flesch score

of 7.3 RGL. A third version involved both vocabulary and sentence

changes and dropped the average Kincaid-Flesch score to 5.5 RGL.

The Navy conducted four different studies with the original and the

three revised versions of these passages. In each study the four versions

were presented to sailors at high (10.5 and above), middle (8.5 to 10.5)

and low (7 to 8.5) RGL. In Study 1 the standard Nelson-Denny time

limits were imposed and questions and text were presented together as in

the usual reading test procedure. Easier versions did not produce

improved comprehension at any of the three reading skill levels. In

Study 2 time limits were removed and questions were previewed prior to

seeing the text. After reading, the text was removed and the questions

were again presented. Results were the same as in the first study, easier

11
versions did not produce improved comprehension. In Study 3 the passages

were presented in the form of cloze tests with no time limits. Again,

easier versions did not produce improved comprehension. And finally,

testing conditions in Study 4 paralleled the "reading to learn" conditions

of a classroom setting. Sailors were allowed to study the passages with

no time limits imposed. Passages were then taken away and the test was

administered. Once again easier passages did not produce improved

comprehension for the high and mid-level reading skill groups. The low

reading skill group (7 to 8.5 RGL) did show an increase in comprehension.

However, the greatest increase (38 percent on original version vs 53

percent) was on the set of passages which had an average readability

score at the 10th grade level and was the most difficult of the easier

versions.

Conclusion: Rewriting To Lower The Formula Score Does Not Increase

Comprehension. Civilian and military research reviewed above suggests

that lowering difficulty of text by rewriting to formula does not enable

readers to comprehend material they could not comprehend from the more

difficult version.

How Will Formula Score Requirements Affect Production Of Written Material?

The Army's major purpose in establishing Reading Grade Level (RGL)

standards for material is to insure a closer match between difficulty

level of the material and reading skill level of the intended readers.

It has been assumed that the RGL scores obtained from readability formulas

identify the reading skill level (also expressed in RGLs) at which

readers will adequately comprehend the material. Research reviewed

above clearly indicates that this is a false assumption. Readability

formulas do not possess the predictive power to identify reading skill

levels at which specific material will be adequately comprehended. Data

reviewed suggests that their predictive power is limited to predicting

that material written above the 10th RGL will generally be too difficult

for soldiers whose reading skills fall below the 10th RGL. Other

research reviewed above indicates that rewriting "difficult" text to

"easier" readability score levels has no practical effect on improving

readers' comprehension of the text. Thus, use of readability formula

scores to determine when reading skill level standards have been met and

when material must be rewritten to satisfy the standards will be ineffective

in achieving Army objectives.

In addition to being inaccurate, use of formula scores to identify

readability standards has other serious drawbacks. Formulas do not

require the writer and editor to examine the writing for clarity in

addressing the targeted readers' information needs. When imposed as a

standard that must be met, attention tends to be focused on meeting the

score requirement rather than on organizing the material to meet the

readers' information needs.

12
Accommodations made to enable writers to meet formula standards may

even promote the production of less readable writing. For example, the

writer is often caught in the crunch of preparing material in a job area

that has a large, well established technical vocabulary. These are

usually multi-syllable words. It is generally considered mandatory that

the writer use these words. The Army Materiel Command (now Army Materiel

Development & Readiness Command) estimated in 1970 that approximately onethird of the words used in their technical manuals were mandatory words

(Army Materiel Command, 1970). This proportion of multi-syllable words

alone would result in a FORCAST estimate of reading difficulty at the

10th grade level. This poses a problem for the writer who must use

these words yet still stay within formula score requirements. A common

solution to this probelm is to keep the same formula -core requirement

but adjust either the formula or the method of counting. The effect of

these adjustments is to pretend the technical terms aren't there for

purposes of computing the score. This solution is justified by the

argument that technical words either are or will quickly become familiar

words to everyone, from entry level on up, working in the technical

area. Apparently, efforts to verify this assumption are rarely made.

Those that have been reported fail to support it (Jablonski, 1971; Hooke

et al., 1979).

Common acceptance of the assumption that technical terms are well

known terms illustrates how the push to meet formula score standards can

actually promote production of less readable writing. Given this

assumption of word familiarity, writers do not have to focus their

attention on how to best communicate technical information to readers

who have limited understanding of the technical area and its jargon.

The Air Force recently completed a research study of the effectiveness

of their readability standards in improving comprehensibility of publications in seven career fields. A major implication for the Air Force cited in

this study is as follows:

The policy of writing to target audience should be continued but

should not be enforced rigidly. Given the relatively crude ways

employed at present to estimate literacy gap, it is probably not

appropriate to insist that writers hit their targets with a great

deal of precision. Additionally, the practical problems involved

in simplifying materials below tenth grade level may be insurtiountable (Hooke et al., 1979).

Conclusion: Requiring that text be written to satisfy a targeted

reading grade level score shifts attention to meeting the score requirement

rather than focusing attention on organizing material to meet the readers'

information needs.

13
A MORE EFFECTIVE APPROACH TO ARMY OBJECTIVES

The recent focus on readability formulas has diverted attention from

developing and improving other approaches to readability standards. As

stated at the beginning of this paper, to achieve Army readability

objectives, publications must satisfy two interrelated conditions. They

must: (a) clearly address the job and training needs of the target

audience, and (b) be written in a style that is easily comprehended by

soldiers in the target audience. A readability formula provides no

information on how well writing meets the job and training needs of the

target audience. Formula scores provide only a gross index to how well

the writer has achieved an easily comprehended style. As numerous

supporters and critics of readability formulas have pointed out the

writing could be completely nonsensical and still obtain an "easily

readable" formula score. Thus, readability formulas can serve at best,

only a minor role in achieving Army objectives for improved readability

of publications. Achievement of Army objectives requires use of a

broader-based method for establishing readability standards and for

evaluating written material against these standards. 41!

A broader-based method, appropriate for the Army's use, does not

exist and will have to be developed. However, general models for such

a method and many of the components that would be used in developing the

method do exist. The basic model is the checklist used by editors and

teachers of technical writing to identify major features of a manuscript

which, by their judgment, are important contributors to its final readability.

Content and level-of detail in existing checklists depends on the developer's

special interests. Checklists ir technical writing texts tend to focus

on general principles of content organization rather than specific types

of material. The usual editor's checklist tends to focus primarily on

form (grammar, punctuation, spelling and format) rather than on function

(organization to convey information, depth and scope of coverage, and so

forth). Checklists developed to accompany Military Specifications

become extremely long and detailed. Figures 1 and 2 present the table

of contents for a checklist prepared for use by the Navy in conducting

quality assurance (Price, 1975). Figure 3 shows the major headings of

a much shorter checklist developed by Air Force researchers to rate

quality of manuals based on the presence or absence of problems users in

the field encountered in trying to use the manuals (Ross, 1959). None

of these checklists are directly appropriate for present Army objectives.

The point is that checklists can be designed to identify major types of

problems judged to make reader comprehension of the message more difficult.

They can be tailored to the special kinds of problems Army writers

encounter in communicating different kinds of material (for example,

administrative regulations versus job training materials). Their effectiveness

can even be submitted to experimental validation. And finally, they can

provide more useful diagnostic guidance to the writer on how to go about

a rewrite than is provided by readability formulas.

14
1. ORGANIZATION ..... ........... ... I... 1.. 

A. Principal Units and Work Packages......................I1

B. Arrangement within Sections or Packages................I1- 1

C. Composition Practices..................................1-~ 2

D. Prose-Graphic Balance .................................. I- 3

2. PROSE COMPREHENSION ........................................I1- 5

A. General Style Principles ...............................I1- 5

B. Instructional Writing .................................. I- 6

C. Paragraphs............................................I- 7

D. Sentences.............................................I - 7

E. Words.................................................I- 8

F. Non-Text Words and Phrases............................. I-10

G. Legibility ............................................ I-11

3. GRAPHICS COM4PREHENSIONT..................................... 1-12

A. General Graphics Principles............................ 1-12

B. Graphic Form Selection ................................. 1-14

C. Schematic and Wiring Diagram Practices................. 1-15

D. Network Diagram Practices.............................. 1-16

E. Block Diagram Practices................................ 1-17

F. Illustrations Practices................................ 1-17

G. Freestanding or Series Pictorials...................... 1-18

H. Tables Practices......................................1-18

I. Graphs Practices.......................................1-20

J. Photograph Practices................................... 1-20

4. READABILITY MEASUREMENT .................................... 1-21

Figure 1. Major Headings From:

IN-PROCESS REVIEW CHECKLIST

PART 1 - PRESENTATION QUALITY

(Price, 1975)

15
Page

1. ACCESS AND SEARCH .......................................... II-1

A. Identification ......................................... II-1

B. Sections or Packages ................................... II-1

C. Table of Contents and Headings ........................ 11-2

D. Index'................................................. 11-2

E. Reference Compliance ................................... 11-3

2. USABILITY AND ACCEPTANCE .................................. 11-5

A. Information Content Adequacy ......................... 11-5

B. Job Relevance and Efficiency ........................... 11-7

C. Workplace and User Compatibility ...................... 11-7

D. Technical Scope and Accuracy ......................... 11-8

Figure 2. Major Headings From:

IN-PROCESS REVIEW CHECKLISTS - PART II

-JOB PERFORMANCE ADEQUACY

(Price, 1975)

16
I. Layout

A. Physical Characteristics of the Manual

B. Order in Individual Operations

C. Index, Glossary, Charts, and Forms

II. Readability and Comprehensibility

A. Readability

B. Comprehensibility

III. Decision Making Procedures

A. Preparation of Decision Making Procedures

B. Trouble Indicators

IV. Diagrams, Photographs and Drawings

A. Diagrams

B. Photographs

C. Drawings

Figure 3. Major Headings From:

CHECKLIST FOR EVALUATION OF TECHNICAL MANUALS

(Ross, 1959)

17
Development of an Army Readability Standards Checklist requires a

research and development effort. This checklist should be developed to

meet the following objectives: (a) identifies specific problems that

need to be corrected when standards are not met; (b) is not limited to

only connected prose but is applicable to the entire manuscript (for

example, illustrations, tables and index); and, (c) does not require

extensive technical editing and writing expertise to apply. Extent of

this effort will depend on the final objectives established for the

checklist. Initial development will focus on working with Army writers

and editors to identify content for the different major types of literature

produced. Draft checklists will be tried out by writers and editors at

different levels in the production chain. Two major criteria will be

employed during this developmental process: (a) extent of agreement

among writers and editors in making specific judgments on the same

material, and (b) relationship of these judgments to users' judgments of

comprehensibility of the material.

Development of the type of checklist described above will require

analyses of the different types of material currently being produced.

An example of Lhis type of analysis and the kinds of problems writers

and editors might expect to find is provided by research conducted for

Army Research Institute (ARI). This research was initiated in the early

1970's in response to the Army's concern over the high reading difficulty

levels of training and job materials entry level enlisted personnel were

expected to use. Part of this research involved performing an extensive

analysis of the contents of Army Field Manuals and Technical Manuals.

This analysis resulted in the identification of a number of writing

problems, summarized under seven major problem areas (Kern et al.,

1977):

(a) Hard words, long sentences.

(b) Topic oriented writing - not directed to any particular job

user.

(c) Main points lost in detail - too vague, too wordy.

(d) Confused development - main points fragmented across paragraphs.

(e) Remote references - text relies on material located elsewhere.

(f) Weak visual aids - absent, illegible, not related to text.

(g) Run together format - lengthy, solid masses of print.

18
Pages from Field Manuals (FHs) and Technical Manuals (TMe) were selected

to illustrate these problem. The final sample of pages reflected

technical, combat and administrative content produced by writers from 11

different Army schools. Each sample was rewritten to provide a clearer,

less difficult version of the material. The rewriting, performed by

research staff members, focused on the seven problem areas listed above.

Note that only one of the seven problem areas listed, the first one, can

be addressed by writing to a readability formula. Preparation of clearer,

less difficult versions of the pages sampled from FMs and TMs required

extensive structuring and reorganization of the way the information was

presented. These changes are illustrated visually by the "Before" and

"After" examples presented in the Guidebook (Kern et al., 1977).

As demonstrated earlier, writing to a formula does not improve

readers comprehension of the material. The criterion for readability

should be one that writers and editors could be encouraged to use as a

guide to improve their writing. Research conducted in 1974 for ARI

indicated that most Army writers are instructors who have not had formal

training in technical writing (Appendix). Editors tended to confine

their review of manuscripts to punctuation, spelling and format regulations.

A checklist which identifies common problems in organizing and presenting

information to meet the readers needs could also assist the editors in

performing their job more effectively.

19
References

Army Materiel Command "Equipment Publications Readability Program."

Paper presented at the AMC/NSIA Equipment Manuals Symposium,

24-26 June 1970.

Caylor, John S., Sticht, Thomas G., Fox, Lynn C. and Ford, J. Patrick

Methodol6gies for Determining Reading Requirements of Military

Occupational Specialities. Technical Report 73-5, Human Resources

Research Organization, Alexandria, VA., 1973.

Headquarters, Department of the Army Improving the Readability of

Army Publications. Circular 310-9, 15 December 1978.

Duffy, T. Oral presentation at US Army Training Development Institute

Conference, Fort Monroe, VA., 24 May 1979.

Hooke, Lydia R., DeLeo, Philip J. and Slaughter, Sharon L. Readability

of Air Force Publications: A Criterion Referenced Evaluation

AFHRL-TR-79-21, Lowry AFB, Colo: Technical Training Division,

Air Force Human Resources Laboratory, May 1979.

Jablonski, Walter A. Improving the Readability of Maintenance Manuals,

Applied Behavioral Sciences, Inc., ABS-TP-1, 1971. AD 735 771.

Kern, Richard P., Sticht, Thomas G., Welty, Diana and Hauke, Robert N.

Guidebook for the Development of Army Training Literature. P-75-3,

US Army Research Institute, Alexandria, VA., November 1977.

Kincaid, J. Peter, Fishburne, Robert P. Jr., Rogers, Richard L., and

Chissom, Brad S. Derivation of New Readability Formulas for Navy

Enlisted Personnel. Research Branch Report 8-75, US Navy Technical

Training Command, February 1975. AD/A 006 655.

Klare, George R. "A Second Look at the Validity of Readability Formulas."

Journal of Reading Behavior, 1976, 8, 129-152.

Kniffin, J. Douglas, Stevenson, Calvin R., Klare, George R., Entin,

Eileen B., and Slaughter, Sharon L. Operational Consequences of

Literacy Gap. AFHRL-TR-79-22. Lowry AFB, Colo: Technical Training

Division, Air Force Human Resources Laboratory, May 1979.

Mathews, John J., Valentine, Jr., Lonnie D. and Sellman, Wayne S.

Prediction of Reading Grade Levels of Service Applicants from Armed

Services Vocational Aptitude Battery (ASVAB). AFHRL-TR-78-82.

Brooks AFB, Tex: Personnel Research Division, Air Force Human

Resources Laboratory, December 1978.

21
Price, Harold E. Development of a Draft Specification For Technical

Manual Quality Assurance, BioTechnology, Inc., prepared for Naval

Ship Research and Development Center, Contract N00167-76-C-0012,

December 1975.

Ross, Donald A. Comprehensibility Evaluation of Technical Manuals, WADC

Technical Note 59-442, Wright-Patterson AFB, Ohio: Aero Medical

Laboratory, Wright Air Development Center, Air Research and Development

Command, July 1959.

22
APPENDIX

The Study of Writing in Functional Contexts: The Army Writer

and the Design of a Guidebook for Developing

Army Training Literature

Richard P. Kern and Thomas G. Sticht

Human Resources Research Organization (HumRRO)

March 1974

Prepared For:

U. S. Army Research Institute

Contract: S73-33 (ATL), DAHCl9-73-C-0051

This is a special report designed for management on information

developed in HumRRO Work Unit ATL (Army Training Literature),

Tasks 4, 5, and 6 (Development of a Guidebook for Writers of Army

Training Literature). The report summarizes interviews with Army

writers. It presents the implications of this information for

the design of the guidebook and for policy regarding Army training

literature development.

23'
GUIDEBOOK FOR THE DEVELOPMENT OF ARMY TRAINING LITERATURE:

RATIONALE AND POLICY IMPLICATIONS FROM INTERVIEWS WITH ARMY WRITERS

Contract: S73-33 (ATL)

DAHC19-73-C-0051

PROJECT REQUIREMENTS: To design a Guidebook for Army writers which will

actually be used, information is needed about the

Army writer and his job.

PROCEDURE: Conduct interviews with Army writers to find out who

they are, how they do their jobs, and what they think

a writer's guidebook should include to be of help

to them.

INTERVIEW POPULATION: Army writers and editors located at five Army Service

School Centers chosen to reflect agencies responsible

for producing literature supporting relatively high

density Military Occupational Specialty (MOS) areas.

School MOS Area No. Interviewed

Infantry School, Fort Benning Combat 19

U.S. Army Signal Center and Technical 24

School, Fort Monmouth

U.S. Army Ordnance Center and Technical 25

School, Aberdeen Proving

Ground

U.S. Army Quartermaster Technical 9

School, Fort Lee

U.S. Army Institute of Administrative 17

Administration, Fort

Benjamin Harrison

Total 94

MAJOR CONCERNS: 1. The Army Writer

Who are Army writers?

What are their qualifications?

How is their time distributed over writing and

instruction?

II. Use of Writers' Guidebooks

Do writers use guidebooks?

What information would writers find useful in a

guidebook?

III. The Current Writing Process

What information is the writer given in a writing

assignment?

jhat types of support does the writer have during

the writing process?

What type of quality control is used prior to

publication?

How is feedback obtained from users?

IV. Nature of Army Training Literature (ATL)

How are Army Training needs identified?

Is ATL topic or job performance oriented?

Are training and reference functions of ATL

distinguished?

25
CONCERN 1: THE ARMY WRITER

QUESTIONS: WHO ARE ARMY WRITERS?

WHAT ARE THEIR QUALIFICATIONS?

SUMMARY OF INTERVIEWS

Writers of Army-Wide Literature are usually personnel who are considered subject matter experts in the assigned writing area, and who

are instructors in the area. At the Infantry School, writers were

predominantly military officers. At the technical and administrative schools writers were predominantly civilian personnel and

either present or former instructors. Qualifications for military

and civilian writers were usually stated as: (a) subject matter experts; (b) experienced as instructors; (c) demonstrated ability to

write instructional materials. Most writers were college graduates

with no formal training in technical writing.

QUESTION: HOW IS THEIR TLME DISTRIBUTED OVER WRITING

AND INSTRUCTION?

The writing assignment is an extra duty for most writers who must

also instruct. Writers pointed out that in the school setting, first

priority is given to instruction and the operational requirements

associated with supporting instruction. Writing manuals is a low

priority effort given little support. The instructor-writer felt

conflicts between the time required to perform his instructional

duties and the time required to research and write an adequate manual, with manual writing suffering from lack of time. Among military writers a high rate of turnover exists, with writing assignments being passed on from man-to-man.

26
IMPLICATIONS

For The Guidebook: The part-time nature of the writing job suggests

that the Guidebook must be designed to be-used by people who cannot

and will not devote considerable time to study to learn to be better

writers. Hence the Guidebook should avoid trying to train writers,

and instead minimize the amount of "mental energy" the writers must

expend to use the Guidebook. As much as possible the attempt should

be to:

* Catalog information in the Guidebook, not in the writer's head.

* Provide aids for decision making and accomplishing recurring

procedures.

* Emphasize intuitive understanding by examples rather than 4

lectures on "how to do it."

For Army Policy: The critical position of Army Training Literature

writers needs considerably more identity and stability if writers

are to effectively produce and also develop skills and techniques to

meet the changing needs of the Army. The Army should consider

creating permanent, full-time technical writers slots for the production of Army Wide training manuals. Then it would not be unreasonable to expect these personnel to undergo formal training in

effective writing and to continue to update their knowledge of ways

to use printed communication to support training and job performance.

27
CONCERN II: USE OF WRITERS' GUIDEBOOKS

QUESTIONS: DO WRITERS USE GUIDEBOOKS?

WHAT INFORMATION WOULD WRITERS FIND USEFUL

IN A GUIDEBOOK?

SUMMARY OF INTERVIEWS

Most of the writers interviewed said they do not use writer's guides

of any type. They get the guidance they need regarding format and

organization by reviewing previous editions of the same manual or a

related manual. Some writers, however, felt the following additional types of guidance would be helpful:

a. identify reference sources for literature research

b. how to conduct literature research

c. outline steps necessary for writing, going from

inception to final review

d. how to develop outlines

e. how to organize material to facilitate learning

f. how to determine reading grade level of material

g. how to write at predetermined grade level

h. when is it appropriate to be redundant?

i. when is it important to summarize?

J. guidance in types and best use of illustrations

k. how to obtain copyright release

1. guidance in determining user, purpose, and scope

CONCERN III: THE CURRENT WRITING PROCESS

QUESTION: WHAT INFORPIATION IS A WRITER GIVEN IN A

WRITING ASSIGNMENT?

Writers appear to be provided with little or no guidance regarding

users, purpose or scope when a new manual is to be written. Because

the writer is a subject matter expert, he is expected to define

these objectives. Lack of consistency in identifying users, purpose and scope was evidenced. Users, when identified, are usually

identified in terms of the organization (e.g., Company or Battalion)

or the Army at large (e.g., everyone from Private to General) and

rarely in terms of specific job positions within any organization.

For revisions, writers generally never see files containing the

previous manuscript, review comments, and change recommendations

that have come in since its publication.

28

*1
IMPLICATIONS

For The Guidebuok: Many writer's guides are available in the Army

today which provide extensive discussions of many of the topics mentioned by writers as candidates for a writer's guide, yet writers

do not use these guides. As suggested above, the ATL Guidebook

should try to capitalize on the fact that writers get ideas about

format and organization by reviewing a previous edition of a manual

or related manuals. Thus, the ATL Guidebook should provide a

"sampler" of various formats in different content areas which writers

can get ideas from for use in their manuals. Guidance should be

given for identifying users, for determining writing objectives, and

for selecting, assembling and organizing content.

For Army Policy: A concerted effort should be made to make current

Army writer's guides available to all writers. Continued education

of the writers regarding the contents of various guides is needed.

This goes hand-in-hand with the recognition of the writing job as

a professional position within the ATL system.

For Army Policy: The writer's writing assignment should specify the

purpose of the manual, the intended major user, the subject area, and what duties and tasks the writer is expected to relate the subject area to. For revisions, the writer should also be given a file

containing all comments and change recommendations that have come in

since the last publication of the manual.

29
CONCERN III: (CONT.)

QUESTIONS: WHAT TYPES OF SUPPORT DOES THE WRITER HAVE

DURING THE WRITING PROCESS?

WHAT TYPE OF QUALITY CONTROL IS USED PRIOR

TO PUBLICATIONS.

SUMMARY OF INTERVIEWS

To research information writers rely heavily on school lesson plan

files and previous editions of the manual. Whenever possible they

"cut-and-paste" from the earlier manuals as well as from current

editions of related manuals. They generally are dependent upon the

routine Post training aid support for illustrators; contact with the

illustrator tends to be of a paper-work, through channels nature and

not face-to-face communication. This, plus the experience of long

time delays cause most writers to avoid any use of illustrations

beyond their own capabilities or beyond those that can be cut-andpasted from other manuals. Writers did not use readability formulas or other methods for testing the difficulty of their writing.

Editorial review of the writer's draft is generally limited to

grammar, proof reading and conformance to format policies. There is

apparently no review by the writer or editor of copy set-up by the K

printer prior to production printing.

QUESTION: HOW IS FEEDBACK OBTAINED FROM USERS?

Generally, the only feedback writers receive from users is from the

students in the classroom. Field reviews were considered ineffective formalities. Suspense dates for review were considered too

immediate to allow adequate review. Writers who had served as reviewers stated that once their written comments were provided, they

learned nothing of the acceptance or rejection of their comments.

Instructors also considered the filling out of recommended change

forms (Form 2028) as futile because they saw the same errors repeated in future revisions. Generally, writers reported they had never

seen Field Review comments on manuals they had written and had no

idea whose comments had even been solicited.

30
IMPLICATIONS

For The Guidebook: Writers need information about the users of their

products. The Guidebook should give guidance in:

* identifying primary and other users

* determining reading abilities of users

The guidebook should give guidance in using information about users

to:

* define purpose and scope of manuals in terms of users

* write at the reading levels of the primary users

The guidebook should acknowledge the writer's tendency to imitate

past work, especially under time pressures, and provide a ready

source of a large set of samples of bad and good writing, formatting,

and illustrating which writers could examine and imitate the good

samples. Guidance should be provided for the use of the FORCAST

readability formula developed for use with Army manuals as a qualitycontrol tool during writing and review of drafts.

For Army Policy: Again, the part-time nature of the writing position places undue pressures on time. Higher priority to the writing

of ATL, including a better defined and more permanent position,

would appear desirable. Research reference files should be maintained

and made available to the writers rather than being an individual

writer's responsibility. These files should also include available

task analyses information on the various users to whom manuals may be

addressed. Provisions should be made for more direct illustrator

support with the illustrator a member of the writing staff. Quality

control during writing and processing of the draft should be reviewed

for approaches to improving it.

For The Guidebook: Since the only feedback writers receive is from

students in the classroom setting, the Guidebook should stress the

use of formal or informal task analysis information to help the writer

in addressing their manuals to user's needs in the field. Contacts

limited to school students also mean that writers have little or no

contact with users in the field who are less capable readers than

they or their students. Therefore, the Guidebook should help the

writer understand the nature of the problems the poorer reader faces

in extracting information frem printed materials.

For Army Policy: The current system does not provide feedback that

writers or literature planners can use to tell whether a manual,

once produced, meets the needs of the intended user, or, in fact, is

even used. The Army should consider a new system to either supplement or replace the Field Review process. The new system should be

a regular system managed by a central authority (e.g., TRADOC). It

might, for example, collect information from users in the field on

the utility and acceptability (readability and readership) of training literature publications and feed this information back to the

training literature branch which prepared the publications. If

this information were collected at the direct user level and not

just at the field command level it should provide useful guidance

for both the planning and writing of ATL.

31
CONCERN IV: NATURE OF ARMY TRAINING LITERATURE

QUESTION: HOW ARE ARMY TRAINING LITERATURE NEEDS

IDENTIFIED?

SUMMARY OF INTERVIEWS

Under the present system school staff carry a major responsibility

for identifying Army Wide literature needs. Each school perodically reviews the Field Manuals, Technical Manuals, etc., for which

they are responsible. From this review recommendations are made

for updating or preparing new material. The school consolidates

these recommendations, including any comments received from the field,

and staffs them through TRADOC for approval, budgeting and priorities. Manuals are then assigned to the appropriate branch or instructional department, where specific manuals are, in turn, assigned to

the person who will do the actual writing. When the assignment

reaches the writer it is frequently simply a directive to write a

manual on "subject X".

QUESTIONS: IS ATL TOPIC OR JOB PERFORMANCE ORIENTED?

ARE TRAINING AND REFERENCE FUNCTIONS OF

ATL DISTINGUISHED?

Writers view their jobs as that of assembling, organizing and recording what is known about a topic. It is assumed that if a manual

contains a reasonably complete exposition of the topic, it can serve

as a general text or reference source and be used by anyone for any

purpose, whether initial learning, looking-up previously learned

information, or consulting for directing the training, learning and

job performance of others. Manuals were only rarely considered in

terms of job duties or task performance.

32
IMPLICATIONS

For Army Policy: A clear identification of the purpose and intended

major user of ATL would i;eem to be basic to planning literature needs

and priorities for the ATL Program. This information should then

become part of the writer's assignment rather than charging the writer

with determining purpose and user after the decision has been made

to write the manual.

For The Guidebook: Guidance should be provided on how to prepare

literature oriented to job performance of users, inzluding design of

manuals for more thin one major user (e.g., commanders, supervisors,

workers, instructors). Guidance should be provided on the design of

manuals for flexibility of purpose (e.g., for initial learning and

for later reference).

For Army Policy: The same manuals designed for presentation of

general, topic-oriented information are not effective as instructional texts for new learners, as job aids for directing job performance, and as reference sources for experienced workers.

Empk. sis should be given to policy which recognizes the different

functions of ATL and the design implications of these functions.

33
DISTRIBUI11ON

ARI Distnbution List

4 OASO (M&RAI 2 HOUSACOEC. Ft Ord. ATTN: Library

2 HODA (DAMI-CSZ) 1 HOUSACDEC, Ft Ord, ATTN: ATEC-EX-E-Hum Factors

I HODA (DAPE-PBR) 2 USAEEC. Ft Benjamin Iarnaon. ATTN: Library -

1 HGOA (DAMA-AR) I USAPACDC. Ft Benjamin Hl'tlon. ATTN: ATCP-HR

1 HQDA (OAPE-HRE-POI 1 USA Comm-Elect Sdc. Ft Monmouth. ATTN: ATSN-EA

1 H0DA (SGRD-ID) 1 USAEC. Ft Monmouth, ATTN: AMSEL-CT--HOP

I HODA (AMI.OOT-C) 1 USAEC. Ft Monmouth. ATTN: AMSEL-PA-P

1 HOA (DAPC-PMZ-Al I USAEC, Ft Monmouth. ATTN: AMSEL-SI-CB

I HODA (OACH-PPZ-A) 1 USAEC, Ft Monmouth. ATTN: C. Fed 0ev Br

I HQOA (DAPE-HRE) I USA Materials S" Anal Agcy. Aberdeen. ATTN: AMXSY-P

I HODA IDAPE-MPO-C) I Edgwood Aw. Aberdeen. ATTN: SAREA-L-H

1 HODA iDAPE-OW) 1 USA Ord Ctr & Sch. Aberdeen. ATTN: ATSL-TEM-C

HOA (OAP-HRL) 2 USA Hum Engr Lab, Aberdeen. ATTN: Library/Dir

I HODA (DAPE-CPS) I USA Combat Arms Tng Sd. Ft Banning. AT'TN: Ad Supervior

HQA (IAFD-MFA) USA Infantry Hum Rich Unit. Ft Seinng ATTN: Chief

1 HODA IOARD-ARS-P) 1 USA Infntry Sd. Ft Benning. ATTN: STESC-TE-T

I HODA IDAPC.PAS-AI I USASMA, Ft ii. ATTN: ATS3-LRC

HOQA (OUSA-OR) 1 USA Air D4 Sch, Ft Olin. ATTN: ATSA-CTD-ME

SHQODA (DAMO-ROR) 1 USA Air Dof Sd. Ft 8in. ATTN: Tech Lib

I HQDA (DASG) 1USA Air Dof Sd. Ft Blis. ATTN: FILES

SHOOA (DA1O-PI) 1 USA Air DOef Bd. Ft Blin, ATTN: STIEO--PO

I Chief. Consult Div IDA-OTSG . Adelpii. MD 1 LSA Crd & General Sd College. Ft Leavenworth, ATTN: Ub

SMil Ant. Hum Re,. OOOR&E, OAD (E&LS) 1 USA Cmd & General Stf College. Ft Leavenworth, ATTN: ATSW-4E-L

I HG USARAL, APO Seattle. ATTN: ARAGP-R 1 USA Cmd & General St College, Ft Leaeworth. ATTN: Ed Adv ior

1 HQ First Army, ATTN: AFKA-O-T1 1 USA Combined Anm Cmbt Dev Act. Ft Leevnworth, ATTN: De%,dr

2 HO Fifth Army. Ft Sam Houston 1 USA Combined Anns Cmbt Dev Act. Ft Leemworth. ATTN: CCS

I Oir, Army Stf Studies Ofc. ATTN: OAVCSA (DSPI I USA Combined Airms Cmbt Do Act. Ft Leswenr d. ATTN: ATCAS.A

SOft: Chief of Stf. Studies Of USA Combined Arms Cmb Dev Act. Ft LUaenworth. ATTN: ATCACO-E

I DCSPER. ATTN: CPS/CP I USA Cousbed Arm% Cmbt 0ev Act. Ft Laaenvwth, ATTN: ATCACC-,I

I The Army Lib. Pentagon. ATTN: R8 Chief I USAECOM. NiVlt Vision Lab. Ft Balvoir, ATTN: AMSEL-NV-S0

1 The Army Lib. Pentagon. ATTN: ANRAL 3 USA Computer Sys Cmd. F, Belvoir. ATTN: Tech ibrary

1 Oft. Ant Sect of the Army (R&O) I USAMEROC. Ft Belvoir, ATTN: STSFS--0O

1 Ted Spor Ofc. OJCS I USA Eng Sch. Ft Belvsir. ATTN: LibrrVy

1 USASA. Arlinton. ATTN: IARO-T I USA Topographic Lab, Ft Selvoir. ATTN: ETL-TO-S

I USA Rech Oft. Durhlm. ATTN: Life Sciences Dir I USA Topographic Lab. Ft Selwor, ATN: STINFO Center

2 USARIEM. Natick. ATTN: SGRO-UE-CA 1 USA Topographic Lab. Ft Selvoir. ATTN: ETL-GSL

I USATTC. Ft Clayton. ATT"N: STETC-MO-A I LSA Intelligence Ctr & Sch, Ft Huachuco. ATTN: CTD-MS

I USAIMA. Ft Brag ATTN: ATSU-CTD-OM I USA Intligence Cr & Sch, Ft Huachud . ATTN: ATS--CTD-MS

I USAIMA. Ft Braen ATTN: Miquat Lib 1 USA Intel'*nice Ctr & Sch, Ft Huachuca, ATTN: ATSI-TE

I US WAC Ctr & Sch. Ft McClellan. ATTN: Ub 1 USA Intelligence Ctr & Sch, Ft Huacuca. ATTN: ATSI-TEX-GS

I US WAC Ctr & Sch, Ft McClellan. ATTN: Tng Dir I USA Intelligence Ctr & Sch, Ft Huschuce. ATTN: ATSI-CTS-OR

I USA ( aermter Sch, Ft Lee, ATTN: ATSM.TE I USA Intelligence Ctr & Sch, Ft Huadues. ATTN: ATSI-CTO--"T

I Intelligece Material 0ev Ofc, EWL. Ft Holabird 1 USA Intelligence Ctr & Sch, Ft Huachuca. ATTN: ATSI-4t -CS

1 USA SE Signal Sch. Ft Gordon. ATTN: ATSO-EA 1 USA Inuligenca Ctr & Sch, Ft Huachuca, ATTN: O.A/SRO

1 USA Chaplain Ctr & Sch, Ft Hamilton, ATTN: ATSC-TE-RO 1 USA Intelligence Ctr & Sch. Ft Huachuca. ATTN: ATSI-TSM

1 USATSC4H. Ft Eustis. ATTN: Educ Advisor 1 USA Intelligence Ctr & Sch. Ft Huacuc. ATTN: Library

1 USA War College, Carlisle 8arracks. ATTN: Lib I CDR. HO Ft Huachuca. ATTN: Tech Ref Div

2 WRAIR, Neurooeychiaury Div 2 CDR. USA Electronic P" Grd. ATTN: STEEP-MT-S

1 OLI. SDA. Monterey I HO, TCATA, ATTN: Tech Libray

1 USA Concept Anel Agcy. Bethesda. ATTN: MOCA-MR 1 HO. TCATA. ATTN: AT CAT-OP-A. Ft Hood

1 USA Concept And Agc , Bethes4da ATTN: MOCA.JF I USA Recruilting Cmd. Ft Sheridan. ATTN: USARC.PM-P

U,. An- Ta Cir, APO S-.-a. ATT?,J: STZAC-PL-MI I Senor Army Adw.. USAFAGOO/TAC, Elgin AF Aux Fid No. 9

I USA Arci Ten Cr, APO Seette. ATTN: AMST-PL.TS I HO. USARPAC. 0CSFE R, APO SF W5ft ATTN: GPPfSE

I USA Armwnent Cmd. Redstone Armnal. ATTN: ATSK-TEM I Stirnacm Lib. Academy of Health Sences. Ft Sam Houston

I USA Armament Crmd. Rock Island. ATTN: AMSAR-TDC 1 Marine Corpe Ins.. ATTN: Dean-MCI

1 FAA-NAFEC. Atlantic City. ATTN: Library 1 HCL USMC. Commandant. ATTN: Code MTMT

I FAA-NAFEC, Athintic City. ATTN: Human Engr Br 1 HO. USMC. Commafant. ATTN: Code MPI.20-28

I FAA Aeronautical Ctv, Oklahoma Cty. ATTN: AAC-440 2 USCG Academy. New London. ATTN: Adlis on

2 USA Fd Arty Sch. Ft Sill. ATTN: Library 2 USCG Academy. New London. ATTN: Litmry

1 USA Armor Scd. Ft Knox. ATTN: Ubrary 1 USCG Training Ctr. NY. ATTN: CO

1 USA Armor Sdh Ft Knox. ATTN: ATSB-DI.E 1 USCG Training Ctr, NY. ATTN: Educ Svc Oft

I USA Armor Sch. Ft Knox, ATTN: ATSS.0T-TP I USCG. Ptychol Re r. DC, ATTN: GP 1/62

I USA Armor Sch, Ft Knox. ATN: ATSS-CD-AO *I HO MId-Range Sr. MC Det Quentico. ATTN' P&S Ole
I US Marine Corps Laiton Otc. AMC. Alaandria. ATTN: AMCGS-F I 0.f & Civil Inii of Enviro Medicine, Canada

I USATRADOC. Ft Monroe. ATTN: ATRO-ED I AIR CRESS w.anrsniltor, ATTN: Info S'!s Br

6 USATRAQOC, Ft Monroe. ATTN: ATPR-AD 1 Militrerpsykologis Tenests. Copenhagen

I USATRADOC. Pt Monroe. ATTN: ATTS-EA Mdiltary Attache. French Emoessy, ATTN: Doc .. -

1 USA Forces Cmd, Ft McPherson, AYTN: Libriry 1 Medgicin Chef. C.E.R.P.A.-Arsenal. ToulonwNavi an:ce

2 USA Aviation Test Bd. Ft Rucker, ATTN STEBG-PO 1 Prn Scientific Oft. AppI Hum Engr Rsch Div. M;r itry

I USA AgcV for Ayataon Safety, Ft Rucker. ATTN: L-ibary of Defense. New Delhi

I USA Agcy for Aviation Safety. Ft Ruijkor. ATTN: Educ Adsor 1 Pers Rsch Ofc Library, AKA, Israel Defense Fofc,

I USA Aviation Scsi. Ft Rucker. ATTN PO Draver 0 I Ministeris van Defrnsie, DOOP/KL Afd SociaaI

I HOUSA Aviation Sys Cmd, St Louis. ATTN: AMSAV-ZDR Psychologtche Zaken. The Hague. Netherlands

2 USA Aviaton SJyi Test Act., Edwyds AFII. ATTN SAVTE-T

I USA Air Del Sh, Ft Bliss. ATTN: ATSA TEM

I USA Air Mobility Rsch & Di.'i Lab, Moffett Fid. ATTN: SAVOL-AS

1 USA Aviation Sch. Res Tng Mgt, Ft Rucker, ATTN: ATST-T-RTI

1 USA Aviation Sch. CO. Ft Rucker. ATTN: ATST-O-A

I HO. DARCOM, Alexandria, ATTN- AMXCO-TL

I HO. DARCOM. Alexandna, ATTN: CDR

I US Military Academy. West Point, ATTN: !;trials Unit

I US Militrwy Academy. West Point. ATTN: Ofc of Milt LdrshP

I US Military Academy. West Point. ATTN: MAOR

I USA StandaIt'izleton Gp. UK. FPO NY. ATTN: MASE-GC

I Ofc of Naval Rscth. Arlington. ATTN: Code 452

3 Of'- of Naval Rwch. Arlington. ATTN: Code 458

1 O, of Naval Rsch. Arligton. ATTN: Code 450

1 Clfc of Naval Rsch. Arlington, ATTN: Code 441

1 Naval Aerospi: Med Res Lab. Pensacola. ATTN: Acous Sch Div

1 Naval Aerospc Med Rts Lab, Pensacola, ATTN: Code L51

1 Naval Aeros.c Mod Res Lab, Pensacola, ATTN: Code LS

1 Chief of NavPer. ATTN: Pert-OR

I NAVAIRSTA. Norfolk. ATTN: Safety Ctr

I Nov Oceanographic. DC. ATTN: Code 6251. Charts & Tsch

I Canter of Naval Anal, ATTN: DOC Ctr

I NavAirSysCom, ATTN: AIR-5313C

1 Nay BuMed. ATTN: ; 13

I NavHelicoptorSubqua 2. FPO SF 9MI

1 AFHRL IFT] WIlliams AFO

1 AFHRL ITT) Lowry AFB

I AFHRL (ASI WPAFB. OH

2 AFHRL.IOOJZ) Brooks AFS

1 AFHRL IDOJN) Lacklarid AFB

1 HOISAF IINYSO)

1 HOUSAF IDPXXA)

I AFVTG (RO1 Randolph AF8

3 AMRL (HE) WPAFB. OH

2 AF Inst of Tesh. WPAFB, ON. ATTN: ENEJSL

I ATC IXPTO) Randolph AF8

1 USAF AeroMed Lib. Brooks AFB (SUL-4). ATTN: DC SEC

1 AFOSR (NL), Arlington

I AF Log Cmd. McClellan AFS, ATTN: ALC/DPCRB

1 Air Force Academy. CO. ATTN: Oe t of Sil Scn

5 NvPser & D0v Ctr. San 014

2 Navy Med Neuropsychiatric Rdici Unit. Son Diego

I Nay Elecronic Lab. Sar Olesgo. ATTN: Res Lab

I Nov isngCn. San Oie*o. ATTN: Code 9000- Lib

1 NavPostGr Sch. Monterey. ATTN: Code 55Aa

I NsvPostGrSch. Monterey. ATTN: Code 2124

1 NavTrngEquioCtr. Orlande, ATTN: Tech Lib

1 US Dept of Labor, DC. ATTN: Manpowe Admin

I US Dept of Justice, DC, ATTN: Orug Enforce Admin

I Nat Bur of Standards, DC. ATTN: Computer Info Section

I Nat Clearing Howue for MH-lnfo. Rockville

I Denver Federal Cir. Lakewood. ATTN: OLM

12 Defense Documentation Center

4 Dir Psvch, Army HQ. Russell Ofcs. Canberra

I Scientific Advsr. Mil 8d. Army Hq, Russell Ofcs. Conoer;

I Mil and Air Attache. Austrian Embassy

I Centre do Rechorce De Factsurt, Hum ine do Is Dafensa

National., Brussels

2 Canadian Joint Staff Washington

I C,'Ait Staff. Royal Canadian AF, ATTN: Pert Std Anal Br

3 Chief, Canadian Def RAch Staff. ATTN: C,'CROSIW)

4 British 0ef Staff. British Emoassy. Washington