.“..

0

f=?j

United States

*i&d

Department of

Agriculture

@” ‘-Q

0

U4.S Forest Service IUlrn

Southern Forest

Research Note

Experiment Station SO-282

September, 1982

Forest Fire Prevention Programs and Their Evaluation

In U.S. Forest Service Region 8

G. Richard Wetherill

SUMMARY

/

LA telephone survey of all national forest ranger districts

in Region 8 obtained daadescribing the status of forest

fire prevention program evaluation. Out of the 396 programs being conducted on the 105 districts in the South,

only one program had undergone any sort of systematic

evaluation. Survey data indicate that ranger district prevention personnel are aware of the lack of evaluation

methods, but are unsure how to go about evaluating a

program without the sole reliance on fire occurrence statistics.

Additional keywords: survey, mass media programs,

midrange programs, personal contact programs.

INTRODUCTION

Despite the easily recognized benefits of program

evaluation, (evaluation provides the feedback necessary

to improve programs), logical, documented evaluation of

the fire prevention programs of forestry agencies is seldom done. The extent of this failure will be discussed in

this report.

THE PROBLEM

In August 1977, an “administrative” study was initiated

by the Southern Forest Experiment Station’s Fire Prevention Research Unit that sought to identify and classify

existing mid-range forest fire prevention programs in Region 8 of the U.S. Forest Service. This program inventory

eventually identified 66 individual prevention programs

then being operated by both State and National forest

personnel (Wetherill 1978). Sponsors claimed that evaluation was underway or planned for 10 of these programs. Informal communication with forestry personnel in

this region had led us to believe that there were substantially more than 66 programs in operation, but it was

reasonable to assume that the proportion of those evaluated to those not evaluated would remain about the

same.

At this point in the process of seeking reliable methods

for evaluating forest fire prevention programs, it was

necessary to establish baselines of current evaluation

practices. The 1977 inventory was a beginning, but its

information was outdated and totally lacking in rigorous

examination of local evaluation methodologies. Metaevaluation (the evaluation of evaluation) was viewed as

the means by which baseline data on current evaluation

practices could be established.

OBJECTIVES

There were two main objectives to this study, (1) to

update the inventory of and to classify all current fire

prevention programs in U.S. Forest Service Region 8,

and (2) to examine both past and present program evaluations (and their techniques) conducted by Forest Service ranger districts to assess procedures and

processes.

STUDYPROCEDURES

We determined that a telephone survey would be the

most effective and efficient way of gathering the data we

needed. A questionnaire was constructed and a Forestry

Aide was hired and trained for the actual data collection.

Southern Forest Experiment Station/T-102lO U.S. Postal Services Bldg., 701 Loyola Avenue, New Orleans, La. 70113

Forest Service, U.S. Department of Agriculture.

Serving Alabama, Arkansas, Louisiana, Mississippi, E. Oklahoma, Tennessee, E. Texas, Puerto Rico, U.S. Virgin Islands

1
Telephone numbers for all Region 8 ranger districts were

supplied by the Forest Service Regional Office.

The interviewees were selected by asking to speak to

“the person responsible for the day-to-day operation of

fire prevention on the district” (or words to that effect).

Each interview lasted approximately ten minutes. Interviewing began on July 15, 1979, and was completed on

July 31, 1979. All 105 national forest ranger districts were

contacted and questionnaires completed during that time

period, then all questionnaires were checked and edited

and the resultant data were tabulated and analyzed.

DATA PRESENTATION

Respondents provided information for this report in two

areas, (1) current prevention efforts and (2) opinions

regarding the state of evaluation.

Current Prevention Programs

Respondents were asked to supply a listing of and

detailed description of each of the forest fire prevention

programs currently being conducted in their district. They

were also asked to informally judge the effectiveness of

each program.

The three major types of fire prevention programs

being conducted by national forests in the South are

mass media programs, midrange programs, and personal contact programs. A separate category was established for law enforcement programs because eight respondents identified it as a current program.’

Mass meda programs consist of (1) television, (2)

radio, (3) signs, (4) print media, and (5) posters.

Midrange programs consist of (1) gimmicks and displays (exhibits at fairs, shopping malls or public meeting

places; parade floats; contests; etc.), (2) school programs (school assembly and/or classroom approaches

to teaching fire prevention, grades K-12), (3) community

group programs (programs in which prevention personnel organize or assist in organizing community-level

groups to do the business of fire prevention), (4) demonstrations (programs in which participants in the educational experience can see or touch objects directly related

to the concept being taught), (5) public information contactor approaches (this category closely approaches the

personal contact approach, but was different enough

from personal contact to warrant inclusion in the mid-

’ It is doubtful that there are only eight law enforcement programs in

existence in Region 8. but since the telephone interviews called for

purely voluntary responses (no prompting) regarding program types,

we can assume that mention of law enforcement programs as a part of

the total prevention spectrum was inadvertently neglected.

range category because the programs operated in field

settings such as visitor centers and combined elements

of classroom teaching, demonstrations, exhibits, and

open forums with individual contact), and (6) open

forums, workshops and training programs (this subtype

includes such programs as fire training, summer camps,

teachers’ workshops, and environmental workshops,

which are more difficult to organize and operate than

most of the other programs, but have many potential

benefits).

Personal contact programs may make use of (1) agency personnel for the direct contact work, (2) local voluntary, (especially local opinion leaders), or a (3) created

opinion leader (which involves the creation and subsequent use of an opinion leader in the target locality).

Table 1 shows the total number of programs for each

national forest major administrative unit. Table 2 shows

the frequency distributions of all reported programs,

broken down by program subtypes and reputed program

effectiveness. The distribution across the various program subtypes proved to be generally as expected; radio,

print media, school programs, and agency personnel

contactor programs were dominant.

Perhaps one of the more interesting uses to which the

admittedly subjective judgments on program effectiveness could be put would be to translate the responses

into a confidence-in-program scale. Taking away the “noconfidence” votes (or the “no” and “don’t know” responses) gives us a rough estimate of the respondents’

confidence in each program type and subtype. Scores of

confidence can be obtained by dividing the number of

“yes” votes by the total number of programs in each

category and multiplying the result by 100. Table 3 shows

Program Confidence Scores and translates them into the

familiar school letter grades. Using this system, three

program approaches rate a grade of A, signs, organizing

community groups, and local voluntary personal contact.

Lest we push this analogy too far, we should not consider these grades as “final” grades. “Semesters” for fire

prevention programs are quite long and, in most case,

these ratings indicate respondents’ perceived successes

and/or failures with the various approaches under widely

varying conditions. Some people are more comfortable

with one approach than with another. These grades may

reflect more upon the respondents’ comfort with the

approach than on the intrinsic worth of the approach.

Without formal, rigorous evaluation it is difficult to determine the true effectiveness of these programs.

Opinions of Evaluation

The dearth of prevention program evaluation became

clear when the questions pertaining to evaluation were

tabulated. Quite frankly, the results were not as we had

assumed they would be prior to beginning this study. We

G. Richard Wetherill is Sociologist, Southern Forest Experiment Station, Forest Service-USDA, Starkville, Mississippi.

2
had expected to find at least the same 10 program evaluations identified in the earlier inventory (see page 1).

Instead, we found the distribution shown in table 4. We

surmised that we would find several formal program evaluations, but we uncovered only eight instances where

program effectiveness was determined by means other

than fire occurrence.2 Additionally, seven national forest

*The on/y case of formal evaluation was one initiated in another study

under the auspices of this unit and a cooperator. The other seven

instances of evaluation consisted of “receiving feedback from target

groups” and “fire prevention unit discussions.”

districts reported no attempts at all to determine program

effectiveness.

One of the more important aspects of the interviews

affecting future programs and their evaluation was establishing what type of acceptance there was for evaluation.

Two questions (table 4) gave us an indication of, (1)

respondents’ opinions on how well the Forest Service is

evaluating prevention programs, and (2) what the future

needs are for evaluation prevention programs in the

agency. A sizable minority (43%) opined that the Forest

Service was doing a good job of evaluation. Ten respondents recognized a future need for specific program evalTable 1 -Region 8 fire prevention program distribution

Administrative unit

National Forests in Alabama

Ouachita National Forest (Arkansas and Oklahoma)

Ozark-St. Francis National Forest (Arkansas)

National Forests in Florida

Chattahoochee and Oconee National Forests (Georgia)

Kisatchie National Forest (Louisiana)

National Forests in North Carolina

Francis Marion and Sumter National Forests (South Carolina)

Cherokee National Forests (Tennessee)

National Forests in Texas

National Forests in Mississippi

Daniel Boone National Forest (Kentucky)

George Washington National Forest (Virginia)

Jefferson National Forest (Virginia)

Total

Number of Number of

ranger districts programs

7 2 2

1 2 4 6

7 2 1

5 2 4

8 2 9

6 2 1

1 0 3 2

7 2 5

6 2 4

8 2 5

1 0 3 5

7 2 2

6 4 2

6 2 8

105 396

Table 2-Subjective evaluation of fire prevention programs’

Program type Is the program effective?

Yes N o Don’t know

Number % Number % Number %

Mass media programs

Television 6 1.5 0 0.0 2 0.5

Radio 38 9.6 0 0.0 4 1.0

Signs 10 2.5 0 0.0 0 0.0

Publications 39 9.8 1 0.2 5 1.3

Posters 9 2.3 1 0.2 1 0.2

Midrange programs

Gimmicks and displays 28 7.1 0 0.0 2 0.5

School 82 programs 20.7 1 0.2 1 7 4.3

Organizing community groups 2 0.5 0 0.0 0 0.0

Demonstrations 5 1.3 0 0.0 1 0.2

Public information contactor 21 5.3 1 0.2 4 1.0

Open forums, workshops, etc. 5 1.3 0 0.0 2 0.5

Personal contact programs

Agency personnel 90 22.7 0 0.0 8 2.0

Local voluntary 3 0.8 0 0.0 0 0.0

Created opinion leader 0 0.0 0 0.0 0 0.0

Law enforcement programs 8 2.0 0 0.0 0 0.0

Totals 346 87.4 4 1.0 4 6 11.6

‘Distributions for individual National Forests are available on request from the author.

Totals

Number %

8 2.0

4 2 10.6

1 0 2.5

4 5 11.4

1 1 2.8

3 0 7.6

100 25.2

2 0.5

6 1.5

2 6 6.6

7 1.8

9 8 24.8

3 0.8

0 0.0

8 2.0

396 100.0

3
uation aimed at identifying the most effective types of

programs, 16 named general evaluation needs, while 61

people had no idea what might be needed in the future.

Only four voted for staying with the current state of

evaluation.

Table 3.-Confidence grades for programs ’

Program Score Grade

Mass media programs 8 8 B

Television 7 5 C

Radio 9 0 B

Signs 100 A

Publications 8 7 B

Posters 8 2 C

Midrange programs 8 4 C

Gimmicks and displays 9 3 B

School programs 8 2 C

Organizing community groups 100 A

Demonstrations 8 3 C

Public information contactor 8 1 C

Open forums, workshops, etc. 7 1 D

Personal contact programs 9 2 B

Agency personnel 9 2 B

Local voluntary 100 A

Created opinion leader N A N A

Law enforcement programs 100 A

‘Grade scale: 94-100 = A, 85-93 = B, 75-84 = C, 68-74 = D.

CONCLUSIONS AND l?ECOMM~NRATIONS

This study was a disappointment in its major realm of

inquiry; where we had expected to find several formal fire

prevention program evaluations, we instead found none

(except for the special case mentioned earlier). However,

null results in a study can sometimes have benefits. In

this vein, it appears that we have a fertile field in which to

establish evaluation. The main difficultly seemes to be a

“language” problem. Very few of our respondents had

any clear idea of what prevention program evaluation is

all about. Most of our respondents gave us the impression that they recognized the need for some type of

evaluation that would go beyond fire occurrence statis- .

tics.

Another bright spot was the region-wide inventory of

forest fire prevention programs. To our knowledge, nothing of this type and scope has ever been done before. We

now at least know the number and type of programs

currently in operation in Region 8.

What of the future of evaluation in prevention programming? The immediate need seems to be to more firmly

establish a climate for evaluation. Even though the language and skills of evaluation are lacking in most field

settings, the need for evaluation has been demonstrated.

The questioning attitude found in many respondents can

be seen as a good sign. Questions lead to the developTable 4.--Responses to questions about program evaluation

Number of

responses’

Question: How do you determine prevention program effectiveness?

Observing reductions in number of fires

Receiving feedback from target groups

Didn’t determine effectiveness

Fire prevention unit discussions

Rigorous evaluation methods

By considering weather factors and number of fires

Guesswork

Trust in prevention methods

No response

Question: In your opinion, is your agency presently doing a good job of prevention program evaluation?

Yes

N o

Don’t know ’

No response

Question: What do you think your agency’s future needs are for the evaluation of prevention programs?

Don’t know

Evaluation (unspecified)

More funding

Specific program evaluation that determines the most effective programs

Maintain the status quo

Financial evaluations

Training

Non-evaluation related responses

No response

‘Totals may exceed the number of ranger districts because of double responses.

8 3

5

5

2

1

1

1

1

1 0

4 5

4 9

9

2

6 1

1 6

1 1

1 0

4

1

1

2

1

4
ment of methods, which leads to answers. Fire occurrence alone is an inadequate indicator of prevention program value even though it is the most commonly used

indicator. The vast judgmental gap between prevention

activities and the benefits of those activities cannot be

bridged by intuition alone. Why must forest fire prevention be unscientific when the rest of our forestry practices

are governed by scientific principles? Evaluation is the

key to unlocking this understanding.

Where do we go from here? This research shows that

our ranger districts are unprepared (but not unwilling) to

do evaluation. Future research (case-study-type efforts)

is planned that will affect a few ranger districts and will

teach prevention personnel the principles of evaluation.

Admittedly, this would only make a small dent in the major

problem. Since forest fire prevention programs are basically an educational process, perhaps an example from

the mainstream of education will provide some direction.

Until about ten years ago, educational evaluation was a

sporadic enterprise. What brought rigorous, systematic

evaluation to the education community was an Act of

Congress requiring that all education programs be evaluated. Perhaps a mandate of this sort would establish the

benefits of program evaluation in fire prevention. It

appears that something of this sort must be done to get us

away from our seemingly sole and shaky reliance on fire

occurrence statistics.

REFERENCES

Wetherill, G. Richard.

1978. An inventory of forest fire prevention programs in

the South: a baseline for evaluation. In Edward L.

McLean (ed.) Rural Sociology in the South: 1978,

Section J. Clemson University, Clemson, S.C.

*U.S. GOVERNMENT PRINTING OFFICE: 1982-569-082138 5