~

LEVEi

~~~ 

Il ‘

7 TschnlcaIPap.r1316 AD 1 ~~~~~~~~ -

—

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

p

~ 

OUTCOME MEASURES FOR CAREER J

COUNSELING RESEARCH.

, 

liVer

1

i

~Z(, ‘/ 

~ 

7 / >

~ ~ 2’ / )

CAREER DEVELOPMENT & SOLDIER PRODUCTIVITY TECHNICAL AREA

w

1= DDC

U. S. Army

Research Institute for the Behavioral and Social Sciences

Approvid for public nh...; distribution unlimit•d. 

~ 

/ O

_ 

‘ItS 12 18 012 ~~~~~

— -

‘ 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
4 

I

-—__________

U. S. ARMY RESEARCH INSTITUTE

FOR THE BEHAVIORAL AND SOCIAL SCIENCES

A Field Operating A gency under the Jurisdi ction of the

Deputy Chief of Staff for Personnel

WILLI AM L. HAUSER

JOSEPH ZEIDNER Colonel , US Army

Technical Director (Designate) Commander

I NOTICES

DISTRIBUflQtl: Prin~~y distribution of this report h been mde by A RI. Plu s. sddr u correspondence concerning distribution of report s to: U. S. Army Rsievcf

~ Institut, for the Seltaviorul end Soclel Sciences, ATTN. PERI.P. 5001 Eiw nhoys.r Aven ue, Aleirendri., Vlrgini. 22333.

FINAL 0I~~OglTION: This report s

~iy be destroyed when it is no longer needed, Pie,, do not return It to

the U. S. Army Ress.rch Institute for the Belisvior.I end Soci.i Sciences.

~~~ 

The (indingi in this report Ice not to bs construed Is n of fid el Depertmsnt of the Army position. unless ce designeted by other suthorized documents.

___________________________ ______ ~~~~~~~~~ _____ .

~~~~~~~

. - - J~~~~~~~ t

~~~-~~_ __
I

Unclassified

SECURITY CLASSIFICATION OF Thus PAGE (Pt,.., bat. Sat.,. ~~

~~~~~~~~~~~~ ~~~~~~ READ INSTRUCTIONS REPORT DOCUMENTA I IUT~ I

~~~ U BEFORE COMPLETING FORM I. REPORT NUMBER 2. GOVT ACCESSION NO 3. RECIPIENT S CATALOG NUMBER

Technical Paper 316

4. TITLE (.,d SubUU.) 5. TYPE OF REPORT I PERIOD COVERED

OUTCOME MEASURES FOR CAREER COUNSELI3~~- RESEARCH -- P

S. PERFORMING ORG. REPORT NUMBER

7. AUTHOR(a) S. CON rRACT OR GRANT NUMSER(.)

Laurel W. Oliver --

I. PERFORMING ORGANIZATION NAME AND ADDRESS tO. PPOGRAM .EM~ NT.PROJEC ’. TASK

Army Research Institute for the Behw1’ioral and 2 762717A766 Social Sciences (PERI—IS) L..

— 5001 Eisenhower_Avenue ,

_Alexandria ,_VA__22333 __________________________

II. CONTROLLING OFFICE NAM E AND ADDRESS 12. REPORT OATS

Army Deputy Chief of Staff for Personnel ~ O~~~~~~ES Washington , DC 20310 48

14. MONITORING AGENCY NAME I AODRESS(U dfII.rret from ControllIng Office) IS. SECURITY CLASS. (of flit, report)

—— Unclassified

IS.. DEC%.ASSIFICATION/OOWHGRAOING SCHEDULE —~~

IS. DISTRISUTIOII STATEMEIIT (of liii Riport)

Approved for public release ; distribution unlimited.

17. DISTRIBUTION STATEMENT (of flu. ab.tr.ct ,t.r,d l

~ Block 20, If dIff.r.nf ho., Repo rt)

IS. SUPPLEMENTARY NOTES

I

IS. KEY WOROS (ContI.... rev.r•. .Id. If nec.a. y mid Identify by block nsmib.r)

Career counseling

F Outcome measurement

Outcome research

20. AeI~ RACT (Contbwe re,.r. .1* If e.ce.. y mid Identify by bloob nu ber)

f\J Thig paper reviews the topic of career counseling outcome measures. The

purpose of the review is to provide background for the construction of instruments used in evaluating a computer—based career information and planning

program for Army officers. After a discussion of general considerations concerning the selection of dependent variables for outcome research , the most

frequently used measures are suninarized. Some of the salient issues that have

I . 

emerged from the career counseling outcome research also are discussed, and

specific recounendationa are made for career counseling outcome researchers .u4

Do 

~~~~ 

1473 ~~~~~~~~~~~ NOV 55 ISO S0I,tTS Unclassified

SECURITY C~ *

ImPlCATvO.. OF THIS PAGE ~~~~~~ oi~7

~~~a.u.

~~~~~~~~~~~~~~~~~~~~~~~~~ - .

~~~-~~~~ -~~- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

-
Technical Paper 316

OUTCOME MEASURES FOR CAREER

COUNSELING RESEARCH

Laurel W. Oliver

Bertha H. Cory, Work Unit Leader

CAREER DEVELOPMENT & SOLDIER PRODUCTIVITY TECHNICAL AREA

Submitted as complete and Approved By: technically accurate, by:

Cecil D. Johnson E. Ralph Ousek, Director

Technical Area Chief INDIVIDUAL TRAINING AND PERFORMANCE

RESEARCH LABORATORY

Joseph Zeidner

TECHNICAL DIRECTOR (DESIGNATE)

U.S. ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AND SOCIAL SCIENCES

5001 Eisenhower Avenue, AI.x.ndrla, Virginia 22333

Office, Deputy Chief of Staff for Personnel

Department of the Army

August 1978

Army Prolect Number Career Progression

20782717A7% Systems

Apprmsd for public rsisss.: dlettibutlon unIIn~tsd.

- — -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~- 

—

~~~~~~~~~~~~~~~~~~~ - 

1
ARI Research Reports and Technical Papers are intended for sponsors of

R&D tasks and other research and military agencies. Any findings ready for

implementation at the time of publication are presented in the latter part of

the Brief. Upon completion of a major phase of the task, formal recommen- dations for official action normally are conveyed to appropriate military

agencies by briefing or Disposition Form.

_______ _____________________ ______ 

‘

1 JL
FOREWORD

An important part of the research of the Career Development and Sol—

dier Pro&ctivity Technical Area of the Army Research Institute for the

Behavioral and Social Sciences (ARI ) supports effective career management

for Army officer personnel. As part of this concern for effective career

management, ARI is conducting research on a computer—based career information and planning system for Army officers. This report reviews the

types of outcome measures previously used in career counseling research.

The immediate purpose of the review is to provide background for the construction of instruments to evaluate a computer—based career information

and planning system for Army officers. Other related reports include API

Research Memorandum 77—13 , which provides an overview of career development theory, and Research Memorandum 77— 14 , which describes a field tryout of the computer—based career information and planning system. Continuing investigations center on coat—benefit analyses of the system. This

research is conducted un der Ar my Project 2Q7627 17A766 , Manpower Systems

Management (FY 78) ; Task C, Career Progression Systems, in basic support

of the Offi ce of Deputy Chief of Staff for Personnel. Dr. John 0. Crites

and Dr. Clara Hill provided comments on this paper.

JO H

nical Director (Designate) i 

____ — 

—____ p

NTIS v.

OOC

u A ~

’Y, r .r:

~ D

JUST$ - —

~~~ N~AUE

Diet. MAft. an~/or SPECIAL

U..— 

-

______ _ ___ 

________________

r

~~~~~~_-..-—— ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

~~ ~~~~~~~~~~~~~~~~~~~~~ 

-=..-~~, :ic~~~~~~
OUTC~ 4E MEASURES FOR CAREER COUNSELING RESEA~~ H

BRIEF

Requirement:

To provide background for constructing outcome measures (that is,

measures of the effect of counseling) to evaluate a career information

and planning system for Army officers.

Procedure:

Previously used career counseling outcome measures (1950—76) were

classified and reviewed , and issues emerging from the review were discussed. A set of recommendations for future career counseling outcome

research addresses these issues.

Findings:

Career counseling researchers have demonstrated a tendency to “re—

invent the wheel” instead of building on each other’s work. A critique

of previous career counseling research revealed the following: (1) use

of inappropriate criteria: (2) preponderance of self—report measures, as

contrasted to the use of measurement approaches such as coat benefit

analysis or behavioral obaervationj (3) relative infrequency with which

reliability and validity data are reported for the instruments used; and

(4) methodological shortcomings concerning randomization of subjects, selection of appropriate comparison groups, and choice of type of statistical analysis. Although much of the career counseling outcome research

has been characterized by one or more of these inadequacies, there are

many examples of good outcome measurement in the literature. In addition,

most published research demonstrates some useful procedure or conceptualization which would be helpful to career counseling researchers.

Utilization of Findings:

This report can be used by career counseling researchers as a re—

source in selecting outcome measures. The specific application is for

constructing instruments for evaluating a computer—based career informa—

tion and planning system for Army officers. The report , however , is expected to have broader application to other types of career counseling

approaches, and the findings presented here are not intended to be lim—

ited to the Army off icer system.
t

OUTCCME MEASURES FOR CAREER COUNSELING RESEARCH

CONTENTS

Page

INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

GENERAL CONSIDERATIONS . . . . . . . . . . . . . . . . . . . . . . 2

SUMMARY OF PREVIOUSLY USED MEASURES . . . . . . . . . . . . . . . 3

Career Decisionmaking Measures . ....... 5

Effective Role Functioning Measures . . . . . . . . . . . . . . 13

Ratings of Counseling Satisfaction and Effectiveness . . . . . 19

Miscellaneous Measures .................... 24

ISSUES IN CAREER COUNSELING OUTCOME MEASUR~~IENT . . . . . . . . . 27

Criteria of Career Counseling Outcome . . . . . . . . . . . . . 27 Instruments . . . . . . . . . . . . . . . . . . . . . . . . . . 29

Design and Analysis . . . . . . . . . . . . . . . . . . . . . . 32

IMPLICATIONS FOR FUTURE RESEARCH . . . . . . . . . . . . . . . . . 35

Recommendations for Criteria . . . . . . . . . . . . . . . . . 35

Recommendations Relating to Inst~uments . . . . . . . . . . . . 36

Recommendations Concerning Design and Analysis . . . . . . . . 37

SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

DISTRIBUTION 47

LIST OF TABLES

Table 1. Self-knowledge as dependent variable 6

2. Appropriateness as dependent variable 8

3. Instr umental behavior as dependent vari able 11

4. Attitude s toward career choice as dependent

variables 14

5. Perfo rmance measures as dependent variables 16

— ~~

, 

~~~~~~~~ ~~~~~~~~~~~~ 

- --- ~~ 

‘

~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~ 

~~~~~~~~~~~~~~~~~~~~~
Page

Table 6. Adjustment measures as dependent variables . . . . . . . 17

7. Ratings of counseling satisfaction and perceived

effectiveness as dependent variables . . . . . . . . . . 20

8. Miscellaneous measures as dependent variables . . . . . 25

(
I

I

OUTCOME MEASURES FOR CAREER COUNSELING RESEARCH

INTRODUCTION

Placing people in the right jobs has been of increasing concern in

both the civilian and military worlds. As stated in DA Pamphlet 600—3,

Officer Professional Development and Utilization, objectives of the Of f i—

cer Personnel Management Systems are to:

1. Develop officers in the right numbers and with the right skills

to satisfy Army requirements, taking maximum advantage of inherent abilities, aptitudes, and interests of the individual officer;

2. Assign officers according to the Army’s neede and the individual’s

competence and desires; and

3. Improve the motivation and professional satisfaction of the of f i-~ cer corps.

One way of implementing these objectives is to study the officer career progression system and design interventions that enhance the career

development of Army officers. Such research has been undertaken by the

Career Progression Systems Work Unit of the Career Development and Soldier

Productivity Technical Area at the U.S. Army Research Institute for the

Behavioral and Social Sciences CARl). Now underway is development of a

computer—aided career planning program to teach officers ca1eer decision—

making skills and also provide relevant career inf’,rmation.

In evaluating any career development program such as described above ,

decisions must be made concerning which of a multitude of dependent vanables are to be used. The decision must be made on which variables are

best suited for evaluating that particular program and population. Therefore, the purpose of this paper is to provide a review of the types of

measures previously used to evaluate career counseling outcome research

in order to provide background for a specific application: constructing

instruments for evaluating a computer—based career information and planning system for Army officers. However, these issues are considered in

the br oader context of the working world, and the findings presented here

are not intended to be limited to the Army officer system.

The discussion begins with some general considerations bearing on se—

lecting dependent variables for outcome research, and a summary of the

most frequently used measures follows this discussion. Next, some issues

1The author wishes to express her appreciation to Dr. John 0. Crites and

to Dr. Clara Hill for their helpful comments on this paper.

1
V

that have emerged from the career counseling outcome research are described. Finally, recommendations for resolving some of the problems of

outcome measurement are considered.

GENERAL CONSIDERAT IONS

Several general factors need to be considered in deciding which of

many dependent variables are to be selected for career counseling outcome

research. One such factor involves the type of behavior to be assessed.

What are the behaviors? Are they short—range or long—range in nature?

Must we rely on self—report , or are unobtrusive measures practical? In

addition, where are these behaviors to be observed? Will this observation be conducted in the field, in the laboratory, or through a mail survey? The location may impose limits in terms of the available sample,

whether randomization is possible, etc. Where the research will take

place may also influence consideration of when it can occur. Timing may

control the number and type of researchers and instruments to be used, as

well as the possibility of followup studies. How these behaviors are to

be observed is also an important consideration, which relates to decisions

about the behaviors selected and where and when the observations are to

be made. What is the availability of current instruments (human observers

as well as paper-and-pencil instruments)? Do instruments of proven reliability and validity exist, or must they be constructed? Is the cooperation of potential observers or sources of information assured? To what

extent are appropriate records obtainable?

Some of the specific factors involved in outcome measures have been

mentioned in general terms of what, where, when, and how. It is the “why,”

however, that often determines what, where, when, and how the research is

to be carried out. Objectives of a field test, for example, may differ

from those of program evaluation. As Helliwell and Jones (1975) have

pointed out, field tests are like “dress rehearsals”; that is, the main

purpose of a field tryout is to “debug” the system. From their experience

with evaluating a model for improving guidance programs, Helliwell and

Jones (1975) made the following suggestions for field tests:

1. To investigate interim rather than terminal behaviors,

2. To use fewer and shorter evaluation instruments,

3. To keep very careful records (including anecdotal ones) concerning the difficulties encountered in conducting the test, and

4. To obtain stron g assurance of cooperation and real involvement

from people at the test site location.

However, if the purpose of the test is not merely to debug a system

but to eva luate the efficacy of the system or to compare it with another

system or systems, the approach may be somewhat different. In a critiq ue

of process and outcome research in counseling, ~.lley, Smits, Leventhal,

2 j - - - . 

- 

~~~c~

--- -.-- - . --_ --—i~

_.. ~~~~~~~~~~~~ —-

~~

-

~~

-

~~~~~~~~~~~~~~~~~~~
and Rhodes ( 1970) concluded that the two major shortcomings of the 73 studies they reviewed were the fai lure to include control groups in the experimental design and the lack of randomization. In terms of the Campbell

and Stanley ( 1966) classification system, most of the research reviewed by

Keiley et al. involved one—shot case studies, one group pretest—posttest,

or static—group comparisons——none of which constitutes a true experimental

design.

Hilton ’s (1974) criteria for adequate evaluation of career education

and guidance outcomes also emphasized random assignment to treatments.

In addition, Hilton suggested pretest—poettest measurement, a representative sample, a sample large enough to obtain statistically reliable results, and an “objectively scored measure of relevant characteristics of

known reliability and validity” (Hilton, 1974). The issue of reliability

and validity of the instruments is an important one in career counseling

outcome research, and the topic will be discussed in some detail later in

this paper.

In this discussion of factors to be considered in selecting career

counseling outcome measures, it has been suggested that the purpose of

the research is of primary importance. The objectives to be accomplished

will help the researcher determine which dependent variables are to be

used. The references mentioned above are good background sources for investigators faced with the problem of selecting career counseling outcome

variables.

In addition , Bingham ’s ( 1974) chapter on assessing counseling outcome

may be helpful. Although the Bingham paper is primarily directed toward

counselors concerned with career guidance and career education in schools,

portions of his discussion are pertinent to any developmental career planning program. Another valuable (and more technical) source for the researcher interested in evaluation is the two-volume Handbook of Evaluation

Research. The first volume (Struening & Guttentag , 1975) emphasizes strategies and methods of evaluation, whereas the second volume (Guttentag &

Struening, 1975) focuses on evaluation in specific content areas.

SUMMARY OF PREVIOUSLY USED MEASURES

In a revi ew of outcome research in educational and vocational counseling, Myers (1971) has classified dependent variables for educational— -

vocational decisionmaking and effective role functioning. For educational

vocational decisionmaking, dependent measures fell into three categories:

accuracy of self—knowledge, appropriateness of career preference or choice,

and instrumental behaviors. Variables related to effective role functioning were classified according to whether they were criteria of adjustment

or perf ormance.

Adapting the Myers classification system somewhat, this report discus—

ses dependent variables for career counseling outcome research in terum of

the following categories:

3

_j * -

. _i -:;~~~~~~~~~~~~~~~~ - ~~~~~~~~~~~~~~~ 

~~~~~~~~~~~~~~~~~~~ -
I

t

1. Career decisionmaking measures

a. Accuracy of self—knowledge

b. Appropriateness of choice

c. Instrumental behaviors (career information seeking, securing

job )

d. Attitudes toward choice (certainty, satisfaction , commitment,

career salience )

2. Effective role functioning measures

a. Performance variables (academic performance, career—related

knowledge)

b. Adjustment variables (career maturity, self-concept changes)

3. Ratings of counseling satisfaction and effectiveness

4. Miscellaneous measures

To clarify the discussion that follows, a table is presented for

each type of dependent measure enumerated above. For each study using

that particular kind of measure, the table identifies the criterion or

specific variable involved and describes briefly how the assessment was

made. (The studies in Tables 1—8 cover the period 1950—76.)

Note that this paper is limited to career counseling outcome research. Research concerned only’ with educational counseling is not included. Also, the focus is on career counseling outcome variables, even

though process and outcome cannot always be easily differentiated. In

fact, Grummon ( 1972) has argued that process studies involving a series

of measurements during the course of counseling are also outcome studies.

Grummon would consider the trend line for an increase in self—accept~ i:ce,

for example, an outcome measpre. However, process variables such as

client-counselor rapport, or the specific topics covered during a series

of career counseling interviews, are not dealt with here. The discussion

focuses on variables that assess the impact of career counseling after

counseling has been completed (although, of course, pretest measures may

be made in order to determine the change that has occurred during the

period of counseling ) .

Also not included are studies that relate to career counseling outcome variables bit that dio not actually involve counseling. One such

type of research compri ses investigations of test score recall. Althou gh

these studies have typically used outcome measures, they are not include d

unless the test interpretation was pa rt of a career counseling treatment.

(T

— I ~~~~~~~~~~~~ - . ~~~~~~~~~~~~~~~~ 

- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~‘• - -
Lastly, the summary of dependent variables used in career counseling

outcome research is not intended to be an exhaustive compendium. Rather,

the purpose of the summary is to describe the kinds of measures typically

used in career counseling outcome research and to present some examples

of each type.

Career Decisionmaking Measures

Ca reer decisionmaking measures constitute the first major category

of career counseling outcome variables. Four types of dependent measures

are ipcluded : (a) accuracy of self—knowledge, (b) appropriateness of

choice, Cc) instrumental behaviors, and Cd) attitudes toward career

choice.

Accuracy of Self-Knowledge. Variables of this type assess the discrepancy between client self—estimates of a characteristic and a criter—

ion, with the criterion consisting of a test score or a rating by judges.

The usua l self—estimate proce dure involves (a) having subjects estimate

their standing on the dimension of interest before any treatment (counseling) takes place, (b) obtaining similar estimates after counseling,

and Cc ) determining the effect of the treatment by analyzing the pretest—

posttest discrepancy (between self—estimates and criterion ) and/or differences between experimental and control groups. Table I summarizes the

studies that have included self—knowledge as a dependent measure.

Many career counseling outcome studies have used self—knowledge variables. Typical of such outcome research is an early study by Johnson

(1953). In this investigation, male subjects were administered tests of

scholastic aptitude (Otis), interests (Kuder), and personality (Bernreuter).

Prior to seeing a counselor for an initial interview, the subject re—

sponded to a questionnaire on which he estimated in which fifth of the

population he stood with respect to intelligence, interests, and personality. Immediately after treatment (vocational counseling), the subject

again completed the questionnaire containing the self—estimates. In a

1—month followup, the subjects filled out the questionnaire for a third

time . At the conclusion of the counseling, the subject’s “true” standing

on the various dependent variables was rated by his counselor, who used

the test results in determining the rating.

As can be seen in Table 1 , outcome studies using self—knowledge measures as depen dent variables have tended to focus on self-estimates of interests and aptitudes. Measures of achievement and personality have been

used less frequently. Most of the studies ask the subjects to compare

- 

- others with the smelves and estimate into which propo rtion (usually quin—

tile) of the comparison populatio n they would fall. A slightly different

procedure was used by Gutt ean and Haase (1972), who had subject s estimate

their three highest and three lowest ( Xudsr ) interest areas.

-. — -

~ - - - - ~~~~~~~~~~~~ - ~~ - -
-- 

I

L 4 J 4) 

.4 ~5 5-’ 4)

K K

o s

S

5 

V

4) I

.4 ‘ 

K K XX N

o .-i

V

44 ..4 ~ 

K I ~5 K N K K K XXK N K

-4 4) .

~~

< I ‘0

‘

0 41

V S

• S ~

-4— 14 -4 4) 

~~

s V S

—~ ~ ~~~~~~~ K N N N K XXX XXX

Z

~ ‘-4 Qs — 4) ‘.4 U

4) 4) I

S S S

~~

. a Dl Dl 5 5)

0 P., 4) V d J

~~~ • E ’ ~ ,

~~

• 5) V V S -4 — — .4

V 4) ~~~~~~~~~~~~~~~~~~~~ 0 0 0 94

5 0 1455 -45 “4) c ecs O s

U •O .

~~~~~~.44)~~~ U 4 ) S S 5) 5) 5) 4) 4J ’.4 4) V .-4 5.4

~l

1J

It) 511 5(U) I

544

-4 c u cs scecs c c 55 • c y es asses • e 14 1414 )4 Is (4 Is 

~~~~~~~ ~c c

4) 4) 1J 1J 4J 4 ) 4)4) 4)4) .4 c ccc cecec c-v

~~~~ ,

~ ~~ g

a 

IS 0 It) — UI

Os m S n” 05 - ~~~~a N ~~~Na N 

_ _ sO 0) ~~ _ 05N 05 OSN

~~~ifl ~~ Q

VI .- 5) Cs ~~ Is

~~~~ 

~ - 

~ 5-4 c — .5 54., 0) ‘I ~ a 5~

) eli —

~ ~ 2 ’ Os ~ 5) ~ *~~~i

Is Os ’- ~ ~~~~~~~~~~~~~ t

~*

! ~~~~~~~~~~~~

~~~I5Ø 4$~~~~~~~~ 5$ s5 _ — o is — 4 8 ~~~~~~~~~~~~~~~~~~~~~ 

~ 

cj is

-

“ I I a

6

— -.~ - - ----- ~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

• • I ~~~~~~~~~~~

—

~~~

-

~~~~~~~ . 

~~~~~~~~~~~~~~~ 

-

~

S~ -
With few exceptions, the studies given in Table 1 used test scores

as criteria. Three studies used judges’ ratings as criteria. Even when

judges’ ratings were used, however, judges had had test data available

to them. Johnson ( 1953), for example, explained that his counselors’ judgments were “guided but not determined solely by the test results” (p. 333). Overwhelmingly, then, career counseling outcome research has

depended on test scores, directly or indirectly, as criteria for client

self—estimates.

In the Kamm and Wrenn (1950) study , a variant of the self—knowledge

criterion was used. Using summaries of pre-interview and post-interview

data , three judges rated client “acceptance” of test information.

Appropriateness of Career Choice. “Appropriateness” of career

choice, sometimes termed “realism” or “wisdom,” is another commonly used

measure of career counseling outcome. Such variables are based on assump—

tions that individuals are better suited for some careers than others to

a measurable degree of “appropriateness.” The usual procedure is: (a) to

assess the client ’s characteristics that are vocationally relevant, (b) to

determine the requirements of the chosen career in ternm of these characteristics, and Cc) to estimate the degree of congruence between client

characteristics and career requirements. Table 2 summarizes some of the

research that has dealt with appropriateness as a dependent variable.

The most common method for determining whether a career is appropriate for an individual has been judges’ ratings. Judges may make their

ratings on the basis of case materials and have no contact with the subjects. Or the judges may be the counselors actually administering the

career counseling and, thus, use interview data as well as the usual test

information and questionnaire responses. Judges’ ratings have been of

various kinds. Some authors have used dichotomous ratings, such as real—

istic/unrealistic (Hewer, 1959, Hoyt, 1955; wright, 1963). Other research

has used scales that represented a degree of appropriateness from realistic to unrealistic (Apostal, 1960; Bivlofsky et al., 1953; Hewer, 1966;

Mencke & Cochran, 1974; Westbrook , 1967). A few researchers (Hanson &

Sander, 1973; Pilato & Myers, 1975) have constructed instruments ranging

from one extreme of inappropriateness to another, with the midpoint representing an optimal degree of realism .

A typical study using appropriateness of choice as a dependent variable is that of Hanson and Sander (1973). In this research, three experienced counselors judged the subjects’ “realism of vocational plans” on a

5—point scale that included the following categories: overshooting, mmd—

erately overshooting, realistic, moderately undershooting, and undershoot—

ing. Overshooting was defined as aspiring to an occupation that was beyond the person’s ability level, and undershooting involved the judgment

that the subject could probably succeed in a higher level occupation. Each

judge was provided with aptitude, interest, and achievement test scores

for the subjects as well as inf ormation on their grades, extracurricular

activities, work experience, health, and family background. Hanson and

18 12 18 012 - - : - - — ~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~ -
‘0 I ~ 1~i Dl V~~~~~ 111 4) 4) - I 14 4)

5

1 00, 5) SI 

• V5 51 .-i 

S

X544 5) .I 5)4) V M 0 >..4 4)~~~~

’0 V ‘05.5.5

‘-I 4)04 ) 5)05 ) 5

~~, 0.4 >1 5 )—. >5) 0 1 4 05 )4 ) 5) 4) ~~ 4) • .0 ~~~ O 0 0 UI 5) 5)0 05) .4

4 p.4 .4 ~~~~~~~~~~ .4 ’D c-V .4 4).0 ~~~~~~~~~ .~~~Dl

~~~~

V a .4

Sr4 0, ‘0 

4 15) 0 0 141 4 VS 

• 1’ 4) • 5 5 5) 5 4)

5)0 0.4 5).4 VW 5)h

1~ 

‘

0.0

Sc- UI .5 .4)4 

4) ’0 ’0 

MAVS O i-S ri ,-4

~~~

,5

544 0 U

0 1 455 ) ‘04) 

0~~~~.-4 5)~

r’ 4)• - 4)0

~~~~-

.

‘-is o w 4 ) 054 ) -I 0.4

5) 

5 .-f ‘-4 ‘D Os 

00 ”” > 14)U 5 . S 0 5 • U ‘.1 14 Os V ’ 44DlDl 145 ) - # 4) 1 4 0 5 ) 4 ) 5)5) USO V 4 ) 51 05) 14050 O A IJ ’.I ~~~0.0 0 Is V4)0,’0 5)4 0 5 V.4 ..I .4 4)4)04) 14 0 5 i-i 5 ) 01(4 V— .4 4) 4)4) 5 V 0 Dl

.4 0 , 5 4 )A 0 

0 ..l5U

-4 ‘.5 -4 .5.~

~

.5 .40 4) 

0

2

~

>

~ ~~~ 0 

..4 >,1-1544 ’14 

S

~~~

0 4 )

~~~

:~ I s O” OW ‘0405 -I i-45’U.-S Is 5) i-s o .C~~~ 5)5) 1414 0.4 0 . 5 SII V O ‘0 0 0 ‘U SW 0.0 4) —S O

V 504)— I IS 

U0

,~~~

5)iI r4 

0 5 ) .4

~~~

0

~ OSli li 

a 5) 0 

5

45 W IJ .4 (4 V ~~~ 

a 14 > 5 5 4 ) V 4) 0’0 °a UI O,’UVV .00 8 S V 4 J (i .5 5) 5) 1.4 4) (iS VS 0 UI ‘r, o 5)5) VVV O V

c-rn .0 4) 5 5

0 (4 ~~~~~~~~~~~~~ 

- - , ~~‘ 00I 0 0 0 0 5) WV “SO 51 —I ‘U 5)5)~~~ 4)0)45) V ~~~~94 ~~~~~~~~~ ‘ 0 0 -4 V 5)4~~4 )0 “ — 0 5.4 ,5 5 ) 4) ~~~ 4 5. 45 5 00 (40 A 0 4)O~~~~ -4 0 (45.4 ~~~~~~~ 455 ) 15 > ’.4 5’.4 I s V V .5 S 4’ 0 94 14 04) -I 5) ’.I V.40 ” V4) 0 >.4 >10.4 - 4)

54 - 4 5 ) - 0 Dlii 5) 04 Do I .4 14~

~4 .4 .4 .4 04V~~~ ~~~~

‘.4 i-I 00, 140.

- S 54 (4 4)04 0 5)044 04) 4) 0.-I

5) 04) 5 4 11 .X V 00 55)’ f l 4) 

D ,-I

blA S 145 )1> 445) 0 04 0 0 )4 4 ) 11 .1 5 5)5 0 0 ‘0.-S O, ~~~~~~~ ~~~~~~~~~~ “V..I ’rs 4) i-S O - 4 0

5 4) U 0 r ~

4 5)145 14 .i

4 .,4 Dl 0 

V o v ’005 55 4 ) 5) 55-4 5) 4 ) 5145 ) a ~~~~~~~~~ ‘

0 945) 0 V 5) V l s S 0 Ha 

~~~~ ~~~

>1 ~~~~~~~~~~ 

‘~~~

% ( 4 ~~~~~~ c - 8~~~

2

~~ 0 0 4) i

~ .-I 0—5 5) 04)

M a •~~~~ U V 5) V .4 4)51 0 (4)4” 45.4 5 544~~~ V 4 ) 5 4) ..4 Sc- i-S 0 5 (4 Dl 5 4) 5) ’ . ~~4J 0 “ Ps ,-5 5 544

V 5)0 “ ISis VV5) VS 110

0 0 5)0 4) 

5

— 

V

~~~~

S.4 0, DlO V 4)~~~ 5)04

) —I ~~~~~~~~~~~~~ V O V4J

D’

0 1 4 0 ‘.4 (4 .4 5) 4) V —I V V — w ‘0.4.40414 ‘0 s ‘ 0 0 i - 4 V’0

Vs s

~~~~ Vo 

~~“io ~~~~~~~~~~~~~~~~~~ 5) 5)4 4)0 0, V .0 014~~~ ,.4 5)5 ~~~— U1i Q, ~~~~~~ S V 0.5 4 )

0

SV

S 

a

V S 

a

4) 4) 

0

5) 5 4) .4 .5

ii

— 

0 O — 

0

.0 0 • 

0

.0 .4

.0

Is 004 00 5 4 ) ~ 

a

04 .50 .,4 a .pI

a

~i14 

~~~

.4 4 ) 5 

4 )0 

0

b 

—

‘~~ -4 

~—5 I—I

1 4 0 8

4) Sc ~~~~~~~~~~~~~ ‘~o

~~ 

S

~ 

V ~~~~~~~~ ~~~~~~~ 14 H H UU

VI rn 0) N 1

~~ — 0) rn

‘0 

— • 0)

.

~ 

Ii

o 

4) 

I

N

• In ‘p

(5 0) 0)

• P.4 S — 

—. -• 

5) 

5 Is Ii

(

S

8

-~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ iI

~

iI ~~~~~~ 

h
-4

‘0 55) ‘i I

0 05 0 1 I ~~~.4 .4.5 5) 5

Dl .9405 05 .4 V V~~) UI 4) — is 5)

5)OV ~~~ 0 5 ) 00 I Os ‘44 IiUV 4) 

~~~5 D ~~~~~~~~ 

‘

~~ ~~~ 0 55 ‘041 V 8 0 0 4)0 .5 ~~~o ‘0)4 ,5(4.4 5) 4)0 0 4) ~~~~~~..4 .-I ,

~ V U 4 ) 5

5 i-s 5)5) ‘

0.0 ~~~~ ~~~~~~~ VO 0

U~~~~ >1 I i A’0 0 0 0 0r ’4 5) i-l 50 -4.1 ‘14 50 “ 4 ) c OW 4) 5) >1 0 ) 4 0 “ 5 ) 

‘0 ’

~~~~ ~~~~~~~~~~ 

c-a ow 4)00 5 5

4) V Ui”s V U ~~~~ V 4)0 04) — 544

Ii’0A 1414 (4 00 .05 ~~~ . 4’0 O U

~~ 4 ) U4J k 51.5 0 5)544 0

~~ .051 0 S V O ‘0>5 .-i U

5 ) 4) 0 ‘0 0> 1 V 5 0 0 .4~~~ c - A owl 14— 0 .-I , - 4c-’0 ..4 0.4 5) ‘0Dl 01 0 5)-I 0 .5 U V ~~~~~~~~~~~~

~~~~

.Za 

~~~~~~~ O U> 1i-’S -I 4) 5) V 0 . 0 4) - S O ‘54

DO ll 544) . 4k m 5 ) 0 5 )

~~~0 ’D U S 50) p 59) ‘0 ‘

~~ 0

~ a~ ~~~~~~~ ‘

~~~o~~~ ‘0 5) , - ~~~~~ p V V a ~~~~~~~~~~~~~

~~D lO .E ’8 1

8 14 0 UV WV MW 45 (11 ‘0 5” c-V-I 5)005 V S ‘ 0 VVO V 0 14 4)4) (4 .4) .5 r4 (f1 5)0 5 — 0 0 5 a 1 4

~ ~s 0 V 4) 0 V 5. 0 4) O~~~~ U 5) V IS ~~~5) 05 5) 1 4 5 V 0) V

I 

.4 I I VV 0 ~~~~~~~ ~~~ ‘0 >1 ~~ .4 V

0 .01 i-I 5.5 - S V Oi,

C 0 0 140 5)’>1 - ~~

a() 4)r1)

S r 4 V 4)Is ’ii 4 ) 5) 5) . - I V S V 4 ) O >’l .00 0 4 1(5 5)594 04) 00W Vii Os ~~~ • rs0005 94~~~~w 4)4)>,c- ‘14 4) . 4V 0 5 4) D 4 ) 0.4 5 ) 00 3

~~~~

I S Osr.4 5)0 0

~~~~~

4 o 5 ) V ..4 ’rs~~ ~

14

0 0 i-sow ~~~~~~~~ 04) 0 .4 1 4 0 0 09414 $4 5) 0

‘00 ‘I4 VVV ~ ~ V (

~~5 44 -4V-S V S ~~~~~~~~~~ 14.4” -’ 0 4) 4 ) V — ..l Ii 4) 0 5) 0 0 5 ‘0 8’ ‘

~ t~’~ 4)4) 0 0 V >1.~~~~~~~~

5)

4) 4)-S S VS 0 ’0055 00.-i 5 ) 1 - 4 4 ) 0 0 (‘4 04)5.4 4) 11.1 05.4 .4 .455 .5 “ .4 c- VS1 J 5 ‘0 —VU .1Ois k V4) 0 14 14 >, .-1 14 544 4)~~ • ‘0 ~~~“

~~ V S ! 5 0(4 Ii Os Qi 0 5) 0 ) 0 4 04) 5)5)00 ( 4 — S ..4 .-4 .l o 5)1 0 5) .-I00 .405 0 ‘.4 5.4 V O ‘0OOsO S 1 4 1,- I .IS V ’r s ~~~i 544I 045(414 

~~~~~~~~ 0 4 V’ . l ‘0~~~~~ 5)S U O —— V A0) 014 — 4 ) 0.5

~~~‘— U 1i 0s U5)O 0 ) 04) ~~~~~~~~ ~~~~~~~~~~~~~

“4

0 a 5) 5

4) -4 5) 4) -I S

8 0 ‘-S 5 0 5 .4 .4-. 5 0 5 4)5 5 V 4) 5) Oh S 0 14 15 ii ( 40 Is 0 0 0— 0 -

~~~~ 0 9 8 - • S S c-c45 1) 4)

I, ”-

‘

0 0 5 5 ‘00 0

• ID

• N N a Cl 0) in • N — — 5

(a Is

0 5) S — 0 “5

I

• . 1 

9

___________ - .

~

_

~~ .•:_ - ~~~~~~~~~~~

- -—- _ !.. • _ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -
J

Sander (1973) also reported interjudge reliabilities, a practice not al—

ways adhered to in career counseling outcome research.

Although judges’ ratings are the most frequently used means of

assessing appropriateness of career choice, some authors have developed

techniques that are objective in nature and avoid the problem of the reliability of such ratings. Zener and Schnuelle (1976) compared the Holland

summary codes of occupations their subjects were considering with the subjects’ codes obtained from the SDS or the VPI. Mencke and Cochran (1974)

used th~ Holland codes for the career choice bit compared them with the

codes for the subjects’ measured interests. Still another measure was

used by Westbrook (1967), who compared aptitude scores with the aspiration

level of the choice. Pilato and Myers (1975) quantified appropriateness

in terms of the client’s measured characteristics (aptitude and interests)

and Roe ’s (1957) classification of occupations by level and field. As did

Hanson and Sander (1973) in the study described above, Pilato and Myers

took into account the fact that an aptitude level higher than that required for a job may be just as inappropriate as an aptitude level that

is too low.

All but one of the studies reported in Table 2 used judges’ ratings

or test scores as the criterion. The single exception was a study by Hewer

( 1966), who used the current occupation of her subjects as a criterion.

Instrumental Behaviors. Almost all measures falling into the classification of instrumental behaviors involve career or educational information—

seeking behavior. As shown in Table 3, the most commonly used dependent

measures are frequency and variety of information seeking. The frequency

variable involves a count of the total number of contacts made by the client to obtain career information, and variety is the number of different

types of sources ( libraries, counselors, etc. ) the subject consulted. Self—

report procedures are typically used with the data obtained by use of a

questionnaire or a structured interview. An unobtrusive measure of inf or—

mation seeking was used by Cooper (1976), who recorded the number of subjects who returned to the counseling center to pick up career information

after career counseling had been completed.

Almost all of the measures of information—seeking behavior appeare d

to be identical to or derive d from those developed by Xrumboltz and his

associates. (Borman, 1972, and Samaan and Parker, 1973, were the only investigators who did not identify their questionnaires as being related to

those used by Krumboltz. ). Most of these procedures involved a structured -

interview such as that described in Krumboltz and Schroeder (1965). An

interviewer who did not know which treatment the person had received in—

tervi ewsd each subject a few weeks after counseling. The intervi ewer followed a detailed questionnaire constructed to elicit report. of 21 cate- - - I gon e. of information-seeking behavior, such as reading books ox magazine

articles about careers, requesting an occupational pamphlet, talking to

persons working in a particular occupation, watching W shows or listening

to radi o programs about occupations, etc. Kruznboltz and Schroeder (1965)

also made a validity check of the interview data by attempting to verify

10

~ 

~~~~~~~~~~~~~~~~~~~~~~~~ 
V

14 .11 Ii 0 094 0 0 0 5

0 0 

‘4

“ 

.

~~~~ 

p

.

04)

~~~~~ . 0” (~~ 5 ~~ i

C94 ) V 0 5) ‘- 41 4)

094 0 14

.4

0 ”05 ,-4 0 O . 4) Ii II Ii 4)

’-~4 

P

‘0 5)>iODO ‘-5 0 5)5)

• (4 0 . 4 5 9 4 0.4 1 4 4 ) O 0 >.4 5 50 0 • .-I aJ 5) ,-4 ..4 00 .-4 V

S . 0 W

050 0 fl2

~~ ~ 

a-

~

0 4) —4

55 .4

~ ~~~~~~~ 

.54

0 5) 0 0 .4 5) “ 

0

-I 0 -I 4) 4) 0 . .4 1 0 0 ~~~~~~~~ 

8..’ 0 .4 5 ) 4 ) Os 0

~~ 

~4

’ V ~ ~0

V 4)4) 11 U 

0

0 5)000 0 .00 ~~~~~~~~ — 5) 0’ ~~ .4 r4 45 U 0 4 ) 0 0 5 5) 5 00 5 0 0 45. - S 5 4) V 0 

0

~ ~~~~~~~~~~ sS 948 

~~~

5 5) 0 

-s -I 

—51.5 .4 ° 

M

.0 51 .0.4 0 14 >14 4) 4)

-I 

U.O IMUO V I45 V 5) 55 • 5 5 I 0)0 0 0 4) V 94 SH4) 0 4) 4) 4) >1 V 0 f55 0

‘-4 0) P’ 

I

~ & 

0

5)

.4

Is - ‘

~~ 0 1 .4

.4 0 

0

5) ‘-54) 

‘0

I V • Dl

.4 V

Dl Dl Dl 0~ Dl ‘0 o .4 a Dl -

~ 14 

~~~~~ 

~~~~Dl ~~~~b’ 5)

Is ~ 

•

~~ 

~~~~5) 

•-.4 

0 .-I V —I • Is m 0

O -I V 0 V

~~~, ~~~V 

V 5) 

V V V ~~~~ V 

U 5) 5 5 ~~ 5

~~~ 00 05 00 

5

II .5 0 

~~~

5) E-s - ~~~~ ~~~~ 

~~~ ° ~~ ~ .2

~ 8 

.~

i Ii 4)0 4)0 4)0 4) 

0 .4 5) 14 ..-I O ~~ •‘4 ! F ~ .0 0 .8 ~~~~~‘•

~ “40

~~~ ~~~ 

~2

I

V 0 5 .U 

.~ 8 .~ 2 

.

~~ .4 4) ‘l-s .4 .4

5) —5 

0

tI’l 

.~~~~

‘ 

0

94.5 l5-5 -.’i (54 .4 114 4)

0 94 0

,~ 

0

O 

~~‘ ‘i-s 04) ~~O ~~0 

>~ O 

~~, .4

0 4) >1 5 >1 >. 5> 1 8 0 4) 4) 4) 4) 4)0 4’ 4) 4)

0 V U

~~~V ~~~V 

~ 

g. 

~

~~~ 5 0’ S .rl “-4 Dl I-I Is V

U 5)

in —

V I

‘C

Cl 0) Ii 

m

a a’ ~~ 0) o ‘0 I’. N — 

a

5 S N — ‘0 5 0) 

0 S ‘- 4)

~~~~ 

- 0 5 

1 a a

(‘I 

‘

0 V

I 

.

~ 

N N N ‘14

,- ,-

S

i

~~

ijj ii I

11

- - 

~~i1~~~

”

~~

’ - ~~~~~~~~~~~~~~~~ 

-
I

‘0 

P

SS

55

5)

14

“-I

55

‘

0 -.-4 -.4 -.4 --4 

0

C V 4) 55 5) 

0 —5 4) 4) 4) 4) V 4) 0 0 0 0 0 C H H H H CI

I 

is

• -2 14

‘-I.0 5 5

- - S 5) .0> .4 5555 

•

-

Ii .5 .0 5) 55 Dl 0’ Dl 0’ >1.5 0 0 o 14 -.40’ -.-5 Dl .4 .40

’ -4) 0 > 1 .540 .540 .54 .540 o 54 4) 14 41.5 5.5 4) 51.4 m V 0 5)0 5.54 V .54 V V .54

‘0 

-.‘i ( 4 4) 55 

O V 0 S W V 0 05) 5 5) 5) .-I V 5) ,-4 h4 0 5 00 0 5 0 .0 0. .5 0.0 0 0 0 0 5) V S X .-4 .4 0 .-5 0 .4 ..-( 5 E-s 0 .0 S 4)0 4)0 4) . 00

55.5 0.5 55 5) -.s o S I sV 5 4) 5 4) 5 5 4 ) 5) 0 55 1 4 5) 155) Is 1 4 5 ) .4 5 15 0 5 0 0 0 0 14 (4 (45) 9454 94(4 94 94 0 4) 554 ) 0 0 0 0 0 0 0 —5 4) 014 .-5 (54 -.4 114 -P-I .-S ’l-5 -‘-5 5) 0 0.0 0 5) (

4 940 94—I 115 .5 0 ‘i-s ‘s-s -. .0 C) 0 0 0 -r~ • 0 0 5) ‘5-4 (5-5 554 94 0) > 10 > 1 0 > 1 0 0’ >, > 1 0

— 

~~~

>1 

~ >1 

~~~~~

> 1-~~ ~~ 5) 5 4 ) 54) V4) 0 51 04) 4) US O W 0 0 . 4 0 055 O Dl—I b -.4 0’.555 0’ 4) 014 Ohs 01s 4) 4) 4) 14

4)S0

H a — 0) (‘4 N ‘00) 

— 0) — a — N

‘0

— N 0 0) N ‘C F- ‘ 5 4) Os Cl — o ,-s ‘- ‘-

>1 4) 0 — — S - 

I

I

‘0 

~4 - ‘-I

I I I

H 

(

12

—

i

i- 

— - - - ~~~~~~~~~~

&iL— 

~~~ 

-

~ 

____
I

the subjects’ reports by checking library records, TV and radio program

guides, etc. Most investigators using structured interviews have attempted to verify the self—reports of random samples of subjects, and

they have concluded that such interviews are highly valid (Krumboltz &

Thoresen, 1964; Samaan & Parker, 1973; Thoresen & Hami lton, 1972; Thoresen

& lCrumboltz , 1967 ; Thoresen & Krwnboltz , 1968; Thoresen , Hosfor d, &

Krumboltz , 1970).

Zener and Schnuelle (1976) modified the Krumboltz and Schroeder

(1965) checklist to expand the range of information—seeking behaviors and

administered the resulting instrument as part of a followup questionnaire.

Cooper ( 1976) used the same instrument, and Krivatsy and Magoon ( 1976)

used an adaptation of the Zener and Schnuelle questionnaire. Jones and

Xruntholtz (1970) also used a questionnaire similar to the Krumboltz and

Schroeder (1965) interview schedule.

Only one of the studies in Table 3 (Thoresen , Xrumboltz , & Varenhorst,

1967) assessed an instrumental behavior other than career information seeking. In the course of individual interviews held 3 week s after a single

career counseling session, Thoresen et al. ( 1967) determined whether each

subject had obtained (or made definite plans to obtain) a summer or parttime job connected with an occupation being considered.

Attitudes Toward Choice. Several attitudes toward career and educational preferences and goals are included in this category. As shown in

Table 4, the most commonly assessed variable is certainty of career choice

(sometimes called “decidedness” or “commitment”). Two studies (Hewer,

1959; Zener & Schnuelle, 1976) also measured satisfaction with the career

choice. Barak, Carney, and Archibald (1975) used a rating technique that

is typical of the certainty and satisfaction measures. Two items on their

career assessment form had 5—point scales ranging from a score of “1” for

“completely undecided” (about a career or major) to a score of “5” representing “completely decided.” Also included in Table 4 is Cooper’s ( 1976)

ca reer salience variable. Career salience, which is the importance of career ( rather than the importance of a specific career) to the subject,

wa s assessed by Angrist’ s ( 1972) Lifestyle Index.

- Effective Role Functioning Measures

A second major category of career counseling outcome variables concerns what Myers ( 1971) has called “effective role functioning.” These

measures are further divided into “performance” variables and “adjustment”

variables.

Performance Variables. Dependent variables of this type are concerned

with academic performance or the assessment of some type of knowledge.

Most counseling research using academic performance as a criterion has been

educationa l counseling, and , thus, it is not covered in this paper. In his

review , Myers (1971) has noted that grade point average (GPA ) prove d to be

a disappointing outcome measure. Hill and Gnieneeks ( 1966), in a comparison

13

~ 4~~~~~~_:- . -

~~~~~L..- - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~ -

~~~-~-- - -
0

.5 ~ 4) 5) 4) 5 4) 0’ S -.4 -

~~ >1 5) Sn 

0 ‘4 Dl ‘0 Is V S V .4 Is 0 0 ‘0 5) —5 0

‘I-I 

S0

0 4) 0 

0 o .i -.4 0

—5 4) 

a

5) 0 

V

0’ - 4) S 5 0 .4 .0

14 

~~, 

0 .-I ~ C) (I-s 0 0 5 94

‘0 ~ 0 •• -.-s 0 

0

S V 4) VI) -P-s 5) a 545 ) 4) 

5a P5 4) 0 -.4 0 0 0 0

0 N -

~

4 V 5) 5) V V 5) 0) .0 0 55 114 0-4

-.4 0 05 0 0 5

)

Is a ow .4 

14 0 .4 _I - 4) 0 0 4) - 0 4) 5) 5

O ‘0 5) 

0 15 0 4) 4) .4 45 151 00 O S

I ~~ 

0 0.5

5 945 4 5 -PS

S ~ 00 -.4 0 

0 — 00 ~~~ .4.5 05) OW S C ) 0 5 i-S 4)~

~

-

94 VV 51 Os’ W V V V

—0 0 -4 

-.4 .4

S V 

‘

0 V’S 0 -4 .-s .-I 0 i-5 .-l ‘4 V V U

0 — .—I 5 

.-S S 5)5) 5) 5)5) 5) -

5) 55 55 H 5)5) 

0 >1 00 U 

0

~~~ 00 0 —I -.4

0

0 0 0 4) 01) 5 0 OS 0 5) 0.0 V V 5 5 4) 00 4 )5 0— 5 5 0 U .-4 -.4 5 -..l .5) 4) 4) -.4 4)4) 4) V .5 4’ >1 4’ a SS 5 4)51 00 0 0 5 1 0 4)

—I -.4 5 

-.I

~~~ 

I

~~~

—I 0 0 0 4) 0 -.-5 .4 -.4 

~~

,-5 .5-

.4 -.-I

• 

O V 00 ~~~~ _ ~~~~ ~~~~ _ 

4)5) 1 )

0 O C) 0 ,5

~ ‘ 

~ 5 ‘5-’ 

_

— - Ft _ . 

! • Sn ~

fl V $ —~~~ ‘- 0 

—~~~ 

‘- Os Os

V

V0

‘0 

5 Is 4) 0 0 0 0 

0

~ ‘0 4) 0 p-5 

-.4 -.4 -.4 .4 -.4 5 5 .4 0 0 0 0 0

0 S 0 ~ >1 

>1 > 1

t

~ ~~‘ ~~ ~~~~~~~ 

~~~~~~~ ‘I-’ >1 54) 4) 4) 0 4)

O 4) 4 )V54 ~~~~~~ 

-

~~ 

.~

~

114~~~~~~~~~9494 ‘4-s .4

1 4) -i-I ‘0 

4) 55 55 555 55 5 05 05 5 5)

~ —5 4) 4) 4) 4) -..5 

4) .4 4) -,4 4 ) .5 -i-I 4) 0 0 Is I-s I-s 144) Ii 4) (44 ) 1 4 4) 4) Is

.4 4’ 0 81 8 8 1) a

o — In ID N 0) N 0) Cl I- — a 5-

— a 5, -. In ~ N

• N Cl 

5

0) 5- a ‘4

,~

~‘ 4) — N 

V a 4) • Sn Cl U • 5) a — N 5- 5

• — 5, ,-5

,-4 N X Sn Cl — 0 0) — 

5

,

~~ 5) 0) N In 5-

• Cl Cl in 4) 

—

~ 4) — 5- 

‘- 

- 45 .-s — — I- 

-

~~ (I

-.4 0 54

~~~~~~~~

/k

14

4

-
of students seeking (and not seeking) educational and vocational counseling, found no differences between the two groups in GPA or rate of graduation. 

Most dependent variables of the performance type have been measures 

of some kind of career-related knowledge. As shown in Table 5, the career knowledge being evaluated may involve career information in terms of 

vocational and educational opportunities and/or requirements, ~nowledge 

of tests and testing, or knowledge of career development theory. Most 

researchers used a quiz of some sort to measure career knowledge. In 

addition to using a quiz, Thoresen and Hamilton (1972) prepared simulation te~t booklets that contained short descriptions of situations involving career exploration. After reading each vignette, subjects were asked 

to specify how they would explore and evaluate the job involved. Both 

Krivatsy and Magoon (1976) and Zener and Schnuelle (1976) asked subjects 

to recall their Holland codes. other measures of knowledge of Holland's 

theory, not clearly described, also were used in these two studies. 

Adjustment Variables. There are two major subdivisions of adjustment variables: measures of career maturity and measures related to 

self-concept. Studies using the two types of variables are reported 

in Table 6. 

Researchers who have used career maturity as a dependent variable 

have tended to excerpt items from instruments designed to measure this 

construct. Crites' Career Maturity Inventory (formerly the Vocational 

Development Inventory), Super's Career Development Inventory, and 

Cooley and Lownes' Readiness for Vocational Planning have been the 

instruments most frequently used. Some investigators have used the 

entire instrument (Myers, Lindeman, Thompson, & Patrick, 1975; Swails 

& Herr, 1976) or have attempted to check the adequacy of the adapted 

instrument (Mencke & Cochran, 1974; Perrone & Kyle, 1975). Smith and 

Evans (1973) used the Vocational Decision Checklist, which classified 

respondents into one of four stages of vocational development (exploration, crystallization, choice, or clarification). 

The other major category of adjustment variables includes measures that relate to the client's self-concept. One such measure, the 

"incorporation score," has been developed by Healy (1968). The incorporation construct is based on the premise that career choice i s an 

implementation of the self-concept (Super, 1957). Healy defines incorporation as the degree of similarity between a person's self-rating and 

his or her rating of an occupation on the same dimensions. Specifically, 

the subjects use 7-point semantic differential scales to rate both themselves and an occupation on (a) 25 occupationally relevant traits and 

(b) 22 bipolar personality traits. The "incorporation score" is obtained by summing the differences between the self-ratings and the 

occupational ratings for each of the two sets of traits, disr~garding 

the direction of the differences and subtracting the total from 100 

(score= 100- ~ lx I). Thus, higher incorporation scores represent 

15 
I

Dl

I 0 

‘

S ~~~~~~~~ I Sn U -.4 5 ,-I .- -.4 0 5) 0 V 0.5 0 0 Os

14 — - .40 0 ‘S (s-i 0 0

o ‘S Os 

0 0 

-‘-5 0 

5 (I-i 0 -.-I • 4-) 5) 4) 0

5) 

4) 0.0 41 (4 

0 Sc ‘-I 0 4 J .0 Sn’S Os >1 0 C ) .4U) VS 0 4~)

4) 5 

0 --I C) 0

-.5 i~

) Is ‘—5

,

~ 0 

0 - ‘-5 - 0 0 5) 0 .0

55 (5) ‘—5

0

,

~~ow 

04) 41 10 4) ,$ 4J Sn Dl 14 .4 O V 4)~~~ ‘S a Is Is 1)54 V 0 0 4 )0 0 41 4) Os — Dl It) 0 .-I ’S 4)55 >1 II Dl — V .4) 0

Is 0 4)XC ) S 0 00 .4 4) 00 4)41 >1 

-

~~ V .5 0 5.5 4) 4) 555) C) V 0 )IsOs -.-S (I-s 4) 4) 0 4) 4 ) 4) —I 0

45>1 00 0 

,~~

5) 

0

S15 .415.5 0 41

5) $ -.4 V 4) 

4 )54 5) 0 Is (5) OUr-S 0 0 V (5) 0 a 0 V S V Ps 0 0 15 (5) 4) -.-I V .p.5 0DlX 54 4) 4) 0 ‘SO C) Is .5 a 

(0 

0 5500 ~5 V U) 4) 0 0 (4 4) 41> 0 —54 )0 Sn .4 ‘05)1 51 0-44) 4)014 - a ~~ 4J 0~~~s 14 .50 5 00 ‘S U)

-

~ 

55>1 0 - 0 >’0 5 1 4 ) 5) 0 

~D

4) C) .0 051 14 54 0 0) 54 WV ,-4 C ) •Is 0 (1-s .-I ‘

5) 51 4 )54 U)0 0 .-S 94 I .5 

( s- sO W 

— 0 W OO (4~

s > 

0

V~~~~ 0 . W

~~~’ -.i

o

--I U) Sn

~~~ 

O~~ 

~~~V --5 4)15 

0 0 Dl 

--I

‘004) 

-

00 4) 

‘S C) 

1)51 0-’ Is 0 0 0 1... .-s o 0 — s .-s s

~~ O4)Ps 

-.4 0 0

0 4) 55 r-S

Qs 0 ODlOW 5)15 .0 0 0

a 0 5 )10 00 

~~~

.0 5.500 544) 4) C) 

V -.-5 .0 14>1 OC) OC ) >- -r~~

O .1-.-l 4) 4) 04) U) -.4 ~~ 0

$ rzs P4 & 0 0

— 

0

.5 I .4) 5) 0 5 0 (0 C-s V -‘-I V 0 

5 (Sl OW 0 .4) V 0 V O W ‘-4 -.4 14 .5 -.4

O Is --5 5 V 5)V ‘0 4)

5) 

04 )0 4) 0 

5 1) 5)0 V 55 _f

n

O~~~~ U V 00 0 1 U 0) ‘

S r-4

~~~)

V 4 ) 00 54 0 C) 0940 0 V 0 

.4) 5 ~~~C) 5 5) 0 0

V 0 S

~~~s 0 41 0 00 -4 

5 p-I 4) -p-5 5 -.4 15 -.4 ~~~5) 4 35 — .545

V 4) 0 4) 

0 > 0 0 41 0.-50-.-4 55 0 >1 5) .00 5.5 U) 940

0 -.4 00 4) 0 Is 5 

—I 04) 0 0 ‘44 14 0 .-I --I ‘0 ~I-5 0 1-5 (s-s O 55 ‘44 .0 54 5) 0 055 5) 0 4 ) 0 0 5 4 0 0 4 ) 0 SC) 0 -40.5 14 .0 94 0 41 0 -.445 00 Dl ~~l

4 

-

~ 

~~

t

Is V 45 51 V 0 0 4) ‘0055 ~~~14 V ‘.-5 4)

~ ~

-

~~

Dl ~ 

iI S) V .-s ’O Is 4) .—s ~~~

Q

55 ~~~~~~ ‘0 00 0,5

‘4-4 

94 -.-I 0

0 

Ct

~ 

If

H

~

P5 a N 

— a ID Cl ID N 5- N a ID Cl — — 0) o 0) In — If) — — N S Cl — Cl 5- .8

-4 -4—

I 1 a 

0V

U)

t

~ 

. -i-I ISl 

~

I Cl U

~ (Sl 5- 11)

— 

4) — (a

55 

(a 4) 0 S 

4)

g S 

( Is

— .1 Is =

/ 16

- 

~~~~ ______ _____________
I

Dl I ( S ) S I

V O 41 -.4 ‘014 I VI) ‘—5 5 055 I Os a 5)) 555 5 4 ,-5 a 4)

O H 0 54 -.4 0 4)

~ 4) (n Is Sn

— ~~ Os V

0 5-s 0Dl

4) -.5 >1 41—4 0 0 0 >1 .0 555 4)5)) V Is (0 00 C) 04) I-s 0 0 0 0 -.-5 0 5 ) 41 1-~ i-S O W -.-4 0 4) 0 1 4

0 0 PsI) 4)4) 45.0 4) 54

14 0 0 0 54 50 0 14.- > ~ ‘8 ~ Os S H~~~ ‘0 5-5 0’I a~~~ 00 

Q 4U ) a 40V

‘0 0 ~~~Is Ps 4) 0 0 (5) 41Q) 0 0 0 14 (fl --S -.4 4554

~~~0 4) ‘5-4 — 5 0

V V ‘4 0 .54 H 0 o 0 a - .54 0 ‘0

‘4 10 ~

4-1 DlCO 45 --S (5) 4) ‘ 0 1 4 4) 0 4) 554 (fl —S 0 .0 5 -.5 0

E 0 50 50 ‘-5 0 -.s 05)) .441 -.4 -4 0 0 o5) - .-I

4) > V I) 0-d .

~~~4) 4) Dl H a 0 14 V .4) Sd’S 5)0 Is 0 0

~~S l I U) -‘4 0 -‘-4 0 -40 0>1 0 .5 > ($5

54— 4) Os~~~ ~~~Is Os .

~~ • ‘ 0— 5 H — Dl 0 5) (s-s O 00 55 S o) Is - S o s e Dl

00 0 -

~I Is 0 (1-5 5 0 ~ -‘-I --- 0 0 0 4 1 0 0

~~~ — 0 (0- i--s --I I 4) -‘4 -.-5 .0 14 •‘-I 0 41 Id -‘-5 0 0 ‘S O 1 0> 1 4) 5 5 4) 545 tn -.-4 O’ U Sn 0 U)O 5)) 4) 14 54 4)0 00 -.44)55 U) -i-I •.4 — 54

‘-5 0 0 C) 055 --I 0544) C) 0 4) 0 0 1454 ~~~Is0 V 0 1400

~

. 0 >5 0 14 0 V 0 4141 - S

.-5 4)0 ,Q ‘5-4 0 ‘-5 0 0 >540 Is Is> g 4) > .4Os 0 4) — U’S

S’S (0 5)X - .-4 -‘-4 5) ‘S CI) H ~~~ Is VV (fl4~ 004) 4) 0 41 V ‘I-s o 55 4) 0 0>, I 4) 1 5 4 ) 0 I S (1) 4-s ~~~0 - . 5 00 Sc .5 05)94 .0 4)45 .-I — 4) 4)0-4 O (5-5 0 45 4)41 ‘00 54094 55 144) 4) V -55 V Sn> H 55 C) -.4 -.-5 H 0 04) 5 0

0 V 4-’~~~~~~ 8 a ~~ U) H H

V

i—S (5) 4) .5 55 Dl 55 45

5-’ 0) 4) V Sn

4~

)

Sn 0 5) 0

V a ‘5 -4 0 4)

I >5 >5 >5 >5 >1 ‘

~

-5

-p-S 4) 4) 4) 4) 4) ~~

‘ Dl

14 -.1 .5 .5 p4 .4 V .5 0 p.4

O ‘0 ~~~~~~~~~~~~

0 4) 4) 4) 4) 4) ‘-S 4) (5 4) -I,

45 0

‘U

Is Is 14 14 14 -‘-I Ii .5 Ii

V V V V 0 4) V CO V

0 V 0 V 4) (

4 0 ..-S V

Is 14 14 14 14 U Is U 14

‘ 

.

—a — 40 ‘0 a ‘0 N N ‘

4’ N 0) Qs N — 0) 5- 5- 05 U) — — — — — — — N Sn ID — — If) 0) N N

0 0 N — Cl 0) V

— — — Sn -4

Cl — 5- ‘

~ i—S

— V N V

~ 

— 0 14 CS 0

U) 5)

>1 Pa 4) Ia

.54 5 V V (a .5 (a 1) 4) V S 5 )

55 .54 5 0 .5 ‘-5 0 14 > 1) 14 4) -.4 _S S

~~ 5) 5) S

55 0) ~ ~ N

55 4)

17

_________________________________________ _____________________

- p

—~ 

. -, ~~~~~~~~~~~~~~~~~ - - ; .

- - - ,

~~~~~~~~~~• .

-

---- 

~~~~~~~~~~~~~~~~ ~~~~~~~ - -

~~
r

I 

I 

~

h D l

14 10 Is~ 

54s 5 0 0 54 --I ’S 

- ‘4(5 

0 O Dl -.-4 --1 ‘

S Os 0) 0 5 —5 

-i-S -.4

5 ‘0 i-4 i-4 5 5a 

.4 0 iS -.4 V i-S r-S (5~

5 (a (5-5 ‘4’ 

--

4 .-S 

0

V 4) Sn -.4

-i-s 

0 ’S 

0 In 

~~

.1

45 iS 0 

05 4 ‘5-5 0) 50

Is V VO V t-S Sn0

I 4) 00 150.- S IsV 550 -.4

0 ‘5-5 0 (U ’S 

0 04) 0(1) - ‘5 4) CO ‘—S O .0 ( 0 00 (0- .5 0 -‘-S V VU 410 0540 ) ID.4Dl 45 0 — V Sn- -S - 4545 4 j V 0 5 5 .4 5545 -i-I 0 4) — ‘5) 55 45 450 ’S (5) 55 .5

0 00 U) (0 - Sn (44 ) ‘00) 

— (0 

0 -.4 V --I 40 4515 H — 4) 014 41) • . VO ‘ Ia 54 ‘4-s 0 ~~~O — 54 0 544 ) 0 ) V 00 0 0

0 4)4) — 4)15 0--S 

14 114 15 VIs

0 4 ) 0 VO ‘S 1 

0 . 40 0 0 (0

.055 >50 

0 0 

0$ V 0 0

.-S -i-S O Dl-.4

Is ‘-I V 

- ‘ 4 Sn 

- . 4 4)0 

0 11)4) (0 0 0

054 i S Is 01 0 4) Sn 00 . 00 4 ) V 4)00 0940) 00 4)0 O 00 0 ) 0 5 0 045.54 0 0 ’ — 05) 5 ) 0

i-I 0 50 4)0 04)- i - S 0 00 1s4 .0 04) (5) 45 0 4 5 Sn 0540 ) 4 ) 0

54 IsO 540 4)~~~ 4 )

0 1 014 4) 100 4)—

S -i-I 41*5 0 0) 1-I (O .” 14 -il ls 015 055

15 5.4 > 

V

-i-S ODl 

O’ SO Dl’4 O -r-s Dl 0 -

4 55 (s-s w 454) -ri 5 -ri 54 ..5 S- Ps O ‘SO -.4 4) -.4 IS ‘ 0 0 ‘0 0 S c O — 0 U) S c 0 Sc 0 0 0

40 0V V V

-4 

0

.0 0 0 Ps 1)

E-~ 0 0 0 —I 4) (5) .4

(5 -4 0) 0 .4

Sn 4) 0 V V Is Is 

-

~ 1.5 — 0 0 0 .0 5) C) ‘5-S Sn 0 0 (5) 0

0 -i-S

0 Is 0 0 (0

55 0 0 4) 4

~

) 4) 4)

4

~

) 4) 

~

s

0 55 55 C) 1) C) C) 0 Is I-s 0 5 0 0

a 0 0 0 

0

0 Is

0 0 0 (5.5 94 ‘5-5 ‘4-5

In P5 50 ‘0 Cl 0) — 5- — N

Sn — 

—

— >1 — 5) P5 ‘

0 -i-I 

‘0 

— 0 — i—I Cl OS

55 — Pa 05

5- 4) 0) 0) — 

— 

I V (a 0 >1 >1 (0 -.4 -i-I

-4 -4 .4 -4 -4

55 55 i-I i-I i-S Os

18

— 

I 

-‘ 

—~
greater “incorporation” (greater similarity between self and occupa—

tion ) because the summed differences are subtracted from 100, a procedure that results in higher scores for lesser differences. Using the

same procedure, Healy, Bailey, and Anderson (1973) used range of

incorporation scores as a dependent variable.

Several studies by Williams and his colleagues (Williams , 1962 :

Hills & Williams, 1965; Williams & Hills, 1962) have used two self—

concept variables based on the Butler and Haigh ( 1954) Q—so rt: the

“self—adjustment score” and the “congruence score. ” Subjects were

asked to sort the 74 Butler and Haigh items into two piles—— ”less

like me ” and “more like me. ” Three sorts were ma de . The first sort

was according to the sub ject ’s current view of himself or herself ,

the second sort was in terms of the “ideal” person, and the last sort

was a description of the “ordinary” person or student. The self—adjustment score was the sum of the number of “good adjustment” items

placed in the “more like me ” category and the number of “poor adjustment” items placed in the “less like me ” direction. The congruence

score was obtained by counting the number of items sorted identically for each pair of concepts (Self—Ideal, Self—Ordinary, Ideal—

Ordinary ) and taking the mean of the three scores.

In Tipton’s (1969) research, subjects were asked to rate eight

concepts relating to their academic ability or to their chances of

achieving personal satisfaction in a given academic area. The

semantic differential technique was used, and analyses were conducted to compare the pretest—posttest differences of the concepts

for various groups of subjects. Another self—concept measure used

to assess counseling outcome was an adaptation of Holland and Baird’s

(1968) Interpersona l Competency Scale. Zener and Schnuelle ( 1976)

used a shortened version of this instrument, which asks the respondent for information or self—ratings concerning traits, life goals,

home background, interests, competencies, and high school

achievements.

Ratings of Counseling Satisfaction and Effectiveness

The counseling , outcome measures described in this section assess

counseling satisfaction and the perception of counseling effectiveness.

As will be apparent from the descriptions of some of the measures, researchers have not always clearly differentiated between the two variables. Measures of perceived effectiveness, for example, have been

interpreted as indicators of satisfaction. Also, a numbe r of different

terms have been used for satisfaction or effectiveness. The “enthusiasm” variable of Barahal, Brainmer, and Shostrom (1950) can be considered a

satisfaction measure, whereas usefulness or helpfulness of information,

e.g. , Babcock and Kaufman (1976) and Holmes (1964), pertains to effectiveness. Table 7, which summarizes outcome research using ratings of

satisfaction or ef fectiveness, shows that almost all the studies have

19

_____________________________________________ __________________ ______________________ _________- p

~ 

‘

~- ~~s-~~~~ - —
.0 VI V o0-.-s I C ) -.4 0 V 0 4)5)14 I 0—I > -‘S 4)4) 0 I

~~~~ 5)94 0

0 15.0 I 5 5) I a -i-I 

5 0 1400

V 5’ V 14 4) 0 -.4 >5 0 ‘4 V

0 45 14 -i-I Os ‘S 0 0 >14 0)45 ’S 4 5 * 0 .-S O 0 -i-I 0 00 4) 14 1 4 0 055

(0 0.4 i-S O 1 4 00 V -.500> WOO

094.0 0 4) -s-l W 14 55~~~~4)• 4) 4) 0 94 (0 ‘00 0 4 N Os 5 0 .4140 ~~~5 594IS 00) 0 5 ) Is 04)-i- S O 0 0 140 0 14(0 -.455>5 V 0 .4 4) ‘0 (5-s E

4)0 14 0’S 94.5 4) 

-i-S ‘0 V V

4-I 4IS 4) 4 ) 54 0 4 ) 0 0 1 ‘00)4) 0 > s 0 00 14.4 ’S H 0 0 )5)0 5 4 )5) V .0 0 .-S 0 .-S 0 4)0 00 45 004)

— 155 ) PS IS Os 4) 0 . 0 0’ S 

1 00) 0 4)0 0 0 ) 0 0) 0 14)0 0’S 0 0 94 i-I ‘S 1 a 0 ° 00 14 i-~ V Q~ ‘4 1 00 El 0 b’ 45 4 5 0

Is V 

‘ -40944 ) 045 9445> 

0 055’S 

0 0 ) .4 ,4 455 ) 4)~~~

0 .-lO IS 

S

-‘-5 0 ‘—I 0 0 OW C) —I V b ’ V 4) 0 4-5

000

(0 >0) 0W ‘ 0 0 (00 55 -’-I ls >5 .545.4- i -

S 4) 5) V D ’->0 V Q ~ ‘-5 >5 V 45550s l4 0’55-4-4

O 51 ..‘S

0 01)

140 -.4

0 .540 ~~ 5 141 45 .-I V 

.4.4 55 W In .,54i--4 ‘S 1400>

‘

~~ 

~~ 4 ) 4)554) > 45 

V .-S 55 0 O s 04) 1 550

5

0 0 05400 

0 0(0 1 4 05 1400— o r ’ s V I s o -.-s 0055 .415.4 45 V i4 

0~

~ ’0Q

-‘-5 0 -i-S 5) 

~ ‘ O s O 4)4) V ‘-I C) 4)~~~~ 4 5 4 ) 0 ( 0 (00 50.4 550 0 0 4 5’ - I 5 0 ’ - 0 145055 45150 5~~~~14 H VOIS

04 500 0 0.5 0 0 

1 4 5 00 04)0 1 4

O 45O.IOV 4)4

)

4) -.-445’.- 014 4 ) 0 4

) 0 45 -.4 5)

• 5) 

00. -SV0 -.4555 0 -i-I 54) 0 14 0V.-5

5494004) 140 —4 V - 0 0 04)’s . s ~~~ 45 000 0

0 00

.- P5045 

0 05) N 4)H 0 -.4 (0 V OsV ’—s sO -‘-I S El000

V ~~~OO VS . 4 0 0) 0 II ..S

V

~~~~ 

V14~~~~~O

(‘

- 00 

0 4 -S W 

0 00 

WO ) 0 0 — 1 4 0 0 5 1 45.0 -is 0 O r-s 45 .-S II ‘4 . 0 0 ’ S 0 4) 45 .-S . 4 O’ 45 b~~5)

‘4 00 05 0 C) 4)000 01455Sn V 5)4 ) 

0 0 0 > 4) 50.0 4) .4 0 .4 5) 0 >5 0 -,-S -i4 ,--I -4 15) 5 5545 .4 ~~~~~~~~~~~~~ .-I .05 0.0 94 5500 . -S 55 0 .-I 4) 0 0 ‘(-s -.4 Os V I s b’ -i-1 0 ~~ 0 V 5)’045 45

00 0’S 0010 010 fl

~~~

S ’ 0~~~

’ 94 4)0 5) 94 ~~~~~~~~~~~~ 514 QSVV .4 4 )11)0 > -.-S . - 4945VV 000

W 0.40 .4.4 0 4 )0 4)0 .0 4 ) 45 55 .-I Is4) 55 V O b

’ 00550.-I 0 5 5 14 IS000 0 ) 55 > V V55O 0>5-i-I S

‘0 0 0 Z C) 0 

~~~

1 5O) 51)

000) 

1

~~~~ 

M 540l4 ’4 01.0 45 -i-S

V

N 

>

-i-I

V V

14

V V V

0 00 45 55 1 140 0 

-

V 555 -‘-4 0 5 0 4 5 ~~ 0 0 ‘4 45 -i-S .4 — 45 ~~ 15 -~S~~~

94 — .4 V 

(

~~~~~ 0 .

~ -4— 5 

0

-i-S 0 

0 0 

0 0 0 0

45 

‘4 0 00 ‘4O -.4 0’S 

-‘4

( 14 V 94 55 45- -S 45 0 5 ~

J 55 1 .4 0 4-5 0 00 0

0 05) 5) 0 55

‘~

i 450 94 14 94 ‘-5 0 94 0 94 0 .5 00 (0 00 0

-.4 Os 04-5 

~~-i-S -4 -.4 (I-s --S 

--S -4 -4 4) 045 4) V V (

0 0) 0 ~~~~~~~~

0 - 

-

U

94 — a

o ‘0 

-I n 1- Cl

— 0 — In ‘0 

‘(0

‘4 0) - 0 50 -In • 45 5 -5- -

~ i-I 

.

- - 

- 

— 

_•1J 4 ) 

4) — _ 55 .54— 5) 50 ‘0 - U

N N 5) ~ (a V (a I - 0)

‘-4 - - -4 (a so 55 14 V

I I I

- .•

~~~~~~

20

-

p

— 

o

~ ~~~~~ ~~~~~~~~~~~~ ~~~~~~~~ ~ ~~~~~~~~~~~~ __ i

_

_

~~~~~~~~~~~~~~~~~~~~

’--

~~~___ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

•-=—

—- 

-
I .0 Os I O w 55 0 I ‘ S ’ 0 0 0 >5 —5 0

~~~ 00 15 ‘40) .0 W O OS-IS O 45’S 4) 0 -i-S 0 94r-4 .-S .4 0)15 0 0

940) O m ’e-s 9 4 - 4 004) 055 04) 044 ‘45- 0 OsO 0 0 . 0 . 45 5 ‘4 94 V -.4 0 45 -4.4 94 5 5 05 0 W V .-S > 0 (5-S O 0 ‘S ‘ -4)0 4)4) .4055 0.5 IS O~~~ .00 0—0.0 C) 15 4)00 ‘S o

~~ 0 ) 15 (Is V 0.4 00) 0 (0 55 55--45 00 a 0) 550 0)

0 .4 W~~~s 015 0 09 40 0 V ’S ‘ -4) 0 ~~ V 45.50) 55 0 .404) b ’55 150 0 0

‘

4 0 ‘SE’S 4) . 4 4) 0 054 1 4) ‘.0-i-S

0 94 .4 00 5514 . 40 >5 045.4 ~~~5 5 I

4) 4)55 0)0 0 .-S 4 5 0 00 . 40 ‘14 —S O

0 .4 - 00 055 0 0 . 5 0 4) ‘-44 )0 4)

0 b’94 0 b94 00 94.40 4)0 15 0045 0 .-S 0.4 55.4 o ’ 0.4IS.—s 45 

-.4 0 V - S C). 45 —5 9 4 > 0 0 0 0 V 00) 04)0 945 5 4 ) 15-.-S ’-I W O . . 0 14 0 (5-5 0 O V .4 0 5500 4) 0.40 140 V ~~~ 0 0 0 0 0 . -S ‘ -15494 .4.40) 0 0 1405 0.40 5 ) 540 55 0’ (0 0 0 ‘000 VI 4)550 0 5 ) 0—S 50 0 ‘S 0.4 4 5 0 0 ‘05 0 51) 0 015.4

V 00 >5 IS’S V 9 4 0-4 . 4 045 004) O 0 .0’S 00 -.-I ’I-s 0 .--SW4) b ’ 550 ‘ 4 E I S 5) V’S V 0 0 0 ‘—S O 0 . 00 0 V 4) 15

14 ,-S ‘ V4) ‘40 0 00 05 - ‘ - 5 0 45 015

0 ‘405 ‘SO ’S 45450 04515’ 0 • 45 0) 55 00.-S 0555 94 0 .-I 00 ~~~~~~~ 0 0 4)

• 0 0 .-S (1)00 - ‘ -5055 0 .401500 ~~~4 5 O

0 5 5 015 a045 450 0 9 4 0 .4.4 ‘0.4 00 00) ‘

00 00 004 5

O -.4 0 0045 1 0.4 450 151454 0 V%O -.-S 0 0 5 )

a 45 -i-I .455 55 m 0 0.4 55 0.54 .45-V 0 0 04) 4-) .5 ~

— W 094 0 0 145 ) ~~54, 0 ,

~~ s

055 C) b ’ 45 055 ~~ -‘-S 0 04) 5) 0 0)5- 0 014 . 44 50

0) 

9 4 . 4V 0’—45 ‘ 0 0 0 00 0 45.4 004) > m i-S W 0 .5~ 0 .-S 15 -.-5.44) 0094 000 ‘

.5 0 .54 -.4 0 -r) 0.4>5 H ’ 4 O

~~~4) 4 5 0 00 ) 45940 — 55 9414 0 15) .-S~~~ - 0 a0 osos 55 0 15 09 40 0 154 )0 5)5) 1 5 O (5)00.4 . 1 5 ’SO 0 0 ) 0 4)0 . 4 0 00 0 0 0 00.- S 0 . - I( 0 0 01545 .-S 0 .-S 0 15. - S O 1455 450 0 4 5 >45 00.4-000 0055 .445 W

0 I 0 0)0 5.5 0 0 0 4 5 VC) > 0 > 45 0) > 94 0 0.0

~~~~

0 000 000 . 4 ‘4 545 5 5 . 0 5 -‘-S .-S 0 .445 , 0 ‘40.4 0 > 0 0 4) 0 00 0 4) 0

‘—1.445 0 .-4 0 .-I 0 9 4 00 0 0 0 0.0 0 . 40 4) U~~~~ .4C) 0 1 5 0 0450S-’4OE 0 5 45 0 ~~~~~~~ N ‘54(Is O 14 — I

~~~~~~~~~~~~ 0 Es .

> 0 N 4 ‘4

0

.4

0 45 94 0 ‘S — 0 V 0 0) .4 45 — 55 —

V 94 (54 0 -4 0 (0 0 0 o .0 0 00 0 0 ‘4 55 0 0 - 0 OW 0 045

t

~ -i-I 0) .4 0 ‘40 0) . 40

V V 45 45 55 0 0 > 94 55.4 -4 55 -_S 0) 9445 0 (5.5 4) —I 94 0 (

4-5 05) 94 05) 45 Os ‘4 Os .40 050 ‘4 0 4)94 ~

- 50 45 (5-s

0’ S -

.4

V

O —

‘0

Cl Cl -

94 5-

0 — — S

0’ §5

, Cl ‘0

5-

(a Cl 5-

II

‘4 I- — p.,

V (a — a 4) a

I’ 

I -‘-I

5,

21

- - ~ - - -

~~ 

-

i 

~~~~~~~~~~~~~~~~~~ - -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~ - . :

~~ .- -~~~~;
‘S 1 45 I (5~5 0 0 V(0V -.44) 0 45 .415 0 ‘ S 9 4’S (0 I 1(00 1 5 000 >5 0 - . 5 00 94.0 0 1 5

~ ~~~~~ 

(5.5~~~ .0 Is ‘ S 45.-4 4510

1 4 0 ( 0.-I 00 (0 55 ‘5 00 0

(I-S i—S W 0 -) 0 094 > 4 )4 5 5 5’ S 0 (055 .0 U) 4 ) 0 — 00 0 0 ‘.5 -i-S .4 - 4 )55 IS> (5-s 4500 0 0 5 - s — 00 0 5-s 0 0 0) 0 0515 0 1 0 0 4594 ‘-5 0 ( 0 0 ‘ S IS-.S I 55-4( 1 )0 -

4 - .40 0 ’S 0 -.S 4) 4) 9415 4) I1

~~~V V 0 U)

0 (4~

4 ‘.5 (14 45 450 5 (0055 (0(0 (5)

O ~~~O sOO O s 55 094 0 -‘-4 > 4) 0 0 0

0 -4 0 0 . 4 5) -.5 0 4 ) 0W O V O V O

0 00- - l U 4 ) 0 

a

~ 

~ 

00 15 000 -i-S

(

5-5.045 . 0V (0 Os 4)45004) I ‘ S O) V I-I (0 ‘-44 00 0 5 5 ‘S ‘

s-s 0 4-i ‘5-s 11)4) ‘44 .-S (5-4050 0 0 05 0

(0 V 0”-.l 0 0 ’S —so 0 55 0 .5 45 -.-5150 0 r-S

V 15 >5940 5 0 0 .00 .—4 - .S ( 05 ) 55 i-S 0 —14 - i - S ’S ‘--‘S W i--S —0) —(0 --S 0’S 0 > 0 0 0150 4) 0 ) 55 V 0 ) 0 I 0 5 - s lO

-S V 55 0 .-S O ‘S’S -i-S W ISO IS -.S (0 .O .-S ‘00 ‘

S ,.545554 ) ‘.4 —S 0 > .-S ’S r-S 04) >59445

15 0 000 00 45 04) 0450 W ’ S r-S -i-S (0> 14 ( 0 0 ‘S’S ( 0 5 ) (0045 V 0 0 0’ S 0 _ V ‘4 154 ) 0 ‘.5 0 0045

0 4 5 1 4 ( 0.0 454) ‘S 45 45094 45’S 0 ~~ .5

45 0) 0 454) 055 (04 -5 00 0 5 - s O 550 1 0

o -s-S O s--S O -‘I C) (00 -.5 (0 -.5 440 ’ 0 00 00 0 ’4-s D-i 0 0 1 50 a

~~~ ’S a> - 00 0515 050450 (0-

~~~~

0 0 45 O 1 4500 .-S Is I Os I --SO .4 ’s-S

a 11)5515 > tn94 0 I n. 0 1 5 -i - 5 C)~~j I

‘

— 0 0 ‘—0 9 4 ’S —0 — 4 J’S45 4)0 55015

‘.5 45.4 0 5 0 0 ~~~ - S 55 -.-S 15~~~~~0 0

(055(0.-S (00) .-4 45 (0 ( 01 4 045 5.5 0 .-S

b’-> VV b’~~ V ’S 0 ) 0) 0)5) 0 V V 000 O 0 1 4 4 5 .5 00 .014 00 0) ‘-44 15 .—I ’ I-SO(0 55 -‘-I ‘.4 94 .4.4 .4 005 ‘400050 45 0’94 - 4)0 ‘4-s O 4)14 4-’ 0 14 I S ç -i400 0 5 ) 0 0’ S ISO 0 .0 5514 55-- -.4 0 0 W O 0 14.4 0 1 5 0 ) 150 140 )4545 0 1515 0 V .—I0.4 0 0’ 0 0550 0 0 0 4)0 09 4 .sJ -.5 0 (0 450 4 ) 05 . 4 ‘4’— ’s-S ’SV 50.4 (0 0’S ‘40 ) 0 0 0.45-5 450 W V

-‘-5 - .-400 45 .4 0’ V’S -i-S b’ -.50940 0550 W O O-i-I 011 0) V 0 ( 0 045 0 V

~~~~

O

.

~ ‘-1 0 0 55 ‘-400 00 ‘-5 0 .-4V 00 000 0 000 0 0 5 - .4 t-.4 -r, U -i-S 0 5.5 45 0 ( 00 0 4 5 V N 94

(5-s

0 14

— 0)

~~~~~~ 

0

-i-S El > 0’ -4 —4 0 0 45 -i-I 0 C) (0 — 5 15 4)0 4? 0

5 ) 0 0 0

u 

0

‘0 — -‘S 0 .0

0 0 - 55 15 C) 45 - 5) — ‘4 V 94 94

O .0 0 O — 0 0 0 0) 0 0 0(0

‘4 5) 00 0 --I -‘-5 0 0 ) 0’ 

(0 0 0 0 0

0 -.S . 40

Is .i

~~ V V S 45 450

0 5.4 00 0 C)> 94

0) ~

:

~‘~

i ~~~~~~~~~~~~~~~~~ 

‘S IS --I

(5-s 94 4)

—I 094 5s-~~ 0) 9414 C) (0 0 ) 0

4) ‘.5 05 O~~

0 V .4 . 44 ) 4 - 5.4 p-S -44) 94 45 45 94

~~~4) V 0 94 0)-i-S 54 0’ S

--Si-S N

V ‘0 0 0) —

— N 5- ‘0

8 — — N 5-

‘1) 45 —

94 ‘0 N i-I

O Cl Cl — 0 5- 5- -4 — .

~ -4

O >. • 0) .14 0) — 0 .4 ‘0 i-I S M S 4) 0 5) 55 ‘-— ‘0 .0

54 Cl U 5- ci)

10 4) (a

‘

~~ 

14

J

22

4) 

_

i p

~~~~-•—s-~~~ - ~~~~~~~~~~~ -

~~~~ 

‘-

~~~~~ iy~~

_

~ - t

~~~~~~~~~~~~—~~ ~~~~~~~~~~~~~~~~~~~~~~~~~ 

~~~~~~~~~~~~~
used client self—ratings. Only a few studies (Barahal et al., 1950;

For gy & Black , 1954; Healy, 1973) used judges’ ratings.

Ratings of satisfaction and effectiveness may involve a global,

overall measure of the variable of interest, or the ratings may assess

specific facets of the variable. A globa l measure may consist of a

single item, or it may represent an average or summation of two or more

items. Barahal et al. (195P) used a global measure of satisfaction defined as an “overall fe~~~ing—tone rating. ” This rating was made by

three judges who listened to recordings of structured interviews in

which the interviewer covered 20 itema that dealt with such aspects as

clarity of future vocational plans and helpfulness of conferences with

the counselor. Although the judges rated each item separately on a

5—point scale (from “very enthusiastic” to “dissatisfied and/or highly

critical”), they also estimated an “overall feeling—tone rating” using

the same 5—point scale. An example of a globa l measure resulting from

the summation of items is Zener and Schnuelle ’s ( 1976) “Evaluation

Scale. ” These researchers combined two highly correlated Cr = .82)

items (“I would recommend the SDS to a friend” and “My summary code

seems reasonable”) to obtain their global measure of satisfaction!

effectiveness.

Ratings may also involve specific rather than global measures of

satisfaction or effectiveness. Examples of specific outcomes used to

assess satisfaction or effectiveness are “to recommend the vocational

exploration experience to a friend” (Cooper, 1976), “setting up goals

consistent with one ’s abilities and interests” (Graff , Danish , &

Austin , 1972), and “clarification of career direction” (Healy , 1973).

Linden , Stone, and Shertzer (1965) h ave devised an instrument

that, although not specifically desi gned for career counseling, could

be used as an outcome measure of satisfaction with career counseling.

The Counseling Eva luation Inventory developed by Linden et al. contains

a subset of -items that assesses client satisfaction. Some satisfaction

items relate to process variables, but four items are concerned with

outcome and are enumerated in Table 7. In addition, the authors have

reported final scoring weights, f actor loadings, test—retest reliability coefficients, social desirability ratings, and validity data for

the instrument.

Also , it might be noted that although satisfaction and perceived

effectiveness are considered outcome measures, some researchers base

their assessment on process variables. Wright (1963), for example,

obtained client ratings of the following warmth of the relationship,

coverage of the test information, clarity of the test interpretation,

and value of the test interpretation for educational and vocational

planning. Of these items, only the last is clearly an outcome measure.

The assuoption is that the warmer the counselor—client relationship,

the mere complete the coverage of the test information , etc. , then the

more effective the counseling.

23 

~- - -- --
Miscellaneous Measures

There are several miscellaneous variables that do not fit into the

categories previously described. These dependent variables are summarized in Table 8.

One measure seldom used in career counseling research is the cost

analysis con ducted by Icrivatsy and Magoon ( 1976). These authors calcu—

lated the mean cost per subject in terms of personnel time , personnel

cost , materials cost, and a total cost (personnel cost plus materials

cost). Barahal et al. (1950) also reported costs for (a) “the mean

dollar estimate of the value of the guidance service” and (b) the

actual costs for tests and personnel time .

Another infrequently used variable is Anderson and Binnie ’s ( 1971)

measure of occupational aspiration, which these researchers adapted from

Haller smd Miller ’s (1963) Occupational Aspiration Scale. On this scale ,

the stuãent is asked to select from jobs at 10 different occupational

levels. Selections are made for realistic and idealistic preferences, as

well as for short—range and long—range choices. The prestige of each of

the subject’s selections then is assessed following the Haller and Miller

procedure.

A more commonly used outcome measure is number of career choice

options. For this criterion, the subject is asked to s

~

tate the career

alternatives he or she is currently considering., and a count is made of

the number of options. Before- and after-measures (Mencke & Cochran ,

1974) of the number of options also may be used. There are some prob—

lems with using this kind of variable as an outcome measur e, because a

larger number of options may or may not be desirable for a particular

client. A somewhat similar measure is to ask the subject to report the

amount of time spent thinking about occupations or about self during a

given period of time (Xrivatsy and Magoon , 1976 ; Zener and Schnuelle,

1976). Aga in , the assumption is that “mere is better ”--that is, the

more time an individual thinks about career-related concerns, the

better that person’s career decisions will be.

24
IV 

VI 

Study 

Barahal et al. (1950) 

Krivatsy & Magoon (1976) 

Anderson & Binnie (1971) 

Babcock & Kaufman (1976) 

Melhus et al. (1973) 

Fore11an & James ( 1973) 

--. 

Table B 

Miscellaneous Measures as Dependent Variables 

Variable 

Cost analysis 

Cost analysis 

Occupational aspirations 

Change in career choice 

Change in career choice 

Change in career choice 

Change in total score on 

relevant questionnaire 

items 

Change in test scale 

relevance 

How m

Mean dollar estimate of va

actual costs for tests and 

Mean cost per subject calc

personnel cost, materials 

Occupational Aspiration Sc

student career selections 

change in expected educatio

Occupational/Educational G

measured pre-post change in 

Pre-post change in expresse

Pre-post change in respons

~lans Questionnaire concer

Amount (but not direction) 

tal score for Occupational 

concerning commitment to, 

ence relevant to, and antic

pational choice; and signif

role in respondent's life. 

Pre-post measures of releva

ality test scales 
0 00 O~~~~~~~ 4) ~~~~~~~~~~~~~~~ 45

— 15 14 14 15 0) 055 0 V — V 0 4? 0 • Os ‘S -

~ ~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~ 

-

0 0 0 ~~ -.4 0 -‘-5 -i-I ’S Z Z Z Z Es -’ s I. E-s .0 ‘.s - -- 14

V

I I

26

— —‘ p

- —.- ~~r-

~

..-~- --- - ~~ - - tI5W~~~~ ~~~~~~~ - “

~~~~~~~~~~~~~

‘ 

‘ - -~
I

ISSUES IN CAREER COUNSELING CUTCOME MEASUREMENT

Several issues emerge from a survey of the outcome measures used

in career counseling research. These issues are neither new nor unique

to career counseling research. In fact, Williamson and Bordin (1941)

touched on many of the same problems facing career counseling researchers

today in their classic article, which critiques the methodology of vocational and educational counseling research over 35 years ago. Problems

encountered in outcome research in psychotherapy often have their counter—

parts in outcome research in career counseling (Bergin & Garfield , 1971 ,

Meltzoff & Kornreich , 1970).

These various problem areas are interrelated. Questions of validity,

for example , are linked to the criterion used, whereas decisions concerning the instruments to be used depend on the operational definitions the

researcher chooses for the constructs to be measured. The issues in this

section relate to criteria , instruments, and design and analysis. Most

problems that the career counseling outcome researcher must resolve can

be subsumed under these general categories.

Criteria of Career Counseling Outcome

The criterion issue has perpetually bedeviled counseling research ,

and there are several aspects of the criterion problem to consider. One

aspect concerns selection of immediate or ultimate criteria. Another

facet of the criterion issue relates to the use of change as a criterion.

In addition, there are infrequently used criterion measures that deserve

special attention. -

Immediate vs. Ultimate Criteria. One aspect of the criterion debate

has involved the relative merits of immediate and ultimate criteria.

Goflyea (1962) pointed out that the attrition problem complicates the collection of long—range criteria data. Because the attrition of subjects

is probably selective , sampling problems become even more complex. In

addition , the longer the time since concluding the counseling, the greater

the probability that factors other than the counseling experience have

affected the outcome. The effect of other factors may account for

research evidence that short—term and long—term measures of the same

variable do not correlate very highly. Hewer ( 1966), for example, used

the current occupation of her subjects as the criterion of appropriateness in a follewup of an earlier study (Hewer, 1959). She compared the

earlier judg es’ ratings of appropriateness with the occupations the subjects pursued some 8 yea rs later and concluded ther e was “no relationship between the judges’ ratings of realism of vocational choice and

realism as determined by similarity of the original choice and current

employment” (Hewer, 1966, P. 292). The quástion posed by Hewer’s research is whether the lack of correlation between the two measures of

appropriateness was caused by the lack of validity of judges’ ratings

or by the various influences on the subjects during the years between

the original and follewup studies.

27

- - - .

I .- ~~~~~~~~~~ -- - - -- - ~~~~~~~~~~~~~~ - -- - - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~- - - —‘i
V

Another facet of immediate versus ultimate criteria has been sug—

gested by Katz (1975), who pointed out that we do not require long—term

payoffs for competencies such as typing, reading comprehension , woodworking, or social studies. Accordingly, Katz questioned the need for assessment of lon g range , real—life outcomes for career decisionmaking. If it

is agreed that career decisionmaking is a desirable skill, Katz ’ argument

would hold that the researcher determines whether the skill has been

acquired , not whether the skill predicted a long—range criterion such

as job success. Myers (1971) came to a similar conclusion from his conceptualization of career development as a series of choice points, with

each choice influencing the one that comes after it. Myers argued that

it is unlikely that short—term career counseling can have much measurable long—term effect , considering the far greater effect each choi ce

actually made will have on subsequent choices. - -

Change as a Criterion. Another aspect of the criterion problem

relates to using change as an outcome measure. Some investigators have

used changes in career choice as outcome measures (Anderson & Binnie ,

1971 ) or changes in the number of career options being considered

(Cooper , 1976; Krivatsy and Magoon , 1976; Mencke & Cochran , 1974; Zener

and Schnuelle , 1976). It is possible, though, that change per se is not

necessarily a desirable outcome. If, in changing, the client ends up

with a less realistic career choice than he or she had before, change

is undesirable . In like manner , an increase in the number of career

options mi ght be a positive outcome for clients who are in the exploratory stage of career decisionmaking whereas a decrease in career options

would be advantageous for clients in a later stage. A similar argument

would hold for other variables, such as certainty ~f career choice.

Higher certainty for an unrealistic career choice might be less desirable than lower certainty for a realistic choice . Change , even if its

direction is predicted, would thus appear to have shortcomings as a dependent variable. The implication of such considerations , then , is that

change criteria should differ from client to client. Some authorities,

e.g., Bergin (1971) and Krumboltz (1966), have already recommended that

the goals of counseling or therapy should be unique to each client.

Neglected Criteria. There are other infrequently used criteria that

seem to have advantages as outcome measures of career counseling. Such

criteria include instrumental behavior8 and cost analysis.

To date, almost all the instrumental behaviors used as dependent

variables for career counseling have been measures of some kind of information, seeking. There is no reason , however, why other instrumental

behaviors could not be used. For example, did the client obtain a job

( Thoresen et al., 1967) or enter a training program? In the Army context ,

did the officer actually complete a career plan? Did the enlisted woman

register for and complete a civilian education course that would enhance

her chances for promotion ?

28
I

Another infrequently used measure is cost—benefit analysis. With

this outcome measure, an assessment is made of the effectiveness of the

career counseling treatment and of its cost. Given two equally effective modes of test interpretation, for example, the one that is avail—

able at lower cost probably will be chosen. On the other hand, a less

expensive approach may not be selected as the treatment of choice if

another, more costly, method is of sufficiently greater effectiveness.

Historically, cost—benefit techniques have been used primarily by

economists in evaluating social change an d the relative desirability of

different social policies (Rothenberg, 1975), With deflated financial

resources of government agencies and other institutions, however, the

goal of such analyses in evaluation research is “to determine that

strategy or combination of strategies that maximizes the des ired result

for any particular resource or budget constraint” (Levin , 1975 , p. 89).

Since accountability is of increasing importance to organizations and

institutions, cost—benefit analyses are likely to become more frequent

in career counseling outcome research.

The advantages of some of these less frequently used outcome

variables in terms of nonreactive measurement will be discussed later.

The point here is that certain criteria not extensively used heretofore——e.g., cost—benefit analysis or archival data on career information seeking——might be considered more frequently by career counseling

researchers as supplementary measures to more traditional criteria.

Instruments

After the criteria of career counseling outcome have been chosen ,

instruments must be selected or constructed. Several problems may - need

to be resolved in making decisions about instruments; these concerns

involve operational definitions, the type of instrument, and the

reliability and validity question.

Operational Definitions. Zytowski and Betz ( 1972) have argued

strongly for greater care in measurement in counseling research. These

authors have pointed out the need for clear, precise definitions of the

constructs to be measured. One of the fuzzy areas of career counseling

outcome research concerns the definition of career “choice” or “preference. ” Ma ny reports of outcome research do not make clear what the

subject was asked to do in specifying his or her career choice. Crites

( 1969) has presented a comprehensive discussion of thiB particular definitional problem (pp. 127—135). In his discussion, Crites reviewed the

ways in which career choice has been defined , made a critical analysis

of these definitions, developed an operational definition of choice , and

enumerated the necessary conditions for the expression of a career choice.

Critea ’ ( 1969) text also contains a section on measures of vocational

choice in which the author has described previously used instruments (pp.

135—148). Crites’ treatment of the problem of defining and measuring

career choice is an example of the type of approach that Zytowaki and

Betz ( 1972) have called for with respect to career counseling constructs.

29

—~~ - - ----- --- --

~~~~~~~~—--— — ________________________________ -- - p

- ~~~~~~~~~~~~~~~~~~~~~ ~~~~~ ~~~~~~~~~~~~ - ~~~~~~ - -

~~
Type of Instrument. After the constructs to be measured have been

defined, the researcher must decide which type of instrument is most

suitable for the objective. As shown in Tables 1—8 , most measures in

career counseling outcome research have involved self—report. Zytowski

and Betz (1972) reported that two—thirds of the instruments they reviewed

were se lf—repo rt measures. Most of the remainder required the quantif ication of judgments about other people or the counseling process.

Self— reports have been obtained by questionnaires, interviews, Q—sort P

techniques, or tests. Judges’ ratings have also been extensively used,

especially for measures of realism and self—knowledge. Even when judges’ ratings comp rised the criterion, self—report measures (such as test results

or expressed career choice) were usually used in making the judgments.

For the same criterion, several different types of instruments could

be used. If the dependent variable involves career information seeking,

for example, data can be obtained through a questionnaire, e.g., Zener

and Schnuelle (1976); an interview, e.g., Krumboltz and Thoresen (1964);

or counseling center records, e.g., Cooper (1976). The ease, precision,

and reactivity of the various types of data collection must be weighed in

selecting the best instrument for any given situation. Although a questionnaire may be easily administered and objectively scored, it may la ck

the flexibility of an interview in following up on aspects of the -research

not anticipated beforehand. Counseling center records may provide the

mnsans for an unobtrusive measure of career information seeking, but similar records may be unavailable in other settings.

To a large extent, dependent variables used in career counseling

research have tended to be “reactive” measures (Webb et al., 1966). Thus,

the measurement process itself may influence the outcome of a career coun—

seling experiment. Such an effect would be especially likely in pretestposttest designs, commonly used in career counseling outcome research.

Although real change may occur, it may be because of the measurement

activity rather than the treatment. Randomization and the inclusion of

control groups, two common techniques used to assure “internal” validity

(Campbell & Stanley , 1966), may be ineffective when reactive measures

are used.

Considering the unknown and possibly sizable affects of reactive

measurement, it is surprising that so few career counseling researchers

have not attempted to assess or control for such effects. One way of

determining the effect of a pretest, for example, would be to employ a

Solomon four—group design (Campbell & Stanley, 1966 , pp. 24—25). A

posttest—only control group design will also eliminate pretest effects by

eliminating the pretest. With respect to this latter design, Campbell and

Stanley (1966) commented that although the pretest is “a concept deeply

embedded in the thinking of research workers in education and psychology,

it is not actually essential to true experimental design s” (p. 25). These

authors have also noted that although the Solomon four—group design is

preferable, its advantages may not outweigh the doubled effort involved

unless randomization is not possible.

30

—- -~~~~~ -~~ - - - - 

-

-~~~~~~~~ - --~~~~~~~~~~ - ~~~~~~~~~~~

— - .

- ~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~ -.-- --
I

An inspection of Tables 1-8 reveals that only two studies used nonreactive outcome measures. Krivatsy and Magoon ( 1976) analyzed the costs

of materials and personnel time to evaluate the relative expense of three

dif ferent career counseling treatments, and Cooper (1976) recorded whether

or not subjects returned to the counseling center for career information.

Dependent variables of the types just described are examples of measures

both objective and ns- nreactive . It is my opinion that a larger proportion of career counseling outcome measures should be objective and nonreactive, especially in cases where randomization cannot be achieved or

the comparison group is not a true control group.

For a comprehensive discussion of the hazards of reactive measurement and a compendium of unobtrusive measures, see Webb et al. (1966).

Weinstein ’s (1975 ) chapter on using information systems to eva luate mental

hygiene programs also contains information pertinent to assessing career

counseling outcomes. Researchers charged with the evaluation of counseling center operations may fin d Weinstein ’s paper of particular interest.

Reliability and Validity . Another problem the career counseling

researcher must consider in devising or selecting instruments relates to

their reliability and validity. Zytowski and Betz (1972), in their review of measurement in counseling research, documented the inadequacy of

the reliability and validity data that have been reported in the literature. The career counseling outcome research that has been surveyed in

this paper has tended to be remiss in presenting reliability and validity

data. Exceptions to this generalization are criteria involving judges’

ratings and career information seeking. About half the studies using

j udges’ ratings presented data on interrater reliability. Rogers ( 1954)

reported test—retest reliability of the instrument he devised for self—

estimates of interests and aptitudes, and Linden et al. ( 1965) provi ded

both reliability and validity data on their Counseling Eva luation

Inventory .

Zytowaki and Betz (1972) concluded that counseling research (either

process or outcome) should not be published without “some minimal indications of the reliability and validity of the instrument on which the research is based” (p. 78). Specifically, they recommended that researchers

report the results of any previous reliability studies, estimates of internal consistency, and test—retest reliability (where indicated). With

respect to the more difficult problem of validity, Zytowaki and Betz suggested that construct validity is perhaps the most feasible type. The

researcher should first write a descriptive statement that clearly and

precisely defines the construct and then generate validity data through

research on group differences, correlation with other variables, treatment effects, and the like. For example, realism of alternate specialty

preference could be defined in terms of a choice that is consonant with

an officer ’s educational level, un dergra duate (or graduate) major, previ—

ous assignments, and primary specialty. Research then could be designed

to determine whether degree of realism of alternate specialty preference

differentiated officers who leave the Army early in their careers from

those who leave it later (group differences). Other studies coul d

31 

~~~- --- -__________

-- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I

investigate such questions as the relationship between realism of the preferences and later satisfaction in those alternate specialties (correlation

with other variables) or the effects of a computer—aide d career planning

intervention on the realism of expressed preferences (treatment effects).

Design and Analysis

Researchers face other difficulties relating to experimental design.

Some of the problems may be particularly troublesome to the person engaged

in career counseling outcome research , because such research frequently

involves real clients, real counselors, and real counseling environments.

In all field research , the researcher exercises less control over the

research design than is usually the case in a laboratory setting. Lesser

control may jeopardize both the internal and external validity of the study .

Design concerns considered here include local control, assignment of sub—

jects, anri comparison groups. Design type is considered in relation to

these concerns, and some associated analysis problems are noted.

Local Control. Voisky et al. ( 1965) have used the term “local control”

for procedures that either eliminate factors contaminating the dependent

variable or incorporate the factors into the research ~~sign. Voisky et al.

called these techniques “controlling out” and “controlling in. ” An example

of the first type would be restricting one ’s sample to self— referred clients, thus “controlling cut” the contaminating influence of persons who

-have been required to come for counseling. The second type would be illustrated by incorporating into the design different client characteristics

such as gender or level of career maturity. When such controls are built -

into the research design, it is possible to detect main effects and interactions due to these variables. -

Assignment of Subjects. Randomization is the sine qua non of experimental research. Assigning subjects randomly does not mean that the

groups are equated on pertinent variables. However , it does assure that

the risk of significant imbalance is the same as. the level of risk assumed by the statistical techniques employed.

Desirable as randomization is, however, it cannot always be attained.

Pa rticularly in fieldwork, randomization may be impractical if not impossible. A researcher may nave to work with intact groups such as orientation classes, for example, selected by the student subjects for nonrandom

reasons such as time of day. Even when clients are randomly assigned to

career counseling groups, the randomization process may be preempted by

the restrictions imposed by the subjects’ course schedules.

When randomization is imperfect, the risk of an interaction be—

tween subject selection and treatment is increased. Kelley at al. ( 1970)

assessed 73 reports of counseling research for sources of invalidity and - f ound that a selection—treatment interaction was a “definite weakness” in

61.6% of the studies and a “probable weakness” in the remaining 38.4%.

Although the investigations reviewed by Kelley at al. were not limited S

32 

-

- - - ~

- ‘

~~~ - -~~ - - ~~~~~~~

,

- -~~~~~ - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - 

-

~ ~---

-

.

~~ -
to career counseling an d include d process as well as outcome research ,

his results illustrate the prevalence of inadequate randomization.

For a more detailed discussion of randomization and its associated

problems for counseling research, the reader is referred to Volsky et al.

(1965, chapter IV ).

Comparison Groups. One major problem in all counseling and therapy

research lies in the difficulty of obtaining appropriate comparison groups.

Usually, the comparison is made between an “experimental” group that received the treatment or intervention and a “control” group that did not.

To deny treatment to individuals who want it in order to obtain a control

group may not be ethically defensible. The solution to this ethical

dilemma in career counseling research has often been to use a “wait—control” group. After the subjects are randomly assigned to two groups, one

group of subjects is given the counseling intervention immediately , and

treatment for the control group is postponed until after posttest

measures are made on both groups.

Unfortunately , denying or postponing treatment does not necessarily

preclude the control sub jects ’ receiving help f rom other sources. People

who are motivated to seek career counseling in the- first place may also

be motivated to seek help f rom other sources, rather than wait until the

termination of counseling for the experimental group. In Campbell and

Stanley ’s (1966) terms, “history ” as a source of invalidity may contaminate the control group or individuals in the control group. Some event

other than the experimental treatment occurs that may have some of the

same effects as the treatment.

In a discussion of the effects of withholding counseling services,

Voleky et al. (1965) reported a study in which all control subjects in

the experiment refused treatment when it was finally available to them.

This situation apparent ly occurred because the experimental treatment

ended near the end of the spring term, with final examinations impending.

Gi ven that there are such effects, Volsky et al. ( 1965) asked , “Wh at can

be used as the placebo comparable in all dimensions except the relevant

treatment variable in subsequent experimentation?” (p. 161). The authors

concluded that the best that can be done in terms of classical experimental methods is to compare treatment methods, using random assignment of

subjects to the various groups.

Control groups may be lass necessary in career counseling outcome

research than in other types of psychological research. Almost all

career counseling outcome studies using a control group found their

experimental groups scored higher on the outcome measures than did the

control groups, e.g. , Hoyt (19 55), Tipton (1969), and Wright (1963).

In fact, this superiority of career counseling over none is about the -

only consistent finding of outcome research . Thus, the suggestion of

Volsky and his colleagues to compare different methods of counseling

instead of comparing experimental and control groups may represent a

practical course for the career counseling researcher.

33

___________________________________________ ________ ______________________________ p

~~ 

- - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

- I --
Type of Design. Campbell and Stanley (1966) and Cook and Campbell ( 1976) have presented in detail the various types of pre—experimenta]., experimental, and quasi—experimental designs available to researchers in the behavioral and social sciences. Because pretest—posttest designs have been used so frequently in career counseling outcome research, the focus of this discussion will be on the pretest—poettest design and some alternatives to it. The researcher’s choice of design and type of data analysis will depend to a great extent on whether subjects can be ran—

iomly assigned to treatxnants and on the type of comparison group(s) available.

A research design proposed by Crites (1964) includes control groups

and allows the researcher to assess the affects (in many instances ) of

pretesting and time, both of which are possible sources of confounding. In his description of this design , Crites has noted possible statistical

tests wh ich can be made and has suggested the possible significance of results in terms of counseling effects.

Though often used, pretest—poettest designs involve other hazards for outcome research (un less controlled for as in the Crites design). Administering a pretest may sensitize subjects to the treatment they are

to receive and may also produce a practice effect. Accordingly, Campbell and Stanley ( 1966) have pointed out the assets of the postteat—only design. Those authors suggested that the advantages of the Solomon f ourgroup design over the posttest—only control group design may not compen- sate for the doubled effort. The Solomon four—group design permits the investigator to assess the effects of the pretest. Nu nnally (1975) has

also recommended the posttest—on ly design , characterizing it as “the workhorse of evaluation research” (p. 123). If certain conditions apply, however, a pretest—poettest design should be considered.

These conditions are that the (a) number of subjects in each cell is small , (b) subjects show a wide variation ~large standard deviation ) on

the dependent variable compared to the expected effects of the independent variable (treatment), and (c) pretest—poettest correlation of the depen- dent variable is expected to be high. When these three conditions apply

and it is reasonable to e

~~ect pretest—treatment interactions and pretest—

posttest interactions to be alight, Nunnally has suggested using a pretest— posttest design.

Although the career counseling outcome researcher often may be able

to assign subjects randomly to different treatments and to a wait—contro l

group, sometimes such procedures are not possible. Intact groups, e.g. , orientation classes, may have to receive the different treatments and to

serve as the control group(s). When intact groups must be used , Nunna lly ( 1975) suggested using a control group and obtaining measures on the dependant variable before and after the counseling treatment. With this

type of quasi-experimen~a1 design, then, pretest-posttest measures are

needed.

J 34

4 .

~ -~- ~~~~~~~~~~~~ 

-- 4 ~~~~- k— ~~~~~~~~ _______
Data Analysis. Whether an experimental or quasi—experimental design is used determines the type of ana lysis to be used. Nunnally ( 1975)

stated that the statistical analysis of data in true experiments is generally not controversial; but such is not the case for methods of analyzing data from quasi—experimental designs. For example, Nunnally challenged the practice of using covariance analysis when an intact group is

used as a control group. Instead, the author suggested using a repeated—

measures ana lysis of variance with the pretest-poattest measures as a

within-subjects factor. In this analysis, the focus of interest would be

the interaction of the treatment condition with the comparison groups.

Nunnally has presented examples of both simple and complex designs of

this type. He also suggested using a conservative approach for any post

hoc tests and recommended obtaining measures of strength of association

to help the researcher decide whether differences between treatments are

meaningful for real world situations. The reader is referred to Nunnally

(1975, pp. 134—136) for recommendations and rationales.

IMPLICATIONS FOR FUTURE RESEARCH

This paper has surveyed the instruments used in career counseling

outcome research , and some of the issues raised by the outcome research

have been considered. What , then, are the implications of the findings

for the career counseling researcher? The intent of the following discussion is to focus on some specific recommendations that emerge from the

shortcomings of career counseling outcome research. Most suggestions

were made previously by authors critiquing research on psychotherapy,

counseling, and guidance. For clarity , the specific points to consider

will be grouped as they relate to criteria, instruments, or design and

analysis.

Recommendations for Criteria

Use Multiple Criteria. Don ’t use only one dependent variable; use

several. Career counseling outcomes are multidimensional; one measure

is not adequate. Also avoid using all the same types of measure. For

example, include behavioral and archival data, as well as attitudinal

variables.

Use Specific Rather Than Global Measures. The use of a global

criterion may not tap the underlying dimensions. Bergin and Garfield

( 1971) stressed the superiority of specific over global measures and also

suggested that perhaps individual criteria should be developed for each

client. Xrumboltz ( 1966) also argued for unique goals for each client.

Career counseling outcome research has not incorporated such criteria

into research designs as yet. An example given earlier concerned change

as a criterion. An increase in career options might be desirable for

one client and a decrease desirable for another, depending on the

client ’s career decisionmaking stage.

35

4_~~~~~~~~~~~~~~ __ _ • ___ ~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~ -
Emphasize Short-Term Measures. It may be unrealistic to expect

long—term effects from the minimal interventions used in most career

counseling research (Myers, 1971). A well—designed, large—scale, longitudinal study obviously will require repeated measures over a long period

of time. Because most career counseling outcome research is not of this

type, short—term measures are usually called for.

Define Constructs Carefully. Mitchell et a].. (1974) expressed the

opinion that most studies on career decisionmaking are of limited use be—

cause constructs and instruments are not comparable across the various

studies. Zytowsk i and Betz (1972) a lso stressed the need for more careful definitions of constructs. An example was given previously from

Crites ( 1969) that illustrated the type of approach needed.

Recommendations Relating to Instruments

Use Instruments From Previous Research. Obviously, one cannot follow

this suggestion if no instrument exists that measures the construct of interest. All too often, however, researchers “reinvent the wheel.” Constructing a good instrument is not an easy task. A, larger poo1 of sensitive and reliable instruments would be available if each researcher built

on the work of predecessors in the area of instrumentation.

Report Reliability and Validity Data for the Instruments You Use. Of

the studies surveyed in this paper, about one—third presented data on reliability and one-sixth on validity. A major point made by Zytowski and Betz

( 1972) was that information on reliability and validity should be reported routinely on all instruments used in counseling research. These authors

also found that such data typically are not included in reports of counseling research. If a researcher uses an already—developed instrument for

whi ch reliability and validity have been established, considerable time

and effort can be saved. An investigator should be aware, however, that

in order to obtain comparable results, it s necessary to use the instrument in the same fa shion that the original researcher used it. A number

of career counseling outcome studies have used only portions of standardized instruments. Unless the reliability and validity of the adapted

instrument can be established, there is no assurance that it measures

the same construct or is as reliable as the original instrument.

Use Objective Rather Than Subjective Measures. In general, ratings

by subjects or judge s are not considered as valid measures as are more

objective types of assessment device.. Not one of the “good” studies

Meltzoff and lCornreich (1970) surveyed used judges’ or patients’ ratings

of therapy outcome. Even if the same instrument (ra ting form) is used,

results may not be reliable if different raters are used across studies.

Judges’ ratings have been used extensively for certain variables, such

as appropriateness of career choice. A more objective method of assessing appropriateness is comparing the Holland code ti-ott a test (such as

(

•

_

36

- I - I - -
I

the VPI , the SDS , or the Strong—Campbell Interest Inventory ) with the

Holland code of the subject’s expressed career choice, e.g., Mencke

and Cochran (1974), Zener and Schnuelle (1976).

Make Use of Nonreactive Measures Whenever Possible. Because nonreactive measurement has so many merits (Webb et a].., 1966), it is somewhat surprising that so few career counseling outcome studies have used

unobtrusive measures. However, such measures are much more difficult to

devise and may require considerably more ingenuity on the part of the

researcher. Archival data such as that used by Cooper (1976) and the

cost benefit ana lysis technique employed by Krivatsy and Magoon (1976)

are nonreactive outcome criteria already in the career counseling

research literature.

Recommendations Concerning Design and Analysis -

Assign Subjects Randomly to Treatment and Control Groups. If randomization is used, the investigator can assume that no systematic bias

was introduced into the composition of the various groups. Thu s, if

random assignment is possible , the researcher can place greater conf i—

dence in the internal validity of his or her study. If intact groups

must be used , employ a pretest—posttest design and analyze the data by

a repeated measures analysis of variance (Nunnally, 1975).

Include a Control Group in the Research Design. A control group is

needed to establish the internal validity of a study. The ethical question on denial of treatment can be resolved by using a wait—control group.

This group will receive career counseling after treatment has been concluded for the experimental subjects. If no control group is possible ,

compare different treatments.

Estimate Sample Size Needed. As Bailey (1971) has pointed out, the

use of more subjects than are needed to detect meaningful differences is “a waste of resources” (p. 323). Career counseling researchers have

limited time as well as money for tests and other materials. Many re—

searchers, thus, omit the step of calculating the size of the sample

they need. One reason for neglect of this procedure is the difficulty

in estimating how large a difference would have to exist in order for it

to matter. Such judgments require the researcher ’s subjective evaluation

of the role of factors such as the kind of outcome measure , the characteristics of the sample, and the level of effort required by the treatment.

More detailed discussion of the problem of estimating sample size and the — methods to use can be found in sources such as Bailey ( 1971) and Winer

(1971).

37

— -.- — - p

___ — -

~~~ ~~~~~~~~~~~~~~~~~~ -- 

- 

~~~~~~~~~~~~~~~~ - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ __________
SUMMA

This paper reviewed measures previously used in career counseling

outcome research. The purpose of the review was to provide background

for the construction of instruments to be used in eva luating a computer—

based career information and planning system for Army officers.

Some of the general considerations relating to career counseling

outcome research were the types of behaviors to be measured, where and

when the assessment would take place, the availability er~~~lid and

reliable instrumentation , and the objectives of the research effort.

Previously used career counseling outcome measures were surveyed using

a taxonomy from Myers (1971), which was expanded slighted to incorporate additional types of measures. Each kind of measure was described,

a typical example of it was presented, and any variants of that type were

noted. For the reader ’s convenience, the various studies were summarized - in eight tables which corresponded to the eight major categories of out—

come measures. -

From this survey of career counseling outcome measures, a number

of interrelated issues emerged. These issues were discussed in terms of

their relationship to criteria, instruments, or design and analysis.

• The implications of these issues for future research were incorporated

into a series of recommendations for the career counseling outcome

research.

-

I

‘

~~~r’ ~~~ ‘t~~ ~ $ir ~~~~~~~~~~~~~~~~
REFERENCES

Anderson , D. G., and Binnie, A. A. Effects of a Group Vocational Guidance

Class with College Community Students. Vocational Guidance Quarterly,

197 1 , 20, 123—128.

Angrist, S. S. Changes in Women’s Work Aspirations During College (or

Work Does Not Equal Career). International Journal of Sociology of

the Family, 1972, 2, 27—37.

Apostal, R. A. Two Methods of Evaluating Vocational Counseling. Journal

of Counseling Psychology, 1960, 7, 171—175.

Babcock , R. J., and Kaufman, M. A. Effectiveness of a Career Course.

Vocational Guidance Quarterly, 1976, 24 , 26 1—266.

Bailey , D. E. Probability and Statistics. New York: Wiley, 1971.

Banducci, R. Accuracy of Stereotypic Perceptions of Types and Levels of

Occupations in Relation to Background and Personal Characteristics

of High School Senior Boys. Unpublished doctoral dissertation, University of Iowa, 1968.

Barahal, G. D., Brammer, L. M., and Shostrom, E. L. A Client—Centered Approa ch to Vocationa l Counseling. Journal of Counsulting Psychology,

1950 , 14, 2 56—260. - 

- -

Ba rak , A , , Carney , C. G., and Archibald, R. D. The Relationship Between

Vocational Information Seeking and Educational and Vocational Decidedness. Journal of Vocation*l Behavior, 1975 , 7 , 149—1 59.

Berdie, R. F. Changes in Self—Rating as a Method of Eva luating Counseling. Journal of Counseling Psychology, 1954, ! , 49—54.

Bergin, A. E. The Evaluation of Therapeutic Outcome. In A. E. Bergin and

S. L. Garfield CEde .), Handbook of Psychotherapy and Behavior Change.

New York : Wiley , 197 1. -

Berg in, A. E. , and Garfield , S. L. C Ede.). Handbook of Psychotherapy and

Behavior Change. New York : Wiley , 1971.

Bingham, W. C. Assessing Career Guidance Outcottes. In R. Wale , L. ~~ith,

and L. Benjamin (Ids.), A Comprehensive View of Career Development.

Washington, D.C.: American Personnel and Guidance Association, 1974.

Bivlofsky , D., McMasters, W. , Shorr , J. E., and Singer, S. L. Individual

and Group Counseling. Personnel and Guidance Journal , 1953, 31 , 363—365.

Berman, C. Effects of a Reinforcement Style of Counseling on Information—

Seeking Behavior. Journal of Vocational Behavior , 1972, 2, 255—259.

39

I.

— 

~~~ - —~~
I

Butler, 1. M., and Haigh , C. Changes in the Re lation between Self—Concepts

and Ideal Concepts Congruent upon Client—Centere d Counseling. In

C. R. Rogers and k, F. Dymond C Ede.), Psychotherapy and Personality

Change. Chicago : University of Chicago Press, 1954.

Campbell, D. T., and Stanley , J. C. ~?cperimental and Quasi—Experimental

Designs for Research. Chicago: Rand-McNally , 1966.

Cook , T. D., and Campbell, D. T. The Design and Conduct of QuasiExperiments and True Experiments in Field Settings. In M. D.

Dunnette (Ed.), Handbook of Industrial and Organizational Psychology .

Chicago: Rand McNally , 1976 .

Cooper, J. F. Comparative Impact of the SCI I and the Vocational Card

Sort on Career Salience and Career Exploration of Women. Journal of

Counseling Psychology, 1976 , 23, 348—3 52.

Crites, J. 0. Proposals for a New Criterion Measure and Research Design.

In H. Borow (Ed.), Man in a World at Work. Boston: Houghton Miff lin,

1964.

Crites, J. 0. Vocational Psychology. New York: McGraw—Hill, 1969.

Dressel, P. L., and Matteson, R. W. The Effect of Client Participation in

Test Interpretation. Educational and Psychological Measurement, 1950 ,

10 , 693— 703.

Folds, .7. H. , and Gazda , G. M. A Comparison of the Effectiveness and Efficiency of Three Methods of Test Interpretation. Journal of Coun—

seling Psychology, 1966, 13, 3 18—324.

Foreman, N. E., and James, L. E. Vocational Relevance as a Factor in

Counseling. Journal of Counseling Psychology, 1973, 20 , 99—100.

Forgy, E. W., and Black , .7. D. A Follow—up After Three Years of Clients

Counseled by Two Methods. Journal of Counseling Psychology, 1954,

1’ 

1—8.

Forster, J. R. Comparing Feedback Methods After Testing. Journal of

Counseling Psychology, 1969 , 16 , 222—226.

rroehlich, C. P., and Moser, W. Es Do Counselees Remember Test Scores?

Journal of Counseling Psychology, 1954 , 1, 149—1 52.

Gilbert, V. H., and Ewing, T. N. Programmed Versus Face-to-Face Counseling. Journal of Counseling Psychology, 1971 , 18, 413—421 .

Gladst.in, G. A. Client Expectations, Counseling Experience, and Satisfaction. Journal of Counseling Psychology, 1969, 16, 476-481.

(

40

•~~-_-

~~~ ~~~~~~~~~~~~~~ - -

~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I

I

Gonyea , C. Appropriateness of Vocational Choice as a Criterion of Counseling Outcome. Journal of Counseling Psychology, 1962, 9, 213—219.

Gonyea, G. Appropriateness of Vocational Choices of Counseled and Uncoun—

seled College Students. Journal of Counseling Psychology, 1963 , 10 ,

269—275.

Graff , R. W., and MacLean, C. D. Eva luating Educational-Vocational Counseling: A Model for Change. Personnel and Guidance Journal, 1970,

48, 568—574.

Graff , R. W., Danish, S., and Austin, B. Reactions to Three Kinds of

Vocational—Educationa l Counseling. Journal of Counseling Psychology,

1972 , 19, 224—228 .

Grummon, D. L. In B. Stefflre and W. H. Grant (Eds.), Theories of Counseling (2nd ed.). New York: McGraw—Hill, 1972.

Gustad, 7. V., and Tuna , A. H. The Effects of Different Methods of Test

Introduction and Interpretation on Client Learning in Counseling.

Journal of Counseling Psychology, 1957, 4, 313—317 .

Guttentag, M., and Struening, E. L. Handbook of Evaluation Research

(Vol. 2). Beverly Hills, Calif.: Sage, 1975.

Guttman, M. A. 7., and Haase , R. F. Effect of Experimentally Induced Sets

of High and Low “Expertness” during Brief Vocational Counseling.

Counselor Education and Supervision, 1972, 11 , 171—178.

Haller, A. 0., and Miller, I. W . The Occupationpl Aspiration Scale:

Theory, Structure and Correlates. East Lansing, Mich.: Mi chigan

State University, 1963.

Hanson, 7. T., and Sander, D. L. Diffe rential Effects of Individual and

Group Counseling on Realism of Vocational Choice. Journal of Counseling Psychology, 1973 , 20, 541—544. -

Harris, 7. The Computer: Guidance Tool of the Future. Journal of Counseling Psychology, 1974, 21 , 331—339 .

Healy, C. C. Relation of Occupational Choice to the Similarity Between

Self-Ratings and Occupational Ratings. Journal of Counseling Psychology, 1968, 15, 317—323.

Scaly, C. C. Toward a Replicable Method of Group Career Counseling. Vo—

cationa l Guidance Quarterly, 1973, 21 , 214—221.

Scaly, C. C., Bailey, N. L., and Anderson, E. C. The Relation of Esteem

and Vocationa l Counseling to Range of Incorporation Scores. Journal

of Vocational Behavior, 1973, 3, 69—74.

41

• 

- - 

- 

~~~~~~~~~~~~~~~~~~~~~~~ - - ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ • -

~ -

•

- ~~~~~~~~~~~~~~~~~~~~~~~ -

~~~~~~~

-

-

~

.- ~~~~~~~~~~~~~ - - ~ gJ
I

Helliwell , C. C., and Jones, G. B. Reality Considerations in Guidance

Program Evaluation. Measurement and Evaluation in Guidance, 1975,

8, 155—162.

Hewer, V. H. Group Counseling, Individual Counseling, and a College Class

in Vocations. Personnel and Guidance Journal, 1959 , 37, 660—665.

Hewer, V. H. Evaluation of a Criterion: Realism of Vocational Choice.

Journal of Counseling Psycholoq,~~ 1966, 13 , 289—294.

Hill, A. H., and Grieneeks, L. Criteria in the Evaluation of Educational

and Vocational Counseling in College. Journal of Counseling Psychol-

~~~ , 1966, 13 , 198—201.

Hills, D. A., and Williams, 3. E. Effects of Test Information upon Self—

Eva luation in Brief Educational-Vocationa l Counseling. Journal of

Counseling Psychology, 1965, 12 , 275—281.

H ilton, T. L. Using Measures of Vocational Maturity in Evaluation. In

D. E. Super (Ed.), Measuring Vocational Maturity for Counseling and

Evaluation. Washington, D.C.: National Vocational Guidance Association, 1974.

Holland, 3. L., and Baird, L. L. An Interpersonal Competency Scale. Educational and Psychological Measurement, 1968, 28, 503—510.

Holmes, 7. B. The Presentation of Test Information to College Freshmen.

Journal of Counseling~ Psychology, 1964 , 11 , 54-58.

Hoyt, D. P. An Evaluation of Group and Individual Programs in Vocational

Guidance. Journal of Applied Psychology, 1955 , 39, 26—30.

Johnson, D. G. Eff ect of Vocational Counseling on Self—Knowledge. Journal of Educational and Psychological Measurement, 1953, t 3 , 330—338.

Jones, G. B., and Krumboltz , 3. D. Stimulating Vocational Exploration

through Film-Mediated Problems. Journal of Counseling Psychology,

1970, 17 , 107—114.

Kama, R. B., and Wrenn, C. G. Client Acceptance of Self—Information in

Counseling. Journal of Educational and Psychological Measurement,

1950 , 10 , 32— 42.

Katz, N. R. Assessment of Career Decision—Making: Process and Outcome.

Paper presented at American Institutes for Research Conference on

Career Decision—Making, New York, March 1975.

Kelley, 7., Smite, S. 3., Leventhal, R., and Rhodes, R. Critique of the

Design of Process and Outcome Research. Journal of Counseling Psychology, 1970 , 17 , 337—341.

42 J

1

_ 

~~~- - - :~ -

~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I

ICrivatsy, S. B., and Magoon, T. N. Differential Effects of Three Vocationa l Counseling Treatments. Journal of Counseling Psychology,

1976, 23, 112—118.

Kru mboltz , 3. D. Stating the Goals of Counseling. Monograph Series of

the California Personnel and Guidance Association, 1966 (No. 1).

Krumboltz , 3. D., and Schroeder , W . W. Promoting Career Planning

Through Reinforcement and Models. Personnel and Guidance Journal,

1965, 44 , 19—26.

Krumboltz , J. D., and Thoresen , C. E. The Effect of Behavioral Counseling in Group and Individua l Settings on Information—Seeking Behavior.

Journal of Counseling Psychology, 1964, 11 , 324—333.

Levin , H. N. Cost—Effectiveness Analysis in Evaluation Research. In

M. Guttentag and E. L. Struening (Eds.), Handbook of Evaluation

Research (Vol. 2). Beverly Hills, Calif.: Sage, 1975.

Linden, J. D., Stone, S. C., and Shertzer, B. Development and Evaluation

of an Inventory for Rating Counseling. Personnel and Guidance Journal, 1965, 44 , 267—276.

Lister, 7. L., and Ohisen, N. M . The Improvement of Self-Understanding

through Test Interpretation. Personnel and Guidance Journal, 1965,

43, 804—810.

Meihus, G. B., Hershenson, D. B., and Vermillion, N. B. Computer Assisted

Versus Traditional Vocationa l Counseling with High and Low Readiness

Clients. Journal of Vocational Behavior, 1973 , 3, 137—144.

Meltzoff , 3., and Kornreich, M. Research in Psychotherapy. New York :

Atherton , 1970. -

Mencke , R. A., and Cochran, D. J. Impact of a Counseling Outreach Work—

shop en Vocational Development. Journal of Counseling Psychology,

1974 , 21, 185—190.

Mitchell, A. H., Jones, G. B., and Krumboltz, 3. D. C Ede.). A Social

Learning Theory of Career Decision Making. AIR—47600—6/30-F R.

Palo Alto, Calif.: American Institutes for Research , 1974 .

Myers, R. A. Research on Educational and Vocationa l Counseling. In

A. E. Bergin and S. L. Garfield CEde.), Handbook of Psychotherapy

and Behavior Change: An Empirical Analysis. New York: Wiley,

1971.

Myers, R. A., Lindeman, R. H., Thompson, A. S., and Patrick, T. A.

Effects of Educ ational and Career Exploration System on Vocational

Maturity. Journal of Vocational Behavior , 1975, 6, 245—254.

_

~~~~~~~~~ r.L 

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -~~~~
I

Nunnally, 3. C. The Study of Change in Evaluation Research : Pr inciples

Concerning Measurement, Experimental Desi gn , and Ana lysis. In E. L.

Struening and H. Guttentag (Eds.), Handbook of Evaluation Research

(Vol. 1). Beverly Hills, Calif.: Sage , 1975.

Perrone , P. A., and Kyle , G. W. Evaluating the Effectiveness of a Grade

7—9 Career Development Program. Vocational Guidance Quarterly,

1975, 23, 317—323.

Pilato, C. T., and Myers, R. A. Effects of Computer-Mediated Vocational

Guidance Procedures: Accuracy of Self—Knowledge. Journal of Vocational Behavior, 1973, 3, 167—174.

Pilato, G. T., and Myers, R. A. The Effects of Computer-Mediated Vocational Guidance Procedures on the Appropriateness of Vocational

Preferences. Journal of Vocational Behavior, 1975, 6, 61—72.

Roe , A. Early Determinants of Vocational Choice. Jou nal of Counseling

Psychology, 1957, 4, 212—217.

Rogers, L. B. A Comparison of Two Rinds of Test Interpretation Interview .

Journal of Counseling Psychology, 1954 , 4, 224—231.

Rothenberg, J. Cost—Benefit Analysis : A Methodological Exposition.

In N. Guttentag and B. L. Struening CEde .), Handbook of Evaluation

Research (Vol. 2). Beverly Hills, Calif.: Sage, 1975.

Samaan , M . K., and Parker, C. A. Effects of Behavioral (Reinforcement)

and Advice—Giving Counseling on Information—Seeking Behavior.

Journal of Counseling Psycholog,

~~ 1973 , 20 , 193— 201.

Singer, S. L., and Stef fire, B. Analysis of the Self—Estimate in the

Eva luation of Counseling. Journal of Counseling Psychology, 1954, 1, 252—255.

Smith, R. D., and Evans, 1. R. Comparison of Experimental Group Guidance

and Individua l Counseling as Facilitators of Vocational Development.

Journal of Counseling Psycholoq,~~ 1973 , 20 , 202-208.

Struening, B. L., and Guttentag, H. Handbook of Evaluation Research

(Vol. 1). Beverly Hills, Calif.: Sage, 1975.

Super, D. B . The Psychology of Careers. New York: Harper & Row, 1957.

Swails, R. 0., and Herr , K. Vocational Development Groups for Ninth

Grade Students. Vocational Guidance Quarterly, 1976, 24, 256—260.

Thoresen, C. B., and Hamilton, 3. A. Peer Social Modeling in Promoting

Career Behaviors. Vocational Guidance Quarterly, 1972 , 20, 210—216.

- -

____

44 

__ 

~~~~~~~~~~~~~ 

• 

• 

- ~~~~~~~~~~~~~~~~~ ~~

-

- - ~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
I

I

Thoresen , C. E., and Kruntholtz, 3. D. Relationship of Counselor Reinforcement of Selected Responses to External Behavior. Journal of

Counseling Psychology, 1967, 14, 140—144.

Thoresen , C. B., and Krumboltz , 3. D. Similarity of Social Models and - - Clients in Behavioral Counseling. Journal of Counseling Psychology,

1968 , 15 , 393—401.

Thoresen , C. H., Hosford , R. B., and Xrumboltz , J. D. Determining Effec—

tiveness Models for Counseling Clients of Varying Competencies.

Journal of Counseling Psychology, 1970, 17, 369—375.

Thoresen, C. B., Krumboltz , 3. D., and Varenhorst, B. B. Sex of Counselors

and Models: Effect on Client Career Exploration. Journal of Counsel—

ing Psychology, 1967, 14, 503—508.

Tipton , R. H. Relative Effectiveness of Two Methods of Interpreting

Ability Test Scores. Journal of Counseling Psychology, 1969, 16,

75—80.

Ullrich, M . F. Several Measures of Expectancy of Vocational Counseling.

Psychological Reports, 1973, 33, 299—304.

Volsky, T. V., Jr., Magoon, T. N., Norman, W. T., and Hoyt, D. P. The

Outcomes of Counseling and Psychotherapy . Minneapolis: University

of Minnesota Press, 1965.

Wachowiak, D. G. Personality Correlates of Vocational Counseling Outcome.

Journal of Counseling Psychology, 1973, 20, 567—568.

Webb, B. J., Campbell, D. T., Schwartz, R. D., and Sechrest, L. Unobtrusive Measures: Nonreactive Research in the Social Sciences. Chicago:

Rand McNally , 1966.

Weinstein, A. S. Evaluation through Medical Records and Related Information Systems. In B. L. Struening and N. Guttentag (Eds.), Handbook

of Evaluation Research (Vol. 1) . Beverly Hills, Calif.: Sage , 1975.

Westbrook , B. W. The Eff ect of Test Reporting on Estimates of Scholastic

Ability and on Level of Occupational Aspiration. Journal of Educational Research, 1967, 60 , 387—390.

Westbrook , F. D. A Comparison of Three Methods of Vocational Counseling.

Journal of Counseling Psychology, 1974 , 21 , 502—506.

Williams, 3. E. Changes in Self and Other Perceptions Following Brief

Educational-Vocationa l Counseling. Journal of Counseling Psychology,

1962 , 9, 18—28.

Wil liams, .7. E., and Hills, D. A. More on Brief Educational—Vocational

Counseling. Journal of Counseling Psychology, 1962, 9’ 366-368.

45
Williamson, B. C. , and Bordin, B. S. The Evaluation of Vocational Educa—

tional Counseling: A Critique of the Methodology of Experiments.

Educational and Psychological Measurement, 1941 , ! , 5—24.

Winer, B. 3. Statistical Principles in Statistical Design (2nd ed.).

New York: McGraw—Hill , 1971.

Wri ght, B. V. A Comparison of Individual and Multiple Counseling for

Test Interpretation Interviews. Journal of Counseling Psychology,

1963 , 10, 126— 134.

Zener, T. B., and Schnuelle , L. Eff ects of the Self—Directed Search on

High School Students. Journal of Counseling Psychology, 1976 , 23, 353—359.

Zytowskj , D. G., and Betz , E. L. Measurement in Counseling Research :

A Review. Counseling Psychologist, 1972 , ! , 72—86 .

- 

-

46 

y

’~~~-~~~~~~~~~

-_—•— 

~~~~~~~~~~~~~~~~~ jI,

~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -
DISTRIBUTION

ARt DIstribution List

4 OASD (M&AA) 2 HQUSACDEC, Ft Ord, ATTN: Library

2 HODA (DAMI-CSZ) 1 HOUSACOEC, Ft Ord, ATTN: ATEC—EX—E—Hum Factors

1 HODA (DAPE.PBR 2 USAEEC, Ft Benjamin Harrison, ATTN: Library

1 HODA (DAMA’AR) 1 USAPACDC. Ft Benjamin Harrison, ATTN: ATCP—HR

1 HODA (DAPE•HR~~PO) 1 USA Comm—Elect $ch, Ft Monmouth, ATIN: ATSN—EA

1 HODA (SGRO-iD) I USAEC, Ft Monmouth, ATTN : AMS EL—CT—HDP

1 HODA IDAMI•DOT-C) 1 USAEC. Ft Monmouth, ATTN: AMSEL—PA—P

I HODA (DAPC-PMZ•Aj 1 USAEC, Ft Monmouth. ATI’N: AMSEL—SI—C8

1 HQDA (DACH-PPZ-A) 1 USAEC, Ft Monmouth, ATTN: C, Fad Des Br

I HODA (DAPE.HRE) - 1 USA Materials Sys Anal Agcy, Aberdeen. ATTN: AMXSY—P

I HODA IDAPE-MPO-CI 1 Edgewood Arsenal, Aberdeen, ATTN: SAREA— BL—H

1 HQDA IDAPE-DW) 1 USA Ord Ctr & Sch Aberdeen, ATTN: ATSL—TEM—C

1 HODA (DAPE.HRL) 2 USA Hum Engr Lab, Aberdeen, ATTN: Library/Dir

I HQDA (DAPE-CPS) 1 USA Combat Arms Tng Bd, Ft Banning, ATTN: Ad Suparviaor

I HODA (DAFO-MFA) 1 USA Infantry Hum Rich UnIt, Ft Banning, ATTN: Chief

— 1 HODA (DARD-ARS-Pl 1 USA Infantry Bd. Ft Binning, ATTN: STEBC—TE--T

1 HODA (DAPC•PAS-Al I USASMA, Ft Bliss, ATFN: ATSS—LRC

1 HODA (DUSA•OR) I USA Air Dii Sch, Ft Bli ss , ATIN: ATSA—CID—ME

1 HODA (DAMO-ROR) 1 USA Air Del Sch, Ft Blite , ATTN: Tech Lib

1 HODA (DASGI 1 USA Air Def Bd, Ft Bliss, ATTN: FILES

1 HODA (DAIO.Pl) 1 USA Air Del Sd, Ft Bliss, ATTN: STEBD—PO

1 Chief . Consult Div (DA-OTSG), Adeiphi, MD 1 USA Cmd & General SrI College, Ft Leavenworth, ATTN: Lib

1 Mu Mat. Hum Ru , ODDR&E. OAD (E&LS ) 1 USA Cmd & General Stt College, Ft Leavenworth , ATTN: ATSW—SE—L

I HO USARAL, APO S.att(e, A1TN: ARAGP-R 1 USA Cmd & General Stf College, Ft Leavenworth, ATTN: Ed Advisor

I HO First Army, AT1’N: AFK A-OI-TI 1 USA Combined Arms Cmbt Dcv Act , Ft Leavenworth. ATTN: DepCdr

2 HO Fifth Army, Ft Sam ‘f ouston I USA Combined Arms Cmbt 0.. Act , Ft Leavenworth, ATTN: CCS

1 DIr, Army SrI Studies Olc, ATTN: OAVCSA (DSP) 1 USA Combined Arms Cmbt Div Act , Ft Leavenworth, ATTN: ATCASA

1 Ofc Chief of Sd, Studies Ofc I USA Combined Arms Cnsbt 0ev Act . Ft Leavenworth, A’fl’N: ATCACO-E

1 OCSPER . ATIN: CPS/OCP I USA Combined Arms Cnsbt Des Act, Ft Leavenworth, ATTN: ATCACC—CI

I The Army Lib. Pentagon, ATTN: RSB Chief 1 USAECOM. Night Vision Lab, Ft Belvoir , AT ’N: AMSEL—NV—SD

I The Army Lib , Pentagon, ATIN: ANRAL 3 USA Computer-Sys Cmd, Ft Belvoir , ATTN: Tech Library

1 Ofc, Asst Sect of the Army (R&D) I USAMERDC, Ft Belvolr , ATFN: STSFB—DO

1 Tech Support Ofc, OJCS I USA Eng Sch, Ft Belvoir , ATTN: Library -

1 USASA. Arlington, ATTN: IA RD-T I USA Topographic Lab, Ft Belvoir , ATFN: ETL—TD—S

1 USA Rich Ofc, Durham , AT1’N: Life Sciancas Dir I USA Topographic Lab, Ft Belvoir, ATTN: STINFO Canter

2 USARIEM , Netick , ATI’N: SGRD.UE-CA 1 USA Topographic Lab, Ft Belvoir, A’fl’N: ETL—GSL

1 USATTC. Ft Clayton, ATTN: STETC-MO-A I USA Intelligence Ctr & Set, Ft Huachuca, ATTN: CTD—MS

1 USAIMA . Ft Bragg, ATTN: ATSU-CTD-OM I USA Intelligence Ctr & Sch, Ft Huachucs, ATTN: ATS—CTD.-MS

I USAIMA, Ft Bragg, A’fl ’N: MarQuet Lib I USA Intelligence Ctr S Set, Ft Huachuca, ATTN: ATSI—TE

I US WAC Ctr & Set, Ft McClellan , ATm: Llb 1 USA Intelligence Ctr & Sch, Ft Huachuca , ATTN: ATSI—TEX—GS

1 US WAC Ctr & Set , Ft McClellan , ATTN: Tog Dir I USA Intelligence Ctr & Sch, Ft Huachuce, ATrN: A’TSI—CTS--OR

1 USA Ouavtarmeetar Set, F: Lea, ATTh: ATSM-TE 1 USA Intelligence Ctr & Sch, Ft Huachuce, ATTN : ATSI—CTD—DT

1 Intelligence MaterIal Des Ofc. EWL . Ft Holabird I USA Intelligence Ctr & Sch, Ft Huachuca, ATTN: ATSI—CTD—CS

1 USA SE Sign.l Sets , Ft Gordon , ATTN: ATSO.EA I USA Intelligence Ctr & Set, Ft Huechuca, ATT’N: DAS/SRD

1 USA Chaplain CarS Sets . Ft Hamilton , ATTN: ATSC-TE.RD 1 USA Intelligence Ctr & Sets, Ft Huachuca, AITh: ATS1—TEM

I USATSCH, Ft Eustle. ATIN: Educ Advisor I USA intelligence Ctr & Set, Ft Huaclsuca, ATrN: Library

I USA War College, Carlisle Barracks, AT1’N: LIb I CDR , HO Ft Huachuci, ATTN: Tech Ref DIv

2 WRA IR, Nauroptychiatry Die 2 CDR, USA ElectronIc Prvg Grd, ATIN: STEEP—MT—S

1 DLI, SDA. Monterey 1 COR. Project MASSTER, A’TTN: Tech info Canter

I USA Concept Anal Agcy, Bethe.da. ATTN: MOCA.WGC I Hq MASSTER , USATRADOC, LNO

I USA Concept Anal Agcy, Bethesda, ATTN: MOCA.MR I Research InstItute, HO MASSTER , Ft Hood

I USA Concept Anal Agcy. B.thaeda, ATTN: MOCA.JF I USA Recruiting Cmd, Ft Sherdlan, ATFN: USARCPM—P

1 USA Artic That Ctr, APO Seattle, ATm: STEAC.MO’ASL 1 Senior Army Mv., USAFAGOD/TAC, Elgln AF Aux FId No. 9

I USA Arsic Test Ct,, APO S.attJe, ATT’N: AMSTE’PL-TS I HO USARPAC, OCSPER, APO SF aeme, ATTN: OPPE-SE

1 USA Ann.nsent Cmd, Radetona Arsenal, ATm : ATSK’TEM 1 Stlmeon Lib, Academy of Health ScIences, Ft Sam Houaton

I USA A,memanl Cmd, Rock Island, ATTN: AM$AR.TOC I Marine Corps Inst., A’T7N: Dean-MCI

1 FAA.NAF EC, Atlantic City, ATTN: Llbraay I I4OUSMC, Commandant, ATTN: Code MTMT 51

I FAA.NAFEC, Atlantic City, ATm: Hum Engr Br I HOUSMC, Commendint, AT1’N: Coda MPI—20

I FAA Aeronautical Ctr , Okietsomi City, ATTN: AAC.440 2 USCG Academy, New London, ATTN: AdmIssion

2 USA FM Any Sets, Ft Sill, AT TN: LIbrary 2 USCO Academy, New London , ATTN: Library

1 USA Armor Seth, Ft Knox, AT TN: LIbrary I USCG Training Ctr, NV , ATTN: CO

I USA Armor $d~

, Ft Knox, A’fl’N: AT$S.DI.E I USCG Training Ctr, NV, ATTN: Educ Sac Ole

I USA Armor Sets, Ft Knois, A1’Th: ATS5.DT.TP I USCG, Psychol Rae Sr. DC, ATTN: OP 1/02

I USA Armor Sals, Ft Knox, ATTN: AT$S-CD’AO I HO MM-Range Br, MC Det~ Gumntico, A1’TN: PU Dlv

~~

‘

- -

~~~ - - - - - ~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 

~~~~~ ~~
- - - -

1 US Marine Corps Liaiiion Of C , AMC, Alexandria, ATTh: AMCG$—F I Del & Civil Inst of Enviro MedIcine, Canada

I USATRADOC, Ft Monroe, AT1’N: ATRO--ED I MR C~E1S, Kenalngtan, ATT’N: Info Sye Br

6 USATRADOC, Ft Monroe, ATIN: ATPR-AD I NIIltieiieytiOIOgiik Tjsnaste, Capeltagen

I USATRADOC. Ft Monroe, ATTN: ArTS-BA I Mllltwy Attedse, Francis Embasey, AIrN: Doe Sec

1 USA Forces Cmd, Ft McPherson, ATTN: LIbrary I Medeciti Chef, CE M.P.A,—A,senel, Toulon/Navel France

2 USA Aviation Test Sd, Ft Rucker, ATTN: STEBG—P0 I Prin Scientific Off, Appi Hum Eage Rids Div. MInidiry

1 L’SA Agey for Aviation Safety , Ft Rueker, ATTN: Libwny of Osfanee. New Delhi

I USA Agcy for Aviation Safety, Ft Rucker, ATTN: Educ Adviior I Firs Rids Ofc Library, AKA. lan aI Defense Forces

1 USA Aviation Sch, Ft Rucker, ATTN: P0 Drawer 0 I Minlstenia van Defanela, 000P/KL Afd Soclaal

1 HOUSA Aviation Sys Cmd. St Louis, ATI’N: AMSAV—ZDR Peytholuglidsa Zekan, The Hague, Netherlands

2 USA AvIation Sys Test Act., Edwards AFO, ATTN: SAVTE—T

1 USA AIr P.15th, Ft Bliss, ATTN: ATSA TEM

I USA Air Mobility Rseh & Des Lab, Moff.tt FM , ATYN: SAVOL—AS

1 USA Aviation Sets, Ru Tng MgI, Ft Ruekar, ATTN: ATS1’—T—RTM

1 USA Aviation Set, CO. Ft Rucker, ATTN: ATST—O—A

I HO, DARCOM , Alexandria, ATTh : AMXCO— FL I HO, DARCOM . Alexandria , ATTN: CDR

1 US Military Academy, West Point, ATm: SerIals Unit

US Military Academy, West Point, ATTN: Ole of Mlft Ldrslsp

1 US Military Academy, West Point, ATTN: MAOR

1 USA Standardization Gp. UK , FF0 NV , ATT’N: MASE—GC

1 Ofc of Naval Rich, Arlington, ATTN : Coda 462

~ Ofc of Naval Rich, Arlington, ATTN: Cods 468

I Ofc of Naval Rsch, Arlington. ATI’N: Code 450

1 Ofc of Naval Ascis, Arlington, ATTN: Coda 441

1 Naval Aerospc Med Ret Lab, Pensacola, ATrN: AcousSdt Div

1 Nasal Aerospc Med Ret Lab, Pensacola, ATTN: Code L5l

I Naval Aerospc Med Rat Lab. Pensacola, A’fl’N: Code LB

I Chief of NavPers , ATTN: Pers.OR

I NAVAIRSTA , Norfolk, A’fl’N: Safety Ctr

1 Nay Oceanographic, DC, ATTN: Code 6251, Charts & Tacit

I Center of Naval Anal, ATTN: Doc Cu

I NavAlrSysCos’n, A1TN: AIR—63 13C

t Nav BuMed,ATTN: 713

1 NavHalicopterSubSqua 2, FF0 SF 98601

1 AFHRL (FT) Wliluam AFB

1 AFHRL (TT ) Loswy AFB

1 AFHRL (AS) WPAFB,OH 2 AFHRL IDOJZ) Brooks AFB

I AFHRL (DOJN) Lackiand AFB

1 HOUSAF IINYSD)

1 HOUSAF (DPXXA)

1 AFVTG (PD) Randolph AFB

3 AMRL (HE) WPAFB, OH

2 AF Inst of Tech, WPAFB , OH , ATTN: ENE/SL I ATC jXPTD) Randolph AF8

I USAF AuroMed LIb, Brooks AFB (SUL— 4), ATTN: DOC SEC

1 AFOSR (ML). Arlington

1 AF Log Cmd. McClellan AFB, ATTh: ALC/DPCRS

I Air Force Academy, CO. ATTh: Dept of Sal Sits

S NavPers & Dav Ct,,San Diego

2 Navy Med Neuropayehiatric Reds Unit, San DIego

I Nay Electronic Lab, San Diego, ATTN: Pee Lab

1 Nay TrngCen, San Diego, ATIN: Coda 9000-Ub

1 NevPoetGraScts, Monterey, AiiM: Code BBAe

1 NavPoetGraScds, Monterey, ATTN: Code 2124

1 NavTrngEquipctr, Orlando, ATTN: Tacit Ub

I US Dept of L.bor, DC, ATTN: Manpower Admln

1 US Dept of Justice, DC, ATTN: Drug Enforce Admin

I Nat Bun of Standards, DC, ATTN: Coisiputer Info Section

I Nat Cleadng House for MM-Info, Rockalile

1 Denver Federal Car, Laltawood. AiiM: Sl,M

12 Defense Documentation Center

4 DIr Psyd,, Army Hg. Russell Olee~ Cesbeira

1 ScIentific Adesr, MIl Sd, Army Hg. RuseaB Ofae~ 

CapiSapre

I Hit and Air Attache, Auatrlen Embassy

I Centre di Rechucelsa Dee Facteure, Humelna deli Difisiss

Netlonale, Ivu

~els

2 Canadian Joint Staff Waitlnglon

I C/Air Staff, Royal Canadian AF. ATYN : Pars Bid Mel I,

3 ChIef, Canadian Del Rich Staff, A’T’TN: C/CRDS(W)

4 IvItlals Del Stuff , StItlit

_________ ~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~ - ~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~