The Efficacy and Cost Effectiveness of

Psychotherapy

October 1980

NTIS order #PB81-144610
Library of Congress Catalog Card Number 80-600161

For sale by the Superintendent of Documents, U.S. Government Printing Office

Washington, D.C. 20402 Stock No. 052-003 -00783-5
Foreword

This volume is a background paper for OTA’s assessment, The lmplications of

Cost-Effectiveness Analysis of Medical Technology. That assessment analyzes the

feasibility, implications, and usefulness of applying cost-effectiveness and cost-benefit

analysis (CEA/CBA) in health care decisionmaking. The major, policy-oriented report

of the assessment was published in August 1980. In addition to the main report and

this background report, there will be four other background papers: 1) a document

which addresses methodological issues and reviews the CEA/CBA literature, 2) a diagnostic X-ray case study, 3) 17 other case studies of individual medical technologies,

and 4) a review of international experience in managing medical technology. Another

related report was published in September of 1979: A Review of Selected Federal Vaccine and Immunization Policies.

Background Paper #.3: The Efficacy and Cost Effectiveness of Psychotherapy w a s

prepared in response to a request of the Senate Finance Committee. OTA established a

special advisory panel to assist in the development of this paper; in addition, OTA

consulted with the members of the advisory panel to the overall assessment and with a

group of ad hoc reviewers from various mental health fields. We are grateful for their

assistance.

JOHN H. GIBBONS

Director

Ill
Advisory Panel on The Implications of

Cost-Effectiveness Analysis of Medical Technology

John R. Hogness, Panel Chairman

President, Association of Academic Health Centers

Stuart H. Altman

Dean

Florence Heller School

Brandeis University

James L. Bennington

Chairman

Department of Anatomic Pathology and

Clinical Laboratories

Children Hospital of San Francisco

John D. Chase

Associate Dean for Clinical Affairs

University of Washington School of Medicine

Joseph Fletcher

Visiting Scholar

Medical Ethics

School of Medicine

University of Virginia

Clark C. Havighurst

Professor of Law

School of Law

Duke University

Advisory Panel on Psychotherapy

Jerome D. Frank

Professor Emeritus of Psychiatry

The Johns Hopkins University School

of Medicine

Donald F. Klein

Director of Research

New York State Psychiatric Institute

Beverly Benson Long

President

National Mental Health Association

Sheldon Leonard

Manager

Regulatory Affairs

General Electric Co.

Barbara J. McNeil

Department of Radiology

Peter Bent Brigham Hospital

Robert H. Moser

Executive Vice President

American College of Physicians

Frederick Mosteller

Chairman

Department of Biostatistics

Harvard University

Robert M. Sigmond

Advisor on Hospital Affairs

Blue Cross and Blue Shield Associations

Jane Sisk Willems

VA Scholar

Veterans Administration

Morris Parloff

Chief, Psychotherapy and Behavioral

Intervention Section

National Institute of Mental Health

Hans H. Strupp

Distinguished Professor of Psychology

Vanderbilt University

Gary R. VandenBos

Director, National Policy Studies

American Psychological Association

Thomas McGuire

Assistant Professor of Economics

Boston University
OTA Staff for Background Paper #3

Joyce C. Lashof, Assistant Director, OTA

Health and Life Sciences Division

H. David Banta, Health Program Manager

Clyde J. Behney, Project Director, Implications of Cost-Effectiveness

Analysis of Medical Technology

Leonard Saxe, Study Director-, Background Paper #3

Kerry Britten Kemp, Writer/Editor

Virginia Cwalina, Administrative Assistant

Shirley Ann Gayheart, Secretary

Nancy L. Kenney, Secretary

Contractors

Brian T. Yates, The American University

Frederick Newman, Eastern Pennsylvania Psychiatric Institute

OTA Publishing Staff

John C. Holmes, Publishing Officer

Kathie S. Boss Debra M. Datcher Joanne Mattingly
Contents

1. SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2. DEFINITIONS AND SCOPE OF REVIEW . . . . . . . . . . . . . . . . . . . . . . .

Theoretical Assumptions . . . . . . . . . . . . . . . . . . . . , . . . . . . . . . . . . . . . . .

Mental Disorders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Practitioners. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Delivery Settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Scope of This Report. ., . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3. METHODS FOR ASSESSING THE EFFECTIVENESS

OF PSYCHOTHERAPY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Measures of Psychotherapeutic Outcomes . . . . . . . . . . . . . . . . . . . . . . . . .

Measurement Criteria ., . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Measures From Patients. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Measures From Family and Friends . . . . . . . . . . . . . . . . . . . . . . . . . . .

Measures From Therapists. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Measures From Community Members. . . . . . . . . . . . . . . . . . . . . . . . .

Summary , . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Research Designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Therapy Versus No-Therapy Designs . . . . . . . . . . . . . . . . . . . . . . . . .

Therapy Versus Therapy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Therapy Versus “Placebo’’ Therapy. . . . . . . . . . . . . . . . . . . . . . . . . . .

Quasi-Experimental Designs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Program Evaluations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Difficulties in Conducting Research. . . . . . . . . . . . . . . . . . . . . ..., . .

Integrating Findings From Psychotherapy Research . . . . . . . . . . . . . . . . . .

Final Comments , . . . . . . . . . . . . . . . . . . . . . . . . . . . ..., . . . . . . . . . . . . .

4. LITERATURE ON THE EFFECTIVENESS OF PSYCHOTHERAPY. . . .

Scope of Review. . . . . . . . . . . . . . . . . . . . . . . . ..., . . . . . . . . . . . . . . . . ,

Efficacy Reviews. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Eysenck’s Reviews. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Meltzoff and Kornreich’s Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . .

Bergin’s Reviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Rachman’s Critique. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Luborsky, Singer, and Luborsky’s Review. . . . . . . . . . . . . . . . . . . . . .

NIMH Synthesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Smith, Glass, and Miller’s Meta-Analysis . . . . . . . . . . . . . . . . . . . . . .

Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5. METHODS FOR ASSESSING THE COST EFFECTIVENESS AND

COST BENEFIT OF PSYCHOTHERAPY. . . . . . . . . . . . . . . . . . . . . . . . .

Cost Assessment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

General Considerations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Methods for Cost Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Discussion of Cost Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Benefit Assessment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

3

9

10

13

14

16

17

21

22

22

22

23

23

24

24

25

26

26

27

27

29

29

31

33

37

37

38

38

40

41

42

43

44

45

47

51

52

52

53

54

54
Methods for CEA/CBA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Cost-Effectiveness Analysis. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

Cost-Benefit Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

Summary . . . . . .. .. .. .. ... . .$....... .. . $ . . . . . . . . . . . . . . . . . . . . 59

6. ILLUSTRATIVE COST-EFFECTIVENESS AND COST-BENEFIT

ANALYSES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

Treatment-Related Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Inpatient Therapy.. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

Community-Based Versus Institutional Therapy.. . . . . . . . . . . . . . . . 6 5

Residential Versus Institutional Therapy for Problem Children . . . . . . 67

Therapist Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Professional Therapists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Nonprofessionals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68

Drug Addiction and Medical Utilization . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Psychotherapy for Drug Addictions . . . . . . . . . . . . . . . . . . . . . . . . . . 70

Psychotherapy and Overutilization of Medical Services . . . . . . . . . . . 72

Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

APPENDIXES

A. Description of Other Volumes of the Assessment . . . . . . . . . . . . . . . . . . . . 7 7

B. Health Program Advisory Committee, Acknowledgments . . . . . . . . . . . . . 79

REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
Summary
Summary

The treatment of mental, emotional, and behavioral dysfunctions has become one of the

most controversial areas of health policy. Even

though the prevalence and the pernicious effects

of mental disorders are well known and have

been documented recently in sources such as the

report of the President’s Commission on Mental

Health (219), opinion about what should be

done to treat these problems is not unanimous.

A number of proposals to expand the mental

health services system and to make treatment

more widely available (e. g., through expanded

insurance coverage) have been made. Yet, there

is not agreement about how to expand mental

health services, nor about what would be gained

by their expansion.

In light of these disagreements, it is perhaps

understandable that policy makers have been reluctant to commit additional public resources to

mental health treatments. To make psychotherapeutic treatments more widely available is potentially expensive. It has been suggested, however, that the provision of psychotherapy reduces other costs currently borne by society. A

central question is the extent to which psychotherapy can be scientifically assessed and its

value demonstrated in a way useful for policymaking. The present report analyzes the current

scientific literature on the evaluation of psychotherapy. It examines the efficacy and cost effectiveness of psychotherapeutic treatments and

considers both the methodological problems of

assessing psychotherapy and the state of current

knowledge about its effects. It is hoped that this

report will inform the developing congressional

debate on Federal research and funding for

psychotherapy.

As a background paper prepared in conjunction with OTA’s assessment The Irnplications of

Cost-Effectiveness Analysis of Medical Technology (see 203), the present report emphasizes

methodological issues related to the assessment

of psychotherapy and the use of cost-effectiveness/cost-benefit analysis (CEA/CBA) to evaluate psychotherapy’s worth. It documents the

use of various evaluation strategies and considers the available evidence concerning psychotherapy’s efficacy and cost effectiveness. The

conclusions of the report relate both to the conduct of research on psychotherapy and the

status of present scientific knowledge about

psychotherapy.

The report discusses four issues centrally related to the evaluation of psychotherapy: 1) the

definition and complexity of psychotherapy; 2)

the degree to which psychotherapy is amenable

to scientific analysis and the availability of appropriate methods for studying psychotherapy;

3) the evidence as to psychotherapy’s efficacy,

including the results of analyses that synthesize

findings across studies; and 4) the appropriateness of CEA/CBAs of psychotherapy and the results of their application. Below, each of these

sections of the report is briefly summarized.

Chapter 2 reviews a number of definitions

and views of psychotherapy. It also attempts to

delimit the scope of this background paper.

Psychotherapy is not a simple treatment, and

part of the confusion about its effectiveness has

to do with the use of different views of what

comprises psychotherapy. In order to represent

the variety of contemporary therapy practices,

the present report adopts a relatively comprehensive definition of psychotherapy. Included

are treatments based on Freudian ideas about

psychodynamics, as well as newer therapies

based on behavioral theories of learning and

cognition. One finding of the report is that

psychotherapies are not distinguishable solely

by their theoretical bases. In addition to the

view of psychopathology adopted by the therapist, therapist variables (e.g., training and personality characteristics), patient variables (e. g.,

seriousness of condition), and the treatment setting (e. g., hospital, private office) affect the nature of psychotherapy treatments. Although the

inclusion of such complex factors makes the

analysis of psychotherapy more difficult, their

inclusion is necessary to adequately assess the

effects of psychotherapy.
Chapter 3 describes the scientific basis of efforts to assess the effectiveness of psychotherapy. Although the definition of psychotherapy

employed in this report is complex, and perhaps

open to dispute, the methods for assessing psychotherapy are better established. The present

report describes and analyzes various methodological strategies for measuring the outcomes of

psychotherapeutic treatment and the ways in

which the reliability and validity of these measures are established. It also describes the variety

of research design strategies that have been used

to establish cause-effect relationships between

psychotherapy and particular outcomes. The

discussion indicates the conditions under which

randomized control group procedures can be

used to assess psychotherapy. Also analyzed are

the use of quasi-experimental and nonexperimental procedures which, depending on what

types of information are required, may provide

useful data. The problems of carrying out

psychotherapy research are also discussed, including the difficulties of withholding treatment

from members of a control group and the

problems of assessing multifactor treatment

programs.

Also considered in chapter 3 are two recent

methodological developments. One is the conduct of program evaluation studies, in which

sets of psychotherapy variables are investigated

simultaneously. Such evaluation studies may be

useful where several psychotherapy-related variables are considered in conjunction with one

another and where it is difficult to separate

treatment components. Another recent methodological development is the use of systematic

procedures for synthesizing the findings of

multiple investigations. These data integration

methods are new and somewhat controversial

procedures for assessing the implications of the

psychotherapy research literature. Both of these

methods, along with scientifically rigorous studies of psychotherapy, may prove useful to developing mental health policy.

Chapter 4 selectively describes the substantive literature on psychotherapy’s effects. A

number of prominent reviews of the psychotherapy literature, and the commentary generated

by these reviews, are analyzed (along with several individual evaluative studies), Despite some

fundamental differences, both in the criteria for

assessing psychotherapy and in the studies included, the reviews all report—under specified

conditions—evidence for psychotherapy’s effectiveness. This finding is stronger the more recent

the literature that is reviewed. In fact, there

seems to be little negative evidence as to the efficacy of psychotherapeutic treatments. Although

it is difficult to make global statements, the evidence seems more supportive of psychotherapy

than of alternative explanations (e.g., spontaneous remission, placebo effects). The available

research, some of which meets rigorous methodological standards, seems to indicate that

psychotherapy treatment is clearly better than

no treatment. However, while the literature supports a generally positive conclusion with respect to the effectiveness of psychotherapy,

there is a lack of specific information about the

conditions under which psychotherapy is effective. It is not clear which aspects of therapy

(e.g., treatment protocol v. the nature of the

therapist’s relationship with the patient) are responsible for particular outcomes.

Methods for assessing the costs and benefits

of psychotherapy and for developing CBAs are

described in chapter 5. The application of CEA/

CBA to psychotherapy is much more recent,

and less developed, than efficacy research. Although the methods for CEA/CBAs of psychotherapy are based on applications of such analyses to other types of health and nonhealth problems, in some instances (e.g., application of the

“willingness-to-pay” concept), the translation to

psychotherapy is difficult. An additional problem with psychotherapy assessments has to do

with the comprehensiveness of cost and benefit

assessment—in particular, our ability to value

in pecuniary terms the effects of psychotherapy.

Because of this difficulty, much of the recent

cost analysis research has involved cost-effectiveness comparisons rather than cost-benefit

comparisons. Potentially, however, both CEA

and CBA techniques may be useful to improve

our understanding of the effects of psychotherapy and the resources necessary for its efficient

use. Such CEA/CBA research may be an important adjunct to effectiveness studies that will

enhance their policy use.
C/I. 1—Summary ● 5

Chapter 6 reviews the available literature on

the use of CEA/CBA for assessing psychotherapy. The literature which reports actual CEA/

CBA studies varies, both in terms of its focus on

different problems and its methodological adequacy. Because of the problem of valuing psychotherapy outcomes, a great number of the

available cost studies focus on low-cost treatments. This may create an incorrect impression

about psychotherapy’s effects. In terms of their

limitation of problem focus, many of these analyses have been conducted on treatment settings

characteristics (e. g., institutional v. noninstitutional care), rather than on different therapies

or therapists. The findings of these studies seem

to indicate that more efficient ways of delivering

psychotherapy can be developed. Unfortunately, it is difficult to interpret the results of many

of these studies, because they ignore important

costs or benefits or because they use inadequate

research designs. In some cases (e. g., the effects

of psychotherapy on medical utilization rates),

at least the potential for psychotherapy to provide society with large net benefits has been

demonstrated. It will be necessary, however, to

validate these findings using better procedures

and a wide range of mental health problems.

In summary, OTA finds that psychotherapy

is a complex—yet scientifically assessable—set

of technologies. It also finds good evidence of

psychotherapy’s positive effects. Although this

evidence may not be generalizable to the wide

range of problems for which therapy is employed, it suggests that additional research may

provide data useful for the development of mental health policy. Given the potential net benefits of psychotherapy, this effort would seem to

be justified.
2m

Definitions and Scope of Review
2

Definitions and Scope of Review

The term psychotherapy has been used to refer to a wide variety of treatments employed to

ameliorate mental distress, mental illness, and

problems of coping with daily life, Psychotherapy is a global term, and one commentator has

found at least 40 different definitions in the

scientific literature (300). The definitions range

from broad and inclusive descriptions of various ways of helping other individuals to more

limiting definitions which are very specific as to

the nature of the problem and treatment (see,

e.g., 118). Differences in the definitions used by

those who discuss psychotherapy are a likely

reason that discussions of psychotherapy have

been so difficult to understand and evaluate.

The present review attempts to specify the nature of psychotherapy, so as to avoid some of

these problems.

To illustrate some of the difficulties of different views of psychotherapy, consider a broad

definition such as “the treatment of emotional

and personality problems and disorders by psychological means” (144). This definition refers

to treatments that may have either a good or

poor scientific basis, as well as to treatments

that are delivered by either professionally

trained or untrained personnel. In contrast, a

more specific definition (which has been favored

by some researchers) describes psychotherapy

in terms of its techniques, the qualifications of

the therapist, and nature of the patient’s problem (181):

Psychotherapy is taken to mean the planful

application of techniques derived from established psychological principles, by persons qualified through training and experience to understand these principles and to apply these techniques with the intention of assisting individuals

to modify such personal characteristics as feelings, values, attitudes, and behaviors which are

judged by the therapist to be maladaptive or

maladjustive.

Apart from their level of specificity, definitions vary with regard to what is included as the

disorders to be treated by psychotherapy. Although some definitions, like the above, are not

specific as to the nature of the disorder (see,

e.g., 84), diagnosis is becoming more systematic; viz, the newly developed Diagnostic and

Statistical Manual of Mental Disorders (63).

The new diagnostic manual includes disorders

that range from organic brain syndromes, schizophrenic and paranoid disorders, to affective

disorders such as depression, anxiety, and disorders such as phobias or posttraumatic stress;

also included are behavior aberrations such as

kleptomania or pathological gambling, personality disorders, alcohol- and drug-related problems, and interpersonal difficulties such as marital or family disturbances or education and

work blockages.

One additional feature of the various approaches to defining psychotherapy is that definitions vary in terms of their underlying theoretical assumptions. These assumptions lead to

differences in defining the goal of psychotherapy, the basic techniques, and the role of the patient and therapist. As Hogan (118) has noted,

psychotherapy can refer to a wide variety of

vaguely defined processes. Attempts to specify

the nature of psychotherapy result, in part, in

definitions that vary in terms of the practitioners and settings which are included as acceptable. Different definitions require different levels of professional training and experience for

psychotherapy practitioners. For example, medical definitions of psychotherapy (see 163) may

limit primary responsibility for psychotherapy

treatment to physicians, while nonmedical models (e. g., 250) may suggest central roles for psychologists, social workers, and nurses, as well

as educational and pastoral counselors. The

various forms of psychotherapy are further affected by the variety of settings in which therapy takes place (e. g., psychiatric hospitals, outpatient clinics, private offices) and the variety of

modalities that are offered (e. g., individual,

family, or group counseling).

In order to adequately summarize psychotherapy research and practice, it is necessary to

Y
10 ● Background Paper #.3: The Efficacy and Cost Effectiveness of Psychpytherapy

reflect components of a number of definitions of

psychotherapy. In this report, the definition of

psychotherapy is limited only in the sense that

an effort is made to view psychotherapy as a

field that is both scientific and professional.

Thus, our definition will emphasize therapies

that have an established scientific base and that

are delivered by trained professionals. In addition, although it seems unnecessary (as well as

difficult) to differentiate between levels of

psychopathology, this report is most concerned

with those disorders that cause severe dysfunction. Although psychotherapy may be useful in

THEORETICAL ASSUMPTIONS

When discussing psychotherapy, one typically refers to differences between treatments based

on different underlying theoretical assumptions.

There is a wide variety of beliefs about people

and pathology, and this has resulted in the development of a number of different psychotherapeutic treatments. The variety of psychotherapeutic treatments, at least in terms of the number of labels used to describe psychotherapies,

is somewhat bewildering. Nevertheless, many

treatments have similar origins, and it is possible to examine the range of therapies by considering only a few major theoretical perspectives.

For present purposes, therapies based on psychodynamic, humanistic/ phenomenological,

and behavioral theories are discussed as the perspectives that underlie most current practice.

These theoretical perspectives do not exist in

pure form, but they illustrate the most important differences between treatment approaches

(see 259).

Psychodynamic. —The psychodynamic perspective, which grew out of the classical psychoanalytic theory of Freud (88), is the oldest

and the most complex of the three orientations.

Although psychoanalytic theory cannot be described here in detail, the role of unconscious

motivation, one aspect of Freud’s theory, can

provide a basis for understanding the nature of

psychodynamic (also called analytic) treatments. The importance of unconscious motivation is central to all of Freud’s psychological inaiding individuals’ adjustment and improving

the quality of life, the primary concern here is

with problems that require some form of outside

intervention.

The following section briefly describes the

scope of psychotherapy practice, in terms of the

definitional variables described above. It is

hoped that this review will orient readers to the

central features of psychotherapy and will identify the central factors that must be accounted

for in an assessment of psychotherapy.

terpretations. The driving forces of an individual’s life are seen as originating in some inner

system of which he or she might, at the most, be

only vaguely aware. Most of the unconscious

cannot be called into conscious awareness; any

move in this direction meets with resistance, or

censorship.

According to Freud’s theory, some unconscious impulses come into conflict with environmental constraints and moral prohibitions.

To keep such threatening impulses from coming

into consciousness, part of the normal energy of

mental life is used to provide a constant defense,

or resistance, against the acceptance of such impulses. These impulses, however, are not eliminated by such a defensive reaction; instead, they

express themselves in indirect ways, leading to

behavior that the individual is sometimes unable to explain. Neurosis occurs when such behavior is especially prominent in the life of an

individual.

Thus, the goal of analytic psychotherapy is

gradually to promote the patient’s insight into

his or her underlying motivations and conflicts

by repeatedly working through the patient’s intellectual and emotional defensive resistances.

Acquisition of insight, along with affective understanding, is presumed to be a sufficient and

necessary condition for restructuring significant

aspects of the patient’s personality and for promoting the development of more adaptive behaviors in real life.
Ch. 2—Definitions and Scope of Review ● 11

The methods that use therapies based on the

psychodynamic approach are intended to aid

the patient in bringing unconscious material to

awareness. The core of therapy lies in transference. Freud suggested that patients inevitably

project into the therapeutic situation feelings

and attitudes from their past, and that they

sometimes reenact with the therapist important

situations and traumatic experiences repressed

since childhood. By becoming aware of these

unresolved conflicts, patients can master them

and liberate their energy to develop a mature

personality. Therapeutic techniques are designed so as to encourage and to analyze this

process of transference.

In Freudian psychotherapy, the therapist also

makes timely interpretations of the patient’s free

associations and dreams and deals with the patient’s resistances to the content of this unconscious material. Psychoanalytic therapy has

provided the model for most “verbal” therapies

(i.e., therapies based on conversations between

therapist and patient). Although infrequently

practiced in pure form, in part because many of

Freud’s students (e.g., Adler, Jung) developed

their own forms of therapy, psychoanalytic theory has had a major influence on other types of

psychotherapeutic treatments.

Humanistic/Phenomenological. —The second

major orientation that has influenced present

day treatment can be called the humanistic/

phenomenological perspective. The approach is

humanistic because it posits that the basic aim

of the individual is the achievement of personal

growth. Personal growth is variously conceived

of as self-actualization (e.g., 234,235,236),

meaning in life (e.g., 87), and cognitive complexity (e.g., 137). The perspective is also called

phenomenological because it holds that the

central focus of treatment should be an individual’s “phenomenal field. ” Thus, a person’s

perceptions about himself or herself and the

world (rather than unconscious thoughts) are

believed to be central to understanding. The

theory further posits that each person has a fundamental urge to preserve his or her self-concept

(i.e., perceptions about self).

This theoretical approach, like psychoanalytic theory, has influenced how psychotherapy is

practiced by a variety of different therapists. A

major form of therapy based on this perspective

is “client-centered” or “nondirective” psychotherapy. This therapy is grounded in the early

theoretical work of Rogers (234; see also 235,

236). According to Rogers, when individuals

have needs, impulses, or experiences that are

not consistent with their self-concept, tension

and anxiety result. More severe problems result

when individuals react with defensive mechanisms to avoid awareness of their contradictory

responses by ignoring important responses o r

by distorting their thinking about them. The

role of the therapist, according to Rogers, is to

bean empathetic responder.

A central tenet of client-centered therapy is

the belief that each individual has the resources

for growth. These resources merely need to be

released for the person to resolve a problem and

achieve maturity. Thus, therapy facilitates the

patient’s growth by promoting the free expression of feelings and by refraining from imposing

patterns and values. Out of the therapeutic relationship, patients (almost always referred to as

clients) evolve a new conceptualization of the

self. This view of themselves is more tolerant,

especially of their failings, and it resolves the

differences between their ideal image and their

actual perception of themselves. As a result,

anxiety and dysfunctional tendencies are lessened, and there is greater objectivity in the

handling of reality. The new self-image should

also be accompanied by a more harmonious expression of attitudes and feelings.

Therapists’ efforts to interpret, evaluate, or

guide the individual (which are central to analytic therapy) are felt by humanistic therapists

to hamper patients’ emerging sense of selfgrowth and self-direction. Techniques commonly employed in client-centered therapy are intended to facilitate individuals who seek help in

achieving their own insight. These techniques

include attentive listening to clients’ communications for content and feeling, reflecting or

verbally focusing clients’ feelings, and encouraging clients in their efforts to manage their own

problems.

Behavioral. —The third major orientation is

the behavioral perspective. It developed, at least
initially, from an extensive body of experimental research and theory associated with animal

and human learning research. Theorists of behavior such as Skinner (263,264,265), Pavlov,

and Hull—who were not psychotherapists or directly interested in the treatment of mental dysfunction—are most responsible for the ideas

that underlie this approach. A wide variety of

therapeutic approaches derived from these basic

research efforts have been developed by later

theorists such as Wolpe (302) and Bandura (11).

The basic assumption of behavioral theories

is that most persistent pathological behavior is

acquired through a process of learning. In essence, experiences in the individual’s life instigate patterns of behavior which remain unless

other conditioning experiences modify them.

Therapies based on behavior theory usually begin with systematic analysis of the problematic

situation and an identification of the specific

behavioral or situational factors to be modified.

Once it is known how and under what conditions these behaviors are maintained in the individual’s behavioral repertoire, the therapist interferes directly with this preestablished order.

Internal (cognitive) or external (environmental)

patterns of behavior contrary to the well-being

of the patient are modified by systematic retraining procedures.

Current clinical practice in behavioral therapy is derived from two laboratory-developed

theories of learning. The first, operant conditioning, is derived directly from research by

Skinner (e.g., 263). Because behavior is presumed to be lawful and predictable, behaviorists believe it is controllable once the relevant

reinforcing variables (those that maintain the

behavior) have been determined. Operant conditioning includes aversive counterconditioning

to eliminate undesirable behaviors and/or positive reinforcement to establish and maintain

new adaptive behaviors. Aversive counterconditioning entails pairing the patient’s behavioral

deviation with an unpleasant stimulus. For example, alcoholism has been treated by administering a substance to patients which would induce violent and uncomfortable physical reactions in the presence of alcohol. Positive reinforcement entails rewarding the patient for producing desired behaviors. For instance, positive

reinforcement has been used to treat autistic

children who are disassociated from reality and

unable to engage in gratifying interpersonal relationships. These children are rewarded with

verbal approval and candy when they show any

evidence of social responsiveness or interaction.

Another form of clinical practice, developed

by Wolpe (302), is reciprocal inhibition. This

therapy is an outgrowth of Pavlov’s classical

conditioning studies and of Hull’s learning

theory. The theory underlying reciprocal inhibition is that neuroses (in particular, phobias) are

characterized by persistent anxiety responses to

situations in which there is no objective danger.

The neuroses are produced when an individual

associates high intensities of anxiety and neutral

stimulus events. Therapy based on principles of

reciprocal inhibition systematically desensitizes

the patient to particular stimulus events. Thus,

a patient with neurotic anxiety is taught to relax. Then, in a state of relaxation, he or she is

asked to think about a mildly anxiety-provoking situation. This technique is then used with

increasingly more disturbing situations. Relaxation inhibits anxiety, and the patient is progressively able to master the situations previously

associated with anxiety reactions.

Although many of the behavioral therapies

rely directly on learning paradigms tested with

laboratory animals, a more recent trend (in

part, exemplified by Bandura’s work (11,12)) is

the development of cognitive behavior therapies. Within such therapies, laws of learning are

used as the basis for modifying cognitive processes (see 164, 180). The patient in cognitive behavior therapy takes an active role in identifying problems and working with the therapist to

develop mechanisms to control desirable or undesirable thoughts. Bandura’s (12) social learning theory is a related approach that emphasizes

the role of observing others (models) who are

reinforced or punished. There are also a variety

of eclectic behavior therapies (e.g., 154) which

employ a varying set of learning principles.

Common Factors.— Psychotherapy, as

should be clear from the above, is derived from

a range of assumptions, Although the three

theoretical perspectives described above under-
lie most forms of therapy, there is considerable

eclecticism in practice. A practitioner is likely to

hold some views or use techniques that are consistent with one dominant theoretical perspective and to hold other views and use other techniques that are quite inconsistent with it.

In part, the reason for this eclecticism is that a

number of important similarities exist across

different theoretical persuasions. Some theorists

such as Frank (84,86), in fact, argue that psychotherapeutic change is predominantly a function of factors common to all therapeutic approaches (e. g., 84). The primary ingredients of

such common, nonspecific, factors are the therMENTAL DISORDERS

Although it is true that one source of confusion about psychotherapies is the divergent

theoretical assumptions of therapists, it should

also be recognized that the nature of the problems dealt with by specific therapies varies

widely. Not only are a broad range of problems

considered psychopathological, but for any one

individual, the identification of a disorder is

often dependent on his or her own reports of

symptoms. In contrast to physical diseases,

which are usually accompanied by measurable

physiological changes, mental disorders must

usually be identified by a patient’s or another

person’s report of a problem. Notwithstanding

the many exceptions to this distinction between

physical and mental illness, the difficulties of diagnosis of mental illness are widely recognized.

An additional problem in diagnosis is that the

severity of a mental/psychological problem is

based on the context in which it occurs. A problem that may be diagnosed as a severe disorder

in one situation (e. g., at work) may be considered a minor disturbance in another context

(e.g., at home).

Widely varying phenomena have been considered mental disorders. Most therapists and

researchers seem to contend that any problem

which causes a patient discomfort and dysfunction is a mental health problem requiring treatment. For present purposes, the most important

aspect of the problem would seem to be the level

apist’s understanding, respect, interest, encouragement, and acceptance. Thus, while the contents and procedures of psychotherapy may differ across theoretical orientations, all forms of

psychotherapy share common “healing” functions. All therapists combat the patient’s demoralization and sense of hopelessness by the relationship they establish with the patient and by

providing an explanation for previously inexplicable feelings and behavior. According to

those who maintain that such nonspecific factors are responsible for psychotherapy’s effects,

one reason for the success of therapy is because

it removes the mystery from the patient’s suffering and supplants it with hope.

of dysfunction. Any person who is unable to

carry out his or her normal responsibilities is a

potential candidate for therapy. Level of dysfunction, however, is a difficult distinction to

make. An individual may appear to be functioning well, but may be extremely troubled. One

would want to alleviate such dysfunction and

prevent more serious dysfunction.

Below, the disorders treated by psychotherapy have been grouped within three general categories: neuroses, psychoses, and conduct disorders. While the most current diagnostic manual

(63) lists mental disorders somewhat differently,

for the purpose of describing the range of mental disorders, this organization seems more

appropriate.

Psychoses are severe disorders characterized

by grossly illogical thought patterns and distorted perceptions of reality. The psychotic individual may have prolonged periods of melancholy or elation and may exhibit extreme isolation or withdrawn behavior. The symptoms of a

schizophrenic disorder, the most prevalent type

of psychosis, include bizarre delusions, auditory

hallucinations, and incoherence. Schizophrenia

is accompanied by a deterioration in level of

functioning in areas such as work, social relations, and self-care.

Neuroses, the most common disorders, are

problems marked by feelings of anxiety, fear,
14 ● Background Paper #3: The Efficacy arid Cost Effectiveness of Psychotherapy

depression, guilt, and other unpleasant emotions. Anxiety, for example, is related to physical symptoms such as trembling, sweating, and

heart pounding. It may result in the individual’s

feeling continually worried and being overly attentive. The individual may feel impatient and

irritable, and may have insomnia. While

associated with discomfort, such anxiety may

not impair social or occupational functioning

more than mildly. In contrast, agoraphobia (an

irrational fear of crossing or being in open

spaces) causes individuals to avoid certain situations, and thus may interfere seriously with

social and occupational functioning. Neurotic

conflicts may also be the basis of other problems, such as divorce and child abuse.

Conduct disorders are characterized by a lack

of normal feelings and conscience that results in

behaviors which violate basic social rules. Conduct disorders are reflected by problems such as

uncontrolled impulses (e.g., outbursts of aggressiveness, failure to resist impulses to steal or to

set fires) and drug and alcohol abuse. Conduct

disorders among children and adolescents include such symptoms as physical violence

against persons or property (e.g., vandalism,

rape, mugging), chronic violations of rules

(e.g., truancy from school, alcohol or drug

abuse), and persistent lying and stealing.

In addition to the above disorders, there are a

variety of conditions which reflect interactions

between different types of problems. Most

prominent here are psychosomatic disorders,

where physical symptoms are thought to have a

psychological base (e.g., 188). Some have suggested (e.g., 152) that over 50 percent of the patients seen in general medical practice have psy -

PRACTITIONERS

As indicated above, the delivery of psychotherapy differs not only in terms of theoretical

assumptions and types of disorders treated, but

also in terms of the personnel who offer therapy. Therapists have an important influence on

the nature of psychotherapeutic treatment. Differences in therapists’ theoretical orientation, or

preferred technique, as well as differences in

chologically based problems. Conditions such

as asthma, colitis, anorexia nervosa, as well as

physical problems such as obesity and headaches due to stress, have been considered suitable problems for psychotherapeutic treatment.

The severity of any psychopathological disorder depends, in part, on the individual’s reaction. For example, whereas depression in some

individuals may be associated with poor appetite, insomnia, and loss of energy, in others it

may be associated with an attempted suicide.

Individuals also differ in the degree of subjective

distress they experience and the urgency with

which they desire and need help. Whereas for

some individuals, mild depression is so bothersome that they cannot function, others who

appear very depressed carry out normal lives.

Whereas some individuals who seem only mildly troubled feel in great need of therapeutic help,

others who have suffered major trauma are resistant to seeking help.

The diagnosis of a disorder, as well as its

seriousness, is thus inevitably part of a subjective decisionmaking process on the part of the

therapist. The therapist is usually responsible

for the diagnostic decision and is influenced

both by the patient’s report of the problem and

by objective evidence (e.g., data on the patient’s

inability to communicate with others). As described in the next section, particular therapists

may view these problems differently, some focusing on behavioral disorders, others on dysfunctional thoughts and emotions. These different emphases sometimes make it difficult to

compare effectiveness across psychotherapies

and psychotherapists.

their training, contribute to the difficulty of describing and assessing psychotherapy.

Professional therapists differ in a number of

ways, both related and unrelated to their training and disciplinary orientation. Thus, therapists may differ with regard to their underlying

assumptions concerning psychotherapy. They
Ch. 2—Definition and Scope of Review ● 15

may adopt a psychodynamic, humanistic/phenomenological, or behavioral perspective, or an

eclectic set of beliefs based on several major

orientations. Therapists also differ in the modalities and techniques they employ to provide

treatment. For example, practitioners may perform therapy mainly with individuals, groups,

or families; as well, they may focus more on

early events in the patient’s life or on recent

events, or lead a patient to insights concerning

behavior or feelings. These differences in approach are not clearly associated with the therapist’s professional training and identification.

The nature of the relationship that a therapist

establishes with a patient may be a particularly

important factor in the nature of changes caused

by psychotherapy. The patient reacts to the

therapist in a general way, not necessarily to

specific characteristics. Thus, therapists and researchers have been very concerned with the

therapist as a human being (see 274). Professional therapists obviously differ in terms of

their personal characteristics. Some of these

characteristics, such as sensitivity, empathy, activeness, and involvement, are seen by some

theorists (e.g., 86) as essential to successful

therapeutic outcomes. Therapists also differ

with regard to age, sex, cultural background,

ethnic factors, level of professional experience,

psychological sophistication, social and cultural

values, and cognitive styles. Any of these characteristics may have a significant effect on the

way in which psychotherapy is delivered. These

characteristics may interact with the practitioner’s theoretical perspective to produce a

unique therapeutic treatment.

For the purposes of this report, it seems important to distinguish professional from nonprofessional helpers. While it is sometimes difficult to make distinctions between what professionals and nonprofessionals actually do in

treatment settings, the professional usually has

a very different orientation and different responsibilities from those of the nonprofessional.

Obviously, persons with mental distress can be

helped by a variety of other individuals, including family and friends. These other individuals

may assist the patient in understanding his or

her problems and in making behavioral adjustments. The unique contribution of professionally trained therapists is important to specify.

The primary professionals who deliver psychotherapy are psychiatrists, clinical psychologists, and psychiatric social workers (see 118).

In 1977, there were approximately 28,000 psychiatrists professionally active in the United

States. Of this number, approximately 24,500

spend at least half of their time in patient care;

the others are primarily administrators, teachers, or researchers (5). Psychiatrists are medical

school graduates who have had a l-year internship and a 2- to 3-year residency in psychiatry

at an approved hospital or clinic. In addition,

they must have practiced their specialty for 2

years and be tested in order to qualify for certification by the American Board of Psychiatry

and Neurology.

One problematic aspect (103) to understanding the delivery of psychotherapy is that many

nonpsychiatric physicians also provide psychotherapy services. Thus, according to the National Institute of Mental Health (see 225), over

half of the patients seen for psychiatric problems are treated by primary care physicians. It is

important to emphasize, however, that while

many nonpsychiatric physicians are involved in

mental illness diagnosis and treatment, their involvement may be very limited (see 60).

In terms of psychologists, according to a 1976

membership survey by the American Psychological Association, there were approximately

23,000 psychologists engaged in mental health

activities. These psychologists worked in individual or group practice settings, mental health

centers, general or psychiatric hospitals, and

outpatient clinics. A 1977 survey (183) found

similar results, although it uncovered a slightly

higher number (approximately 26,000) of clinical psychologists (because nonmembers of the

Association were located).

Generally, clinical psychologists hold a doctoral degree (either a Ph. D. or a Psy. D.),

although some (less than 10 percent) hold only a

master’s degree. An integral part of the training

of clinical psychologists is a l-year internship in

a clinic or hospital setting approved by the
16 ● Background Paper #.3: The Efficacy and Cost Effectiveness of psychotherapy

American Psychological Association; in addition, clinical psychologists typically have

several years supervised experience, and Ph.D.s

receive training in research (which culminates in

a dissertation). A psychologist who has had at

least 5 years of postdoctoral experience may apply for advanced certification from the American Board of Professional Psychology.

A large number of psychotherapists are clinical social workers. In 1978, 10,922 social workers were included in the Register of Clinical

Social Workers (see 2); in addition, many other

social workers who are members of the National

Association of Social Workers practice psychotherapy. Psychiatric social workers have had 2

to 3 years of postgraduate training in a school of

social work which typically includes an internship. Psychiatric social workers with a masters

degree in social work and 2 years of experience

may take a test for certification by the Academy

for Certification of Social Workers.

Although the largest number of identifiable

psychotherapists are either psychiatrists, psychologists, or psychiatric social workers, various other professionals are trained in psychotherapy and administer it to patients. These

other professionals include psychiatric nurses,

DELlVERY SETTINGS

Variations in psychotherapy are also seen in

the diversity of settings in which therapy takes

place. These settings may have an important impact on the nature of psychotherapy. In some

cases, the setting influences the type of therapy

available to the patient. Settings may include

traditional inpatient mental hospitals, community mental health centers (CMHCs), private offices, residential treatment centers, and other

locations such as schools, offices, and military

facilities.

Prior to the establishment of CMHCs in 1963,

most psychotherapy was provided in mental

hospitals; certainly, these settings provided

much of the societally supported mental health

treatment. Although there is some controversy

pastoral counselors, educational psychologists,

and occupational therapists, as well as a number

of individuals from other disciplines. Nurses, in

particular, play an important role in the provision of psychotherapy. According to a 1972 survey (see 2), almost 38,000 registered nurses

reported working in a “psychiatric/mental

health area;” about 4,000 of them had master’s

degrees. These nurses and other professionals

have had unique disciplinary training, but they

have also completed internships with (or under

the supervision of) psychiatrists and psychologists, and they often provide treatment in combination with other professional therapists. To

assess the effects of psychotherapy, the role of

these professionals needs to be recognized.

In addition, the work of paraprofessionals

(i.e., psychiatric aides and trained volunteers)

should be noted. The Alcohol, Drug Abuse, and

Mental Health Administration (2) has estimated

that over 150,000 paraprofessionals work in

direct contact with mental patients. Many psychotherapy settings rely on the services of personnel who, though not formally trained as

therapists, have a great deal of contact with patients. Such paraprofessionals may play a role

in identifying patient problems and maintaining

therapy begun by the professional therapist

(4,65).

as to the relationship between the development

of CMHCs and deinstitutionalization of mental

hospital patients, there has been a sharp decline

in the number of hospital beds for mental patients. In 1962, psychiatric hospitals had a total

of 717,000 beds with 91.0-percent occupancy

(194); in 1978, there were only 219,517 beds

with 80.7-percent occupancy (121). Although

large numbers of the severely disturbed are still

hospitalized, their stays are generally much

shorter than was previously the case. The nature of treatment has also changed. Patients today have more frequent access to specific treatments (i. e., treatment other than being “housed”

in the hospital), including group therapy, individual therapy, and psychoactive drugs (i. e.,

substances that alter mental states). In addition,
patients are usually exposed to a number of different therapists (e.g., a psychiatrist who supervises their use of drugs, a psychologist who does

individual counseling, and a psychiatric nurse

or social worker who leads group therapy).

CMHCs and other outpatient services represent the fastest growing segment of the mental

health delivery system. Since their inception in

1963, CMHCs have been made available to almost 50 percent of the U.S. population. In 1975,

according to the National Institute of Mental

Health (197), patient visits to CMHCs totaled

1,961,000. As do outpatient mental hospitals,

CHMCs use a variety of mental health professionals, and a patient is likely to have exposure

to several modes of treatment. CMHCs tend to

emphasize short-term treatment, even though

they often deal with chronic patients.

A large segment of mental health treatment

takes place in private offices. Once available

only to the well-to-do, private practice has

flourished recently as public and private insurance coverage for mental health disorders

SCOPE OF THIS REPORT

As should be clear at this point, psychotherapy refers to a broad range of treatments used to

ameliorate a number of different kinds of conditions, by different therapists, and in a variety of

settings. The present report, whose goal is to describe the scientific basis for assessing psychotherapy, attempts to delimit its discussion in

two ways.

First, the report attempts to describe those

aspects of psychotherapy that have the most important implications for financing and reimhas become more widespread. It is estimated

that there are now 15,562 physicians, 4,70 0

psychologists, and 2,189 social workers engaged

more than half-time in the private practice of

psychotherapy (2).

Several other settings for psychotherapy are

worth noting. Outpatient mental health services

are growing in acceptance as part of medical

primary care. Health maintenance organizations, hospital-based outpatient clinics, and

even some small physician groups are including

mental health professionals on their staffs (e. g.,

26). In general, such services are used to supplement medical treatments and/or to reduce reliance on physical health care. Much psychotherapy also goes on in nonpsychiatric residential

treatment centers. For example, under some circumstances, homes for children with behavioral

and learning problems and prisons can be considered treatment centers. In addition, many organizations such as schools, industries, and the

military provide in-house resources to deliver

psychotherapy for members with mental health

problems.

bursement policy. This is reflected in an emphasis on severe dysfunctions (which are, potentially, the most costly) and an effort to distinguish

the applicability of psychotherapy to particular

problems. Second, the report attempts to describe the current range of practice and research

on psychotherapy by referring to the four

categories of variables described above. Thus,

the theoretical assumptions of therapy, and the

characteristics of patients, therapists, and

delivery settings serve as categories that need to

be considered in psychotherapy assessments.
3.

Methods for Assessing the

Effectiveness of Psychotherapy
3 ■

Methods for Assessing the

Effectiveness of Psychotherapy

Given the diversity of theoretical approaches

to psychotherapy, as well as the range of mental

health problems, therapists, and delivery settings, it is difficult to give simple answers to

questions about psychotherapeutic efficacy. Although the position adopted here will be that it

is possible to answer efficacy questions through

scientific research, such questions are undoubtedly difficult to answer. Both because of

the inherent nature of psychotherapy and because of the nature of the scientific research

process, answers to questions about psychotherapy will require a complex sequence of steps and

the adaptation of several research technologies

to the problem of psychotherapy.

Questions about psychotherapy are complex,

because, for policy purposes, they have been

stated very globally. It must be recognized that

general statements as to whether psychotherapy

is effective are necessarily equivocal and must

be tempered by information as to the specific

conditions under which particular treatments

are efficacious. The way in which the level of

specificity of one’s question affects research on

psychotherapy and the methodological considerations that affect the conduct and interpretation of efficacy research are described below.

Most research assessing the effectiveness of

psychotherapy has examined very specific issues. Which technique is more effective and how

effectiveness is moderated by differences among

patients, therapists, and settings are the typical

foci of psychotherapy outcome research (see

207,287). Much of this research shows that

some techniques are more effective than others,

although, unfortunately, no-treatment or placebo treatment control conditions are not always

included as part of these studies. Without such

comparison

the ultimate

to assess.

conditions, their implications for

effectiveness questions are difficult

Questions about the effectiveness of specific

types of psychotherapy usually deal with the

conditions under which therapy is provided.

Thus, the type of mental dysfunction, the characteristics of the patient, and the characteristics

of the delivery system are the central variables

being tested. Although this research can yield

generalizations to policy about psychotherapeutic treatment, the inherent limitations should be

recognized. Unless generalizability has been empirically established by tests conducted with a

range of patients in an actual treatment site, the

conclusions must be regarded as tenuous.

A number of methodological issues that arise

in assessing the efficacy of psychotherapy are

described in the following sections. The first

issue has to do with how one measures the outcomes of therapy. There is a substantial literature describing procedures for determining the

presence and strength of particular results of

therapy (see 293). This literature is examined

below in terms of the development of useful policy data about psychotherapy. A second set of

methodological issues concerns the design of

psychotherapy research and the confidence one

can have that obtained changes are a result of a

particular psychotherapeutic intervention. The

research design problems have to do with determining the reason for outcomes and organizing

research so that extraneous factors can be ruled

out as the cause of treatment effects (see 41,

105). Also considered below are the problems of

actually conducting psychotherapy research, including the ethical and pragmatic issues of employing random assignment procedures, A separate set of methodological problems having to

do with the synthesis and interpretation of multiple efficacy studies is also described. A number

of techniques have been developed for reviewing and integrating findings, and, potentially,

these methods may allow more definitive assessments of the psychotherapy literature.

21
22 . Background Paper #3: The Efficacy and Cost Effectiveness of psychotherapy

MEASURES OF PSYCHOTHERAPEUTIC OUTCOMES

Measuring the outcomes of psychotherapeutic treatment has been a major focus of psychotherapy theory and research for at least the last

two decades. Developing a technology for measuring outcomes is the first step in determining

psychotherapy’s efficacy. It involves decisions

about what variables are important to assess, as

well as the development of measurement techniques that can be used in actual treatment settings. Because psychotherapy takes a number of

forms (e.g., treatment based on different theoretical assumptions), much of the literature

deals with the selection of outcome measures

that are appropriate to the goals of the treatment under study. One recent focus has been the

development of measurement procedures that

can be used across different types of psychotherapy. Some of this literature has proposed

“batteries” of instruments (e.g., 293), whose intent is to capture (through the use of several

types of measurement procedures) the core

changes that result from any application of

psychotherapy.

Underlying the effort to develop such common measures is a belief that no one instrument

or set of procedures can measure all the outcomes of psychotherapy (21). Different psychotherapeutic techniques applied by a range of

therapists to various patient populations may

require different measures of outcome. Thus,

functional measures of behavioral effects, although perhaps appropriate to assess behavioral

therapies, might be inappropriate to assess psychodynamically based therapies. Similarly, cognitive measures might be seen as inappropriate

to evaluate behavioral therapies.

Notwithstanding the unique goals of particular therapies, there seems to be support for the

concept that many of the changes produced by

psychotherapy can be assessed along some common matrix. Probably, this implies the use of a

matrix that includes both behavioral and cognitive variables. Any single study of psychotherapy, thus, would incorporate a number of measures, not necessarily tied to the goals of the

therapy. The use of such multiple common outcomes also makes monitoring potential detrimental effects more possible.

not be detectable if unique

measures are used.

Such effects may

therapy-relevant

Measurement Criteria

To be useful in an effectiveness analysis,

measures of psychotherapy outcome must be

both reliable and valid (see 201). Reliability

means that the measure gives the same finding

over multiple uses (assuming no change in what

is being measured) and provides the same findings when used by different researchers. Validity means that the measure assesses the outcomes that it is supposed to measure and provides data that are generalizable (see, e.g., 48).

Neither reliability nor validity is intended as a

theoretical concept; each is established by pretesting the measuring instrument.

The use of reliability and validity criteria

results in particular measurement processes’ being validated to measure specific therapeutic effects. Thus, for example, a single instrument

might not be reliable and valid for assessing improvement in both depression and agoraphobia;

reliability and validity, at least, would have to

be separately established for each condition.

Moreover, outcomes have different meanings to

patients, the therapist, and interested third parties. Validity may depend on who’s perspective

one adopts in assessing the measurement instrument. Below, some of the differences in measures designed to collect data from various individuals affected by therapy are described.

Measures From Patients

Reports and ratings by patients of their behavior, thoughts, and feelings represent one

typical, and usually important outcome measure. Often, such data are collected on questionnaires or through interviews. These measures

structure verbal reports of the patient’s ability

to cope with various problems, and include

paper-and-pencil measures of personality and

adjustment (31). Although patient assessment

measures yield important data, they have obvious limitations, These limitations include
Ch. 3—Methods for Assessing the Effectiveness of Psychotherapy ● 23

social desirability effects (patients’ responding

to create a certain impression) and response bias

(e.g., positive responding to express appreciation to the therapist).

types of measures can

such response patterns.

One type of patient

self-report instrument

report on aspects of

Validation using- other

determine the effects of

measure is a functional

which asks patients to

their daily functioning

(123). Although narrative reports of functioning

may be collected as a part of any therapeutic

treatment, instruments that incorporate standardized questions have been used to systematize

this data collection. Such an instrument may include a series of structured questions about time

lost from work, feelings of guilt, and satisfaction with therapy. Careful pretesting of these

questions (e.g., by comparing responses of individuals known to be psychologically impaired

with those not impaired) yields a subset of these

questions. Responses can be quantified and

summed to form composite scores (total and

subset) of social adjustment and functioning. A

number of these instruments have been developed which show high reliability and validity

(see 123).

Another type of patient measure is represented by “psychometric” questionnaires and

personality tests, such as the Minnesota Multiphasic Personality Inventory (MMPI). Such instruments, which are perhaps the most common

type of measuring tool for both diagnostic and

assessment purposes, ask a variety of questions

about the respondent’s thoughts. Patients, for

example, those suffering from depression and

anxiety (210), have been shown to respond to

these questions according to particular patterns.

For MMPI, a subset of questions has been developed which tends to be answered differently by

those who are trying to fake responses and those

who are not; thus, social desirability and other

biased response patterns can be detected (see

55).

Measures From Family and Friends

Friends, work associates, and relatives of patients are often asked to supply information

about a patient’s functioning, and such data

concerning patient behavior and inferred mental

states have been used to assess therapy outcomes (e.g., 116; see, in particular, a report by

the National Institute of Mental Health, 293).

The value of these measures may be due to the

great amount of information that family,

friends, and work associates have about the

functioning of the patient. These individuals

have a chance to observe the patient in a variety

of situations and may thus have a great deal of

“data” on which to base their responses to questionnaires. Although these individuals have selfinterests which can bias the data, they may have

less motivation than the patient to reflect socially desirable responses and less need to show

gratitude to the therapist.

The outcome variables included on these

questionnaires also assess outcomes that may be

more important to “society” than the variables

included on patient self-report questionnaires.

For example, questions about how much disturbance the patient causes in family life or

work routines, or how often the patient has

secured employment or performed satisfactorily

in school are typically included in these instruments. Such questions may be more important

for societal evaluations of the usefulness of

therapy than questions about coping with daily

activities that are typical of self-report measures. These techniques are validated both by

comparing the observations of those who know

the patient against one another (e. g., family

members compared to work associates) and by

comparing these data with other available information about the patient.

Measures From Therapists

Measures taken from the therapist and others

involved in the therapeutic process are another

frequently used source of outcome data. While

the therapist can provide a first-hand perspective on the therapeutic process, such data may

be biased because the therapist has a vested interest in producing positive outcomes. Nevertheless, a substantial research literature on such

“clinical judgments” exists, and there is some

evidence that therapists can provide useful and

relatively unbiased reports. Particularly in

terms of assessments of patient functioning,

there is evidence that therapists can provide

valid data (e.g., 67,108,161,184,198).
24 ● Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

Generally, the more specific and concrete the

observations required of therapists, the greater

the resulting interrater reliability and validity

(186,187). These observations can range from

“counts” of behavior exhibited by patients during therapy to therapist ratings of the patient’s

functioning with his/her family (based, perhaps, on how the patient has reported family

relationships). To assess some therapies, where

the concern is more with mental phenomena

than behaviors, instruments have been designed

to capture nonobservable behavior. While such

instruments can provide one type of information about the effect of therapy, their usefulness may depend on how the goals of therapy

have been described and the availability of

validation data.

A focus on the specific behaviors of the patient may allow therapists to provide easily validated data about patient functioning; however,

the therapist’s desire to show improvements as a

result of therapy may bias the use of such measures. In an attempt to remove this bias from

functioning judgments, some researchers have

trained members of a therapy staff who are not

individuals actually involved in therapy to conduct such assessments (e.g., 209). A variety of

such “blind” data collection techniques have

been employed.

Measures From Community Members

Information on the outcome of psychotherapy can also be collected, more broadly, from

community members or agencies. These measures may include patient data on criminal arrest

rates, measures related to the patient’s work, or

rates of medical utilization (from nonpsychiatric

illness). The range of potential community

measures is very large and, though often unwieldy, seems to reflect important information

that is needed to assess adequately the effects of

psychotherapy. As is described later in the discussion of cost-benefit measurement (ch. 5),

such variables are important to assess in order

to conduct comprehensive cost-effectiveness

and cost-benefit analyses (CEA/CBAs).

Many of the data collected on community

variables can be assessed in relatively direct

ways. The information often can be gleaned

from existing court records, hospital charts, insurance claims, and similar records. To be utilized as part of an effectiveness assessment,

however, such measures must be tested for reliability and validity. Oftentimes, the validity of

record data is easy to establish because of its obvious relationship to desirable psychotherapy

outcomes (referred to as content validity). It

should be noted that much of this information

can also be collected directly from patients (e.g.,

data on medical utilization, if it can be shown

that patients report these data accurately).

Community data have also been taken to

mean a patient’s economic contribution to his or

her community (see, e.g., 305). Thus, the patient’s net monetary contribution in terms of

earnings and taxes may be used as one measure

of psychotherapeutic outcome (see ch. 5). These

outcomes have been assessed for psychotherapy

in a number of investigations

are described later in terms

psychotherapy.

Summary

(e.g., 51,114) and

of CEA/CBAs of

The outcomes of psychotherapy can be measured in a variety of ways. It is probably impossible to develop any single measure of outcome which would reflect the diverse changes

that might be brought about by psychotherapy.

At present, there exists a diverse set of procedures for eliciting information from patients,

family, therapists, and others. Data from each

of these sources, if properly assessed for reliability and validity, provide information needed to

assess psychotherapeutic effectiveness.
RESEARCH DESIGNS

The development of reliable and valid outcome measures represents only one part of the

assessment problem. One must be able to determine which of the many processes utilized in

therapy are responsible for improvements and

must also be able to test the possibility that nontherapeutic components are responsible for obtained effects. The particular research designs

and techniques used to test psychotherapeutic

efficacy are, in part, determined by what questions are being asked. These questions depend

on the goals of both the patients and the therapy

program. The present discussion focuses on the

possible ways of carrying out psychotherapy

outcome research and how questions about psychotherapy can actually be tested.

A basic element of good research design is to

frame the questions to be tested in a very specific way. Whereas the global question of psychotherapy’s efficacy may be obvious (i. e., “is

therapy effective?”), the questions asked of

research need to be more circumscribed

(21,207). Such circumscribed questions usually

develop as research progresses through a series

of stages (105). At a formulative stage, research

is based on extensive observation and summary

descriptions of these observations. Then, the

focus shifts to a description of patterns among

data elements. At a later stage, explanations and

theories of the observed patterns are formed. It

is usually these later explanations that are tested

in formal experiments and that represent the

bulk of the outcome research literature. Although effectiveness studies can be conducted

on a post hoc basis (i. e., a design constructed

after the data have been collected), such research usually can be interpreted only when

theory and formal experimental evidence are

available (e. g., 47,1.51 ).

The purpose of the present report is to understand psychotherapeutic effectiveness; thus, the

focus is primarily on data collected in actual

treatment settings (in vivo). A great deal of psy -

chotherapy research has been carried out under

laboratory or analog conditions, however, and

at least some researchers (e.g., 10) view the data

from this research as very important. To the extent that such data provide theoretical support

for in vivo findings, they are probably necessary to consider. OTA, in previous discussions

of the evaluation of medical technology (202),

has regarded this as efficacy rather than effectiveness research (ideal v. actual conditions).

When applied to psychotherapy, however, the

efficacy/effectiveness distinction seems ambiguous because of the variety of factors (therapist,

patient, setting) which affect the outcome of a

particular treatment and the absence of a clear

demarcation between laboratory and nonlaboratory conditions. It seems desirable in the case

of psychotherapy, instead of trying to differentiate between assessments of efficacy and effectiveness research, to note clearly the different

conditions under which research is conducted.

In terms of developing research tests of therapeutic effects, some limitations of prevailing scientific logic should also be noted (see 41), Testing particular hypotheses does not permit the

psychotherapy researcher to prove that a particular therapeutic effort causes a particular outcome; instead, one tests whether alternative explanations can be disregarded (47,105). Resulting inferences are probabilistic and indicate,

within identifiable error rates, the likelihood

that generalization can be made from the study’s

sample to other populations.

The inherent variability of human behavior,

thoughts, and feelings often makes the findings

of psychotherapy research equivocal. A great

deal of variability will exist under any conditions, and these variables must be separated

from the effects of the treatment. In addition,

because patients can be affected by many variables (i.e., factors other than psychotherapy)

that cannot be controlled, there may be numer-
ous alternative explanations for any apparent

improvements produced by psychotherapy. The

number of alternative explanations can be reduced somewhat by careful design of psychotherapy research and by the inclusion of control

conditions that hold all elements constant, except some aspect of a treatment. A variety of experimental designs have been developed to reduce the plausibility of a number of standard

alternative explanations. These research designs

are not unique to psychotherapy (see 27,41,

94,165), but their use in assessing the efficacy of

psychotherapy raises a significant set of unique

problems. Several types of research designs that

can be used to assess psychotherapy are described below.

Therapy Versus No-Therapy Designs

The classical research design, as applied to

psychotherapy, assigns patients either to receive

psychotherapy or not to receive psychotherapy

according to a random selection procedure. It

can be expected that randomization will distribute differences in patient characteristics (e.g.,

level of mental dysfunction, amenability to

treatment) equally between the psychotherapy

and no-psychotherapy conditions. Typically,

the psychotherapy condition is referred to as the

“experimental” condition, while the no-psychotherapy condition is referred to as the “control”

condition. The functioning of each subject in the

experiment is measured following psychotherapy (or following an equal interval of time for

control subjects). Although measures can be

taken at other times (e.g., pretherapy), additional design problems (having to do with the

effect of taking a test or responding to a questionnaire more than once) result with such

procedures.

The measurements of functioning (including

cognitive and behavioral outcomes) obtained

from the treatment condition contain three possible “effects:” 1) “effects” of treatment, 2) “effects” due to whatever nontherapy factors are

affecting the patient, and 3) “effects” of the

haphazard fluctuations in functioning measures

caused by imperfections in measurement instruments and the variability of behavior. The

measurements of patient functioning obtained

from the control condition are used to “subtract” the “effects” due to treatment from the

“effects” due to other factors. A control condition is necessary to perform the above “subtraction” because it provides the only empirical way

of knowing how nontherapy factors and measurement problems affected the outcome.

When this type of therapy versus no-therapy

research design is used, explanations of apparent improvements in patient functioning that

are actually due to factors other than therapy

itself can be rejected with reasonable confidence. As long as patients have been assigned

randomly to therapy and no-therapy conditions, it can be assumed (within identifiable

probability limits) that the obtained “effects”

are due to the psychotherapeutic treatment. If

patients are not randomly assigned, but are

“matched” on various characteristics, such an

unequivocal statement is not possible. It can be

argued that differences existed between experimental and control group subjects that were not

controlled by matching and that these characteristics are responsible for differences obtained

between therapy and no-therapy groups. The

absence of a control group makes such inferences about causal factors extremely difficult

to develop.

Therapy Versus Therapy

The basic rationale used to distill the effects of

psychotherapy from effects of measurement and

factors unrelated to therapy can be extended to

test for the superiority of one psychotherapy

over another. In such a therapy comparison

study, patients are assigned randomly to therapy A, therapy B, . . ., and, perhaps, to a notherapy or delayed-therapy group. In effect,

such a design results in the use of multiple treatment groups. The use of no-treatment control

groups is not ruled out, but such groups are

often not employed in therapy comparison studies because the purpose of such studies is the

assessment of the best therapy. It should be

noted that comparisons across therapies are

often made without use of an experimental research design. When that is done, patients who

have received different therapies are compared

without regard to the selection factors that in-
fluenced which patients received which therapy.

In such nonexperimental research, differences

may be due to a variety of factors (e.g., preexisting differences between patients in each

group), and these variables need to be controlled before such data are useful.

In another common design used to compare

therapies, two potentiality effective therapy

techniques are presented separately to two

groups of subjects and in combination to a third

group (yielding individual and combination

therapy conditions). Statistical analyses are

used to separate the effects of the therapies and

to compare them with no-therapy conditions. A

particular concern of the statistical analysis is to

test for interaction effects (e. g., where therapies

combine to produce an effect that is different

from the sum of the two effects alone). A variant of this type of design has been used to assess

the joint, as well as separate, effects of chemotherapy and psychotherapy (e.g., 142).

Therapy Versus “Placebo” Therapy

One problem that has plagued much psychotherapy research (as well as other research on

medical interventions) is that some of the effects

obtained by psychotherapy researchers may be

due to placebo effects. The “aura” of being in

therapy and the expectancy that one is finally

about to be “cured” may be a form of treatment

(136,256). The problem of separating these effects from those of formal therapies is analogous

to the use of sugar pills in controlled medical

research and involves the use of placebo-control

conditions. In such conditions, the patient may

receive attention from a therapist, but therapeutically meaningful discussion is avoided and no

specific techniques are used (e.g., 176). It is

easier to employ such control procedures when

testing the efficacy of behavioral, or even psychodynamic, therapies, in which the therapist

plays an active and directive role.

Probably, because of the nature of psychotherapy and the difficulty of specifying the

precise ingredients of therapy, it is impossible to

control all placebo effects (207,280). The relationship that a therapist establishes with a patient is acknowledged (e. g., 84) to be an important component of therapy, and it is not clear

how such effects should be distinguished from

treatments per se. It should also be recognized

that while placebo effects may inflate the true effects of therapy, they may often distort the data

obtained in control conditions. Few control conditions can be “pure” in the sense that patients

do not interact with a therapist. If for no other

reason than to monitor the patient, those in control conditions must usually be supervised by a

therapist. The effects of this supervision may

make therapy appear to be less effective.

Control conditions may be introduced to

assess the effects of such factors as therapist “demand characteristics. ” An ardent researcher or

therapist may communicate to patients what the

results of an experiment are “expected” to be

(e.g., 240). Experiments using se] f-report measures obtained from patients are especially prone

to this problem. To examine the strength of

these demand characteristics, psychotherapy

researchers can use designs in which patients are

told that therapy will produce a temporar y

worsening of functioning, when in actuality the

researcher expects no worsening. Alternatively,

the researcher may tell patients in some conditions that therapy should not be effective for

several months, when gradual improvement actually is expected. If the “demanded” effects are

not found in these conditions, the researcher can

have some confidence that demand characteristics are not causing therapy effects. Although

these controls may not be employed in every

outcome study, they are often used in developing research designs.

Quasi-Experimental Designs

For a variety of reasons, including ethical

problems of withholding treatment (see discussion below), the experimental designs described

above are often not employed. The real or perceived difficulties of assigning patients on a random basis to therapy or no-therapy conditions

make other types of comparisons necessary.

There are a number of quasi-experimental designs that can be used in such circumstances (see

41), Such designs are sometimes considered to

be poor substitutes for “pure” experimental

designs, but this may not be the best way to

view them. Quasi-experimental designs, if care-
28 ● Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

fully constituted, eliminate the most important

plausible alternative explanations for the results

of an experiment and can provide useful information. Quasi-experiments based on these

designs, however, may involve complex statistics and require the collection of many more

data (perhaps from fewer subjects) than a “true”

experiment would.

One common quasi-experimental design in

psychotherapy is referred to as an intensive

design (97). Typically, intensive designs use

fewer subjects than experimental designs, but

compensate by obtaining more measures on a

frequent basis and by obtaining clear “baseline”

measures of patient functioning. The intent is

that sudden improvements will be dramatic and

closely correlated with the onset of therapy. To

the extent that these changes are sudden, many

of the alternative explanations can be ruled out

(despite the lack of a control group). The use of

intensive designs, however, depends on the type

of therapy and the outcomes that are expected.

Only therapies that posit observable and rapid

changes (typically, the more behavioral therapies) can be tested with such designs.

Return-to-Baseline Designs.—In this design,

patient functioning is measured several times

before therapy (the baseline period) and several

times during therapy (the manipulation period).

Therapy is then withdrawn abruptly (reversal to

second baseline), and it is expected that there

will be a return to lower levels of functioning

similar to those found during the first baseline

period. Repeated measurement continues during

this period and during a return to therapy

(second manipulation). Therapy is then phased

out with the hope that improvements will be

maintained.

An example of this type of design is Allen,

Hart, Buell, Harris, and Wolf’s (3) study of a

single child. The child had isolated herself from

other children, causing severe psychological

problems. The amount of time that the child

spent with adults and other children was measured during the baseline period. A simple therapy that provided reinforcement for her interactions with other children was then provided.

Measurement continued during this period, as

well as after therapy. Later, there was an abrupt

withdrawal of the reinforcement (reversal to

second baseline), followed by its reinstitution.

The results showed very noticeable changes in

the girl’s interactions. In the absence of other

factors (besides therapy), the effects were seen

as demonstrating therapeutic efficacy.

Intrasubject Multiple Baseline Designs.— To

obtain information analogous to that obtained

from a separate no-therapy control group, it is

also possible to separately treat and assess functioning in different areas of the patient’s life.

Thus, treatment is designed to improve one

aspect of functioning, and the patient’s other

behavior is used as a control. Such a design,

while having advantages because changes can

be compared with only one subject, suffers from

the obvious problem that effects may generalize

across function areas. In fact, for some therapies

generalization of effects could be expected, and

an intrasubject multiple baseline design would

lead to “missing” the effects of therapy.

An example of this type of design is provided by Morganstern (189). The effectiveness

of aversive conditioning on obesity (pairing

nauseous smoke with eating certain foods) was

examined by conditioning a patient to one type

of food, then after several days, to another. Because eating of other foods was found not to

change until aversive conditioning was applied

to the particular foods, it was possible to infer

that the therapy produced changes.

There are a number of variations on this

intrasubject multiple baseline design. Multiple

subjects can be used, and different combinations

of treatment can be tested. This design does not

eliminate alternative explanations due to placebo or demand characteristics, but can eliminate

some problems inherent in nonrandomized experiments. Thus, additional subjects can serve

as controls to assess the importance of factors

such as spontaneous remission (improvement

without treatment). However, the use of this

design may be severely limited by the types of

problems (very specific) and the therapies

(mostly behavioral) which one tests.
Program Evaluations

The design considerations described above

implicitly refer to psychotherapy as a unitary

treatment that can be applied or not applied

to form experimental and control conditions.

However, as chapter 2 points out, psychotherapy treatments are comprised of a number of

factors. The theoretical orientation of the therapy, usually the basis of a label for the treatment, may not adequately describe the treatment as it is actually delivered. Depending on

the nature of patients’ problems, therapists’ orientation and skill, as well as aspects of the delivery setting, therapy may have different outcomes. One possibility is to treat each of these

factors as an independent variable and to construct multifactor designs. In such designs, subjects are randomly assigned to different therapies, therapists, and settings. A more recent

trend, though, has been to conduct such outcome research through program evaluations

(see 212). In such evaluative research (e.g., 9,

159,252), one evaluates a complex of treatment

variables that have been organized as a program

(e.g., a community mental health center

(CMHC)).

Thus, for example, a Program evaluation

study of a CMHC tries to assess how and to

what extent patients who receive treatment at

the center are aided. The CMHC, in addition to

being community based (which is hypothesized

to be an adjunct to treatment), may offer patients a number of therapies, and patients may

be treated by multiple professionals and paraprofessionals. Under such circumstances, where

joint effects of these treatments are expected and

where it is extremely difficult to separate—for

research design purposes—the components of

treatment, program evaluation yields a design

that may be more compatible with the actual

circumstances. Evaluative research does not

preclude the conduct of experiments where individual aspects of the treatment are assessed.

The designs for program evaluation studies

of psychotherapy can include aspects of the

true and quasi-experimental designs described

above. In general, the same methodological

considerations for research designs apply to

program evaluations (229,298,303). There is, in

fact, a substantial literature on the use of the experimental designs in program evaluation (e.g.,

230,249). The literature describes both how

complex variables such as psychotherapy can be

conceptualized and the conditions for implementing randomized designs. Similarly, the

literature on quasi-experimental designs has

been related to program evaluation (e.g., 41,

230). The principal difference between these designs and traditional research design is in how

one conceptualizes the treatment. In a program

evaluation study, the treatment includes a number of elements. These elements, at least in the

initial stages of such an evaluation, are not separately tested.

Even though program evaluations are designed specifically to aid in policy decisionmaking, there are a number of endemic problems.

Just as it is difficult to organize a traditional psychotherapy outcome study (i. e., randomly or

otherwise assign patients to treatments), it is difficult to organize program evaluation studies. In

fact, because the randomization units are more

complex, such evaluation studies are often very

difficult to carry out well, and there is a substantial literature about implementation failures. In addition, program evaluation studies

may not resolve the underlying conceptual

problems involved in assessing psychotherapy.

It may be difficult to determine through a program evaluation what factors were responsible

for the success or failure of the program.

Difficulties in Conducting Research

There are a variety of problems which make it

difficult to conduct psychotherapy outcome research. Some of these problems, which relate to

the inappropriateness of some methods to test

particular therapies (e.g., intensive designs) and

the problem of multiple factors affecting outcome (e. g., the role of therapist variable), have

already been described. There are also problems, however, which are perhaps more important, having to do with the pragmatic and ethical difficulties of conducting experimental

research.
30 Background Paper #3: The Efficacy and Cost Effectiveness of psychotherapy

While the advantages of experimental methods to develop unequivocal data are well known

and widely accepted, there are obvious ethical

problems connected with decisions to withhold

treatment (e.g., 45,229) or to make treatment

available on a random basis. Especially if one

believes that therapy is efficacious, it is difficult not to allow certain patients to receive

treatment for research purposes. Several considerations are important in thinking about this

dilemma.

One important consideration is the necessity

of conducting research, especially that which

employs randomized control groups. Such studies (which, in medicine, are referred to as randomized clinical trials) make it possible to assess

causality in the most unambiguous way (see,

e.g., 36,41,165). Although, from a theoretical

point of view, the value of randomized control

group studies has long been recognized, there is

now some evidence that such studies better enable researchers to detect inefficacious treatments (see, e.g., 94). If less adequate designs are

used (i.e., designs not employing randomized

control conditions), decision errors may result.

Thus, it may appear, perhaps because of the effects of other variables, that the treatment is effective when it is not.

One resolution to the ethical problems presented by needing to withhold treatment is suggested by the nature of the dilemma. If a treatment has not been demonstrated to be effective,

then it may not be unethical to deny treatment

to some individuals. The treatment, in the

absence of empirical data, may not be accomplishing anything. Actual treatment providers

(i.e., therapists) may not share this view and,

therefore, may be reluctant to participate in this

form of experimental research. At the very

least, practitioners might want to be able to

supervise control group subjects and, in effect,

provide partial treatment.

One pragmatic resolution to the ethical dilemma of this research is to provide control group

subjects with access to treatment once the experiment has been concluded. This is often referred to as the “waiting list” approach. It is

made a more attractive option in some studies

(which compare several therapeutic approaches)

by offering the delayed control group subjects

the form of treatment which the experiment

demonstrated as best. Obviously, this option

can be provided only when therapy is of a relatively short duration. It is, however, easy to implement when resources are very limited. If only

a small group of patients can be given treatment

at any one time, a waiting list (established by

random assignment) may be practicable (although it might be viable only when patients do

not have other treatment options).

Whatever the justification for random assignment to treatment and control conditions, explicit guidelines (established by the Department

of Health and Human Services and professional

societies) must be followed to protect patients’

rights. One central principle of these guidelines

is that participation be voluntary, based on patients’ (or legal surrogates’) “informed consent”

(see, e.g., 6). Informed consent procedures require that subjects be informed as to the purposes of the experiment, known risks, data to be

collected, and how the data will be used. Subjects also have the right to leave the experiment

at any time and, usually, must be told of other

treatment resources. The procedures also require that review panels be established to approve research protocols and monitor the conduct of this human research.

Although such procedures may create differential attrition across conditions of an experiment, and perhaps result in only certain types of

patients’ or treatments’ being involved in research, it does not rule out the use of experimental designs. Under a variety of circumstances,

individuals will agree to participate in experimental research, and the problems may have

more to do with the researchers or therapists involved than with patients. As noted above, until

unequivocal evidence is available about psychotherapy and more resources are available for

such treatments, it seems necessary to conduct

such experimental studies. These studies are

critical to providing unequivocal tests of the

theoretical hypotheses suggested by basic research. For many therapies, without experimental evidence, there will be no way of resolving questions about alternative explanations for

their effects (see 207). In order to carry out such
studies, it may be necessary for the Government

to develop special rules for their conduct. Thus,

for example, special reimbursement procedures

may need to be applied so that treatment and research costs can be separated.

Given the difficulties of conducting in vivo

experimental research, it is perhaps not surprising that much of the best controlled research has

been laboratory analogs to clinical settings. In

these analog settings, psychotherapy is offered

to patients who often have less severe dysfunctions than are seen in psychotherapy clinics.

However, these patients can be assigned to notreatment and placebo conditions because the

psychotherapy researcher controls the program

and because the loss of the therapeutic benefit

has less impact. This type of research may yield

more rigorous experimental findings, but has

low external validity; that is, the patients (often

undergraduate students), therapists (often doctoral students), and procedures (more theoretically oriented and less eclectic than is typical)

are not representative of those used in “real”

psychotherapy (e.g., 162,169). It should be

noted that some researchers suggest that this is

the best way to conduct rigorous investigations

of the effectiveness of psychotherapy and that

the innovative techniques developed and tested

in such settings can be transferred to more

“messy, ” real world settings (10, 135, 136). In

essence, they argue that the differences between

actual and analog settings are not as great as

might be anticipated and that generalizability

can be demonstrated for these analog studies (at

least for some therapies and conditions of

treatment ).

INTEGRATING FINDINGS FROM PSYCHOTHERAPY RESEARCH

Although the key to evaluating the efficacy of

psychotherapy lies in the conduct of welldesigned research which uses multiple measures

that are both reliable and valid, this may not be

sufficient for policymaking purposes. No one

research design (including true experiments and

program evaluations) will enable the development of definitive information about the effects

of psychotherapy, nor will the measures used in

any one study be adequate for all purposes. In

part, this is because of the current state of

theorizing about psychopathology, which encompasses a number of approaches each with

different ideas about research. The problems,

however, are not only theoretical, but also reflect the limitations of the scientific process with

respect to developing unambiguous conclusions

on the basis of individual research studies.

Given the diversity of criteria against which

such psychotherapy research must be evaluated,

as well as the divergent theoretical views,

methods are required to synthesize, aggregate,

or integrate the findings of multiple studies.

Despite the fact that it might be desirable to

have a few studies which “settle” the effectiveness question, for the most part, it is necessary to treat cautiously the results of individual

studies. Their results must be judged against

other research designed to test the same or similar hypotheses. Traditionally, such judgments

have been made through literature reviews,

where scholars analyze a body of research.

These scholarly analyses are reviewed by peers

and pub] i shed so that o t her researchers can

comment. Because of the scientific standards

underlying peer review (see, e.g., 39), most such

reviews reflect honest efforts to weigh the appropriate evidence.

However, even though the evidence may be

reported accurately, it is fairly easy to be selective about the research data one includes. It is

rarely possible, especially when considering the

host of problems for which psychotherapy has

been applied, to include all potentially relevant

research. Any researcher who conducts a review

makes a number of implicit and explicit choices

about what will be included. As well, choices

are made about what elements of the studies will

be discussed and those that will be ignored; in

effect, the author “slants” the review to support

a particular hypothesis or viewpoint. Clear criteria are not always provided for judging the
methodological adequacy of the studies included in the review.

To make this process more scientific, or at

least more systematic, a number of procedures

have recently been developed and applied to

assessments of the psychotherapy outcome literature. These procedures, although they have

their own limitations and are by no means

unanimously accepted, represent attempts to

make sense of the burgeoning and often contradictory research about psychotherapy. Two

such procedures are discussed below: “boxscore analyses” and “meta-analyses.”

Box-Score Analyses.—The procedure of

“box-score analysis” begins with identification

of a population of research studies (see, e.g.,

162). Usually, the reviewer establishes certain

standards and excludes studies that are not sufficiently rigorous by methodological criteria, or

are otherwise not appropriate. The latter category might include studies that are well designed but lack sufficiently reliable or valid outcome measures. It might also include studies

that do not have patients assigned randomly to

conditions. The difference between this procedure and that used in literature reviews is not

sharply defined and is only the greater degree to

which criteria are explicitly described in boxscore analyses and used to select research for the

review.

All studies meeting the reviewer’s criteria are

culled from some defined set of sources (e. g.,

journals) and are then sorted into categories.

Typical categories might be dysfunction treated,

therapy technique, and/or training of therapist.

Finally, the reviewer evaluates each study’s outcome (e. g., “yes, ” “no, “ “equivocal”) and tallies

scores using the predefine categories (e. g., dysfunction, technique, therapists).

This method of categorization and then tallying of effects is designed to systematize the literature review process. However, it still leaves a

great deal of room for individual judgment and

typically uses a rather simplistic measure of

treatment outcome. Importantly, it does not

take account of the strength of findings within

particular studies. A box-score analysis does not

take account of the methodological rigor of a

study, except in global fashion (e. g., by excluding studies without a particular kind of control

condition). It is, nevertheless, a useful procedure, and a number of important box-score-type

reviews of the psychotherapy literature are reported in chapter 4.

Meta-Analyses.—As described by Smith and

Glass (267) and Smith, Glass, and Miller (268),

“meta-analysis” is a procedure for integrating

research findings (see also 239). It is a more recent, and undoubtedly controversial, set of procedures (see ch. 4). Meta-analysis employs statistical techniques for aggregating data and for

determining relationships between causal variables and outcomes. Usually, the first step in a

meta-analysis is the precise description of a population of studies on which the analyses will be

based. In this respect, a meta-analysis is conducted similarly to a box-score analysis.

In a meta-analysis, however, studies are then

coded on a set of variables that are thought to be

related to outcomes—the number of these variables is at the discretion of the analyst. The experience of the therapist, the patients’ symptomatology, the quality of the research design,

and the setting of the treatment are examples of

the categories of variables that would be coded

in an analysis of psychotherapy. These measures are later correlated with the outcomes

(usually quantified) and used as the basis for

organizing outcome results in terms of aspects

of the studies. In Smith, et al. ’s (268) work, an

outcome measure for psychotherapy based on

the size of the effect (standardized) was computed for almost 500 studies. The studies came

from a population of controlled (i.e., comparison group) investigations. Although a global effect size measure, as calculated by Smith, et al.,

does not differentially weigh studies according

to the quality of the measures employed or the

design, such factors are controlled by coding

each study in terms of instrument and design

validity. If, for example, there is a difference between the size of the effects in studies that used

poor measures and those that used good measures (i. e., low valid v. high valid), then one

knows that a bias of some sort is operating.

Meta-analysis uses systematic methods to uncover trends in the available research literature.
Assuming that there are some good studies and

coding criteria can be agreed on, it should prove

to be a useful tool in understanding the research

on the effects of psychotherapy. A potential

problem, however, is that by focusing on available research, one ignores the fact that only research that reports positive and/or statistically

significant findings may be published. Although

some believe (241 ) that this could be a problem

FINAL COMMENTS

This chapter has described the methodological strategies that can be used to assess psychotherapy. Implicit in this discussion was an assumption that psychotherapy represents a researchable intervention that can be evaluated

using scientific criteria of measurement and

design. Valid measures of psychotherapy outcome can be developed, and designs that allow

relatively unequivocal assessment of psychotherapy can be constructed. Although there are

some who maintain that it is not possible to

assess psychotherapy because of its inherent

complexity, it is not clear what types of information would be excluded by the scientific

analyses described here.

Whether the methods described here have

been applied appropriately and to what ends

they have been used, however, is a different

question. Despite the possibility of conducting

research on the outcomes of psychotherapy, as

is noted in the next chapter, well-conducted reonly when there are a few studies reporting significant findings and when the magnitude of the

differences between groups is very small, obviously, the quality of available research is a critical factor in the usefulness of meta-analysis.

Other problems, such as the compatibility of

studies, are described in detail in the next chapter as part of a substantive review of Smith, et

al.’s work.

search is inadequate to answer at least some of

the important questions about psychotherapy

(207,277). The reasons for the lack of research

are not clear, but probably have to do with both

substantive features of psychotherapy and a

complex set of pragmatic factors.

These factors include the difficulty of conceptualizing the multiple factors that are part of

psychotherapy, as well as attitudinal differences

on the part of researchers and therapists. In

part, the development of program evaluation

techniques may facilitate the conduct of policyrelevant research. In addition, more attention,

by researchers, therapists, and the Government

to the issues of conducting ethically acceptable

controlled experiments, may help to develop

better research. Finally, attention to the issue of

conclusion-making, based on an analysis of

multiple investigations, may lead to better understanding of exactly what is achieved by psychotherapy under what conditions.

I
“Literature on the Effectiveness

of Psychotherapy
4.

Literature on the Effectiveness

of Psychotherapy

Undoubtedly, there is confusion about the effectiveness of psychotherapy; both in the professional and popular literature, a variety of

claims have been made about the worth of psychotherapeutic treatments. Although it is not

possible here to resolve fully the differences between conflicting viewpoints, the present chapSCOPE OF REVIEW

Although no one has classified all the research

conducted on psychotherapy, it is clear from

even a cursory review of the literature that there

is a plethora of research (see, e.g., 208,276). The

research ranges from very specific analyses

of psychotherapeutic procedures to large-scale

program evaluations. Clearly, there are differences in the usefulness of particular types of research for the general question of psychotherapy’s efficacy, but it is difficult to differentiate

research according to a relevance criterion.

While some commentators have made a distinction between basic and evaluative research

(e.g., 66, 268), the utility of any research to

answer efficacy questions is usually a matter of

degree. Basic research on psychotherapy does

not indicate whether or not treatments are effective as practiced, but does provide some understandings about the nature of the intervention.

For those concerned with understanding the

processes underlying psychotherapy, such data

are very important. Conversely, evaluative

studies, although they provide direct information about the actual effectiveness of a treatment, often yield equivocal analyses of causality and must be interpreted in conjunction with

basic research data. In the present chapter, categorizations are avoided; the analysis includes a

variety of research studies which have been considered, by at least some reviewers, to be releter briefly reviews the major areas of dispute

and attempts to apply the methodological principles described in the previous chapter to assess

the various claims about psychotherapy. The legitimate disagreements among reviewers are acknowledged, and an effort is made to assess the

differences objectively.

vant to understanding the efficacy problem.

Several additional caveats should also be noted.

One important issue relating to how one assesses the efficacy literature is the nature of the

problems for which psychotherapy treatments

are offered. Thus, a great deal of research examines the treatment of very specific nondisabling mental/behavioral problems, and it is

difficult to assess the generalizability of this

literature to understanding the efficacy of

treatments for very severe and disabling conditions. Unfortunately, the problem is even more

complex, for much of the psychotherapy literature deals with conditions that are difficult to

classify. In part, this problem has to do with the

fact that definitions of severity are relative and

dependent on the context in which the problem

exists. The present chapter attempts to clarify,

wherever possible, the degree to which findings

about particular problems are generalizable to

other mental health problems.

In terms of the scope of the present chapter,

the main focus is on a description and analysis

of previous reviews of the literature. As discussed in chapter 3, no matter how well conceived and executed, single studies have limited

use within a scientific framework. A number of

very extensive reviews of the literature (i e.,

discussions of multiple studies) have already

been conducted, and an effort is made in the

37
38 ● Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

present chapter to describe these reviews as fully

as possible. Where appropriate, as part of this

analysis, the data and methods of particular

studies discussed by these reviews are presented.

OTA’s decision to focus on reviews has to do

both with a goal to best represent the available

literature and with the fact that these reviews

comprise a focal point for much of the current

debate about psychotherapy’s effectiveness.

One omission from the literature reviews in

the present chapter are popularized summaries

of the psychotherapy literature. For example,

Gross’ recent book The Psychological Society

has not been included. Gross’ treatise is an interesting (although critical) report on psychotherapy, but is primarily a secondary source review

of the literature (see also 279,281). Instead of

discussing such work directly, the chapter describes the scholarly reviews on which Gross

based his book. Not included here either are

EFFICACY REVIEWS

In the reviews described below, the data reported are reviewed first; then, some of the

commentary that has been stimulated by the review is discussed. The methods used by the reviewer to select studies and analyze them are

compared against the methodological criteria

described in chapter 3. As noted above, the goal

is to describe the research that is most relevant

to the efficacy question and to assess what appear to be the most reasonable implications of

this research literature.

Eysenck’s Reviews

The earliest and probably the most controversial review of the psychotherapy outcome literature was conducted by Eysenck almost 30 years

ago (70; see also 71,72), Eysenck, in his early

review, considered 24 research articles which included 8,053 cases of psychotherapy conducted

with neurotic patients. He divided the studies,

on the basis of his assessment of their therapeutic approach, into two groups: 1) psychoanalytic therapy and 2) eclectic therapy. The

principal criterion for assessing effectiveness

some of the scholarly literature reviews which

have as their primary focus the contrast of one

therapy or approach with others. We were centrally interested in reviews that dealt, most

generally, with the problem of whether psychotherapy is efficacious under what conditions.

The central focus of a review on this question

guided OTA’s selections.

The reviews discussed below are presented in

approximate chronological order according to

their appearance in the literature. It is hoped

that this ordering will give readers a sense of

how research and thinking about effectiveness

of psychotherapy have evolved during the past

several decades. The chapter also describes the

findings of a number of the most important

studies on outcome. These descriptions are presented, where necessary, to facilitate understanding of the issues raised by reviewers.

was a rating Eysenck developed of improvement

following therapy.

The most controversial aspect of Eysenck’s report was his comparison of improvement rates

in therapy against an improvement rate that he

calculated for two no-treatment groups. In the

eclectic therapy studies, Eysenck calculated an

improvement rate for therapy of 64 percent

within 2 years. He compared this rate with what

was achieved in comparison groups of patients

who did not have therapy. Eysenck found that

the no-treatment improvement rate was approximately the same as that achieved in the eclectic

therapy studies. The improvement rate calculated for psychoanalysis was lower, approximately 44 percent, and was below that of the

no-treatment groups, One source of Eysenck’s

data on no-therapy improvement was a study

by Denker (61) of patients with emotional problems who were treated by general medical practitioners (another source was Landis (153)).

Denker’s data consisted of insurance company

records on 500 individuals who had submitted

mental disability claims. He had found that
Ch. 4—Literature on the Effectiveness of Psychotherapy ● 39

within 1 year, without receiving any specific

psychotherapy, 44 percent had returned to

work, and that by the end of 2 years, an additional 27 percent had returned to work.

Eysenck’s work unleashed an extraordinary

reaction, some of which still affects thinking

about psychotherapy research (17, 18,20,160,

181 ,242,268). Much of the reaction has been

negative, and, in response, Eysenck and his

colleagues have tried to refute claims of his

critics and to provide additional data (e. g.,

70,71 ). The debate between Eysenck and other

researchers is important to unravel, both because of its centrality in discussions about psychotherapy, and because of its implications for

the most important questions about psychotherapy. In terms of Eysenck’s original report, three

types of problems seem to be important in interpreting his analysis: 1) the nature of the data

that Eysenck reviewed, 2) the utility of Denker’s

comparison data, and 3) Eysenck’s interpretation of the data.

The first problem, that of specificity of the

data, has been raised by a number of Eysenck’s

critics (e. g., 160). In essence, these reviewers

have claimed that it was inappropriate to generalize very widely about psychotherapy on the

basis of Eysenck’s data. The most important difficulty is that it is not clear what was done in

each of the studies. One cannot determine how

much therapy was actually received by patients

and what the quality of this treatment was (19).

It also seems important to note that the research

reported by Eysenck was conducted prior to

1950, at a time when nonpsychodynamically

based therapies were only beginning to be used.

The generalizability of this work to the types of

nonpsychodynamic therapies carried out today,

as well as to currently practiced psychoanalytic

treatments, is not at all clear (see, e.g., 269).

The second problem has to do with the “control” group data used by Eysenck in his analysis. Eysenck’s control data for the effects of

treatment were drawn from a nonrandomly selected control group. It is difficult to determine

how this nontreated group might have differed

from individuals who received therapy. They

may have been more or less troubled and dysfunctional than those who sought treatments.

Not only are the differences between the control

and treatment group subjects not clear, but as

noted by Meltzoff and Kornreich (181), neither

Denker’s (61) nor Landis’ (153) data were representative (i. e., the data were not derived from

a survey).

Meltzoff and Kornreich (181) also note that

the control group data did not really represent

no-treatment data. Especially in Denker’s study,

where patients were diagnosed by general medical practitioners, a form of treatment was provided. Patients were provided “sedatives, tonics, as well as reassurance, and a placebo type of

treatment. ” Furthermore, since none of these patients was severely disturbed to begin with, the

return-to-work measure may not indicate an actual remission of the symptoms that first

brought the individual to the practitioner.

Finally, a number of commentators have

pointed to errors in Eysenck’s categorizing of

studies and errors in the way he handled data.

Bergin (19) reanalyzed the data used in Ey -

senck’s original report and finds a different rate

of recovery for psychoanalytic treatment. What

Bergin did, among other procedures, was to

eliminate cases where individuals left therapy;

Eysenck had, instead, counted these cases as

failures. Bergin also claims to have found coding errors in Eysenck’s original report (e. g., outcome data incorrectly represented). Bergin’s reanalysis had yielded an improvement rate of 65

percent for the eclectic therapies (similar to

Eysenck’s). He notes that improvement rates for

all types of treatment may be deceptive, because

there was a great deal of systematic variation

across personality types, therapists, and clinics.

Bergin concludes that global questions of psychotherapeutic efficacy are “silly” and that one

must analyze specific therapies for specific

problems.

Eysenck acknowledges many of the methodological problems with his data and has argued

that the burden of proof to provide evidence

should be on those who seek to promote psychotherapy (see 223). He has also updated his

original research, and, in later reviews (e. g.,

69), cites an additional 11 studies. Although his

initial conclusion about the lack of effectiveness

of psychotherapy remains, he finds supportive
evidence for at least one type of psychotherapy,

a behavioral approach based on the work of

Wolpe. It should also be noted that a number of

commentators (e.g., 181) have criticized Ey -

senck for selectively reviewing the literature. By

the time of his 1965 paper (which was published

along with 17 critiques by prominent researchers), there were at least 70 control group studies

of psychotherapy. Most of these were not included in Eysenck’s review, although his reasons

for not selecting these studies are not clear.

Although Eysenck’s work is often interpreted

as an indictment of all psychotherapy, as noted

above, Eysenck (69) cites evidence as to the

efficacy of therapies based on behavior theory.

Thus, he describes a study conducted by Lazarus (155) of phobic patients who were treated

either by group desensitization (a behavior therapy), group interpretation (a psychodynamic

therapy), and a combination of nonbehavioral

methods. The results showed a 72-percent recovery rate for the behavior therapy versus a

12-percent recovery rate for the verbal therapy.

Eysenck also relies on an unpublished report by

Wolpe (the developer of desensitization therapy) in which the results of Wolpe’s investigations (122 cases) were compared with reported

results from two large psychoanalytic institutions (approximately 400 cases). Wolpe had reported that 90 percent of his cases were cured or

much improved versus 60 percent of the psychoanalytically treated patients.

In trying to make sense of the controversy

over Eysenck’s research, a number of aspects of

his work should be highlighted. First, whether

one accepts his findings or not, the primary purpose of the original review was to assess the

data for treatment of neurotics. Severe mental

disability was considered only secondarily in

later reports. Second, while Eysenck’s critique

has been interpreted as a critique of all psychotherapy (and psychotherapy research), his comments are most critical of psychoanalytically

derived therapies. He is supportive, as described

above, of therapies based on behavior theory.

Finally, it should be noted that Eysenck did not

find psychotherapy to be harmful and, in fact,

found that eclectic therapies were associated

with fairly high rates of improvement. The critical question—whether improvement was caused

by psychotherapy or resulted from other factors

(e.g., spontaneous remission, placebo effects)

—is not resolved by Eysenck’s work.

Meltzoff and Kornreich’s Analysis

According to Meltzoff and Kornreich (181),

their work was stimulated by a belief that there

had been advances in the development of research on the effectiveness of psychotherapy

since Eysenck (who initially published in the

early 1950’s), but that this work was underrepresented in the literature. Meltzoff and Kornreich, in their book-length discussion of the psychotherapy research literature, made an important departure from earlier reviews and classified studies by methodological adequacy.

Groups of studies were formed according to

their methodological quality and compared in

terms of the positiveness of findings. Meltzoff

and Kornreich wanted to come up with a “fair”

assessment of what was contained in the available literature.

In their category of “adequate” research

designs, they included studies which used a

control group condition and adequate outcome

measures. In their “questionable” research design category, they included studies with control groups that may not have been comparable

and/or used poor outcome measures or analysis

procedures. They also included in this latter

category studies that used analog designs. The

results of the studies they assessed were categorized as either “posit ive,” “negative,” or null.

Since most studies included multiple outcome

measures, their subdivision of outcomes was a

judgment of the balance of statistically significant findings. Their analysis included approximately 100 control group studies.

According to Smith, et al. (268), who tabulated the results of Meltzoff and Kornreich’s review, 80 percent of the controlled studies they

reported yielded positive results, while the other

20 percent had null or negative results. There

was also a positive relationship between research quality and positive findings. Thus, 8 4

percent of Meltzoff and Kornreich’s adequately

designed studies yielded positive results about
Ch. 4–Literature on the Effect/vc)wss of Psychotherapy ● 41

the effectiveness of psychotherapy, while only

33 percent of the studies of questionable design

yielded positive results.

Meltzoff and Kornreich (181) also assessed

the degree to which results were “major” or “minor.” The studies categorized as “adequately designed, with major outcomes” varied on a number of dimensions. These studies included research such as Grace, Pinski, and Wolff’s (107)

assessment of group therapy for individuals

with ulcerative colitis. In this study, patients in

the treatment group received psychotherapy to

alleviate stress. Over a 2-year period, it was

found across a variety of “hard” criteria (e.g.,

operations required) that those who received

group therapy were less likely to require medical treatment than those who did not (there was

a lower morbidity rate for therapy patients, as

well).

A different type of study included by Meltzoff

and Kornreich (181) was Morton’s (190) study

of college students who were referred to psychotherapy for severe personal problems. A randomized control group design was implemented

by having some subjects placed on a waiting

list. Raters, who were blind to information

about the students’ treatment (they were the

counselors who made the actual referral), talked

with students after 3 months, On a number of

measures, including those that assessed cognitive and behavioral variables, 93 percent of the

treatment group students and only 47 percent of

the control group students showed improvement. The students who received psychotherapy

were considered to have received a major

benefit.

Although it is difficult to systematically

analyze the types of studies included in Meltzoff

and Kornreich’s (181) review, most of the studies they report appear not to be of severely disabled patients. In fact, almost all the studies in

which there were adequate designs and major

positive benefits were studies in which the problems were relatively specific and the patients

were not institutionalized. The reason for this

may be connected with the problems of doing

control group research, as well as with the nature of psychotherapy. As noted in chapter 3, it

is easier to conduct well-designed control group

research when problems are specific and the

potential harm of withholding treatment is not

severe. The hypothesis must be left open that

the more general and unspecified the problem,

the less efficacious is psychotherapy.

This point has been made by Malan (168)

about Meltzoff and Kornreich’s work (181).

Malan noted that, although he agreed with the

Meltzoff and Kornreich conclusion about the

beneficial effects of psychotherapy, its generalizability for a variety of patient groups was not

clear. There is also a problem, in generalizing

from Meltzoff and Kornreich, of not knowing

what specific techniques work with what specific problems. Although their methodological focus is useful for assessing hypotheses about the

relation between research quality and outcome,

it obscures the synthesis of findings about

specific treatments. From a more optimistic perspective, it is comforting that their positive findings rate is a bit higher than that of Eysenck,

and further, that these positive results were obtained in well-designed studies.

Bergin’s Reviews

Bergin’s 1971 review (19) of psychotherapeutic effectiveness was described briefly above in

reference to his critique of Eysenck’s work.

Bergin’s analysis is important (see also 18,21)

because, unlike earlier critics of Eysenck, he actually examined the studies used in the original

report (72) and recomputed Eysenck’s treatment

remission rates. Bergin used different assumptions and procedures for establishing improvement rates, and, as noted above, he found a discrepancy in Eysenck’s analysis of psychodynamic therapy. In addition, Bergin (19) calculated a different remission rate for no-treatment

(based on data not discussed by Eysenck), and

found that only about 30 percent of the patients

would have recovered had there not been psychotherapy. Within the framework of Eysenck’s

nonexperimental data base, there are limits to

one’s certainty about any conclusions.

Bergin (19), in part to remedy this problem,

reviewed 52 studies of psychotherapy outcome,

He claimed that these studies represented a

“cross-section” of the available literature (see

276). Approximately half of the studies had control groups, and Bergin categorized the studies

on the basis of a number of variables, including

.
42 ● Background Paper #3 The Efficacy and Cost Effectiveness of Psychotherapy

the adequacy of the research design, duration of

therapy, type of therapy, and therapist experience. Studies were judged and cross-tabulated in

terms of the results of the therapy: “positive,”

“negative, “ or “in doubt. ”

Of the 52 studies, Bergin (19) judged 22 to be

positive (indicating psychotherapy was effective), he judged 15 as having negative results,

and 15 were judged “in doubt. ” Bergin concluded that “psychotherapy, on the average, has

moderate positive results. ” Typical of the studies Bergin classified as positive was Gottschalk,

Mayerson, and Gottlieb’s (106) study of brief

therapy. Using a no-treatment control group design, Gottschalk, et al. studied neurotics who

were given emergency outpatient therapy. The

treated patients improved significantly more

than the controls on a variety of behavioral and

cognitive measures. Another positive study

cited by Bergin was Vorster’s (288) investigation

of neurotics treated by eclectic therapies.

Vorster found the treatment group significantly

improved, although this study’s design did not

employ a randomized control group.

In a more recent extension of Bergin’s (19)

review, Bergin and Lambert (21) discuss additional evidence to support their hypothesis that

psychotherapy is efficacious. They also reply to

the criticism (e.g., 222) of Bergin’s original

work. Bergin and Lambert cite as one of the best

examples of more recent research a study by

Sloane, Staple, Christol, Yorkston, and Whipple (266). That study used 90 outpatients, of

whom two-thirds were neurotic and one-third

had personality conduct disorders. The patients

were randomly assigned (they were also

matched with respect to sex and severity of

symptoms) to short-term analytically oriented

therapy, behavior therapy, or a minimal treatment wait-list group. Treatment outcome was

assessed after 4 months. On average, all groups

improved significantly across a set of target

symptoms, but the two therapy groups (analytically-oriented and behavior) improved significantly more than the minimal treatment waitlist group. No significant differences were found

in outcomes between the actual therapy groups.

Bergin and Lambert (21) conclude that, “Psy -

choanalytic/insight therapies, humanistic or patient-centered psychotherapy, many behavioral

techniques and, to a lesser degree, cognitive

therapies rest on a reasonable empirical base. ”

They contend that these therapies achieve results that are superior to no-treatment and to

various placebo treatment procedures. Bergin

and Lambert also assert that even if one accepts

Eysenck’s “two-thirds remission within 2 years”

formula, there is still positive evidence for psychotherapy. This is because treatment effects of

the same magnitude are frequently obtained in 6

months or less in formal psychotherapy, “. . . a

considerable evidence of therapy’s efficiency/

efficacy over no treatment. ” In effect, psychotherapy hastens the recovery period and reduces

both suffering and the effects of disability.

Rachman’s Critique

As can be seen from the above two reviews,

the early 1970’s was a fertile time for psychotherapy assessments. Rachman (222), a frequent

collaborator of Eysenck’s, provided another review which served, in part, to refute Eysenck’s

critics. Rachman, in particular, responded to

Bergin’s (19) reanalysis of Eysenck’s data and to

the dispute about what studies should be included in a review of psychotherapy efficacy.

Rachman, in analyzing Bergin’s work, disallowed a number of studies (5 out of 14), because the subjects were delinquents or had psychosomatic complaints, rather than being neurotic. If Rachman’s analysis had been extended

to Meltzoff and Kornreich’s (181) review, the

same problem would have existed.

Rachman (222) analyzed 23 studies which he

considered to be relevant for assessing the efficacy of “verbal” psychotherapy (i.e., therapies

based on conversation between therapist and

patient—primarily, humanistic and psychodynamically based). Of these studies, Rachman

found that only one provided tentative evidence

of the effectiveness of psychotherapy. Five studies produced negative effects, where treatment

group results failed to exceed control groups or

base-line remission rates. It is important to note

that Rachman excluded 17 studies for a variety

of reasons, only 2 of which showed negative effects. The reasons for excluding the 15 studies

that showed positive results varied, and in-
Ch. 4–Literature on the Effectiveness of Psychotherapy ● 43

eluded the use of unacceptable outcome measures (e. g., a projective test), the exclusion of

subjects who left therapy before treatment was

completed (which Rachman said inflated improvement rates), and inconsistency across outcome measures (i. e., some measures showed

positive effects, while one or more measures

showed negative effects). The two negative

studies were excluded because randomized control group conditions were not employed.

A number of commentators (e.g., 257,268)

have reviewed Rachman’s (222) critique and

analysis, and they claim that he used inconsistent standards for evaluating evidence. In particular, they assert that Rachman used criteria for

assessing the verbal psychotherapies that were

different from the criteria he used for the behavioral therapy experiments. Smith, et al. (268)

have criticized Rachman because he selectively

chose studies to review. They indicate that

many more studies were available to Rachman

than he included, and add that it cannot be inferred from his discussion why particular studies were not referenced. Smith, et al. also criticize Rachman for “ex post facto” exclusion of

studies based on methodological criteria. They

argue that he should have compared the good

and poor designs to determine whether or not

they yielded different kinds of conclusions, as

Meltzoff and Kornreich (181) did in their

analysis.

Luborsky, Singer, and

Luborsky’s Review

In a more current assessment, Luborsky,

Singer, and Luborsky (162) tried to evaluate all

reasonably controlled studies of psychotherapy

on “real” patients (i. e., analog studies were excluded). Luborsky, et al. examined available

studies that had assessed therapy for treating

recognized problems of individuals who sought

psychological treatment. Many of these studies

were ones that had been used by Bergin (19) and

by Meltzoff and Kornreich (181). Although Luborsky, et al. ’s definition excluded some behavioral research (primarily because it was not conducted in actual clinical settings) and some patient populations, their scope was wider than

that of other reviewers.

Each of the studies Luborsky, et al. (162) assessed was categorized on a number of dimensions and then summarized in a “box-score”

analysis (see ch. 3). One central dimension was

research quality, which was determined on the

basis of 12 criteria. The criteria included the

study’s method for assigning subjects to comparison groups, the procedures for dealing with

premature therapy termination, experience of

therapists, tailoring of outcome measures to

therapeutic goals, and the adequacy of the sample size. Luborsky, et al. summarized the codings by “grading” each study on a 5-point scale

of research quality. They then categorized results in terms of whether there were significant

differences showing better effects for the treatment group ( + ), the comparison group ( – ), or

no significance between the groups (0).

Luborsky, et al. (162) found 33 studies in

which psychotherapy treatment groups were

compared with no-treatment control groups. Of

these, 20 studies yielded psychotherapy treatment groups which were significantly better off

than control groups, and 13 showed no difference. Luborsky, et al. did not find any instance

of a control group being better than a psychotherapy treatment group. They found 19 studies

in which schizophrenic populations (i. e., severely disabled individuals) were studied. Of these,

11 yielded results in favor of the psychotherapy

condition, and 8 yielded no differences. Luborsky, et al. also found that in a majority of comparisons (13 out of 19), there were no significant

differences in outcomes to patients between behavior therapy and other psychotherapies.

While the reviewers note the positive research

findings on behavior therapies, they suggest the

need for more studies in which behavior therapies are applied to patients who have generalized maladjustments.

Luborsky, et al. (162) conclude that control

studies find that patients who go through psychotherapy do, in fact, gain. Because they

used “box scores” where effect sizes were not

estimated (see 221), it is not possible to determine how strong these effects will be. However,

according to Smith, et al. (268), there has been

no published substantive criticism of Luborsky,

et al. and (as is described below) Luborsky, et
—- ——-—— —-—--

44 ● Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

al. ’s general finding has been substantiated

other reviews.

NIMH Synthesis

Perhaps the most comprehensive review

by

of

psychotherapy outcome research has been conducted at the National Institute of Mental

Health by Parloff and his colleagues (208). This

review provided an assessment of psychosocial

treatments for mental disorders and was prepared for the Institute of Medicine (National

Academy of Sciences) as part of that Institute’s

work for the President’s Commission on Mental

Health (219). The Parloff, et al. review differs

from earlier works in that it uses a more narrowly defined treatment and is organized primarily according to disabling conditions. Thus,

for each of a variety of mental disorders, the

available evidence as to the effectiveness of psychotherapy was assembled and analyzed. The

reviewers tried to make some general statements

about psychosocial therapies (psychotherapies

that do not use drug treatments) and how these

therapies are affected by other variables, such as

therapist and patient characteristics. Parlofff et

al. ’s work is very extensive, so a detailed summary is not attempted here; only the work’s general conclusions are described below.

Parloff, et al. ’s (208) general finding (see also

207) was that “patients treated by psychosocial

therapies show significantly more improvement

in thought, mood, personality, and behavior

than do comparable samples of untreated patients. ” These reviewers found that spontaneous

remission rates developed from separate samples provide evidence that psychosocial treatments seem to result in greater improvement

than would be expected without psychotherapeutic treatment. Their finding is supported

most clearly for disorders such as “anxiety

states, fears, and phobias. ” Parloff, et al. ’s

relatively positive assessment, however, was accompanied by a number of caveats. For many

disorders, psychotherapy alone (i.e., without

other treatments such as drugs) has not been

demonstrated to be effective; nor does it appear

to be effective for particular populations (e.g.,

children). Finally, the review notes, although effectiveness evidence exists for a number of disorders, only some types of therapies (with particular therapists) may be effective.

The central aspect of Parloff, et al.’s (208)

review was a summary, by each psychopathological condition, of the available treatment

research evidence. To appreciate the complexity

of this task, consider their discussion of severe

mental disorders such as schizophrenia. For

these disorders, Parloff, et al. found that individual and group psychotherapy provide an

ambiguous amount of improvement for institutionalized patients; however, in conjunction

with drug therapies and other psychological

treatments, they appear to have important effects. The authors note that “. . . drugs do not

teach individual social and interpersonal skills.”

For such hospitalized populations, however,

Parloff, et al. found considerable evidence that

a specific type of therapy (behavior based) improves social adjustment (on a variety of social

and interpersonal variables). They also found

that the return of severely disturbed patients to

their community had positive effects on treatment outcome, although this finding was limited to patients with certain interaction skills,

and under the condition that the patient returns

to a “good” family situation.

One important feature of the Parloff, et al.

work, both as it was presented in the original

form (208) and as it was summarized by the

President’s Commission on Mental Health

(219), is that this report examines the evidence

for alternative hypotheses about the effects of

psychotherapy. In particular, Parloff, et al.

found a variety of reported spontaneous remission rates; that is, different improvement rates

are obtained for disturbed patients who do not

receive therapy (see, e.g., 280). Despite the fact

that patients improve to some extent without

therapy, however, the hypothesis that such

spontaneous remission effects account for

changes in treated patients cannot be validated.

Parloff, et al. (208) report that studies which

have been controlled for placebo effects find

that changes associated with treatment are

greater than those associated with the placebo.

Unfortunately, there is an inherent problem in

theoretically identifying the nature of a psychotherapy placebo (see ch. 3) and separating it
Ch. 4–Literature on the Effectiveness of Psychotherapy ● 45

from the effects of the treatment, so these results

must be regarded as tentative.

Parloff (207), in a discussion of the review

(208), calls for the development of clinical trial

research to assess more widely and systematically the effects of psychotherapy. While Parloff

adopts a relatively positive view of the effects of

psychotherapy, he finds the current research literature limited. He proposes the use of experimental designs to assess the specific conditions

(including aspects of the treatment, therapist,

and patient) under which therapy procedures

yield particular outcomes. Although one view

of such clinical trials would be to test different

theories of psychotherapy, an implication of

Parloff, et al. ’s work is that other factors need to

be incorporated in the design of these trials.

One additional note about Parloff, et al.’s

(208) findings has to do with their review of behavior-based therapies. Parloff, et al., along

with Eysenck, Rachman, Bergin, and to some

extent, Meltzoff and Kornreich, report clear-cut

evidence that behavior-based therapies are effective treatments for specified conditions.

Although its generalizability has not been

established across the range of disorders, this

finding suggests one important focus for future

research.

Smith, Glass, and Miller’s

Meta-Analysis

The final review considered here, that by

Smith, Glass, and Miller (268), has the potential

to be as controversial as Eysenck’s original

work, and may stimulate an entire new set of

psychotherapy outcome analyses. The metaanalytic methods used by Smith, et al. (see ch.

3) are at the heart of the controversy about this

work. A preliminary published report of the

meta-analysis of 375 control group studies of

psychotherapy (267) has already stimulated a

variety of critiques (e.g., 73,90,218). Smith, et

al. ’s goal was to determine the state of knowledge about the effects of psychotherapy, using

systematic scientific procedures. Their metaanalytic procedure was used to integrate the

findings of a disparate set of studies on psychotherapy. In addition, each of the studies in the

sample was classified to enable determination of

the factors that influence outcome findings.

In order to conduct their review, Smith, et al.

(268) tried to include all controlled studies of the

effectiveness of any form of psychotherapy.

Controlled studies were defined as investigations where a treatment group received psychotherapy, and another group which was comparable did not receive treatment. In some cases,

receiving treatment meant a placebo or alternate

form of psychotherapy. Smith, et al. considered

a study to be relevant if the study investigated

therapy that involved: 1) patients who were

identified as having an emotional or behavioral

problem, 2) treatment that was psychological or

behavioral, and 3) therapists who were identified professionals. The definition resulted in the

analysis’ including a variety of investigations of

treatments applied to problems of different degrees of severity. Smith, et al. excluded some

types of treatment, including those in which

psychoactive drugs were used (these studies

were analyzed separately), those which were

primarily educational, and those which were

not, essentially, psychosocial treatments. They

surveyed a great number of sources to identify

studies, including published journals, dissertations, and clearinghouses that identify professional publications. Their final sample included

475 controlled studies of psychotherapy.

Probably the most important aspect of Smith,

et al.’s (268) analysis was the way they classified

the research studies. Each study was coded on a

number of dimensions related to the characteristics of the researcher, therapist, and the patient (including diagnosis); most importantly,

the studies were classified on a series of methodological criteria. The methodological categories included the nature of the assignment to

conditions (e. g., random v. matching) and such

factors as experimental mortality and internal

validity (see 41,165). Smith, et al. ’s principal

dependent measure was a standardized score for

the size of the effect. From the data reported in

each study, Smith, et al. calculated scores for

the size of the effect; to allow comparison across

studies, they computed each as a standard

score. Studies often included more than one outcome measure, so Smith, et al. ’s analysis treated

each variable as a separate case. Thus, from the

475 studies Smith, et al. analyzed, they found

1,766 effect size measures. In addition, they
46 ● Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

coded outcome measures in terms of the type of

measure, instrument, and its reactivity (i. e.,

susceptibility to social desirability, faking, etc.).

Smith, et al.’s (268) principal finding was

that, on the average, the difference between

average scores in groups receiving psychotherapy and untreated control groups was 0.85 standard deviation units (i.e., the effect size difference

was 0.85). According to Smith, et al., this average effect size can be translated to indicate that

the average person who receives therapy is better off than 80 percent of the persons who do

not. They found little evidence for the existence

of harmful effects of psychotherapy (i. e., very

few cases where the mean of the control group

was higher than the treatment group). Smith, et

al. found some significant differences across the

types of therapies whose effects were studied

(the range was 0.14 to 2.38), but these effects are

confounded by variables such as patient and

therapist characteristics which were distributed

unequally among the therapies. Finally, their

methodological categories proved not to correlate with effect sizes; thus, for example, the better designed studies did not yield less positive

findings.

OTA has had an opportunity to review

Smith, et al.’s (268) coded data of the 475 studies. A sample of studies was drawn, and their

reliability was checked by comparing our blind

codings with their original data. In particular,

effect size ratings for a dozen studies were

recomputed. This analysis indicated that the

Smith, et al. codings were both easily replicable

and apparently reliable. Their validity is more

difficult to establish.

A key question is whether the effect size

scores calculated by Smith, et al. (268) are a

valid measure. There are reasons to suggest that

their average effect size is inflated, but there are

also reasons to suggest that it is conservative.

Inflation in the effect size measure may have

come about because only well-designed studies

were included in Smith, et al. ’s analysis. Thus,

to the extent that the control group studies are

completed only for successful psychotherapies

or for a limited range of psychotherapy treatments, the sample may be biased positively

when compared to psychotherapy research. It is

difficult to determine the exact nature of the bias

that may result from this problem, There are

also some aspects of Smith, et al. ’s procedures

which suggest that their average effect size

measures may be conservative. Smith, et al.

considered placebo treatments, as well as actual

forms of therapy and counseling, in their treatment group means. These are all not legitimate

therapies, and, in fact, when separately analyzed, showed lower effectiveness as compared

to actual psychotherapy.

Most critics of Smith, et al. (268), in particular Eysenck (73), have disputed the metaanalysis approach (actually, the published critiques refer to Smith and Glass (267)), because it

lumps together too many things and includes

studies of poor design, as well as good design.

This criticism is potentially justifiable, but a

close review of Smith, et al. indicates that they

control for this problem by their classification

of studies according to methodological criteria.

Not only do they start with a group of relatively

well-designed studies (in terms of their definition of controlled psychotherapy research), but

they provide analyses of the relationship between effect size and classification variables

such as internal validity (a measure of the quality of the research design—see 41). The correlations are close to zero, indicating that studies

that use randomized control group designs find

the same effect as studies that use poorer

designs. There may still be a sampling problem

in that the published literature only reports

well-controlled studies, but this problem is different from that on which most of the criticism

of Smith, et al. ’s meta-analytic procedures is

based.

At this point, it is difficult to know how to

utilize the results of Smith, et al. (268). Their

work is certainly more systematic than that of

other researchers; however, their methods are

not yet widely accepted and their work has not

been available long enough for comprehensive

reviews to appear. Perhaps, the most important

limitation of such meta-analytic research is that

it relies on the existing literature. It seems clear

that much better research on psychotherapy can

be done, and a more definitive meta-analysis

may have to await the completion of this new

research.
Ch. 4—Literature on the Effectiveness of Psychotherapy ● 47

DISCUSSION

If one considers only the trend of findings

reported by scholarly reviews and analyses of

the psychotherapy outcome literature, it would

appear that psychotherapy treatments, under

some conditions, have been shown to be efficacious (see, e.g., 287). Although the evidence is

not entirely convincing, the currently available

literature contains a number of good-quality

research studies which find positive outcomes

for psychotherapy. There are also a large

number of studies which report positive effects,

but whose methods or generalizability are difficult to assess. The quality of the evidence

varies in terms of the nature of the treatment

and the patient’s problem. One difficulty with

the available literature is that, while a host of

factors have been identified as important to psychotherapeutic outcomes, the role of these factors (e.g., characteristics of the patient, therapist, setting) has not been assessed in any definitive way.

Frank (85), for example, in commenting on

the state of psychotherapy outcome research,

reports a disappointment that research has not

produced more specific understandings of what

occurs during therapy. While he supports the

conclusion that research demonstrates that psychotherapy is somewhat better than no therapy,

he also finds that much of the literature indicates equivocal outcomes. One reason for

these equivocal findings, according to Frank, is

that the personal qualities of the therapist and

patient and their interaction may be more important than the therapeutic method. Because

these factors vary so widely and are not typically controlled for, variability is built into

psychotherapy studies, making it more difficult

to detect significant treatment effects. Frank

urges more research that clearly assesses these

components of therapeutic treatments.

It is likely, given the theoretical potential to

conduct such research on the ingredients of

psychotherapy, that a more extensive research

literature could be developed (see also 207). It

probably means that psychotherapy evaluations, unlike evaluations of new drugs, need to

pay equal attention to the conditions of treatment and the treatment per se. Research done on

psychotherapy in settings different from actual

practice will probably have limited utility and

will represent only one stage of a testing program. Although it would be hoped that psychotherapy research could lead to better understandings of the role of therapist characteristics

and skills, it might only be possible to assess

how these factors interact with treatments as

they are actually delivered. Finally, it should

also be noted that some forms of potential outcome research, such as program evaluations, do

not seem to have made a substantial impact on

the literature (cf. 181,212). While program evaluations have been done of a variety of process

factors in mental health delivery systems, these

studies have less frequently measured outcomes

using experimental designs. In part, this reflects

the developing nature of the methods for program evaluation (see ch. 3) and the slow shift to

the funding of such research. It is possible that

this situation will change with increased pressures for accountability and with increased emphasis on understanding complex sets of psychotherapy technologies.
5m

Methods for Assessing

Cost Effectiveness and

Cost Benefit of Psychotherapy
5.

Methods for Assessing

the Cost Effectiveness and

Cost Benefit of Psychotherapy

The preceding chapters illustrate some of the

complexity of assessing the outcomes of psychotherapy. Although such assessment is undoubtedly difficult, it nevertheless seems possible to

evaluate psychotherapy using scientific methods

of analysis. Assessing the costs and benefits of

psychotherapy, and developing comparisons

among effects, costs, and benefits—that is, conducting cost-effectiveness and cost-benefit analyses (CEA/CBAs)—is a natural next step in this

research process. To conduct such CEA/CBAs

encourages the explicit analysis of the resources

used in psychotherapy and the effects (positive

and negative) of different resource allocation

decisions.

Potentially, CEA/CBA is a set of procedures

that can aid in decisionmaking about the use of

psychotherapy. Increasingly severe economic

constraints, as well as the call for an expansion

of mental health services (see 219), make it especially important to understand how the effects

of psychotherapy are related to the resources it

consumes. There are a great number of competing pressures for health care resources, and,

ideally, CEA/CBA applied to psychotherapy

can serve as an aid to resolving these conflicts.

Although the application of CEA/CBA is, perhaps, not as well developed in psychotherapy as

in other areas (see 203), the current policy controversy about mental health treatments has

created increased interest in its use. The present

chapter describes the methods underlying the

conduct of CEA/CBA in psychotherapy and indicates both the potential for its use and the

problems associated with its application to mental health treatments.

It is important to note that many of the issues

of CEA/CBA of psychotherapy are closely related to the problems of assessing efficacy. Economic analyses of psychotherapy are dependent

on the quality of research data pertaining to

psychotherapy’s effects. The unique problems of

CEA/CBA of psychotherapy have to do with

the difficulty of comprehensively assessing and

valuing the effects of psychotherapy (see 98,

157,206). Such effects include the reduction of

pain and suffering and enhancement of “wellbeing. ” These effects are very difficult to measure and even more difficult to value in monetary

terms. This difficulty may result in CBAs of

psychotherapy consistently undervaluing the

benefits of psychotherapy. CEA, in contrast to

CBA, does not require that such effects be expressed in monetary units, but does rest on the

premise that they can be valued in some manner. Such difficulties restrict the usefulness of

cost analyses.

The methodological issues involved in the development of CEA/CBAs of psychotherapy are

described in the following sections. For a more

complete description of CEA/CBA, the reader

should consult OTA’s main report on CEA

(203). ] The present analysis begins with a discussion of the methods for assessing costs, and

that is followed by a discussion of the methods

for assessing benefits. In the third section, the

actual conduct of cost analyses is described. The

discussion below emphasizes the relationship of

CEA/CBA to efficacy assessments and the usefulness of CEA/CBA in aiding policymaking

about psychotherapy.

‘In addition to the main report ot OTA’S assessment of CEA,

Backgroutlci Pup(Jr #1. Met}loJologicul Issues a~ld Literature Re-

~Il[Ju) describes, in detail, the use of CEA , CBA methods.

51
52 ● Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

COST ASSESSMENT

For the purpose of CEA/CBA, the cost of psychotherapy may be conceptualized as the value

of various resources consumed in the process of

therapy (see 6,157,173,237). These costs include

the value of a variety of resources used to provide treatments, such as the value of the therapist’s time and the value of the use of the treatment facility. They may also include the value

of the patient’s time. Inevitably, decisions are

required about whether to include or exclude

costs and how to value resources appropriately.

These decisions often reflect the subjective judgments of different interest groups involved in

the cost analysis (see 203,277) and the purposes

for which the analysis is being conducted. Just

as different theoretical perspectives on psychotherapy may result in different measures of therapeutic effectiveness, so too may different perspectives yield alternative ways of defining therapy costs.

General Considerations

There are a number of general considerations

relevant to assessing the costs of resources used

in psychotherapy. Some of these general issues

have to do with the data that are used in developing cost estimates. Others concern the use of

procedures to transform available data into useful indices of resources consumed. In a later section, the application of these procedures to the

collection of specific cost data is described.

Accounting Methods.—The most readily accessible source of data to assess the costs of

psychotherapy are the entries in the accounting

records of a treatment facility (e.g., a practitioner’s office, a mental health center). Because

they are usually highly organized and accessible, accounting records give a ready definition

and a reasonably reliable record of the moneys

required to deliver therapy. These direct costs

are the ones that are most often referred to in

simple cost analyses of psychotherapy. A caveat

that should be noted, however, is that the costs

that result from accounting tabulations may not

include all or even most of the resources used by

therapy. Costs from such tabulations, therefore,

may not be an accurate reflection of the resources consumed.

In addition, a number of costs may not appear in accounting records. Thus, for example,

volunteered time and donated facilities may

play a large, but unaccounted for, role in the

provision of psychotherapy. Therapies that attract more volunteers owing to location or type

of patient treated may appear less costly than

therapies located in areas where volunteers are

scarcer (e.g., in impoverished neighborhoods)

or than therapies treating less “attractive” persons (e. g., sex offenders compared to children

who are autistic). Resources contributed to therapy by the family or others connected with the

patient may also be considerable, but would not

be recorded in an accounting system.

Opportunity Cost. —A more correct approach to assessing the costs of psychotherapy

than using accounting costs involves the opportunity costs concept. In using opportunity

costs, one calculates the value of a resource as if

it were applied to a best alternative use. For

psychotherapy, determining opportunity costs

allows one to consider more completely the

value to society of various resources consumed

by psychotherapy treatments and significantly

alters the analysis (25,40,283,305). Consideration of opportunity costs avoids problems created by different accounting procedures and, at

least conceptually, the problems due to the use

of volunteers and donated facilities.

Discounting. —The actual calculation of opportunity costs involves the use of discounting

procedures, which provide a present value of future costs. Thus, if costs will be incurred at a

future time, they will appear to be less costly if

valued in the present. The discount rate is usually based on the prevailing interest rate. Probably, discounting procedures are more important for properly valuing benefits (which are

more likely to occur over time) them for valuing costs. Obviously, though, discounting procedures are necessary when dealing with costs

such as those for a treatment facility.
Methods for Cost Assessment

The valuation procedures used to assess the

most common cost elements in psychotherapy

are considered in this section. The use of opportunity cost procedures and the complete

assessment of relevant cost elements are emphasized. Also discussed is the assignment of costs

to specific treatment.

Personnel Costs.—The cost of personnel is

usually the largest therapy cost (between 60 and

80 percent, according to Levin (157)), Personnel

costs can usually be estimated simply by multiplying the sum of salaries and benefits of the

personnel employed in the therapy process by

the time involved. Personnel may include professional therapists and paraprofessionals, as

well as support staff. In cases where salary data

are difficult to analyze or where time is volunteered (e.g., a therapist donates time in a teaching facility), personnel costs can be estimated by

other means. Thus, for example, hours of therapy can be multiplied by standard hourly salary

figures.

Several “time accounting systems” have been

specifically developed to collect personnel cost

data in mental health treatment facilities (30,

150,205). Some of these data are available in accounting records, but oftentimes they are not

available in sufficient specificity (i. e., broken

down by tasks). In the time accounting system

described by Carter and Newman (30), all personnel, including volunteers, patients, and

salaried staff, were required to keep daily records of time spent in therapy-related activities.

This information formed the basis of a costaccounting system.

Facilities and Equipment Costs.—Also included in the valuation process are the costs of

operating facilities, most correctly expressed in

terms of market rent, plus overhead such as

cooling and building maintenance costs. Market

rent is used in order to value the resource in

terms of its “opportunity cost, ” discussed

above. This procedure corrects for the problem

of valuing donated facilities and for valuing

government facilities which may be leased (to

the treatment agency) at artificially low rents.

The procedure would include the estimated value of donated or loaned rooms and buildings. It

would also adjust the rent or mortgage to assess

their true, rather than paid-for, value.

Costs for the equipment and materials used in

psychotherapy also have to be calculated. These

costs, which are usually available in accounting

records, include such resources as office supplies, food, laundry, and telephones. They may

also include the value of such specialized materials as psychological tests and computer scoring services. Usually, the difficulty with assessing the cost of these specialized resources is the

lack of records as to their use.

Other Costs. —In addition to personnel and

facilities/equipment costs, some analysts include the costs of the patient’s time and/or the

costs of therapy to a patient’s family. Although

such costs can, alternatively, be considered as a

negative benefit (i. e., subtracted from benefits),

it is sometimes useful for them to be included

with the actual costs of providing therapy.

Thus, for example, when a patient loses time

from work activities or when an employer has

to give release time to a patient for therapy, it

may be useful to value the patient’s time and

consider it as a cost.

Assessment of the costs of the patient’s time is

similar to the way other personnel costs are calculated. Usually, an accounting can be be made

of the amount of time that the patient spends in

therapy. Assuming that this time could be used

productively, it is multiplied by the patient’s

salary. Parallel calculations can be made for

family members who become involved in the

therapy or are required to spend time with a patient as a result of therapy. One problem with

this aspect of the costing process is the problem

of equity, since some groups of people earn

more than others and, thus, their time could be

valued more highly.

While a number of other costs could be included, such as the psychological cost of therapy, there is no agreement as to whether they

should be included or how they can be valued

(see, e.g., 126,172). The development of psychological cost measures would involve the

assessment of the suffering or pain of a patient

as a result of therapy. However, such costs, as
well as the costs of mental illness are usually

considered as a reduction in the benefit of a

treatment.

Specifying Costs.—For a simple CEA or

CBA, the sum of personnel, facilities, equipment, and materials costs may be adequate. For

an analysis which seeks to identify the procedures, therapists, patient types, or settings that

consume more costs than others, however, procedures have to be devised for assigning costs to

individual components of therapy. Analyses of

this type may become even more complex when

distinctions have to be made among various

types of costs (157,248).

Cost data that can be used for analyses of the

resources used for specific components of therapy can be collected either during therapy or

after it is completed. If collected after therapy,

the analysis requires summary data to be

broken down into different costs corresponding

to therapy components. Because many assessments of psychotherapy have been introduced

only after a therapy program has begun, the

breakdown method has received considerable

attention. Analyses that examine the cost effectiveness of treating individual patients may

divide overhead costs such as salary, rent, and

basic supplies equally among patients, according to the amount of their therapy (e. g., number

of therapy hours).

Some cost assessment procedures assign therapist costs to patients according to the amount

of time that therapists spend working with different patients; then, the costs of overhead, personnel, and other resources are divided equally

across patients (e. g., 30,149,199,305). To compare the relative cost, cost effectiveness, or cost

benefit of different components of therapy programs, the difference in direct costs of the various treatment components is calculated. In

such analyses, the overhead costs that are the

same for each component are ignored. These

direct costs are then used in CEA/CBAs, as described below.

Discussion of Cost Assessment

Despite some longstanding interest in assessing the cost of psychotherapy (e.g., 76) and the

apparent ease of applying standard valuation

methods to psychotherapy, the assessment of

psychotherapy’s cost is not widespread, nor

has such assessment been evaluated. Certainly,

techniques for the measurement of effectiveness

are much better developed than those for measuring costs. Most psychotherapy research—perhaps 95 percent—neglects the cost of the treatment. The implications of this neglect for CEA/

CBAs of psychotherapy are important to consider (16). It should be noted for example, that

no standards as to what should be included in

cost analyses have been developed. More importantly, the available studies may reflect a

narrow range of treatments (probably those

treatments which are either very costly or very

low in cost). In addition, cost data may be

derived from existing studies even if not explicitly included in the original analysis. Usually,

the available information on numbers of therapists, patients, and treatment length is detailed

enough for rough cost estimates to be made.

Combined with secondary analysis procedures,

the availability of these data may provide a

promising opportunity for further R&D of psychotherapy CEA/CBAs.

BENEFIT ASSESSMENT

The valuation of “benefits” resulting from to do with selecting effects to be valued and

psychotherapy which can be used in CBA, un- determining appropriate ways of translating effortunately, is even more problematic than cost fects into benefits (i.e., valuing effects in moneassessment (e.g., 140,237). The translation of ef- tary terms). Below, these problems are disfects into benefits is necessary to provide a com- cussed in terms of the types of benefits produced

mon metric for comparing resources used with by psychotherapy—to patients, to those assoeffects. The problems of benefit assessment have ciated with patients, and to society.
Benefits to Patients.—The most obvious benefits of psychotherapy accrue directly to the patient, although different effects of therapy have

somewhat different problems attached to their

valuation. Thus, while a change in earnings

may be valued in a relatively straightforward

way, other benefits of psychotherapy, such as a

reduction in pain or anxiety, are more difficult

to measure and value. Because these intangibles

are so difficult to quantify in monetary units,

any CBA of psychotherapy may undervalue the

benefits. It is also possible that some of the

negative benefits of psychotherapy (perhaps,

the effects of stigmatization of being a patient)

will not be calculated. To avoid this problem,

many studies separately analyze tangible and

intangible benefits, comparing only tangible

benefits to costs.

From the perspective of some economists

(e.g., 251), it is argued that a patient’s willingness to pay for psychotherapy reflects its

value to the patient. Thus, that which patients

are willing to pay for therapy is the net value of

the expected health, social, and economic

benefits, minus the psychological suffering, lost

time, and personal costs that are incurred as a

result of undergoing psychotherapy. Underlying

the use of the “willingness-to-pay” concept is the

assumption that the patient has made an informed decision to pay the required fee. However, the willingness-to-pay assumption, while

theoretically possible, assumes that the mentally

distraught person has made a rational decision.

Since such individuals seldom have access to

clear information about psychotherapy’s costs

and benefits, this assumption is probably unlikely and has not really been applied.

Some economists, for example, Weisbrod

(296), recognizing the inherent problem with applying the willingness-to-pay criterion to psychotherapy, have proposed more direct assessment of patient benefits. In Weisbrod’s research,

the variety of effects of treatment are identified

and, to the extent possible, transformed into

monetary values. For some effects, such as improved quality of life or absence of mental illness, no pecuniary value can be assigned. These

factors are therefore not included in an assessment of benefits, but can be contrasted with the

costs of particular therapies.

Aside from such problems of valuing certain

effects, there is also a problem of identifying

which effects should be attributed to psychotherapy. This is accomplished by comparing

treatment and control group data, To the extent

valid control group data are not available, resulting benefit estimates may be in error. While

the availability of appropriate effectiveness data

is a problem for the assessment of any type of

benefit, it is a particular problem for assessing

the effects on a patient. For most effects, there

will be a variety of other possible causes which

will be difficult to separate without control

group data.

Benefits to Those Associated With Patients.—

The effects of psychotherapy may extend beyond the patient. Thus, the family and friends

of a patient may have their own quality of life

improved if therapy is successful. In a more tangible way, they may also achieve more productivity in their own work and have more time

available for their own needs. The opportunity

value of their improved productivity and time

can be calculated and, where appropriate, considered as a benefit of psychotherapy.

As a result of therapy, patients may also be

more productive workers, and the benefits of

this productivity may accrue to their employer

(over and above the wages paid to the employee). Absenteeism may be reduced, accidents be

fewer, and a host of other benefits are potentially the result of psychotherapy (e.g., 138).

These benefits, of course, must be reduced by

costs for an employer to provide psychotherapy

or to allow employees release time to undergo

psychotherapy. As noted earlier, however,

these costs can either be considered a direct cost

or subtracted from the benefits. It is important

to ensure that an analysis uses consistent procedures. It is also important that the same benefits

are not counted twice; thus, for example, the

analyst must be careful not to count the same

wages as a benefit to the patient and to the

employer.

Benefits to Society .—Some of the most tangible benefits of psychotherapy may accrue to so-
56 ● Background Paper #3, The Efficacy and Cost Effectiveness of Psychotherapy

ciety. Thus, the maintenance of employment or

a reduction in criminal activities may yield a

savings directly to society. These effects, reflected in such savings as reduced unemployment payments, may be over and above the

benefits to patients and employers. Although

such outcomes may be relatively easy to value,

a problem is that the benefits, if they exist,

probably accumulate over a long period of time.

In most cases, these benefits would have to be

very large to offset the impact of discounting

their value over the time they are received.

It should be noted that most often the benefits

described above will be in the form of expected

cost savings. Therapy-related benefits, such as

reductions in work absenteeism, physician

visits, drug abuse, and arrests, have each been

considered in CEA/CBA studies (e.g., 51,228,

244,245). In these studies, the cost of each unit

of service is estimated from average rates or

from accounting records, and the reduction in

use of units of social services is multiplied by the

METHODS FOR CEA/CBA

The purpose of carefully measuring the costs,

as well as the benefits, of psychotherapy is to be

able to conduct CEAs and CBAs. CEA differs

from CBA and it is essential to recognize the difference (see 203). Cost-effectiveness studies require only that the costs of psychotherapy be

valued in dollars. In CEA, the effects are not

valued in monetary terms and can be expressed

in any units. In contrast, CBA requires that

both the costs and outcomes be valued in monetary terms. (Theoretically, costs and benefits

need only be expessed in the same unit for CBA,

but this unit is nearly always monetary. ) It

should also be noted that informal cost analyses

can be conducted where a researcher selectively

pays attention to some aspects of resource use

or benefits.

Although CEA usually requires a simpler set

of procedures and yields less comparative information than CBA, CEA is often considered a

more appropriate tool; in other cases, when certain comparative information is necessary and

when valuation problems can be overcome,

unit cost to estimate monetary cost-savings benefits. Usually, the validity of control group data

to estimate these savings is critical in order to

separate the effects of psychotherapy on these

variables from other causes.

Discussion. —In several ways, developing

benefit measures is more difficult than the comparable procedures for cost assessment. Because

of the problems associated with using willingness to pay as the basis for valuing benefits,

methods have to be developed to transform effects into benefits. If errors are made, they will

probably result in some analyses’ not taking

benefits (especially the psychological ones) into

account and, thus, understating the benefits of

therapy. In addition, it is probably easy to err

by not including some benefits. Because benefits

potentially accrue to a large number of people

and societal agencies, often more so than the

number of people or groups who incur costs,

problems in analyses may result.

CBA is better suited to the problem (see, e.g.,

98,157,206,305,307). In still other cases, when

comprehensive data are not available, actual

CEA/CBA is not done, but the costs and benefits are compared informally. Most commonly,

CEA and CBA have been used to compare the

“worth” of different psychotherapies as provided in different settings. Also frequently employed are formal and informal CEA/CBA

studies designed to yield information to those

directly responsible for the therapy. Such analyses often try to assign costs and outcomes to

specific treatment processes, so that the optimal

mixture of processes can be obtained (see 305).

Below, some procedures associated with the

conduct of CEA and CBA studies are described.

Cost= Effectiveness Analysis

When benefits and costs cannot be valued in

the same units, or when the outcomes of a treatment seem more valid when expressed in their

natural units (e.g., reduction of a anxiety), then

CEA is probably more appropriate than CBA
(e.g., 157,237). In addition, when treatments

are compared that have similar goals, CEA may

provide an adequate and appropriate methodology. In conducting CEA, outcome data are divided by costs to form cost-effectiveness ratios.

The procedure allows, for example, the comparison of several therapies to determine which

therapy produces the greatest amount of change

for the least cost. A variety of examples of this

procedure are given in chapter 6.

Tabulation and Matrix Methods.—One of the

simplest ways to analyze effectiveness and cost

relationships is to display the data in an array.

A simple tabulation model described by Krumboltz (150) provides a rudimentary example of

how the basic direct cost, processes, and effectiveness can be arrayed in a useful way. A table

is developed with the sideheadings “Problem

Identification, “ “Method,” and “Outcome.” Under a supraheading, “Cost,” are the headings

“Activity, “ “Hours,” and “Dollars.” Cost breakdowns, problems to be worked on further, and

the other essential information for simple analysis are contained here.

Newman (30,198) has developed a CEA procedure that has been adopted in a number of

psychotherapy settings. An instrument is used

to measure level of functioning along a range of

dimensions. Newman arrays his outcome data

in a matrix, using dimensions such as level of

functioning before treatment and level of function after therapy, The cells of the matrix are

completed with the number of patients who

functioned at that level before treatment and

who moved to another level of functioning by

the end of therapy (or who stayed in the same

level). Next, the cost of treating each patient

in that cell of the matrix is summed, and divided

by the number of patients whose functioning

change is described by the cell. The resulting

cost-per-patient ratio reflects cost effectiveness in terms of its position in the matrix (see

also 270).

Linear Functions.—To describe and predict

relationships between effectiveness and cost, it

is possible to develop equations that describe

cost-effectiveness relationships (see 305). The

techniques that would achieve a given level of

effectiveness also can be chosen from this

graphic model of the cost-effectiveness relationship. Not only can effectiveness be predicted,

but the effectiveness of particular techniques for

different costs can be determined. In an actual

situation, a number of possibilities exist for a

function to describe this relationship (e.g.,

linear or exponential). Ideally, if this method

were chosen before therapy, several plausible

models would be chosen, such as the linear and

exponential, and the average of predictions

generated from them would be used in decisions

until further information supported one model

over the other.

Linear Programing.—Finally, some applications of CEA techniques attempt to incorporate

information on cost limits, as well as on the factors that determine the effectiveness of psychotherapy. The basic concept underlying linear

programing is to consider not only the factors

that contribute most to therapy outcomes, but

also the cost of less effective factors and budget

restrictions. Linear programing is a statistical

procedure used to find the exact mixture of the

most contributory factors that are possible

within budget constraints.

The equations for linear programing bring

together information on which therapy techniques, delivery systems, or therapists work

best and on the amounts of each resource

needed to implement each technique or delivery

system, or to hire each therapist. Equations can

be used to minimize the total costs of achieving

a prescribed degree of effectiveness or benefit.

The equations can also be manipulated further

to discover which cost constraint could be fitted

to yield the maximum improvement of effectiveness or benefit. A number of psychotherapy researchers have advocated use of these and related techniques to conduct cost analyses in the

human services delivery such as psychotherapy

(e.g., 1,14,72,115,193,310).

Cost= Benefit Analysis

In CBA, benefits are summed using the same

units (e. g., dollars, person-hours) and costs are

summed using the same units as benefits. A

ratio is then derived by dividing total benefits

by total costs. If benefits exceed costs, the ratio

is larger than 1, and if benefits are less than
58 ● Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

costs, the ratio is less than 1. Such benefit/cost

ratios provide a convenient economic index of

the net benefit of an activity.

One reason that CBA may be more useful

than CEA is that decisions which compare benefits to costs of a treatment program are seldom

made in isolation. Often, benefits and costs of a

given treatment are most useful when compared

with benefit/cost ratios of other treatments

competing for the same funds. Thus, an alternative treatment program may be available which

could generate superior benefit-to-cost ratios.

The important factors are the difference in the

costs of alternative treatments, the difference in

benefits of alternative treatments, and the ratio

of benefit to cost. An argument can be made

that effectiveness measures should be used, in

some cases, instead of benefit measures, because

effectiveness data probably retain more accurate and valid information on treatment outcomes. That very much depends on the situation, however, and it may be necessary to present both types of data.

To illustrate the problem, consider the applications of CEA methods to psychotherapy by

Halpern and his colleagues (22,113,114). Their

approach views improvement in therapist ratings of patient functioning as a monetizable increment in the economic value of the patient.

This increment is contrasted, in simple benefit/

cost ratios, to the monetary cost of treating the

patient. Potter, Binner, and Halpern (216) have

made their models somewhat more sophisticated by considering benefits according to the

amount of time the patient stays in the community as well as the improvement noted by the

end of therapy. Although perhaps useful, this

method suffers from potential bias in therapist

ratings of improvement in patient functioning

and in assignment of a somewhat arbitrary

value (e. g., $10,000) to each functioning unit

improvement. It may be more accurate to use

the actual functioning unit scores in a CEA.

Depite such problems, there are still reasons

to prefer benefit measures, especially when the

units in which benefits have been measured are

more directly meaningful than those used by

Halpern, et al. (22). Ratios of effectiveness divided by cost may not provide as much information as benefit divided by marginal cost, because if the latter is greater than 1, the additional benefits of one of the alternative treatments can be said to be “worth” the additional

costs it requires. It is difficult to make a similar

statement about effectiveness/cost ratios.

Fishman’s research (77,78), developed to assess the effects of a community mental health

center, illustrates the use of similar CBA procedures. Fishman views CBA as an experiment in

which outcome and cost data are gathered using

the research designs for assessing efficiency

described earlier. If the effectiveness of one program or program component is shown by statistical analysis to be significantly superior to the

effectiveness or benefit of another program, and

the costs of the two do not differ significantly,

then the program with superior effectiveness is

more cost effective. If the costs of alternative

programs differ significantly, but their effectiveness or benefit does not, then the least costly

program is adopted. Fishman acknowledges

that his model breaks down in situations where

the significantly more effective or beneficial

program is also significantly more costly. The

question as to whether the increment in effectiveness or benefit is “worth” the increment in

cost is, as noted earlier, a question for marginal

benefit-cost analysis.

Net Benefit Analysis.—It should be recognized that ratios of benefit to cost or effectiveness to cost do not really yield information

about the absolute amount of benefits and total

costs involved. This information may be important in decisions that must deal with limits on

the maximum cost allowable and on the minimum benefit that should be produced. The

amount that benefits exceed costs also may be of

concern. A benefit/cost ratio of 2 can be produced by a benefit of $200 and a cost of $100 for

one program, or by a benefit of $200,000 and a

cost of $100,000 for another program, but these

two programs are not the same. If no more than

$2,000 can be spent, the former treatment program is the only feasible one; if benefits must exceed $2,000, then the second treatment program

is the only one possible. To aid in maximizing

benefits, analysts often calculate net benefits,

which are present-valued benefits minus pres-
ent-valued costs. This is one of the advantages

in expressing outcomes and costs in the same

units. If the net benefit is negative, then the program is not worthwhile; if net benefits exceed

zero, the program is worthwhile. This information may be more useful than ratios in many instances, although it might also be desirable to

consider calculations of net benefit per patient.

Sensitivity Analyses.—In any CBA, it is important to consider the impact of alternatively

valuing benefits and the impact of measurement

error. For example, if a benefit of $100,000 is in

error by + 10 percent and the cost of $90,000 in

error by + 10 percent, the benefit/cost ratio

might vary from 0.91 ($90,000)/($99,000), t o

1.36 ($110,000)/($81,000), instead of the 1.11

($100,000)/($90,000) calculated originally. Providing these alternative calculations would give

the interpreter of the CBA an idea of the possible error.

SUMMARY

There exist a variety of methods to assess the

costs and benefits of psychotherapy and to compare the data generated by various studies. Although in the case of psychotherapy, unique difficulties arise (in particular, having to do with

the valuation of benefits), the problems of conducting CEA/CBAs are not necessarily unique

to assessments of psychotherapy. In every instance, the usefulness of such analyses is very

Statistical analyses can also be applied to such

problems, Thus, for example, one can calculate

standard error scores which provide a precise

statistical measure of error (see, e.g., 201).

Statistical procedures can also be used to test the

significance of different cost/benefit ratios (30).

These ratios are typically calculated for the

treatment as a whole, in which case possible

error in measurements of costs, effectiveness,

and benefits cannot be treated as variance about

a mean, but instead as an absolute error. In such

analyses, the degree to which measurement

error may influence the benefit /cost ratio can be

investigated only by first establishing a reasonable range of possible error and next calculating

(benefit given error) /(highest cost given error),

and (highest benefit given error) /(lowest cost

given error). Some of these procedures are also

applicable to CEA.

much dependent on the quality and availability

of outcome data. It would seem, however, that

much more methodological development needs

to take place with respect to CEA/CBA before

these techniques can be used with known reliability and validity in psychotherapy assessments. The substantive literature describing the

application of these methods to psychotherapy

is examined in the next chapter of this report.
. . . .

6.

Illustrative Cost-Effectiveness

and Cost-Benefit Analyses
6

Illustrative Cost- Effectiveness

and Cost= Benefit Analyses

As noted in the previous chapter, the application of cost-effectiveness and cost-benefit analysis (CEA/CBA) to psychotherapy is fairly recent

and probably far less developed than other

areas of psychotherapy research. Not surprisingly, therefore, there are a number of problems

with the available CEA/CBA research. Some of

these problems have to do with the methodological limitations of the ways in which psychotherapy’s outcomes have been assessed (see chs.

2 and 3); additional problems relate to the way

CEA/CBA methods have been employed. CEA/

CBAs of psychotherapy, for example, focus

solely on the costs of treatment and pay minimal attention to outcomes. Others use single or

insensitive measures of outcome. Still others fail

to measure costs or outcomes comprehensively,

and many CEA/CBA studies use accounting

costs, rather than opportunity costs. CEA/CBA

studies that use only accounting costs potentially underestimate the actual amount of resources

required to provide psychotherapy treatments

and the benefits that may accrue from their application. At present, CEA/CBA analyses of

psychotherapy seem to have neither the methodological rigor, nor the breadth of application,

to be centrally useful in assessing psychotherapy’s value (see 2).

Despite the problems with CEA/CBA research, however, it does not follow that CEA/

CBA studies of psychotherapy cannot or should

not be conducted. As described in chapter 5,

there are methods for valuing the resources used

by psychotherapy; in addition, methods for

valuing the outcomes of psychotherapy, though

less well developed, can be applied. To the extent that CEA/CBA is a useful adjunct to the

health policy decisionmaking process (see 203),

its application to psychotherapy seems as reasonable (see, e.g., 179) as its application to

other health problems. The use of CEA/CBA

studies may provide information about the use

and impact of psychotherapy that is otherwise

unavailable. At the very least, such studies may

help structure the questions and data about

psychotherapy in a way that should aid policymakers in choosing among alternative programs. CEA/CBA may also be useful in stimulating better research about the effects of

psychotherapy. When resource-effect relationships are considered as in CEA/CBA, aspects of

psychotherapy are highlighted which otherwise

might be ignored; thus, for example, cost analyses may suggest assessing a wider range of variables than would usually be considered theoretically relevant.

Although CEA/CBA studies of psychotherapy currently appear to have limited policy

usefulness, it should be recognized that data

from such studies sometimes provide helpful

policy information. There are reliable and valid

data which indicate that some conditions of offering psychotherapy are more cost effective or

cost beneficial than others. Although these data

may have limited generalizability, they may be

helpful in making specific resource allocation

decisions (e.g., reimbursement policy for mental

health treatments). The data also suggest promising areas for future research.

The review below describes selected aspects

of a number of recent cost-effectiveness and

cost-benefit studies of psychotherapy. The focus

is on individual studies, because syntheses of the

CEA/CBA literature in psychotherapy are not

yet available. In general, the available CEA/

CBA studies do not compare different psychotherapy treatments—instead, they deal with the

factors that affect the provision of psychotherapy treatments. Thus, a major emphasis of

CEA/CBA studies conducted to date has been

differences among treatment settings. Another

emphasis has been the analysis of conditions

under which either low-cost therapies or potentially high-benefit treatments have been used.

63
64 ● Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

Because of the difficulties associated with

measuring psychotherapy’s benefits (see ch. 5),

the literature is probably biased toward assessing low-cost treatments. This characteristic of

the available studies should be kept in mind.

The attempt in the present review is not to

provide a comprehensive analysis of the CEA/

CBA literature, but to illustrate the types of

studies available and their findings. A number

of the studies reported here were included, in

some form (e.g., were considered within a review that was analyzed), in the discussion of effectiveness in chapter 4. They are described in

more detail in the present chapter because of

their explicit consideration of resource issues

(i.e., assessment of cost and/or benefits). Included in the present chapter are a number of

studies that assess effectiveness poorly and contain other methodological flaws. These studies

are included to suggest the potential contribution of CEA/CBA research on psychotherapy

and to identify some of the problems of conducting CEA/CBA.

The discussion below is organized according

to the treatment factors and patient conditions

to which CEA/CBA has been applied. First, the

review considers a number of treatment-related

characteristics that have been subjected to

CEA/CBA. Next, it considers therapist factors

as they relate to costs of treatment and outcomes. Finally, several specific patient conditions and CEA/CBA studies of psychotherapy

treatments applied to these conditions are

reviewed.

TREATMENT-RELATED CHARACTERISTICS

As noted above, a focus of psychotherapy

CEA/CBAs conducted to date has been characteristics of the treatment system in which psychotherapy is provided. Some studies have focused on the provision of psychotherapy under

various forms of institutional care, and some

have compared, for example, institutional and

outpatient/community-based treatments. The

results of these comparisons are not easy to

summarize, although it does seem clear that the

conditions of treatment can vary widely in

terms of the resources they use and their impact.

Inpatient Therapy

One area in which CEA/CBAs of psychotherapy have frequently been done concerns institutionalization of mental patients and the provision of inpatient psychotherapy. The costs of

institutionalizing a patient are high, and the

costs of providing therapy (in addition to caring

for the patient) are potentially even higher. A

number of studies have examined the costs and

benefits of various treatments offered patients,

in particular, the provision of therapy along

with custodial or milieu care. The conclusion of

a number of these studies is that the provision

of psychotherapy can provide more cost-effective or cost-beneficial treatment (e.g., 24,130,

175,273,289). These findings were obtained because therapy may reduce the amount of hospitalization or achieve larger benefits in some

other area.

McCaffree (175), in one study of the benefits

of “intensive” inpatient treament, reviewed the

costs and results of treatment in the Washington

State hospital system during two periods. During the two periods, there was a shift in the

treatment of institutionalized patients. In the

first period, only custodial care was provided;

in the second period, various forms of psychotherapy (which was called “intensive therapy”)

were provided as well. McCaffree assessed the

costs for both types of treatment. He included

both public costs (e.g., subsistence and treatment) and private costs (e. g., loss of patient income). McCaffree found that the costs per patient in intensive therapy were about 50 percent

less than the costs of custodial care. Moneys

were saved in intensive therapy primarily

because patient stays were much shorter (22 v.

42 days). Unfortunately, McCaffree’s data are

nonexperimental, and some of his data indicate

that the intensive therapy patients may have

been less disabled than the custodial care group.
Ch. 6–Illustrative Cost-Effectiveness and Cost-Benefit Analyses ● 65

In addition, McCaffree ignores psychological

measures of effectiveness. It is possible that the

intensive therapy group was no better after

receiving treatment, but that criteria for patient

discharge had been altered between the two data

gathering periods. Also important is the fact

that not only was more psychotherapy offered

in intensive treatment, but (see 173) drug treatment was introduced at the same time.

A later study by May (173) assessed the cost

effectiveness of milieu care (see also 24) and four

other treatment methods. A patient in milieu

care is provided a therapeutic environment

(nurses and others who deal with the patient are

specially trained). May used experimental procedures to assign schizophrenic patients to treatment conditions and assessed both costs and effectiveness. He found that milieu therapy was

almost as costly as psychotherapy, but was the

least effective of the five treatments. Drugs

without psychotherapy were the least costly

treatment, and drugs with psychotherapy were

the most effective. It is difficult to determine

whether May’s results are generalizable to other

patient conditions and psychotherapy/chemotherapy as they are currently provided.

Community= Based Versus

Institutional Therapy

Although it appears to be more cost beneficial

and cost effective to provide inpatients with intensive treatment, other alternatives have been

studied. Thus, a major focus of CEA/CBA studies has been community-based approaches for

providing psychotherapy. Community care has

the potential to provide major cost savings,

because such care reduces the costs not directly

associated with treatment (i. e., the costs of

housing, feeding, and supervising an institutionalized patient are higher than the costs of the

same services provided in the community).

Thus, for example, Binner, Halpern, and Potter (22) used data from almost 600 patients

who had received either inpatient or outpatient

treatment. Binner, et al. quantified intensity according to a factor-analyzed combination of

resource utilization, therapeutic involvement,

support from the therapeutic environment, and

length of stay (see 247). Binner, et al. found that

less intensive treatment resulted in higher benefit/cost ratios. Categorizing patients by severity

of dysfunction at admission, and stratifying

therapy intensity at four levels, Binner, et al.

showed that benefit/cost ratios were greater

than 1 (benefits exceeded costs) for all combinations of impairment and intensity, but were significantly larger for low-intensity therapy provided to less impaired patients. Average benefit/cost ratios ranged from 2.28 for patients

treated more intensively to 5.23 for patients

given the least intensive therapy. These results

suggest that therapy may not be effective for

severely impaired populations and that resources should be directed to outpatient

treatment.

Although Binner, et al.’s (22) study collected

data on outcomes as well as cost, it is nevertheless difficult to generalize from it. One problem

is that patients were not randomly assigned to

different levels of therapy intensity. The classification scheme for intensity, though mathematically sophisticated, cannot control for

possible differences in the type of patient who

was assigned to more versus less intensive

therapy. It is also not clear whether the patients

who participated in community-based treatment had dysfunctions as severe as those of the

inpatients. Furthermore, the outcome measure

used by Binner, et al. was a gross measure based

on a therapist rating. The valuing of this benefit

was done by assigning an arbitrary dollar

amount to changes in therapists’ ratings. Finally, Binner, et al. used operations costs, rather

than opportunity values, and this may have led

to either an underestimate or overestimate. It is

difficult to determine what bias was incorporated and how this relates to the procedure they

used to value therapist-rated changes.

In another study of an outpatient treatment

program, Washburn, Vanicelli, Longabaugh,

and Sheff (292) studied a “daycare” treatment.

Daycare is actually a modified form of inpatient

treatment where patients return to their own

homes (or families) each evening. The researchers collected a variety of effectiveness data, including checklists completed by patients, those
66 Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

who knew them well, and their therapists. Patients were assigned randomly to daycare or

standard inpatient care. Analyses of effectiveness data collected at 6-month intervals during

therapy and at a 2-year followup found only a

few significant differences. Daycare patients

evidenced somewhat less subjective distress,

functioned better in the community, and were

less of a burden to their families than patients

who received standard inpatient therapy. Cost

data also collected at 6-month intervals, however, showed significantly lower costs for daycare. The lower cost of daycare treatment seems

to be the primary determinant of cost effectiveness, although daycare also seems to have improved effectiveness to some degree. What is

missing from Washburn, et al. ’s study, unfortunately, is a no-treatment or a placebo treatment condition, which would allow inferences

about how much of the outcome is caused by

factors not related to treatment procedures (i.e.,

spontaneous remission), or by the patient’s

expectations.

In another comparative study of community

versus institutional treatment, Murphy and

Datel (192; see also 57) projected the costs and

benefits for 52 mentally ill and mentally retarded patients who were placed in the community from State institutions. The costs of community care included housing and subsistence,

as well as the costs of community treatment.

Benefits included the cost savings of not having

to provide institutional care and the wages/

fringe benefits received from jobs. Costs and

benefits were adjusted for present value (a 0.08

discount rate was used) and inflation. Murphy

and Datel’s results were organized in terms of 12

patient categories. Their findings indicated that

10-year projected benefits exceeded 10-year projected costs of community care, yielding benefit/cost ratios of between 0.99 to 11.86. T h e

average ratio was substantially greater than 1,

indicating that community care was superior.

Murphy and Datel’s (192) study, although it

considered a broader array of benefits and costs

than most CBA studies and used present value

and inflation adjustment procedures, does have

some methodological problems. Although it

was recognized that there are psychological benefits to patients and their families and data on

such things as marriage, normality of appearance, mobility in the community, and employment were presented, these data were not used

in the benefit/cost ratios. Valuing such benefits

is admittedly difficult, but Murphy and Datel’s

analysis, as presented, may underestimate therapy’s benefits. Another problem with this study

that limits its validity is that patients who refused community placement were not included

in the analysis. These individuals were considered treatment failures and excluded from the

analysis. If a significant number of these patients returned to the institution, costs of establishing them in the community (“set-up” costs)

and of readmitting them to institutions should

have been incorporated in the CBA. The study

also didn’t consider cost ratios for an alternative

program. As discussed in the preceding chapter,

there always is an implicit alternative to which a

cost-benefit calculation is compared and not all

programs with benefit/cost ratios exceeding 1

should be continued. An obvious comparison

group for the Murphy and Datel study would

have been a randomly selected group from the

potentially deinstitutionalizable population that

was retained in the institution or was released to

another program.

Cassell, Smith, Grunberg, Bean, and Thomas

(32) obtained cost and effectiveness data for

almost 500 deinstitutionalized patients, most of

whom had been diagnosed as chronic schizophrenics. These patients had resided in mental

hospitals for an average of 18.2 years prior to

deinstitutionalization, and the costs of hospitalization had totaled $70,000 per patient before

release. After deinstitutionalization, the costs of

welfare, followup, drugs, and rehospitalization

for those who could not adapt to community

living was only $1,575 annually, or 2.44 times

less than the cost of institutionalization. Although very few of the patients would have

been discharged had their deinstitutionalization

program not begun, 49 percent of the men and

38 percent of the women under age 65 were employed at least 3 months during the 2-year

postrelease period. More than 20 percent of the

men and 10 percent of the women were employed for at least 13 months following their
Ch. 16—Illustrative Cost-Effectiveness and Cost-Benefit Analyses ● 67

release. Utilization of medical care by the deinstitutionalized patients was also significantly

less than for normally adjusted persons in similar groups. In sum, the cost was less and effectiveness greater for noninstitutional compared

to institutional care provided to severely disturbed patients. Other studies of reinstitutionalization, which similarly have tracked patients

after release, have yielded similar findings (see,

e.g., 79,156,191).

An important methodological limitation of

these deinstitutionalization studies is their use

of nonexperimental designs. Typically, these

CEA/CBA studies follow deinstitutionalized patients, accounting for the resources they require.

This is potentially problematic given the fact

that new treatments were simultaneously being

introduced within mental hospitals (see 173,175)

and the fact that patients who were reinstitutionalized are likely to be less disabled than

those who remain in the hospital. There is also a

problem of adequately assessing the costs of

community care. Although most studies include

major housing and support costs for maintenance of patients in the community, a comprehensive accounting of community costs may be

difficult (especially in comparison to the obviousness of many costs borne in a hospital). As

a result of these factors, the high benefit/cost

ratios for community treatment may be overly

optimistic.

The study reported by Weisbrod (296; se e

also 297) illustrates the implications of considering these factors comprehensively in a CBA.

Weisbrod compared traditional therapy in a

mental institution against a community care

program using random assignment of patients.

Therapy in the mental institution was brief

(usually less than 1 month) and was followed by

community care delivered by local mental

health agencies. The usual stay in the hospital

was 17 days, but many patients returned to the

institution. Therapy in the alternative community-based program involved relatively little

time in institutions (i.e., every effort was made

not to hospitalize patients). During this 14-

month “community living” trial, patients had to

cope with normal living conditions and were required to take responsibility for the problems

that might, in other circumstances, result in

their return to an institution.

Sixty-five patients were included in each

therapy condition and data were collected for

the first 12 months of treatment. Weisbrod’s

(296) cost assessment procedures were comprehensive and included a variety of direct and indirect costs of treatment. Included also were law

enforcement costs and family burden costs (e. g.,

lost wages caused by the patient). Weisbrod’s

results indicated that community care was

slightly more costly, but yielded higher benefit

than the institutional treatment program. The

costs were higher in the community program because patients were closely supervised and received intensive therapy. The benefits of community treatment were higher primarily because

the outpatients earned more from employment.

Despite the use of random assignment and

comprehensive assessment of cost and benefits,

there are important limitations to the generalizability of Weisbrod’s (296) findings. Although

data were collected over a relatively long period

of time (4 years, from 1972 to 1976), each patient participated for only 14 months. That may

be too short a time to assess the effects of treatment. In addition, only a rather small number

of patients were included (less than 150), all

from the same geographic area. It is not clear

whether the program actually studied a representative sample of those eligible for reinstitutionalization. There is also no evidence that

either community or institutional treatment is

superior to no treatment, since a no-treatment

control condition was not included. Of course,

the patients in Weisbrod’s study were severely dysfunctional, and withholding treatment

would have been difficult.

Residential Versus Institutional

Therapy for Problem Children

Most of the aforementioned studies involved

schizophrenics and other severely disturbed

adults. Another typical application of CEA/

CBA in assessing mental health treatments are

studies which have examined the use of psychotherapy for “problem populations” (see 63). For

example, there has been great interest in study-
68 ● Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

ing the use of psychotherapy with juvenile delinquents. There is evidence that a substantial

number of serious crimes are committed by juveniles under 14 years of age (see 286), and there

is considerable correlational evidence that criminal behavior in youths is related to adult crime

and mental dysfunction (see, e.g., 233). T h e

magnitude of the costs associated with these

problems is very great.

A number of studies have investigated the use

of behavior modification procedures with troubled youths. Phillips (213), for example, presented data on the cost and effectiveness of a

residential token economy therapy for juvenile

delinquents called Achievement Place (see also

243). Token economy therapy is an application

of learning theory to establish a therapeutic

milieu. Along with the direct costs of treatment,

Phillips assessed effectiveness data on several

variables, including police and court contacts,

school attendance, and grades before, during,

and after therapy. Before therapy, Achievement

Place youths averaged about 3.8 contacts per

year with the police or courts. After 1 year of

therapy, Achievement Place delinquents averaged only 0.75 police and court contacts and no

contacts 2 years after therapy. In comparison to

a traditional program for juvenile delinquents (a

THERAPIST VARIABLES

Because the major proportion of resources

used in psychotherapy relates to the cost of personnel time, a number of cost comparisons of

different therapists have been conducted. Included in these studies is an analysis of the differences in fees charged by various professionals

and paraprofessionals and the impact of therapist variables on outcome.

Professional Therapists

Karen and VandenBos (132) report one of the

few available studies on the cost effectiveness of

different professional therapists. A small number of patients diagnosed as schizophrenics were

treated by psychologists, by psychiatrists, or by

special school), Achievement Place was much

more effective and one-third as costly ($6,000 v.

$20,000 to $30,000 per delinquent).

Although Phillips, et al. (213) provide very

positive cost-effectiveness data, it should be

noted that their findings are in contrast to those

of Powers and Witmer (217). In what has become a classic study (known as the CambridgeSomerville youth study), 650 youths were

matched and randomly assigned to either longterm counseling and supervision or no mental

health treatment. The participants in this study

were tracked from 1939 to 1976, and data on a

variety of psychological and social dimensions

were collected (see also 178). Unfortunately, no

long-term benefits can be demonstrated, and

there is some disturbing evidence that the therapy-treated youths have faired worse than those

who received no treatment (on such criteria as

alcoholism and later mental health). It is important to recognize, however, that the treatment

received by these youths may be far different

from that provided by current standards (see

269). In some respects, the Cambridge-Somerville approach is not comparable in rigor to

treatments such as Achievement Place (which

was based on a clear-cut theoretical rationale).

drugs without therapy. Karen and VandenBos

found that the cost of therapy provided by psychologists was substantially less than the cost of

therapy provided by psychiatrists, primarily

owing to lower hospitalization and drug use in

therapy provided by psychologists. The average

cost per patient treated by psychologists was

$7,813, but the average cost per patient was

$12,221 when psychiatrists provided treatment

and $17,234 when drugs alone were used in

therapy. Long-term costs due to recidivism also

favored psychologists over psychiatrists. Patients treated by psychologists were hospitalized

an average 7.2 days in a 2-year followup, compared to 93.5 days for patients treated by psychiatrists and 99.8 days for patients treated by

drugs alone.
Although these findings suggest that psychologists may be more cost-effective therapists

for schizophrenics than psychiatrists, the generalizability of these data is hard to determine. Fee

schedules for psychologists vary widely, and the

specific conditions of Karen and VandenBos’ research setting may have affected the results in

unknown ways. One important outcome was

that therapy given by either type of professional

was less costly than drugs alone: 17 percent less

when provided by psychiatrists, and 33 percent

less when provided by psychologists. While,

again, it is difficult to know the generalizability

of these findings, these data suggest an area for

further CBAs.

In a related study, Karen and VandenBos

(133) report that therapy provided by better

trained therapists is more effective, though

more costly, than therapy provided by less

trained therapists. The reduction in costs results

from generally shorter lengths of stay and other

savings in types of services required. This finding was obtained despite a gross underestimate

of the actual cost of hospitalization (132). The

generalizability of such studies, however, may

be limited to the type of patient they used.

There is evidence, for example, that hospitalized neurotics can be treated by nurse therapists

with better cost/benefit ratios than with therapists who have more training (i.e., a doctorate),

Ginsberg and Marks (95) assessed the effectiveness of treatment by nurse therapists for

brief behavioral psychotherapy (patients used

an average of 9 sessions) for neurotic and

phobic patients. Ginsberg and Marks used data

from 1 year prior to therapy and 1 year posttherapy to estimate the impact of therapy. The

data indicated that therapy significantly reduced symptomology and resulted in a number

of tangible (i. e., valued) benefits. The benefits

included reduced use of medical services and improved work productivity. Since the study did

not employ comparison groups, it is impossible

to estimate the comparative effects of nursetherapists; however, the study does suggest a

potentially useful direction for future analyses.

Gabby and Leavitt (89) report similar findings

for therapy on a population of neurotics. A selfsupporting nonprofit clinic provided long-term

therapy and was able to charge an average of 50

percent of the fees charged by private practitioners in the area ($13.50 to $15/hour v.

$19.95/hour at that time). A key factor in the

clinic’s being able to offer lower fees was its extensive use of nonpsychiatric staff. Psychiatrists

were called in only when necessary for consultation. Although Gabby and Leavitt provide an

example of how costs can be reduced, they do

not report outcome data in any detail, it is unclear whether the effects of treatment in the

clinic setting were as good as those in traditional

settings. In addition, it is unclear whether patients at the clinic were more or less disturbed

than other patients in the community. There is

also some reason to suspect that they attracted a

select population of patients.

Nonprofessionals

One focus of CEA/CBA studies has been the

training of parents to provide therapy to disturbed children (see, e.g., 111,211,290). In one

study, Rubenstein, Armentrout, Levin, and

Herald (243) placed 36 emotionally disturbed

children in normal homes. Parents residing in

these homes had received training in child management skills and received salaries and expense

reimbursement for caring for the disturbed

children. Lengths of stay ranged from 9 to 26

months, and the patients were supervised by

other “parent therapists. ” Professional mental

health specialists also met weekly with parents.

Psychometric tests, grades, and a behavior

checklist showed that children treated by parent

therapists were as improved as comparable

children treated in two residential programs.

Costs of parent-therapist treatment were only

half of residential treatment costs ($30.60 v.

$63.77 per child per day).

What seems to have reduced costs was the

distribution of professional expertise over a

number of “helpers. ” This simple modification

of the traditional way of providing psychotherapy (direct contact between professional therapists and patient) would be expected to reduce

costs. However, the cost reductions suggested

by the Rubenstein, et al. (243) study probably

are overestimated because they do not include

many of the costs of training parents. The cost
70 ● Background Paper #3: The Efficacy and cost Effectiveness of Psychotherapy

findings also are limited by Rubenstein, et al.’s

use of “comparable” rather than randomly selected comparison groups. Perhaps the children

assigned to the parent-therapist program were

less severely disturbed or had characteristics

that predisposed them to benefit more from

parent therapy than other children. As is typical

of many CEA and CBA studies, costs used in

Rubenstein, et al.’s study were accounting costs,

rather than opportunity costs. These costs may

well underestimate the use of resources.

Other CBA and CEA studies have explicitly

compared the use of professional versus paraprofessionals. Yates (306), for example, reports

a study of paraprofessionals who conducted

therapy for obese patients. The therapy was less

than one-tenth as costly as therapy provided by

psychologists and psychiatrists, but appeared to

be equally effective. Effectiveness was measured

directly in terms of the number of pounds lost.

Yates’ study, however, used a quasi-experimental design in which patients selected their own

therapy, and it may be that those who selected

the paraprofessional treatment condition believed they needed less help to lose weight (i.e.,

were most motivated). Also, while it can be assumed that nonprofessionals are less costly than

professionals, costs were not directly reported

(see also 282).

Self-directed therapy, with or without consultation with a professional, has also begun to

receive some attention (see 96). Many of these

studies have focused on dysfunctions typically

not treated by professional psychotherapy, but

there are some examples with depression, anxiety, phobias, and sexual dysfunctions. Such therapies involve the use of a book or manual to

direct the individual and have been mainly

based on learning and phenomenological theories. Case studies of therapies directed by

manuals have yielded results indicating that

positive effects can be obtained with such lowcost therapies (e. g., 294,299), but a thorough

evaluation of the effectiveness and cost effectiveness of these procedures has not been done.

The lack of comprehensive cost data in such

studies further limits any conclusions about cost

effectiveness or cost benefit.

One formal cost-effectiveness study of selfdirected therapy was conducted by Marston,

Marston, and Ross (170). Marston, et al. mailed

obese patients weekly weight-reduction readings (“bibliotherapy”) and had them respond to

written questions about the readings and

weight-loss problems. They found their form of

therapy to be as effective as weekly visits with a

professional therapist and much less costly. Unfortunately, the range of cost and effectiveness

data collected was limited; in addition, these

programs have not been compared with traditional psychotherapy treatments. Further studies have shown that some contact with a professional or paraprofessional therapist is necessary

for these bibliotherapies to be effective compared to no-treatment groups, even if this contact is made by phone or mail (305,307).

DRUG ADDICTION AND MEDICAL UTILIZATION

The studies cited above principally focus on

CEA and CBA of different ways of offering psychotherapeutic treatments. CEA and CBA reports have also been developed to study treatments for problems such as alcohol and heroin

addiction. These are high-cost problems to society and are also problems that have highly visible costs and benefits/effects. Overutilization

of medical services, also a costly problem, has

also received considerable attention (e. g., 60,

129), and it has a number of implications for

how and under what conditions psychotherapy

should be evaluated in a national health insurance program. Below several CEA/CBA studies

of treatment for these high-cost problems are

described.

Psychotherapy for Drug Addictions

The obvious relationship between alcohol

and opiate addictions and subsequent psychological dysfunction, coupled with the high

societal costs of such addictions, has stimulated

a number of CEA and CBA studies of therapies
Ch. 6–Illustrative Cost-Effectiveness and Cost-Benefit Analyses ● 71

for drug addictions. Involvement of multidisciplinary teams in therapy for drug addictions

seem to have generated more comprehensive

CEA and CBA in this area than in any other

and, in general, the findings of these studies is

that therapy is beneficial. This conclusion must

be tempered, however, by recognizing that

treatments for drug addiction almost always involve more than psychotherapy.

Rufener and his colleagues (244,245), for example, conducted a CBA of five different therapies for heroin addiction: 1) methadone maintenance, 2) therapeutic community, 3) outpatient

drug-free treatment, 4) outpatient detoxification, and 5) inpatient detoxification. Benefits

were calculated by estimating social costs incurred directly and indirectly as a result of heroin use (e. g., crime, incarceration, court costs),

the reduction in costs produced by rehabilitating a heroin abuser, and adjustment of benefits

for the relative effectiveness of different programs. Costs were assessed from accounting

records of providing each therapy. Rufener, et

al. considered benefits under three assumptions

of the size of heroin abuser population and three

different discount rates for present-valuing. The

resulting ratios were all greater than 1 (benefits

exceed costs) and showed that outpatient drugfree therapy to be the most cost beneficial.

Drug-free therapy yielded a 12.82 benefit/cost

ratio for the intermediate abuser population size

(and included discount rate assumptions). A

possible problem of this has to do with estimated adjustment cost savings for differential

relapse rates which may have been overly optimistic. In addition, the study did not use random assignment of patients to different treatment techniques, and it is difficult to determine

the possible effects of the research design.

In another study, Hall, Bass, Hargreaves, and

Loeb (112) report a 20-percent reduction in the

use of opiates and barbiturates for outpatient

detoxification patients. Random assignment of

subjects to behavior therapy and no treatment

conditions was accomplished, and effectiveness

was assessed by urine tests. The behavior therapy consisted of reinforcing patients (by paying

them up to $10/day) for drug-free urines. Depending on the day urines were collected, 40 to

50 percent of the paid subjects had not used

drugs. Since most estimates of the daily cost to

society of heroin use are five or more times that

cost, this reinforcement contingency seems

promisingly cost beneficial, even when personnel, facility, and testing costs are added. Other

data suggest that patients did not use their

payments to purchase illegal drugs.

Sirotnik and Bailey (262), in a similar study,

conducted a CBA of heroin addiction therapies.

Their investigation involved 285 patients over a

1.5-year period. They found that comprehensively defined benefits exceeded costs by more

than a 2.5 ratio under each of a range of costsavings assumptions. This analysis probably is

conservative, because it did not consider benefits that might accrue to patients and society

after the program. Unfortunately, nonrandom

assignment to therapies and the absence of a

control group limit, the implications of this

study.

Even larger benefit/cost ratios may be produced by long-term therapy for addicts, although there is much debate on this point. Aron

and Daily (8), for example, found that longterm therapy was more effective and, in total,

less costly than short-term therapy for drug addiction when costs of re-entry to therapy and

long-run effectiveness were assessed. Cost-effectiveness ratios were $4,624 per successfully

treated addict in long-term therapy, but $5,988

per successful patient in short-term therapy.

Maidlow and Berman (167) contrasted the

cost effectiveness and cost benefit of a drugfree therapeutic community and methadone approaches to treating heroin addiction. The average stay of 4 years in drug-free communities was

found to generate a 65-percent success rate,

resulting in a direct cost per successful patient of

$17,760. Methadone or other drug substitution

had a higher, 87-percent success rate at a lifetime cost effectiveness of $45,000 per successful

patient. Considering the probability that former

addicts would leave therapy, and present-valuing future benefits and costs (thus reducing lifetime cost of drug substitution), it would seem

drug substitution is more cost beneficial than

drug-free therapeutic communities. However,

Maidlow and Berman’s study, although it uses
72 ● Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

sound CBA methodology, did not use random

assignment or a no-treatment control group and

there may be important biases in who selects

particular treatment.

Goldschmidt (101) described a comprehensive

model of CEA for health care, applying it to

analysis of heroin addiction therapies. Sampling

1,640 patients over a 6-month period he found

1,241 who could be interviewed and used data

from them to compare the cost effectiveness of

drug substitution (methadone) to the therapeutic community. Using several variables to assess

effectiveness and operations cost for the two

approaches, high variability was found in nonrecidivism measures of effectiveness (10.9 t o

33.7 percent for methadone therapy, depending largely on length of stay, v. 12.5 to 47 percent for therapeutic communities). Goldschmidt

found that the annual cost per patient of the

therapeutic community was about four times

the cost of drug substitution. Because therapeutic communities were found to treat more

patients and have somewhat more persistent effects, the average cost/successful patient ratio

was 1.7 times higher for therapeutic communities than for drug substitution, thus favoring

drug substitution as more effective for its cost.

This finding was maintained for both “normalcy” (i. e., some use of drugs, but normal

functioning) and “heroin-free” criteria of effectiveness. This study shows the impact that treatment system costs can have on CEA findings.

Consideration of lifetime costs of methadone

maintenance, however, might reverse the direction or magnitude of the findings. No information was provided on opportunity value and

comprehensive costs and, as well, no-treatment

control groups were not evaluated.

To summarize the status of CEA and CBA for

drug addictions, while this area has received

much attention, there are still serious shortcomings with the available research. These

shortcomings have to do primarily with the research designs used. Although some of the findings are impressive in that consistent cost/benefit ratios greater than unity are found, there

may be other explanations for these results. In

addition, it should be recognized that drug

abuse remains a serious problem and it is not

clear that the available studies are representative of the treatment programs being conducted.

Psychotherapy and Overutilization of

Medical Services

Depending on the definition used, it has been

estimated that medical services are overused (or

in some other way abused) by between 20 and

60 percent of those who seek them (e.g., 54,

174,182). Various researchers and analysts have

concluded that medical services often are used

to ameliorate mental problems, rather than the

treatment of physical dysfunctions. Regier, et

al. ’s (225) data on the role of primary care

physicians in the delivery of mental health services is one indication of this problem (see ch. 1).

It has been suggested that psychotherapy may

be used to reduce individuals’ dependence on

medical services. Thus, for example, a 4-year

study by Cavanaugh (cited in 119) found that

hospitalization for physical ailments was reduced from an average 111 to an average 53

days by psychotherapy, resulting in a $1.1 million savings (which was greater than psychotherapy costs).

Cummings and Follette (53; see also 99) reported that a single session of psychotherapy reduced utilization of medical services for high

utilizer patients by 60 percent over a 5-year

period. Additional sessions further reduced

medical care utilization: patients attending two

to eight sessions subsequently decreased medical

use by 75 percent. Continuing their study on use

of medical services in the large Kaiser-Permanente health insurance plan, Cummings (51)

used findings from Cummings (50) and Cummings and Follette (52,53) to contrast the benefits of medical cost savings to costs of four durations of psychotherapy.

Cummings formed benefit/cost ratios by dividing medical care utilization (number of visits) for the year preceding therapy by the sum of

subsequent medical care and psychotherapy visits. Very brief psychotherapy (one to four sessions) generated “cost-therapeutic effectiveness”

(actually crude benefit/cost) ratios of 2.59. Psychotherapy lasting from 1 to 15 sessions had a

similarly positive benefit/cost ratio of 2.11.
—

Long-term therapy (more than 16 sessions) and

interminable therapy were found to be less costbeneficial (benefit/cost ratios of 1.14 and 0.91,

respectively). Not receiving any psychotherapy,

however, generated the worst ratio of all: 0.88.

Other investigators have found specific reductions in use of physician services, especially

laboratory and X-ray services. In one report,

Goldensohn and Fink (100) found that these effects followed after the administration of psychological tests. A number of other studies

show similar, and often monetized, reductions

in medical care utilization following a variety of

forms of psychotherapy, including brief treatment (e. g., 125). In many of the studies, reductions in medical utilization have been directly

related to reduced costs.

Jones and Vischi (129), at the National Institute of Mental Health, have conducted a comprehensive review of the literature on the “offset” benefits of psychotherapy treatments. They

reviewed 25 studies that examined the impact of

therapy for mental illness, alcohol abuse, and

drug abuse on medical utilization. Their conclusion was that, under certain circumstances,

medical care utilization does “appear” to be

reduced as a result of therapy. In particular, for

treatments of mental illness, they found 12 studies (out of 13 available studies) which demonstrated reduced medical care utilization following therapy. The median reduction was 20 percent, and the range (for the 12 studies) was 5 to

85 percent. The reduction in medical care utilization was accompanied by parallel reduction

SUMMARY

This chapter has provided a number of illustrations of the use of CEA/CBA methods for

assessing a variety of psychotherapy treatment

alternatives and mental health problems. The

conduct of CEA/CBA studies of psychotherapy

research is not widespread, and many important

areas and types of psychotherapy have been

ignored to date. The review in this chapter suggests that, for some treatment system characteristics and for some problems, psychotherapy

in cost (mental health treatments being typically

less costly than medical treatment). The investigators report a variety of benefit/cost ratios,

which range from 0.95 to 2.11.

Jones and Vischi (129) also reported a variety

of studies in which psychotherapeutic treatments were applied to alcoholism and drug

abuse. These studies report similar findings (of

reduced medical care utilization) although they

tended to have more methodological problems

than the mental illness studies. A particular

problem for all of the studies was the frequent

lack of appropriate control group conditions.

Another problem was the selection of time periods for analysis. Typically, patients entered

these studies after a period of high use of services and it is difficult to attribute the reduction

solely to the mental health treatment (it could

reflect a natural change in medical needs). Jones

and Vischi, although optimistic about the use of

psychotherapy to reduce medical utilization,

recommend more rigorous research that includes better cost-benefit data.

In a number of cases, psychotherapy has been

employed as an adjunct to medical services

(e.g., 143,204,215,238). Although there is not

yet a substantial literature describing the effectiveness of psychotherapeutic treatments used in

this way, such fields as “behavioral medicine”

are being developed. It would seem that as the

effectiveness literature on such behavioralmental treatment matures, cost and benefit data

should also be assessed.

appears to be cost beneficial and that it can be

made more efficient (improved cost effectiveness). These findings must be considered tenuous, however, given the methodological problems inherent in many of these studies. What is

needed is a more systematic application of

rigorous research designs that employ no-treatment and placebo treatment conditions, as well

as more comprehensive measures of outcome

and cost.
74 ● Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

It would seem that we possess the scientific potential to improve well-being and to be cost

tools to assess psychotherapy and to use this in- effective. Although one will probably never be

formation in making policy decisions about totally satisfied with the answers provided by

societal support for these treatments. It would research, that seems a poor reason not to atalso appear that we have substantial knowledge tempt” more rigorous and comprehensive scienwhich indicates that these treatments offer the tific analyses.
Appendixes

.*‘
Appendix A.— Description of

Other Volumes of the Assessment

The overall OTA assessment, The lmplications of

Cost-Effectiveness Analysis of Medical Technology,

consists of a main, policy-oriented report plus five

background papers. The present volume, The Efficacy and Cost Effectiveness of Psychotherapy, is

one of the background papers. The main report and

the other background efforts are briefly described

below.

The main report examines three major issues: 1)

the general usefulness of CEA/CBA in decisionmaking regarding medical technology, 2) the methodological strengths and shortcomings of the technique,

and 3) the potential for initiating or expanding the

use of CEA/CBA in six health care programs (reimbursement coverage, health planning, market approval for drugs and medical devices, Professional

Standards Review Organizations, R&D activities,

and health maintenance organizations), and most importantly, the implications of any expanded use.

The prime focus of the report is on the application

of CEA/CBA to medical technology (i. e., the drugs,

devices, and medical and surgical procedures used in

medical care, and the organizational and support systems within which such care is provided). With the

exception of the present background paper on psychotherapy, the report does not address psychosocial

medicine. Other aspects of health, such as the environment, are not directly covered either. The findings of the assessment, though, might very well apply

to health care resource decisionmaking in general,

and with modification, to other policy areas such as

education, the environment, and occupational safety

and health.

The main report contains chapters on methodology, general decisionmaking, each of the six health

programs mentioned above, and the general usefulness of CEA/CBA. It contains appendixes covering a

survey of current and past uses of CEA/CBA by

agencies (primarily Federal), a survey of the resource

costs involved in conducting CEA/CBAs, a discussion of ethical issues and CEA/CBA, and a brief discussion of legal issues.

Background Paper #I: Methodological Issues and

Literature Review, includes an in-depth examination

of the decisionmaking context and methodology discussions presented in this report. A critique of the literature, a bibliography of over 600 items, and abstracts of over 70 studies and other articles are also

included.

In order to help examine the applicability of techniques to assess the costs and benefits of medical

technology, 19 case studies were prepared. All 19 are

available individually. In addition, 17 of the cases are

available collectively in a volume entitled Background Paper #2: Case Studies of Medical Technologies. Some of the cases represent formal CEAs (e.g.,

the case on bone marrow transplants), and some represent net cost or “least cost” analysis (e. g., the case

on certain respiratory therapies). Other cases illustrate various issues such as the difficulty of conducting CEA in the absence of adequate efficacy and safety information (e.g., the case on breast cancer surgery), or the role and impact of formal analysis on

policymaking (e.g., the case on end-stage renal disease interventions). The 17 case studies in Background Paper #2 and their authors are:

Artificial Heart

Deborah P. Lubeck

John P. Bunker

Automated Multichannel Chemistry Analyzers

Milton C. Weinstein

Laurie A. Pearlman

Bone Marrow Transplants

Stuart O. Schweitzer

C. C. Scalzi

Breast Cancer Surgery

Karen Schachter

Duncan Neuhauser

Cardiac Radionuclide Imaging

William B. Stason

Eric Fortess

Cervical Cancer Screening

Bryan R. Luce

Cimetidine and Peptic Ulcer Disease

Harvey V. Fineberg

Laurie A. Pearlman

Colon Cancer Screening

David M. Eddy

CT Scanning

Judith L. Wagner

Elective Hysterectomy

Carol Korenbrot

Ann B. Flood

Michael Higgins

Noralou Roos

John P. Bunker

End-Stage Renal Disease Interventions

Richard A. Rettig

Gastrointestinal Endoscopy

Jonathan A. Showstack

Steven A. Schroeder

.-/,
78 . Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

Neonatal Intensive Care

Peter Budetti

Peggy McManus

Nancy Barrand

Lu Ann Heinen Nurse Practitioners

Lauren LeRoy

Orthopedic Joint Prosthetic Implants

Judith D. Bentkover

Philip G. Drew

Periodontal Disease Interventions

Richard M. Scheffler

Sheldon Rovin

Respiratory Therapy

Richard M. Scheffler

Morgan Delaney

The 18th case study is the present volume, Background Paper #3: The Efficacy and Cost Effectiveness

of Psychotherapy, The 19th case study was prepared

by Judith Wagner and is published separately as

Background Paper #5: Assessment of Four Common

X-Ray Procedures.

Background Paper #4: The Management of Health

Care Technology in Ten Countries is an analysis of

the policies, programs, and methods, including costeffectiveness and cost-benefit techniques, that nine

industrialized nations other than the United States

use to manage the effects of medical technology. The

experience of these nine countries in managing medical technology is compared to that of the United

States. The paper on the United States and the comparative analysis were prepared by OTA staff, assisted by Louise Russell. The authors of the papers on

the nine foreign countries are:

United Kingdom

Barbara Stocking

Canada

Jack Needleman

Australia

Sydney Sax

Japan

Joel Broida

France

Rebecca Fuhrer

Germany

Karin A. Dumbaugh

Netherlands

L.M.J. Groot

Iceland

David Gunnarson

Duncan vB. Neuhauser

Sweden

Erik H. G. Gaensler

Egon Jonsson

Duncan vB. Neuhauser

A related report prepared by OTA and reviewed

by the Advisory Panel to the overall assessment is A

Review of Selected Federal Vaccine and Immunization Policies. That study, published in September of

1979, examined vaccine research, development, and

production; vaccine efficacy, safety, and cost effectiveness; liability issues; and factors affecting the use

of vaccines. Pneumococcal vaccine was used as a

case study, and a CEA/CBA was performed.
Appendix B.— Health Program Advisory Committee,

Acknowledgments

HEALTH PROGRAM ADVISORY COMMITTEE

Frederick C. Robbins, Chairman

Dean, School of Medicine, Case Western Reserve University

Stuart H. Altman

Dean

Florence Heller School

Brandeis University

Robert M. Ball

Senior Scholar

Institute of Medicine

National Academy of Sciences

Lewis H. Butler

Health Policy Program

University of California, San Francisco

Kurt Deuschle

Professor of Community Medicine

Mount Sinai School of Medicine

Zita Fearon

Consumer Commission on the Accreditation of

Health Services, Inc.

Rashi Fein

Professor of the Economics of Medicine

Center for Community Health and Medical Care

Harvard Medical School

Melvin A. Glasser

Director

Social Security Department

United Auto Workers

Patricia King

Professor

Georgetown Law Center

Sidney S. Lee

Associate Dean

Community Medicine

McGill University

Mark Lepper

Vice President for Znter-institutional Affairs

Rush-Presbyterian Medical School

St, Luke’s Medical Center

Frederick Mosteller

Professor and Chairman

Department of Biostatistics

Harvard University

Beverlee Myers

Director

Department of Health Services

State of California

Mitchell Rabkin

General Director

Beth Israel Hospital

Kerr L. White

Rockefeller Foundation

Acknowledgments

Appreciation is expressed to Dr. Brian Yates (American University) and Dr. Frederick Newman (Eastern

Pennsylvania Psychiatric Institute), who provided materials which were useful in preparing portions of this

report. In addition, thanks are expressed to the following students and staff from Boston University and the University of Pittsburgh, who assisted in various aspects of the report’s preparation: Alice Grossman, Gary

Menster, Amy Sonka, and Lois Weithorn.

Finally, appreciation is expressed to the individuals listed below, who, along with members of the OTA Advisory Panels on CEA and on Psychotherapy, provided reviews of various drafts of the report:

Ellen A. Andruzzi Patrick H. DeLeon Charles Kiesler

Psychiatric Nurse Office of Senator Inouye Carnegie-Mellon University

Arthur H. Auerback David Ehrenfried Daniel Koretz

University of Pennsylvania Blue Cross/Blue Shield Congressional Budget Office

Irving Chase Michelle Fine Marvin Kristein

National Mental Health Association Columbia University Economic Research Bureau

79
Mack Lipkin, Jr. Peter D. Richman Mary L. Smith

Rockefeller Foundation Clinical Social Worker University of Colorado

Beryce MacClennan Jeanne M. Safer David Upton

General Accounting Office Postgraduate Center for Mental Health Psychiatrist

John C. Nemiah

Beth Israel Hospital, Boston
References
References

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

Alberts, D. S,, A Plan for Measuring the Performance of Social Programs: The Application

of Operations Research Methodology (New

York, N. Y.: Praeger Publications, 1970).

Alcohol, Drug Abuse, and Mental Health Administration, National Data Book, ]anuary

2980, DHEW publication No. (ADM) 80-938

(Washington, D. C.: U.S. Government Printing

Office, 1980).

Allen, K. E., et al., “Effects of Social Reinforcement on Isolate Behavior of a Nursery School

Child, ” Child Dev. 35:310, 1964.

Alley S., et al. (eds. ), Paraprofessionals in

Mental Health: Theory and Practice (New

York, N. Y.: Human Sciences Press, 1979).

American Medical Association, Physical Distribution and Medica/ Licensure in the U. S,

(Chicago, 111.: AMA, 1977).

American Psychological Association, Ethical

Principles in the Conduct of Research With Human Participants (Washington, D. C.: APA,

1973).

Standards for Providers of Psychological Se~ices (Washington, D. C.: APA, 1977).

Aron, W, S., and Daily, D., “Short- and LongTerm Therapeutic Communities: A Follow-Up

and Cost-Effectiveness Comparison, ” Znt. ],

Addic. 9:619, 1974.

Attkisson, C. C., et al., “Evaluation: Current

Strengths and Future Directions, ” in Evaluation

of Human Service Programs, edited by C. C.

Attkisson, et al. (New York, N. Y.: Academic

Press, 1978).

Bandura, A., “On Paradigms and Recycled Ideologies,” Cog. Theory & Res. 2:79, 1978.

Principles of Behavior Modification

(New York, N. Y.: Holt, Rinehart and Winston,

1969. )

Social Learning Theory (Englewood

Cliffs, N. J.: Prentice-Hall, 1977).

, et al., “Relative Effectiveness of Desensitization and Modeling Approaches for Inducing Behavioral, Affective, and Attitudinal

Changes,” J, Pers. Soc, Psycho/. 13:173, 1969.

Barnhart, G., “Social Design and Operations

Research,” Pub, Health Rep. 85:247, 1979.

Beck, A. T., Diagnosis and Management of

Depression (Philadelphia, Pa.: University of

Pennsylvania Press, 1973).

Beck, N., “Cost-Benefit and Cost-Effectiveness

Analysis in Psychotherapy Outcome Research, ” manuscript submitted for publication,

University of Missouri, 1979.

17.

18.

19.

20.

21,

22.

23.

24.

25.

26.

27.

28.

28.

30.

31.

Bergin, A. E., “The Effects of Psychotherapy:

Negative Results Revisited, ” J. Colin. Psychol.

10:244, 1963.

“An Empirical Analysis of Therapeutic

Issues, ‘

; in Counseling and Psychotherapy: An

Overview, edited by G. Arbuckle (New York,

N. Y.: McGraw Hill, 1967).

, “The Evaluation of Therapeutic Outcomes, ” in Handbook of Psyclzotherapy and

Behavior Change, edited by A. E. Bergin and S.

L. Garfield (New York, N, Y.: Wiley, 1971).

“Some Implications of Psychotherapy

Resear;h of Therapeutic Practice, ” ], Abn.

Psycho/. 71:235, 1966.

/ and Lambert, M. J., “The Evaluation

of Therapeutic Outcomes, ” in Handbook of

Psychotherapy and Behavior Change: An Empirical Analysis, edited by S. L. Garfield and

A. E. Bergin (New York, N. Y.: WiIey, 1978).

Binner, P. R., et al., “Patients, Programs, and

Results in a Comprehensive Mental Health

Center,” ]. Cons, C/in, Psychol. 41:148, 1973.

Blanchard, E, B,, and Epstein, L. H., A Biofeedback Primer (Reading, Pa.: Addison-Wesley, 1978).

Boudewyns, P. A., “Is ‘Milieu Therapy’ in

a Short-Term Inpatient Psychiatric Setting

Worth the Money?” Znt. Merit. Health Res.

News. 16;7, 1974.

Bowman, M. J., “The Costing of Human Resource Development, ” in The Economics of

Education, edited by E. A. Robinson and J. E.

Vaizey (New York, N. Y.: St, Martins, 1966).

Budman, S. H,, et al., “Adult Mental Health

Services in a Health Maintenance Organization, ” Am. }. Psychi. 136:392, 1979.

Bunker, J. P., et al., Costs, Risks, and Benefits

of Surgery (New York, N. Y.: Oxford University Press, 1977).

Burgess, J., System Approaches to Public Ser-vice (Teaneck, N, J.: Fairleigh-Dickenson University Press, 1978).

Campbell, D. T., and Stanley, J. C., Experimental and Quasi-Experimental Designs for Research (Chicago, Ill.: Rand McNally, 1963).

Carter, D. E., and Newman, F. L., A ClientOriented System of Mental Health Service Delivery and Program Management: A Workbook and Guide, DHEW publication No.

(ADM) 76-307 (RockvilIe, Md.: National Institute of Mental Health, 1976).

Cartwright, D. S., “Patient Self-Report Measures, ” in Psychotherapy Change Measures,

83
84 ● Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

32.

33.

34.

35.

36.

37.

38.

39.

40.

41.

42.

43.

44.

45.

DHEW publication No. (ADM) 74-120, edited

by I. E. Waskow and M. B. Parloff (Washington, D. C.: DHEW, 1975).

Cassell, W. A., et al., “Comparing Costs of

Hospital and Community Care,” Hosp,

Comm. Psychi, 23:197, 1972.

Claiborn, W. L., and Kayton, R., “Analysis of

Mental Health Service Delivery: Profession,

Patient, Treatment, ” 1. Comm. Psychol. 2:18,

1974.

, and Stricker, G,, “Professional Standards Organization, Peer Review, and CHAMPUS, ” Prof. Psychol. 631, 1979.

and Zaro, J. S., “The Development of

a Peer ‘Review System: The APA/CHAMPUS

Contract, ” in Psychology and National Health

Insurance: A Sourcebook, edited by C. A.

Kiesler, et al. (Washington, D. C.: American

Psychological Association, 1979).

Cochrane, A. L., Effectiveness and Efficiency:

Random Reflections on Health Services (London: Nuffield Provincial Hospitals Trust,

1972).

Cohen, J., and Cohen, P., Applied Research/

Correlation Analysis for the Behavioral Sciences (Hillsdale, N. J.: Lawrence Erlbaum Associates, 1975).

Cohen, M., and Ewalt, P. L,, “An Intensive

Program for Severely Retarded Children, ” Soc.

Casework 56:337, 1975,

Cole, S., et al., “Peer Review and the Support

of Science, ” Sci. Am, 237:34, 1977.

Conley, R. W., et al., “An Approach to Measuring the Cost of Mental Illness, ” Am. /.

Psychi. 124:63, 1967.

Cook, T. D., and Campbell, D. T., Quasi-Experimentation: Design and Analysis Issues for

Field Settings (Chicago, Ill.: Rand McNally,

1979).

Cooper, B. S,, and Rice, D. P., “The Economic

Cost of Illness Revisited, ” Soc. Sec. Bull.,

39(2):21, 1976.

Cooper, E. M., Guidelines for a Minimum Statistical and Accounting System for Community

Mental Health Centers, series C, No. 7 (Rockville, Md.: National Institute of Mental Health,

1977).

Cowen, E, L., “Some Problems in Community

Program Evaluation Research, ” ]. Cons. Clin.

@cho1. 46:792, 1978.

“Some Problems in Community Program Evaluation Research, ” }. Cons. Clin.

Psychol. 46:792, 1978.

46.

47.

48.

49.

50.

51.

52.

53.

54.

55.

56.

57.

58.

59.

60.

Craighead, W. E., et al., Behavior Modification: Principles, Issues, and Applications (Boston, Mass.: Houghton Mifflin, 1976).

Cronbach, L. J., Designing Evaluations (Stanford, Calif.: Stanford Evaluation Consortium,

1978).

et al., The Dependability of Behavioral

Measurements: Theory of Generalizability for

Scores and Profiles (New York, N. Y.: Wileyr

1972).

Crow, W. J., “The Effect of Training on Accuracy and Variability in Interpersonal Perception, ” ]. Abn. Soc. Psychol. 55:355, 1957.

Cummings, N. A., “The Health Model as Entree to the Human Services Model in Psychotherapy, ” Clin. Psychol. 29:19, 1975.

“Prolonged (Ideal) Versus Short-Term

(Realis~ic) Psychotherapy, ” Prof. Psycho/.

9:491, 1977.

, and Follette, W. T., “Brief Psychotherapy and Medical Utilization: An Eight-Year

Follow-Up, ” in The Professional Psychologist

Today: New Developments in Law, Health Insurance, and Health Practice, edited by H.

Dorken and Associates (San Francisco, Calif.:

Jossey-Bass, 1976).

, and Follette, W. T., “Psychiatric Services and Medical Utilization in a Prepaid Health

Plan Setting (Pt. 2),” Med. Care. 5:31, 1968.

CupIan, R., and Davies, B., “Psychiatric Illness

at a Medical and a Surgical Outpatient Clinic, ”

Comp. Psychi. 1:228, 1960.

Dahlstrom, W. G., and Welsh, G. S., MMPZ

Handbook: A Guide to Use in Clinical Practice

and Research (Minneapolis, Minn.: University

of Minnesota Press, 1960).

Dahms, W. T., et al., “Treatment of Obesity:

Cost-Benefit Assessment of Behavioral Therapy, Placebo, and Two Anorectic Drugs, ” Am,

]. C/in. Nut. 31’:774, 1978.

Datel, W. E., and Murphy, J. G., “A ServiceIntegrating Model for Deinstitutionalization, ”

Admin. Merit. He. 3:3s, 197s.

Dawes, R. M., “The Robust Beauty of Improper Linear Models in Decision Making, ”

Am. Psychol. 34:s71, 1979.

Delaney, J. A., et al., “Crisis Intervention and

the Prevention of Institutionalization: An Interrupted Time Series Analysis, ” Am, ].

Comm. Psychol, 6:33, 1978.

DeLeon, P. H., and VandenBos, G. R., “Political Factors Related to the Coverage of Mental

Services in Federal Programs: Implications for
References ● 85

61.

62.

63.

64.

65.

66.

67.

68.

69.

70.

71.

72.

73.

74.

75.

76.

77.

the Reimbursement of Psychotherapy, ” in Psychotherapy: From Practice to Research in Policy, edited by G. R. VandenBos (Beverly Hills,

Calif.: Sage Publications, in press).

Denker, P. G., “Results of Treatment of Psychoneuroses by the General Practitioners, ”

N. Y. State ]. Med. 46:2164, 1946.

Department of Health, Education, and Welfare, National Health Survey, Runaway Youth:

A Status Report and Summary of Projects

(Washington, D. C.: DHEW, 1976).

Diagnostic and Statistical Manual of Mental

Disorders (3d cd., Washington, D. C.: American Psychiatric Association, 1980).

Dorken, H., “CHAMPUS Ten-State Claim Experience for Mental Disorders: Fiscal Year

197s, ” in Psychology and National Health insurance: A Sourcebook, edited by C. A.

KiesIer, et al. (Washington, D. C.: American

Psychological Association, 1979).

Durlak, J. A., “Comparative Effectiveness of

Paraprofessional and Professional Helpers, ”

Psychol. Bull. 36:80, 1979.

Edwards, A. L., and Cronback, L. J., “Experimental Design for Research in Psychotherapy, ”

J. Clin. Psychol. 8:51, 1952.

Endicott, J., et al., “The Global Assessment

Scale, ” Arch. Gen. Psychi. 33:766, 1976.

Epstein, S., “The Stability of Behavior: I. On

Predicting Most of the People Much of the

Time,” ]. Per. Soc. Psychol. 37:1097, 1979.

Eysenck, H. F., The Effects of Psychotherapy

(New York, N. Y.: International Science Press,

1966).

“The Effects of Psychotherapy, ” in

Handb;ok of Abnormal Psychology, edited by

H. J. Eysenck (New York, N. Y.: Basic Books

Inc., 1961).

“The Effects of Psychotherapy, ” Int. }.

Psychi.’ 1:97, 1965.

“The Effects of Psychotherapy: An

Evalua~ion, ” ). Cons, Psychol. 16:319, 1952.

“An Exercise in Megasilliness, ” Am.

Psycho’l. 33:517, 1978.

I “Note on ‘Factors Influencing the Outcome of Psychotherapy ’,” Psychol. Bull. 78:

403, 1972.

Fairweather, G. W., et al., Community Life for

the Mentally 11/: An Alternative to Institutional

Care (Chicago, Ill.: Aldine, 1969.

Fein, R., Economics of Mental Zlbzess (New

York, N. Y.: Basic Books, 1958.

Fishman, D. B., A Cost Effectiveness Method78.

79.

80.

81.

82.

83.

84.

85,

86.

87.

88.

89.

90.

91.

92.

ology for Rational Decision-Making in Mental

Health Services, paper presented at the Meeting

of the American Psychological Association,

San Francisco, Calif., Aug. 26, 1977.

“Development of a Generic Cost-Effectivenes~ Methodology for Evaluating the Patient Services of a Community Mental Health

Center, ” in Evaluation in Alcohol, Drug

Abuse, and Mental Health Service Programs,

edited by J. Zusman and C. R. Wurster (Lexington, Mass.: D. C. Heath and Co., 1975).

Foreyt, J. P., et aI., “Benefit-Cost Analysis of a

Token Economy, “ Prof. Psychol. 6:26, 1975.

Fox, H., “Toward an Understanding of Operations Research Concepts, ” Mgt. Sem, 7:23,

1970.

Fox P, D., and Kuldau, J. M., “Expanding the

Framework for Mental Health Program Evaluation, ” Arch. Gen. Psychi, 19:538, 1968.

Foxx, R. M., and Azrin, N. H., “Dry Pants: A

Rapid Method of Toilet Training Children, ”

Behav. Res. Ther. 11:43s, 1973.

Frank, J. D., “Methods of Assessing the Results

of Psychotherapy, ” in Ciba Foundation Symposium on the Role of Learning in Psychotherapy, edited by R. Porter (Boston, Mass.: Little

Brown and Co., 1968).

Persuasion and Healing (2d cd., Baltimore, Md.: Johns Hopkins, 1973).

, “The Present Status of Outcome Studies,” 1, Cons, Clin. Psychol. 47:31o, 1979.

“Therapeutic Components of Psychotherap~, ” ]. Nerv. Merit. Dis. 159:325, 1974.

Frankl, V. E., “Basic Concepts of Logotherapy, ” Confina Psychiatrical, 4:99, 1961.

Freud, S., Standard Edition of the Complete

Psychological Works of Sigmund Freud, edited

and translated by J. Strachey (vol. 17, London:

Hogarth, I$IS.S). (Originally published, 1918. )

Gabby, J. I., and Leavitt, A., “Providing Low

Cost Therapy to Middle Income Patients, ”

Comm. Merit. Health]. 6:210, 1970.

Gallo, P. S., Jr., “Meta-Analysis-A Mixed

Meta-Phor?” Am. Psychol. 33:515, 1978.

Garfield, S. L., “Research on Client Variables

in Psychotherapy, ” in Handbook of Psychotherapy and Behavior Change: An Empirical

Analysis, edited by S. L. Garfield and A. E.

Bergin (2d cd., New York, N. Y.: Wiley, 1978).

and Bergin, A. E. (eds. ), Handbook of

Psycho’therapy and Behavior Change: An Empirical Analysis (2d cd., New York, N. Y.:

Wiley, 1978).
86 ● Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

93.

94.

95.

96.

97.

98.

99.

100.

101.

102.

103.

104.

105.

106.

, and Kurtz, R., “A Survey of Clinical

Psychologists: Characteristics, Activities, and

Orientation,” C]irz. Psycho]. 28:7, 1974.

Gilbert, J. P., et al., “Assessing Social Innovations: An Empirical Basis for Policy, ” in Evacuation and Exper-irnent, edited by Bennett and

Lumsdaine (New York, N. Y.: Academic Press,

1975) .

Ginsberg, G., and Marks, L., “Costs and Benefits of Behavioral Psychotherapy: A Pilot

Study of Neurotics Treated by Nurse-Therapists, ” Psychol. Med. 7:685, 1977.

Glasgow, R. E., and Rosen, G. M., “Behavioral

Bibliotherapy: A Review of Self-Help Behavior

Therapy Manuals, “ Psychol. Bull. 85:1, 1978.

Glass, G. V., et al., Design and Analysis of

Tirne-Ser-ies Experiments (Boulder, Colo.: Laboratory of Educational Research Press, 1973).

Glass, M. G., and Goldberg, S., “Cost-Benefit

Analysis and the Evaluation of Psychiatric

Services,” Psycho/, Med. 7:701, 1977.

Goldberg, I., et al., “Effect of a Short-Term

Outpatient Psychiatric Therapy Benefit on the

Utilization of Medical Services in a Prepaid

Practice Medical Program, ” Med. Care 8:423,

1970.

Goldensohn, S. S., and Fink, R., “Mental

Health Services for Medicaid Enrollees in a Prepaid Group Practice Plan, ” Am. ]. Psychi. 136:

160, 1979.

Goldschmidt, P. G., “A Cost-Effectiveness

Model for Evaluating Health Care Programs:

Application to Drug Abuse Treatment, ” inquiry 13:29, 1976.

Goodwin, I., and Rosenblum, A., “A Method

of Measuring and Comparing Cost in Mental

Health Clinics, ” Hosp. Comm. Psychi. 23:47,

1972.

Gottfredson, G. D., and Dyer, S. E., “Health

Service Providers in Psychology, ” Am. Psychi.

33:314, 1978.

Gottman, J. M., and Leiblum, S. R., How To

Do Psychotherapy and How To Evaluate It

(New York, N. Y.: Holt, Rinehart and Winston,

1974).

and Markman, H. J., “Experimental

Design; in Psychotherapy Research, ” in Handbook of Psychotherapy and Behavior Change:

An Empirical AnaZysis, edited by S. L. Garfield

and A. E. Bergin (2d cd., New York, N. Y.:

Wiley, 1978).

Gottschalk, L. A., et al., “Prediction and Evaluation of Outcome in an Emergency Brief Psychotherapy Clinic, ” ]. Nerv. Merit, Dis. 144:77,

1967.

107.

108.

109.

110.

111.

112.

113.

114.

115.

116.

117.

118.

119.

120.

Grace, W. J., et al., “The Treatment of Ulcerative Colitis: II, ” Gastroent, 26:462, 1954.

Green, B. L., et al., “Relationships Among

Diverse Relationships of Psychotherapy Outcome, ” J. Con, Clin, Psychol, 43:689, 1975.

Green, P. E., and Carmone, F. J., Multidimensional Scaling and Related Techniques in Marketing Analysis (Boston, Mass.: Allyn and

Bacon, 1970.

Gruber, A. R., “The High Cost of Delivering

Services, ” Soc. Work 18:33, 1973.

Guzzetta, R. A., “Acquisition and Transfer of

Empathy by the Parents of Early Adolescents

Through Structured Learning Training, ” ].

Coun. Psychol. 23:449, 1976.

Hall, S. M., et al., “Contingency Management

and Information Feedback in Outpatient Heroin Detoxification, “ Behav. Ther. 10:443, 1979.

Halpern, J., “Program Evaluation, Systems

Theory, and Output Value Analysis: A Benefit/Cost Model,” in Program Evaluation for

Mental Health, edited by R. D. Corusey (New

York, N. Y.: Grune and Stratton, 1977).

and Binner, J. R., “A Model for an

Output’ Value Analysis of Mental Health Programs,” Admin. Merit, He. 1:40, 1972.

Halpert, H., “Models for the Application of

Systems Analysis to the Delivery of Mental

Health Services, ” in Community Psychology

and Community Mental Health, edited by D.

Cooks (San Francisco, Calif.: Holden-Day,

1970).

Hargreaves, W. C., et al., Resource Materials

for Community Mental Health Program Evaluation: Part 3—Evaluating Effectiveness of Services, DHEW publication No. (ADM) 75-222

(Rockville, Md.: National Institute of Mental

Health, 1975).

Hiebert, S., “Who Benefits From the Program?” in Evaluation of Behavioral Programs:

in Community, Residential, and School Settings, edited by P. O. Davison, et al. (Champaign, 111.: Research Press, 1974).

Hogan, D. B., The Regulation of Psychotherapists: A Study in the Philosophy and Practice

of Professional Regulations (vols. 1-111, Cambridge, Mass.: Ballinger, 1979).

Holden, C,, “Senators Hear Case for Psychotherapy, ” Science 201:794, 1978.

Hollon, S., and Beck, A. T., “Psychotherapy

and Drug Therapy: Comparisons and Combinations, ” in Handbook of Psychotherapy and

Behavior Change: An Empirical Analysis,

edited by S. L. Garfield and A. E. Bergin (2d

cd., New York, N. Y.: Wiley, 1978).
121.

122.

123.

124.

125.

126.

127.

128.

129.

130.

131.

132.

133.

134.

135.

136,

Hospital Statistics (Chicago, Ill.: American

Hospital Association, 1979).

Hupmans, J. H., The lmplernentation of Operations Research (New York, N. Y.: Wiley,

1970) .

Imber, S. D., “Patient Direct Self-Report Techniques, ” in Psyc]zotherapy Change Measures,

edited by 1. E. Waskow and M. B. Parloff,

DHEW publication No. (ADM) 74-120 (Washington, D. C.: DHEW 1975).

James, L. R., and Jones, A. P., “Organizational

Climate: A Review of Theory and Research, ”

Psychol. BL{ll. 81:1096, 1974.

Jameson, J., et al., “The Effects of Outpatient

Psychiatric Utilization on the Costs of Providing Third-Party Coverage, ” Research Series

28, Blue Cross of Western Pennsylvania, 1976.

Jeffrey, D. B., “Treatment Evaluation Issues in

Research on Addictive Behaviors, ” Addic.

Behav. 1:23, 1975.

Jeffrey, R. W., et al., “Behavioral Treatment of

Obesity: The State of the Art, 1976,” Behau.

T{ler. 9:189, 1978.

Joint Commission on Accreditation of Hospitals, Principles for Accreditation of Community Mental Health Service Programs (Chicago,

Ill.: JCAH, 1976).

Jones, K., and Vischi, T., “Impact of Alcohol,

Drug Abuse, and Mental Health Treatment on

Medical Care Utilization: Review of the Research Literature, ” Med. Care (suppl. ) 17:12,

1979.

, et al., “Cost and Efficiency in Mental

Hospitals, ” The Hospital 57:23, 1961.

Jung, C., Collected Works, edited by H. Read,

et al. (New York, N. Y.: Pantheon, 1953).

Karen, B. P., and VandenBos, G. R., “Cost/

Benefit Analysis: Psychologist Versus Psychiatrist for Schizophrenics, ” Prof. Psycho/. 7:107,

1976.

, and VandenBos, G. R., “Treatment

Costs of Psychotherapy Versus Medication for

Schizophrenics, ” Prof. Psycho/. 6:293, 1975.

Karush, A., et al., “The Response to Psychotherapy in Chronic Ulcerative Colitis. I. Pretreatment Factors, ” Psychosorn, Med. 30:225,

1968.

Kazdin, A. E,, “Evaluating the Generality of

Findings in Analogue Therapy Research, ” ].

Cons. Clin. Psychol. 46:673, 1978.

“Therapy Outcome Questions Requiring Co~trol of Credibility and Treatment-Generated Expectancies, ” Behav. Ther. 10:81,

1979.

137.

138.

139.

140.

141.

142.

143,

144,

145.

146.

147.

148.

149.

150.

151.

152.

Kelly, G., The Psychology of Personal Constructs (New York, N. Y.: Norton, 1955).

Kennecott Cooper Corp., Insight, unpublished

report (Salt Lake City, Utah: Utah Copper

Division, 1975).

Kiresuk, T. J., and Lung, S. H., “Goal Attainment Scaling, “ in Evaluation of Human Service

Programs, edited by C. C. Attkisson, et al,

(New York, N. Y.: Academic Press, 1978).

Klasman, H. E., “Application of Cost-Benefit

Analysis to Health Systems Technology, ” ].

Occup. Med. 16:172, 1974,

Klein, D. F., “A Proposed Definition of Mental

Illness, ” in Critical Issues in Psychiatric Diagnosis, edited by R. L. Spitzer and D. F. Klein

(New York, N. Y.: Raven Press, 1978).

Klerman, G. L., et al., “Treatment of Depression by Drugs and Psychotherapy, ” Am. ].

Psychi. 131:186, 1974.

Knapp, T. J., and Peterson, L. W., “Behavior

Management in Medical and Nursing Practice, ”

in Behavior Modification: Principles, Issues,

and Applications, edited by W. E. Craighead,

et al. (Boston, Mass.: Houghton Mifflin, 1976).

Kolb, L. C., Modern Clinical Psychiatry (Philadelphia, Pa.: Saunders, 1968).

Noyes’ Clinical Psychiatry (7th cd.,

Philad~lphia, Pa,: Saunders, 1968).

Krantz, D., et al., Foundation of Measurement

(vol. 1, New York, N. Y.: Academic Press,

1972).

Krapfl, J. E., “Accountability Through CostBenefit Analysis, ” in Behavior Analysis and

Systems Analysis: An Integrative Approach to

Mental Health Programs, edited by D. Harshbarger and R, F. Maley (Kalamazoo, Mich.:

Behaviordelia, 1974).

Krowinski, W. F., and Fitt, D. X., “Day Treatment vs. Brief Hospitalization: A Controlled

Study of Clinical and Cost Effectiveness, ” Am,

). Comm. Psycho/., in press.

, and Fitt, D. X., “A Model for Evaluating Mental Health Programs: The Functional

Baseline System, ” Admin. Merit, He. 6:22,

1978.

Krumboltz, J. D., “An Accountability Model

for Counselors, ” Pers. Guid. ]. 52:639, 1974.

Lachenmeyer, C. W., “Experimentation-A

Misunderstood Methodology in Psychological

and Socio-Psychological Research, ” Am. Psy -

chol. 25:617, 1970.

Lalonde, M., A New Perspective on the Health

of Canadians (Ottawa, Ontario: Government

of Canada, 1974).

d
88 ● Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

153.

154.

155.

156.

157.

158.

159.

160.

161.

162.

163.

164.

165.

166.

167.

168.

169.

Landis, C., “A Statistical Evaluation of Psychotherapeutic Methods, ” in Concepts and

Problems of Psychotherapy, edited by L. E.

Hinsie (New York, N. Y.: Columbia, 1937).

Lazarus, A., Behavior Therapy and Beyond

(New York, N. Y.: McGraw Hill, 1972).

“Group Therapy of Phobic Disorders

by Sys~ematic Desensitization, ” ], Abn. Social

Psychol. 63:504, 1961.

Levenson, A. J., et al., “Acute Schizophrenia:

An Efficacious Outpatient Treatment Approach as an Alternative to Full-Time Hospitalization, ” Dist. New. Sys, 38:242, 1977.

Levin, H. M., “Cost-Effectiveness Analysis in

Evaluation Research, ” in Handbook of Evaluation Research, edited by M. Guttentag and E.

Struening (vol. 2, Beverly Hills, Calif.: Sage

Publications, 1975).

Levine, D. S., and Winner, S. G., The Cost of

Mental Illness, 2974, DHEW publication No.

(ADM) 76-158 (Rockville, Md.: National Institute of Mental Health, 1976).

Liptzin, B., et al., “A Federal View of Mental

Health Program Evaluation, ” Prof. Psycho/.

8:543, 1977.

Luborsky, L., “A Note on Eysenck’s Article

‘The Effects of Psychotherapy: An Evaluation’,” Bri, ]. Psychi. 45:129, 1954.

, and Bachrach, H., “Factors Influencing

Clinical Judgment of Mental Health,” Arch.

Gen. Psychi. 31:292, 1974.

, et al., “Comparative Studies of Psychotherapies: Is It True That ‘Everybody Has

Won and All Must Have Prizes’?” Arch, Gen,

Psychi. 32:995, 1975.

Macklin, R,, “The Medical Model in Psychoanalysis and Psychotherapy, ” Comp, Psychi.

14:49, 1973.

Mahoney, M. J., Cognition and Behavior Modification (Cambridge, Mass.: Ballinger, 1976).

“Experimental Methods and Outcome

Evalua~ion,” ]. Cons. C/in. Psycho/. 46:660,

1978.

“The Self-Management of Covert Behavior: A Case Study,” Behav. Ther. 2:575,

1971.

Maidlow, S. T., and Berman, H., “The Economics of Heroin Treatment, ” Am. ]. Pub.

Health 62:1397, 1972,

Malan, D. H., “The Outcome Problem in Psychotherapy Research: A Historical Review, ”

Arch. Gen. Psychi, 29:719, 1973.

Marks, I. M., “Empirical Psychotherapeutic

Methods,” Psychoth. & Psychosom. 24:222,

1974.

170,

171.

172.

173.

174.

175.

176.

177.

178.

179.

180.

181.

182.

183.

184.

185.

Marston, A. R., et al., “A Correspondence

Course Behavioral Program for Weight Reduction, ” Obesity/Bariatric Med. 6:140, 1977.

Maslow, A. H., The Farther Reaches of Human

Nature (New York, N. Y.: Viking, 1971).

Mason, P, E,, and Stitham, M. D., “The Expensive Dalliance: Assessing the Cost of Patient-Therapist Sex, ” Bull. Am. Acad. Psychi.

Law 5:450, 1977.

May, P. R., “Cost Efficiency of Treatments for

the Schizophrenic Patient, ” Am, ], Psychi.

127:118, 1971.

Mazer, M., “Psychiatric Disorders in the General Practices of an Island, ” Med. Care 7:372,

1969.

McCaffree, K. M., “The Cost of Mental Health

Care Under Changing Treatment Methods,”

Am, ], Pub. Health 56:1013, 1966.

McCardell, J., and Murray, E. J., “Nonspecific

Factors in Weekend Encounter Groups, ” ].

Cons. Clin, Psychol. 42:337,1974.

McClean, P. D., and Hakstian, A. R., “Clinical

Depression: Comparative Efficacy of Outpatient Treatments, ” ]. Cons. Clin. Psychol., in

press.

McCord, J., “A Thirty-Year Follow-Up of

Treatment Effects, ” Am. Psycho/. 33:284,

1978.

McGuire, T., National Health Insurance and

Issues and Evidence (Cambridge, Mass.: Ballinger, forthcoming).

Meichenbaum, D., Cognitive Behavior Modification: An Integrative Approach (New York,

N. Y.: Plenum Press, 1977).

Meltzoff, J., and Kornreich, M., Research in

Psychotherapy (New York, N. Y,: Atherton,

1970).

Mendelssohn, F., “Long-Term Psychotherapy

Coverage Under National Health Insurance, ”

Soc. Sec. Bull. 18:1, 1976.

Mills, D. H,, et al., “The National Register

Survey: The First Comprehensive Study of All

Licensed/Certified Psychologists, ” in Psychology and National Health Insurance: A Sourcebook, edited by C. A. Kiesler, et al. (Washington, D. C.: American Psychological Association, 1979).

Mintz, J., et al., “Measuring the Outcomes of

Psychotherapy: Findings of the Penn Psychotherapy Project, ” ~, Cons. Clin, Psychol.

47:319, 1979.

Mischel, W., Introduction to Personality (2d

cd., New York, N. Y.: HoIt, Rinehart and Winston, 1976).
References ● 89

186.

187.

188.

189.

190.

191.

192,

193.

194.

195.

196.

197.

198.

199.

200.

201.

Personality and Assessment (New

York: Wiley, 1968).

“Toward a Cognitive Social Learning

Reconc’eptualization of Personality, ” Psychol.

Rev, 80:252, 1973.

Moos, R., Coping With Physical Illness (New

York, N. Y.: Plenum Press, 1977).

Morganstern, K. P., “Cigarette Smoking as a

Noxious Stimulus in Self-Managed Aversion

Therapy for Compulsive Eating: Technique

and Case Illustration, ” Behav. Ther. 5:255,

1974.

Morton, R. B., “An Experiment in Brief Psychotherapy,” Psych. Monogr. 69(386):1, 1955.

Mosher, L. R., et al., “Soteria: Evaluation of a

Home-Based Treatment for Schizophrenia, ”

Am. J. Orthopsychi. 4s:4ss, 197s.

Murphy, J. G., and Datel, W. E., “A CostBenefit Analysis of Community Versus Institutional Living, ” Hosp. Comm. Psychi. 27:165,

1976.

Nagel, S. S., and Neef, M., Operations Research Methods (Beverly Hills, Calif.: Sage

Publications, 1976).

National Health Insurance Resource Book,

House Committee on Ways and Means (revised

edition, Washington, D. C.: U.S. Government

Printing Office, 1976).

National Institute of Drug Abuse, Management

Effectiveness Measures for NZZ3A Drug Abuse

Treatment Programs, Volume 1: Cost-Benefit

Analysis, Volume 2: Costs to Society of Drug

Abuse (Washington, D. C.: NIDA, 1977).

National Institute of Mental Health, Draft Report: The Financing, Utilization, and Quality

of Mental Health Care in the United States

(Washington, D. C.: DHEW, Office of Program

Development and Analysis, 1976).

, Mental Health Statistical Note No.

239, August 1977.

Newman, F. L., and Rinkus, A. J., “Level of

Functioning, Clinical Judgment, and Mental

Health Services, ” Eval. & the Health Prof.

1:175, 1978.

, et al., “Program Analysis Using the

Client-Oriented Cost Outcome System, ” Eval.

& Z%og. P!ng. 1:19, 1978.

Noble, J. H., Jr., “The Limit of Cost-Benefit

Analysis as a Guide to Priority-Setting in Rehabilitation,” Eval. Qtrly. 1:347, 1977.

Nunnally, J, C,, and Wilson, W. H., “Method

and Theory for Developing Measures in Evaluation Research, “ in Handbook of Evaluation

Research, edited by E. L. Streuning and M.

202.

203.

204.

205.

206.

207.

208.

209.

210.

211.

212.

213.

214.

215.

Gutenberg (vol. 1, Beverly Hills, Calif.: Sage

Publications, 197s).

Office of Technology Assessment, U.S. Congress, Assessing the Efficacy and Safety of

Medical Technologies, GPO stock No. 052-

003-00593-0 (Washington, D. C.: U.S. Government Printing Office, September 1978).

U.S. Congress, The Implications of

Cost-E~fectiveness Analysis of Medica l

Technology, GPO stock No. 052-003 -00765-7

(Washington, D. C.: U.S. Government Printing

Office, August 1980).

Olbrisch, M. E., “Psychotherapeutic Interventions in Physical Health: Effectiveness and Economic Efficiency,“ Am. Psychol. 32:761, 1977.

O’Neil, H. F., et al., “A Systems Analysis of a

Campus Community Mental Health Facility: A

Preliminary Investigation, ” Am. 1. Comm.

Psychol. 1:362, 1973.

Panzetta, A. F., “Cost-Benefit/Studies in Psychiatry,” Comp. Psychi. 14:451, 1973.

Parloff, M. B., “Can Psychotherapy Research

Guide the Policymaker? A Little Knowledge

May Be a Dangerous Thing, ” Am. Psycho/,

34:296, 1979.

, et al., “Assessment of Psychosocial

Treatment of Mental Health Disorders: Current Status and Prospects, ” report to the National Academy of Sciences, Institute of

Medicine, Washington, D. C., 1978.

Patterson, G. R., “Intervention for Boys With

Conduct Problems: Multiple Settings, Treatments, and Criteria, ” ]. Cons. Clin. Psychol.

42:471, 1974.

Payne, F. D., and Wiggins, J. S., “MMPI Profile Types and the Self-Report of Psychiatric

Patients,” J. Abn. Psychol. 79:1, 1972.

Perkins, J. A., and Wicas, E., “Group Counseling With Bright Underachievers and Their

Mothers,” ). Coun, Psychol. 18:273, 1971.

Perloff, E. and Perloff, R., “Selected Processes

for Evaluating Service Delivery Programs:

Overview,” Prof. Psychol, 8:389, 1977.

Phillips, E. L., “Achievement Place: Token Reinforcement Procedures in a Home-Style Rehabilitation Setting for ‘Pre-Delinquent’ Boys, ” ].

Appl. Behav. Anal, 1:213, 1968.

Piasecki, J. R., et al., Determining the Costs of

Community Residential Services for the Psychosocially Disabled, DHEW publication No.

(ADM) 7-5041 (Rockville, Md.: National Institute of Mental Health, 1977).

Pomerleau, O. F., “Behavioral Medicine: The

Contributions of the Experimental Analysis of
90 ● Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

216.

217.

218.

219.

220.

221.

222.

223.

224.

225.

226.

227.

228.

229.

230.

231.

232.

Behavior to Medical Care, ” Am. Psychol. 34:

654, 1979.

Potter, A., et al., “Readmission Discount Factors in Program Evaluation: An Output Value

Analysis of an Adult Psychiatry Program, ”

Am. ]. Comm. Psychol. 3:3o3, 197s.

Powers, E., and Witmer, H., An Experiment in

the Prevention of Delinquency (New York,

N. Y.: Columbia University Press, 1951).

Presby, S., “Overly Broad Categories Obscure

Important Differences Between Therapies, ”

Am. Psychol. 33:514, 1978.

President’s Commission on Mental Health, Report to the President, GPO stock No. 040-

000-00390-8 (vol. 1, Washington, D. C.: U.S.

Government Printing Office, 1978).

Prest, A. R., and Turvey, R., “Cost-Benefit

Analysis: A Survey, “ Econ, ]. 75:683, 1965.

Rachman, S., “Double Standards and Single

Standards, ” Bull, Brit. Psycho/. Soc. 30:295,

1977.

“The Effects of Psychological Treatmerit, ” ‘in Handbook of Abnormal Psychology,

edited by H. Eysenck (New York, N. Y.: Basic

Books, 1973).

The Effects of Psychotherapy (Oxford,

England: Pergamon Press, 1971).

Rathus, S. A., and Nevid, J. S., Behavior

Therapy: Strategies for Solving Problems in

Living (New York, N. Y.: Signet, 1977).

Regier, D. A,, et al,, “The De Facto U.S. Mental Health Services System, ” Arch. Gen.

Psychi. 35:685, 1978.

Reiss, B., “Changes in Patient Income Concomitant With Psychotherapy, ” ]. Cons. Psychol.

31:130, 1967.

Reiss, M. L., et al., “Behavioral Community

Psychology: Encouraging Low-Income Parents

to Seek Dental Care for Their Children, ” ].

Appl. Behav. Anal. 9:387, 1976.

Rice, D. P., Estimating the Cost of Mental Illness, Health Economics Series No. 5 (Washington, D. C.: DHEW, 1966).

Riecken, H. W., “Principal Components of the

Evaluation Process, ” Prof. Psycho/. 8:392,

1977.

and Boruch, R. F. (eds. ), Social Experimentat;”on: A Method for Planning and Evaluating Social Intervention (New York, N. Y.:

Academic Press, 1974),

Rivlin, A. M., Systematic Thinking for Social

Action (Washington, D. C.: The Brookings Institution, 1971).

t and Timpane, P. M. (eds. ), Ethical and

233.

234,

235,

236.

237,

238.

239.

240.

241.

242.

243.

244.

245.

246.

247.

Legal Issues of Social Experimentation (Washington, D. C.: The Brookings Institution, 1975).

Robins, L. N., Deviant Children Grownup

(Baltimore, Md.: Williams and Wilkins, 1966).

Rogers, C. R., Client-Centered Therapy: Zts

Current Practice, Implications, and Theory

(Boston, Mass,: Houghton Mifflin, 19s1).

“Client Oriented Psychotherapy, ” in

Comprehensive Textbook of Psychiatry, edited

by A. M. Freedman, et al. (vol. 1, 2d cd., Baltimore, Md.: Wilkins and Wilkins, 1975).

On Personal Power: Inner Strength

and Zt; Revolutionary Impact (New York,

N. Y.: Delacorte, 1977).

Remans, J. T., “The Economic Evaluation of

Mental Health Programs, ” Znt. ], Merit. Health

2:38, 1973.

Rosen, J. C., and Wiens, A. N., “Changes in

Medical Problems and Use of Medical Services

Following Psychological Intervention, ” Am,

Psycho/. 34:420, 1979.

Rosenthal, R., “Combining Results of Independent Studies, “ Psychol. Bull. 85:185, 1978.

Experimenter Effects in Behavioral Research ‘(New York, N. Y.: Appleton-CenturyCrofts, 1966).

, “The ‘Filedrawer Problem, ’ and Tolerance for Null Results, ” Psychol. Bull. 86:638,

1979.

Rosenzweig, S., “A Transvaluation of Psychotherapy—A Reply to Hans Eysenck, ” J, Abn.

Soc. Psycho/. 49:298, 1954.

Rubenstein, J. S., et al., “The Parent-Therapist

Program: Alternative Care for Emotionally

Disturbed Children, ” Am. ], Orthopsychi.

48;654, 1978.

Rufener, B. L., et al., Management Effectiveness Measures for NIDA Drug Abuse Treatment Programs, Volume 1: Cost Benefit Analysis, GPO stuck No, 017-024-00577-1 (Washington, D. C.: National Institute of Drug

Abuse, 1977).

, et al., Management Effectiveness

Measures for NIDA Drug Abuse Treatment

Programs, Volume 2: Costs to Society of

Drug Abuse, GPO stock No. 017-024-00578-9

(Washington, D. C.: National Institute of Drug

Abuse, 1977).

Rush, A. J., et al., “Comparative Efficacy of

Cognitive Therapy and Pharmacotherapy in

the Treatment of Depressed Outpatients, ” Cog,

Ther, Res. 1:17, 1977.

Rutledge, L., and Binner, P. R., “Readmission

to a Community Mental Health Center, ”

Comm. Merit, Health J. 6:136, 1970.
References ● 91

248.

249.

250.

251.

252.

253.

254.

255.

256.

257.

258.

259.

260.

261.

262.

Salsbery, D. L., Accounting Guidelines for

Community Mental Health Centers and Related Facilities, DHEW publication No. (HMS)

73-9068 (Rockville, Md.: National Institute of

Mental Health, 1972).

Saxe, L., and Fine, M., Social Experiments:

Methods for Design and Evaluation (Beverly

Hills, Calif.: Sage Publications, forthcoming).

Scheff, T. J., Being Mentally 111: A Sociological

Theory (Chicago, Ill.: Adeline, 1966).

Schelling, T. C., “The Life You Save May Be

Your Own, ” in Problems in Public Expenditures, edited by S. B. Chase, Jr. (Washington,

D. C.: The Brookings Institution, 1968).

Schulberg, H, C., “Issues in the Evaluation of

Community Mental Health Programs, ” Prof.

Psychol. 8:560, 1977.

“Quality-of-Care Standards and Profession~l Norms, ” Am. ], Psychol. 133:1047,

1976.

Seres, J. L., and Newman, R. I., “Results of

Treatment of Chronic Low-Back Pain at the

Portland Pain Center, ” ]. Neuro. 45:32, 1976.

Shanteau, J., and Phelps, R. H., “Judgment and

Swine: Approaches and Issues in Applied Judgment Analysis,“ in Human Judgment and Decision Processes: Applications in Problem Settings, edited by M. F. Caplan and S. Schwartz

(New York, N. Y.: Academic Press, 1977).

Shapiro, A. K., and Morris, L. A., “The Placebo Effect in Medical and Psychological Therapies, ” in Handbook of Psychotherapy and l?ehavior Change: An Empirical Analysis, edited

by S. L. Garfield and D. E. Bergin (2d cd., New

York, N. Y.: Wiley, 1978).

Shapiro, D., and Shapiro, D., “The ‘Double

Standard’ in Evaluation of Psychotherapies, ”

Bull. l?rit. Psychol, Soc. 30:209, 1977.

Sharfstein, S. S., “Third-Party Payers: To Pay

or Not To Pay, ” Am, J, Psychi. 135:1185,

1978.

and Nafziger, J. C., “Community

Care: Costs and Benefits for a Chronic Patient, ” Hosp. Comm. Psychi. 27:170, 1976.

Shephard, R. N., et al. (eds. ), Multidimensional Scaling: Theory and Application in the

13ehavioral Sciences (vols. I and 2, New York,

N. Y.: Academic Press, 1972).

Siegert, F., and Yates, B. T., “Behavioral-Child

Management: A Comparison of Individual InOffice, Individual In-Home, and Group Delivery Systems, ” Eual. & the Health Prof. 3:123,

1980.

Sirotnik, K. A., and Bailey, R. C., “A CostBenefit Analysis for a Multimodality Heroin

263,

264.

265.

266.

267.

268.

269.

270.

271.

272.

273.

274.

275.

276.

277.

278.

279.

Treatment Project, ” lnt. ]. Addict. 10:443,

1975.

Skinner, B. F., About Behaviorism (New York,

N, Y.: Knopf, 1974).

Beyond Freedom and Dignity (New

York, N. Y.: Knopf, 1971).

Science and Human Behavior (New

York, N. Y.: MacMillan, 1953).

Sloane, R. B., et al., Psychotherapy Versus

Behavior Therapy (Cambridge, Mass.: Harvard University Press, 1975).

Smith, M. L., and Glass, G. V., “Meta-Analy -

sis of Psychotherapy Outcome Studies, ” Am.

Psychol. 32:752, 1977.

et aI., The Benefits of Psychotherapy

(Baltimore, Md.: Johns Hopkins University

Press, in press).

Sobel, S. B., “Throwing the Baby Out With the

Bathwater: The Hazards of Follow-Up Research, ” Am. Psychol. 33:290, 1978,

Sorenson, J. E., and Grove, H. D., “Using

Cost-Outcome and Cost-Effectiveness Analyses for Improved Program Management and

Accountability, ” in Evaluation of Human Seruice Program, edited by C. C. Attkisson, et al.

(New York, N. Y.: Academic Press, 1978).

and Phipps, D. W., “Cost-Finding: A

Tool fir Managing Your Community Mental

Health Center,” Admin. Merit, He. 1:68, 1972.

and Phipps, D. W., Cost-Finding and

Rate-S; tting for Corr,munity Mental Health

Centers, DHEW publication No, (ADM)

76-291 (Rockville, Md.: National Institute of

Mental Health, 1975),

Stern, B. E., and Stern, E. S., “Efficiency of

Mental Hospitals, ” Brit. ]. Prezl. & Soc. Med.

17:111, 1963.

Strupp, H. H., “Psychotherapy Research and

Practice: An Overview, ” in Handbook of Psychotherapy and Behavior Change: An Empirical Analysis, edited by S. L. Garfield and A. E.

Bergin (2d cd., New York, N. Y.: Wiley, 1978).

, “Success and Failure in Time-Limited

Psychotherapy, ” Arch. Gen. Psychi. 37:595,

1980.

and Bergin, A. E., A Bibliography

of Res;arch in Psychotherapy (Washington,

D. C.: NIMH, 1969).

and Hadley, S. W., “A Tripartite

Model ‘of Mental Health and Therapeutic Outcomes, ” Am. Psychol. 32:187, 1977.

, et al., Psychotherapy for Better or

Worse: The Problem of Negative Effects (New

York, N. Y.: Jason Aronson, 1977).

Stuart, R. B., Trick or Treatment: How and
92 ● Background Paper #3: The Efficacy and Cost Effectiveness of Psychotherapy

280,

281,

282.

283.

284.

285.

286.

287.

288.

289.

290.

291.

292.

293.

294.

When Psyclzotherapy Fails (Champaign, Ill.:

Research Press, 1970).

Subotnik, K. L. “Spontaneous Remission: Fact

or Artifact, ” Psychol. Bull. 77:32, 1972.

Tennov, D., Psychotherapy: The Hazardous

Cure (Garden City, N. Y.: Anchor Press, Doubleday, 1976).

Tharp, R. G., and Wetzel, R. J., Behavior

Modification in the Natural Environment (New

York, N. Y.: Academic Press, 1969).

Thomas, J. A., The Productive School: A Systems Analysis Approach to Educational Administration (New York, N. Y.: Wiley, 1971).

Thoresen, K. E., et al., “Learning House: Helping Troubled Children and Their Parents Help

Themselves,“ in Progress in Behavior Therapy

With Delinquents, edited by J. S, Stumphauzer

(Springfield, Ill.: Thomas, 1979).

Ullmann, L. P., and Gurel, L., “Validity of

Symptom Ratings From Psychiatric Records, ”

Arch. Gen. Psychi. 7:130, 1962.

United States Bureau of the Census, Characteristics of American Children and Youth, 1976

(Washington, D. C.: U.S. Department of Commerce, Bureau of the Census, 1978).

VandenBos, G. R., and Pines, C. D., “Research on the Outcome of Psychotherapy, ” in

Psychotherapy: From Practice to Research in

Policy, edited by G. R. VandenBos (Beverly

Hills, Calif.: Sage Publications, in press).

Vorster, D., “Psychotherapy and the Results of

Psychotherapy,” S. Afr. Med. ]. 40:934, 1966.

Wadsworth, W. V., et al., “Cost of Treatment

of Affective Disorders: A Comparison of Three

Hospitals, ” Lancet 273:533, 1957.

Walder, L. O., et al., “Teaching Behavioral

Principles to Parents of Disturbed Children, ” in

Psychotherapeutic Agents: New Roles for NonProfessionals, Parents, and Teachers, edited by

B. G. Guerney (New York, N. Y.: Holt, Rinehart and Winston, 1969).

Wallace, J., and Sechrest, L., “Frequency Hypothesis and Content Analysis of Projective

Techniques,” ]. Cons. PsychoZ. 27:387, 1963.

Washburn, S., et al., “A Controlled Comparison of Psychiatric Day Treatment and Inpatient Hospitalization, ” 1. Cons. Clin, Psychol.

44:665, 1976,

Waskow, I. E., and Parloff, M. B. (eds. ), Psychotherapy Change Measures, DHEW publication No. (ADM) 74-120 (Washington, D. C.:

DHEW, 1975).

Watson, D. L., and Tharp, R. G., Self-Directed

Behavior: Self-Modification for Personal Adjustment (Belmont, Calif.: Brooks/Cole, 1972).

295.

296.

297.

298.

299.

300.

301,

302.

303.

304.

305.

306.

307.

308.

309.

Watts, C. A., et al., “Cost-Effectiveness Analysis: Some Problems of Implementation, ” Med.

Care 17:430, 1979.

Weisbrod, B. A., A Guide to Benefit-Cost

Analysis, as Seen Through a Controlled Experiment in Treating the Mentally Zll, discussion

paper SS9-79 (Madison, Wis.: Institute for Research on Poverty, University of Wisconsin,

1979).

, et al., “An Alternative to the Mental

Hospital—Benefits and Costs, ” paper presented

at the meeting of the American Psychiatric

Association, Miami Beach, Fla., April 1976.

Weiss, C. H., Evaluation Research: Methods of

Assessing Program Effectiveness (Englewood

Cliffs, N. J.: Prentice-Hall, 1972).

Williams, R. L., and Long, J. D., Toward a

Self-Managed Life Style (2d cd., Boston,

Mass.: Houghton Mifflin, 1979).

Wolberg, L. R., The Technique of Psychotherapy (New York, N. Y.: Grune and Stratton,

1977).

Wolins, M., A Manual for Cost Analysis in Institutions for Children (New York, N. Y.: Child

Welfare League of America, 1962).

Wolpe, J., Psychotherapy by Reciprocal Inhibition (Stanford, Calif.: Stanford University

Press, 1958).

Wortman, P. M., “Evaluation Research: A

Psychological Perspective,” Am. Psycho/,

30:562, 1975.

Yates, B, T., “Benefits and Costs of Community-Academia Interaction in a Paraprofessional

Training Course,” Teach. Psychol. 7:8, 1980.

Improving Effectiveness and Reducing

Costs ‘in Mental Health (Springfield, 111.:

Thomas, 1980).

“Improving the Cost-Effectiveness of .

Obesit~ Programs: Three Basic Strategies for

Reducing the Cost per Pound, ” lnt. ], Obes.

2:249, 1978.

, and Newman, F., “Approaches to Cost

Effectiveness Analysis and Cost Benefit Analysis in Psychotherapy, ” in Psychotherapy: From

Practice to Research in Policy, edited by G, R.

VandenBos (Beverly Hills, Calif.: Sage Publications, in press).

, et al., “Toward More Cost-Effective

Delivery Systems in Behavioral Medicine: Self

Management Institute’s Obesity, Insomnia,

and Smoking Divisions, ” in Behavior Group

Therapy, 1981: An Annual Review, edited by

D. Upper and S. M. Ross (Champaign, Ill.: Research Press, in press).

Young, F. W., “Nonmetric Multidimensional
References ● 93

Scaling: Recovery of Metric Information, ” Psy - Counselor-Training Systems: An Operant and

ckometrika. 35:455, 1970 . Operations Research Perspective, ” Coun. Psy31o. Zifferblatt, S. M., “Analysis and Design of chol, 3:12, 1972.