Bridging Science and Service

A Report by the National Advisory Mental Health Council's

Clinical Treatment and Services Research Workgroup

National Institutes of Health * National Institute of Mental Health

TABLE OF CONTENTS

PREFACE

EXECUTIVE SUMMARY

TREATMENT AND SERVICES RESEARCH: AN ALLIANCE FOR PROGRESS

Introduction

The Clinical Treatment and Services Research Workgroup

Definitions

Integration of Research Areas

Status of the NIMH Extramural Research Program

Defining the Issues

ENHANCING THE PUBLIC HEALTH VALUE OF MENTAL HEALTH RESEARCH

Introduction

Key Perspectives and Priority Setting

Issue: Integrating Public Health Benefit and Scientific Merit to Establish a Research Agenda

Issue: Research-Based Consensus Development

Issue: Creating an Infrastructure for Monitoring Public Mental Health and Assessing the Impact of Research

Summary

MAKING CONNECTIONS

Introduction

Fostering the Dynamic Nature of Public Health Research

Issue: Building the Bridges Among Efficacy, Effectiveness, Practice, and Service Systems Research

Efficacy Research

Effectiveness Research

Practice Research

Service Systems Research

Summary

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
METHODS DEVELOPMENT AND INNOVATION

Introduction

Issue: Incorporating Measures of Costs Associated with Interventions

Issue: Tailoring Interventions to Communities

Issue: Tailoring Interventions to Distinct Subgroups

Issue: Considering Treatment Preferences and Decision Making

Issue: Confidence in Drawing Conclusions from Findings

Issue: Assessing External and Internal Validity

Issue: Fostering Innovative Methods

NIMH's LEADERSHIP ROLE IN PROGRAM DEVELOPMENT, REVIEW, AND ADMINISTRATIVE

ACTIVITIES

Introduction

Program Development: Selecting or Creating an Appropriate Funding Mechanism

Issue: Mechanisms for Training

Issue: Mechanisms for Creating Collaborative Research at Provider Sites

Issue: Requirements for Meeting the Broader Scope of Public Health Research

Issue: Review Criteria

Issue: Expediting Administrative Decisions

Review Considerations

Issue: Review Structure

Issue: Ensuring Expertise in Peer Review

Issue: Conflicts of Interest

Issue: Role of the Scientific Review Administrator (SRA)

Administration

REFERENCES

APPENDIX A: National Advisory Mental Health Council (NAMHC)

APPENDIX B: NAMHC Clinical Treatment and Services Research Workgroup

APPENDIX C: Roster of Consultants

| Home | Research | Top | 

PREFACE

An urgent task before the National Institute of Mental Health (NIMH) is to generate information that, properly used, will

better enable people with mental illnesses to receive optimal care. Such information is needed by mental health care

providers, general health care personnel, service systems administrators, policymakers, and, importantly, patients and their

families. Through research we continue to develop and refine an array of safe and efficacious interventions, yet more

emphasis must be devoted to translating the yield of basic and clinical research into effective treatments for patients

encountered in non-research settings. More attention must be paid, also, to designing treatment delivery systems and

policies that facilitate provision of optimal care. Persons with mental disorders and their care providers are entitled to the

same high quality of research-based information upon which to make treatment and service decisions as persons with heart

disease, cancer, or other general medical conditions. 

The NIMH in recent years has targeted resources to increase substantially the breadth and depth of the mental health

services research field. Our growing services research capacity is an important counterpoint to our strong basic research

portfolio in neuroscience, molecular biology, and basic behavioral research. These two arenas–basic science and service

systems research–bracket a third component of the NIMH research program: clinical research and, particularly, treatment

trials. The pace of advance in basic and clinically inspired research, and the extent of change in service systems and the

questions posed by the various types of patients, providers, and settings demand a thorough reappraisal of the manner in

which we explore and evaluate clinical innovations and move them into the hands of service providers.

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Recognizing this need, the NIMH leadership requested the National Advisory Mental Health Council (NAMHC) to

establish a Clinical Treatment and Services Research Workgroup that would undertake an in-depth review of the pertinent

issues. Under the insightful and energetic leadership of A. John Rush, M.D., a core group of the Nation’s most

distinguished treatment and services researchers met over the course of a year, among themselves, with outside

consultants, and with members of the NIMH and other Federal officials. The outcome of these deliberations is seen in the

report that follows, and specifically in the recommendations made for modernizing and revitalizing our approaches to

treatment and services research.

No less critical than the calls for action at institutional levels is the open, inviting tone of the report. Unlike laboratory

research, relevant treatment and services research demand the broadest possible levels of interest and participation on the

part of the clinical community, policymakers, and consumers of services. Special efforts were made by the

Workgroup–and will be continued by the NIMH–to solicit the needed breadth of involvement. In conjunction with the

completion of the work of the Clinical Treatment and Services Research Workgroup, for example, we were pleased to

announce that NIMH is establishing a new Office of Communications and Public Liaison, one strategy for broadening the

interface between our scientific programs and the many "stakeholders" in our research. We will continue to seek more

innovative ways of achieving productive interactions with individuals and organizations who have contributions to make to

our research planning and specific projects. A user-friendly NIMH home page on the World Wide Web

(www.nimh.nih.gov) offers one approach; active solicitation of committed and creative individuals to serve as scientific

and public reviewers on the Institute’s advisory committees or as members of local Institutional Review Boards offers

another.

We are pleased to make this document available to all interested parties and we welcome your comments, including

suggestions for additional steps we might take to enhance the relevance of clinical and services research and the power of

such research to put vitally needed information in the hands of those who will use it to the benefit of Americans with

mental illnesses. 

Steven E. Hyman, M.D.

Director, National Institute of Mental Health 

| Home | Research | Top | 

EXECUTIVE SUMMARY

Historically, the vulnerability and suffering of persons who live with disabling mental illnesses too often were ignored or

misunderstood by much of society. Yet over the past half century, the people of the United States, through the National

Institute of Mental Health (NIMH), have supported medical, neuroscientific, and behavioral research on mental illnesses

consistently and generously. With that support, and that of others (e.g. foundations, industry, and other Federal agencies),

a remarkable scientific effort has demonstrated convincingly that mental illnesses affect a specific organ-the brain-and that

in the vast majority of instances, mental illnesses can be treated successfully using an array of specific interventions. 

Researchers, policymakers, health care providers, and most critically, individuals with mental illnesses and their families

today recognize that translating the remarkable breakthroughs into procedures and policies of everyday clinical practice is

an urgent, essential, and achievable task. It is a challenge that has profound implications for the quality of the lives of

Americans with mental illnesses and for the health of the Nation. 

The Clinical Treatment and Services Research Workgroup of the National Advisory Mental Health Council (NAMHC)

was charged by the Director of the NIMH to advise the Council on strategies for increasing the relevance, speeding the

development, and facilitating the utilization of research-based treatment and service interventions for mental illnesses into

both routine clinical practice and policies guiding our local and national mental health service systems. 

The Workgroup reviewed the NIMH research portfolio that extends from academic research settings to large, State-wide

service systems, to the moving target of "front-line" clinical care. The review made vividly clear the need for mutually

enriching interaction between research and both practice and service systems. In addition, the Workgroup consulted with

representatives from advocacy groups, insurers, public and private purchasers, researchers, and State and Federal

policymakers. Although these perspectives were not all concordant, they did highlight the fields' capacity to enhance the

delivery of treatment and services available to individuals with mental illnesses. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Toward this end, the Workgroup shaped an action plan with 49 recommendations for fulfilling the Nation's commitment to

individuals with mental illnesses. This action plan is structured by the goals of informed priority setting, using a dynamic

and rapidly growing knowledge base, as well as methodological innovation, and administrative and infrastructure

enhancements. The specific recommendations follow. 

Increase the usefulness of NIMH research for individuals with mental illnesses, clinicians, purchasers, and

policymakers through informed priority setting. 

1. NIMH should establish an ongoing priority-setting process that integrates the perspectives of

patients/consumers, providers, purchasers, researchers, and policymakers in determining long-term

initiatives and responding to scientific opportunities to improve the relevance of mental health intervention

research. 

2. NIMH should renew its role in providing a forum for focusing the key perspectives in the mental health

research enterprise to develop clear and productive lines of bi-directional communication. 

3. NIMH should support the synthesis of available information on mental illnesses, their treatment, and

service needs to enhance priority-setting meetings. 

4. NIMH should routinely organize consensus development conferences on specific mental illnesses to

synthesize the research findings. 

5. NIMH should conduct these activities in conjunction with appropriate partners, such as the Center for

Mental Health Services and the Agency for Health Care Policy and Research, to provide the necessary

integration for the Federal effort. 

6. NIMH should collaborate with Federal, State, and private agencies to establish a mechanism to monitor the

value (as defined by an equation of quality, access, and cost) of mental health care delivered across the

Nation and the impact of policies on practice and service systems. 

Selectively expand the NIMH portfolio in the domains of efficacy, effectiveness, practice, and service systems

research to foster integration across these fields and to expedite the implementation of research-based practices and

policies. 

7. NIMH should augment its efficacy portfolio with research that assesses the generalizability of interventions

across diagnostic complexities (e.g., comorbidity or chronicity), as well as individual, social, and

demographic factors. 

8. NIMH should enrich its efficacy portfolio with trials that measure change in terms of both symptoms and

function over a meaningful period of time. 

9. NIMH should incorporate rehabilitation into its intervention portfolio. 

10. Research should be encouraged that better characterizes promising interventions, patient/consumer

populations, ongoing treatments, and service settings. 

11. NIMH should expand its effort in clinical epidemiology through workshops and through developing

research addressing the issues in the epidemiology of care. 

12. NIMH should expand its effort in generating research to inform and evaluate treatment guidelines in

conjunction with other Federal agencies. 

13. NIMH should review the current measures of functional status and support tests of which measures or items

are reliable, valid, and have clinical utility. 

14. NIMH should expand its research portfolio to include the interface between the architecture of services (i.e.,

the structure, organization, and financing of services) and its effects on the quality of care and clinical

outcomes (symptoms and functioning). 

15. NIMH should support targeted research on how to synthesize and incorporate existing knowledge into

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
clinical practice better, with particular emphasis on understanding factors and mechanisms involved in

changing practice and systems of care. 

16. NIMH should encourage and sponsor more research that links service systems changes with quality of care

and other clinical indicators of patient/consumer status. 

17. NIMH should sponsor validation studies of assessment tools designed to measure the quality of health care

across systems of care to assist policymakers and patients/consumers with decision making. 

18. NIMH should encourage the use of existing databases for understanding service systems. 

Identify research innovations in design, methods, and measurement to facilitate the translation of new information

from bench to trial to practice. 

19. NIMH should improve measures and analysis of costs in intervention studies, with special attention to

successful examples from general medicine. 

20. Large intervention studies should include a cost-effectiveness component that uses the best methodology

available. 

21. Approaches to conceptualize and assess key characteristics of intervention settings should be developed, as

should models for understanding the effects of these characteristics on outcomes. 

22. NIMH should encourage the development and evaluation of key research measures for assessing "usual

care" and develop analytic methods to adjust for variation in components of usual care. 

23. NIMH should promote development of innovative sampling strategies for inclusion of underrepresented

groups and sufficiently large intervention studies to incorporate representative community populations. 

24. NIMH should encourage the development of methods to study and incorporate clinician and

patient/consumer decision-making processes into intervention research. 

25. NIMH should support research to identify common practices believed to be helpful and bring them under

research scrutiny, that is, ascertain what is going on in the practice community and determine how much of

that is beneficial. 

26. NIMH should encourage the improvement of methods for both evaluating clinician implementation and

patient/consumer adherence to treatment recommendations and estimating the consequences of these

variations on the effectiveness of treatment. 

27. NIMH should explore new methods for analysis of data from studies that incorporate innovative

combinations of research designs. 

28. NIMH should encourage development of methods to explicitly evaluate trade-offs in alternative design

features that differ in their implications for internal and external validity. 

29. NIMH should convene a methods workshop to identify options for advancing intervention and service

systems research. The results of this workshop should be debated broadly and options tested in appropriate

follow-up activities. 

Strengthen NIMH's leadership and administrative activities to provide the infrastructure to achieve the goals stated

above in a timely manner. 

30. NIMH should develop additional training and career development programs that offer hands-on experience

in diverse research settings in order to provide researchers with an enriched training experience. 

31. NIMH should revise and renew program announcements (PAs) in the spirit of the Public-Academic Liaison

(PAL) Program to maintain and promote existing partnerships between academic researchers and public

care systems, health plans, both carve-outs and health maintenance organizations (HMOs), and employers

providing health benefits and their representative groups. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
32. NIMH research centers, which demonstrate great potential to secure partnerships with service delivery

systems and to utilize these systems to conduct intervention research, should be supplemented to develop,

implement, and sustain such partnerships. 

33. NIMH should issue a request for applications (RFA) or PA to encourage secondary analyses of service

systems data and to establish accessible formats for such data to maximize the use of databases in practice

settings. 

34. NIMH should stimulate new alliances by providing developmental funds to establish shared research

resources such as data banks, staff time, consultant time, etc. 

35. NIMH should commit resources to identify, describe, and disseminate models of successful partnerships. 

36. NIMH should consider contracts as a mechanism for supporting clearly articulated research needs and

deliverables. 

37. The importance of public health relevance should be specifically added to the review criteria in new PAs,

RFAs, and requests for proposals in the relevant areas to emphasize its significance for applicants and

reviewers. 

38. NIMH should develop a mechanism for responding to unique, but fleeting research opportunities with great

public mental health significance. 

39. NIMH is asked to work with the National Institute on Alcohol Abuse and Alcoholism (NIAAA) and the

National Institute on Drug Abuse (NIDA) to encourage timely joint funding decisions for comorbid

substance abuse and mental illness research. 

40. Efficacy, effectiveness, practice, and service systems research applications should remain at the Institute for

review, and the clinical and services study sections reconstituted, including an appropriate mix of expertise

among reviewers. 

41. NIMH staff should attend carefully to planned evaluations of the Center for Scientific Review

neurosciences and AIDS review mergers, as well as the upcoming behavioral sciences merger, to enhance

future restructurings of NIMH review. 

42. NIMH should explore shared review opportunities with NIAAA and NIDA to enhance the review of

applications on comorbid substance abuse and mental illness applications. 

43. Review panels should have patients/consumers, providers, or policymakers as members to ensure expertise

on the utility of research findings in intervention research. 

44. NIMH staff should explore how other institutes deal with conflicts of interest and consult with the National

Institutes of Health on revising and implementing the conflict-of-interest rules for intervention research. 

45. Scientific Review Administrators (SRAs) should strive to maintain a link between special emphasis panels

and the regular review committees through a subset of the regular reviewers and, whenever possible, the

special emphasis panels should be arranged to precede or follow immediately the regular review meeting to

facilitate the use of committee members and their scoring standards. 

46. SRAs and panel chairs should actively consider when scientific breakthroughs require a substantive update

for the review panel. The SRA and chair may convene a workshop to brief the committee. 

47. NIMH should make judicious use of its ability to make funding decisions out of percentile order to achieve

its programming objectives. 

48. Grant budgets must be based upon providing essential support as determined by review groups and NIMH

program staff for achieving research's scientific and public health aims. 

49. NIMH should increase its use of administrative and competitive supplements to provide additional funds

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
and time to researchers to test the generalizability of their findings or to disseminate important research

findings. 

| Home | Research | Top | 

CHAPTER 1 

TREATMENT AND SERVICES RESEARCH:

AN ALLIANCE FOR PROGRESS

INTRODUCTION

Recent decades have brought remarkable advances in our abilities to diagnose and treat mental illnesses. The National

Institute of Mental Health (NIMH) has played a pivotal role in supporting the research that has made it possible to

enhance, often dramatically, the lives of persons with these illnesses. Research has contributed to the quality and breadth

of treatments available and has reduced significantly the stigma attached to mental illnesses. Scientific and clinical

advances have helped the public to understand that mental illnesses are treatable medical conditions. Ongoing research is

elucidating the complex interplay between brain and behavior and forecasts even more significant developments in

treatment and prevention strategies and in public understanding. 

The scientific grounds for a sense of accomplishment and optimism are firm. Still, satisfaction with what is known about

mental illnesses and their treatments must be tempered. Mental health service settings and systems of care are evolving

rapidly, often outpacing the rate at which new knowledge enters clinical practice. Also, the economics of mental health

services delivery is being transformed. And, importantly, individuals with mental illnesses and their families have

achieved a strong voice in helping to set priorities and processes of care. 

Amidst these changes, many people are unable to obtain, for themselves or for one close to them, appropriate,

state-of-the-art treatment for a mental illness. All too often, clinical practices and service system innovations that are

validated by research are not fully adopted in treatment settings and service systems for individuals with mental illnesses.

The substantial disparity between what is known through research and what is actually provided in routine care is not

limited to mental illnesses. Significant gaps between what is known and what is practiced have been documented in other

areas of medicine as well, including cancer (Cronin et al. 1998), diabetes (McClellan et al. 1997), and cardiac care

(Howard and Duncan 1997). 

Rapid changes across health care, juxtaposed against continuous scientific advances, raise urgent questions for all parties

involved in the Nation’s mental health: 

For patients/consumers and their families, the questions are straightforward. What treatments will help? What

treatments are best? How can we be sure that a specific treatment is both appropriate and of high quality? How can

we afford to pay for the treatment? 

Clinicians have different, but related questions. What is the best treatment for this patient/consumer? What

treatment should be recommended next if the first is unsuccessful? What are the clinical and financial barriers to

applying different approaches to care? 

Health care administrators must address questions at a different level. For example, what types and levels of care

are appropriate for specific patient/consumer groups? What resources are needed to provide such care, and how

should they be delivered and integrated? 

Policymakers, purchasers, and insurers must make important decisions regarding access to and coverage for

patient/consumer care. For them, the questions are no less urgent. What will it cost to pay for effective treatment?

What will it cost, in terms of protracted disability and related costs, not to pay for treatment? What delivery system

and financial incentives will provide optimal cost-effectiveness of care? How do purchasers receive value for their

investment? 

The NIMH, along with other Federal agencies, is responsible for the research that is generating answers to these questions.

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
In 1991, the Institute issued Caring for People with Severe Mental Disorders: A National Plan of Research to Improve

Services. Prepared under the auspices of the National Advisory Mental Health Council (NAMHC), this benchmark report

was one of the factors that invigorated the field of mental health services research in recent years. Since 1991, NIMH

support for treatment and services research has increased from $82 million to $170 million in fiscal year (FY) 1997. Years

of accelerated growth now have made it timely and important to examine the Institute’s progress and, where necessary, to

adjust the Institute’s efforts in light of rapidly evolving science, systems of health care, and technologies for information

dissemination. 

THE CLINICAL TREATMENT AND SERVICES RESEARCH WORKGROUP

In September 1997, Dr. Steven E. Hyman, Director of NIMH and Chair of the NAMHC (see Appendix A for roster),

convened a workgroup of researchers, policymakers, and mental health advocates to advise the Council on the array of

issues that are the domain of clinical treatment and mental health services research. The director’s charge to the NAMHC

Clinical Treatment and Services Research Workgroup (see Appendix B for roster) was to speed the development and

routine utilization of research-based treatment and service interventions in daily practice. Dr. Hyman asked the Workgroup

to consider factors that influence the development, translation, and implementation of research findings into practice. He

urged the Workgroup to consider how to build most effectively on the experiences of other institutes within the National

Institutes of Health (NIH). He also encouraged development of strategies to improve the synergy of the NIMH’s

relationships with patients/consumers and families, managed care organizations (MCOs), public and private purchasers,

and other major “stakeholders” in the mental health system. 

DEFINITIONS

As the Workgroup began its deliberations, members had to come to terms with the lexicons of many different research

fields. Although the Workgroup’s name indicates a focus on treatment and services research, these two terms meant

different things to different members of the Workgroup. Other terms used in the separate literatures also were ambiguous,

so the Workgroup gradually developed its own working lexicon. These definitions are provided for clarification and have

the additional benefit of surveying the many exciting areas that mental health researchers are exploring. These distinct

groupings are provided for discussion purposes, because as discussed later in this report, these areas do or should overlap. 

Efficacy Research

The purpose of efficacy research is to examine whether a particular intervention has a specific, measurable effect and also

to address questions concerning the safety, feasibility, side effects, and appropriate dose levels. As a consequence, the

classic efficacy study is a clinical trial in which an experimental treatment is compared to a control treatment that can be a

standard treatment and/or a placebo. Because the chief concern is to detect any treatment effect, researchers try to

eliminate or hold constant any factors that may obscure this effect. For instance, to reduce the influence of researchers’

and participants’ expectancies on the assessment of effect, clinical efficacy trials are often double-blind. This means that

both the research staff and the participant do not know who is getting the experimental treatment and who is getting the

control treatment. Also, efficacy trials usually employ highly restrictive inclusion and exclusion criteria for selecting

participants. This is because the more similar the participants are, the easier it is to detect the effect of a treatment as well

as its side effects. Complicating factors such as co-occurring substance abuse or other illnesses are typical exclusionary

criteria. Despite these criteria, not all factors can be anticipated or controlled. For this reason, efficacy trials typically

randomly assign participants to the experimental or control condition. The goal is to be sure that condition assignment is

based on “the luck of the draw” rather than an undetected confounding factor. 

In addition to double-blind, randomized, clinical trials, efficacy research typically has other common features: (1) study

settings are highly controlled to ensure optimal treatment delivery (e.g., with clinicians who follow a strict set of

procedures in providing the experimental and the control treatment); (2) treatment is provided without cost to participants;

(3) the “effect” is usually defined by shorter-term clinical outcomes such as symptom reduction after a month or two rather

than longer-term outcomes such as the ability to function at home or work over months to years; and (4) economic

variables such as costs or health care utilization are not typically measured. 

Effectiveness Research

The principal aim of effectiveness research is to identify whether efficacious treatments can have a measurable, beneficial

effect when implemented across broad populations and in other service settings. For instance, any person seeking help

with a particular mental illness, regardless of other co-occurring conditions or the duration of the illness, might be eligible.

Treatments are administered by clinicians who have not necessarily been specially trained in the research protocol;

patients/consumers and clinicians exercise choices over treatments; and the frequency and duration of visits, how and

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
when outcomes are gauged, and the use of adjunctive services are dictated by local practice patterns or administrative

policies. Effectiveness studies can be randomized controlled trials, but as in some types of efficacy work, blinding of the

participants and/or researchers is not always possible. Traditionally, effectiveness studies focus on more broadly defined

outcome measures such as disability and quality of life, and have placed less emphasis on detailed evaluation of clinical

status. With the increasing emphasis on cost of care and efficient service delivery, many effectiveness studies now

incorporate analyses of the cost-effectiveness of various mental health interventions when compared to care as usual or an

alternative treatment. Of course, design elements for effectiveness trials can include random assignment and

double-blinding, as discussed under efficacy. 

Practice Research

Practice research examines how and which treatments or services are provided to individuals within service systems and

evaluates how to improve treatment or service delivery. The aim is not so much to isolate or generalize the effect of an

intervention, but to examine variations in care and ways to disseminate and implement research-based treatments.

Although some studies may have randomized designs, currently most are observational. An emergent field, practice

research is built on and encompasses at least three established areas of research inquiry: 

Clinical epidemiology represents a broad field that addresses what happens to people with illnesses who are seen

by providers of clinical care. Studies use traditional epidemiological methods and are conducted in groups that are

usually defined by illness or symptom or by a diagnostic procedure or treatment given for the illness or symptom.

Thus, it includes studies of the natural history of an illness, studies of diagnostic and screening tests, and

observational and experimental studies of interventions delivered to people with illnesses or symptoms. 

Quality of care research is concerned with defining and describing the care received in clinical settings and

establishing and testing standards for quality of care. Areas of study include processes and outcomes of care;

research on the structure of health organizations as it pertains to processes and outcomes; and the relationship

among structures, processes, and outcomes. Components also may include research on treatment guidelines and

clinical decision making. Considerations of economic impact in decisions about use of medical technologies and

pharmaceuticals are important factors in this process. 

Dissemination research evaluates methods by which appropriate interventions are introduced and adopted in

clinical practice, including factors that inhibit or facilitate such adoption. This research includes studying

mechanisms to change clinician or patient/consumer behaviors, and to improve the delivery of clinical care. Quality

improvement studies and practice guideline dissemination evaluations are currently main components of

dissemination research. A relatively new field of research in mental health, dissemination research study designs

may be either observational or randomized, depending on the nature of the question asked. 

Service Systems Research

Service systems research addresses large-scale organizational, financing, and policy questions. This includes the cost of

various care options to an entire system; the use of incentives to promote optimal access to care; the effect of legislation,

regulation, and other public policies on the organization and delivery of services; and the effect that changes in a system

(e.g., cost-shifting) have on the delivery of services. At the interface of service systems research and other areas of

practice and treatment research are studies that integrate these different levels of inquiry (e.g., studies of how variations in

service system characteristics affect quality of care). Studies of how treatment effects differ, in the context of different

systems, integrate effectiveness and service systems research. Cost-effectiveness analysis has a very important role in

evaluating various forms of treatment delivery systems. Service systems research study designs may incorporate elements

of efficacy and effectiveness research. 

INTEGRATION OF RESEARCH AREAS

In an effort to bridge the existing divisions between the domains of efficacy, effectiveness, practice, and service systems

research, the Workgroup developed a model that took into account these four types of research and the potential users of

research (i.e., patients/consumers and families, clinicians, public and private purchasers, insurers and MCOs, and

policymakers). This model (see Figure 1) demonstrates the interconnectedness of these domains, which are commonly, but

inappropriately, viewed as discrete or linear. 

Figure 1 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Relationships Among Research Domains

and Users of Findings

Treatments do not occur in isolation from patients/consumers, clinicians, and service setting factors. As research findings

are made available to “end-users,” investigators in all four domains should consult with end-users to identify the next

generation of research questions. Treatments are always administered in a context, and it is that emphasis on context that

marks an important change in the treatment research paradigm. Further, a service system must have a strong method for

understanding the clinical nature of an individual’s illness and functional status for the most appropriate provision of care.

Accordingly, the four areas must be mutually informative, each building on the understanding provided by the other and

addressing pressing public mental health concerns. 

The model emphasizes the need for interaction across these domains of study and between research results and users of the

research. Further, there are important cross- cutting areas of research such as health economics, epidemiology, and

prevention. As a case in point, treatments designed without attention to costs may generate interventions that are only

unaffordable and therefore impossible to implement. Economic studies are best focused on questions that arise regarding

current interventions or those under consideration for implementation, ideally with these latter interventions being

evidenced- based. 

STATUS OF THE NIMH EXTRAMURAL RESEARCH PROGRAM

The Workgroup conducted a comprehensive review of all NIMH-supported treatment and services research. Aims of this

review were to identify gaps in the research portfolio, ascertain how to develop needed knowledge more effectively, and

how to use available knowledge most efficiently and appropriately (i.e., how to foster awareness, acceptance, and use of

new knowledge by practitioners and health care systems). 

Within the NIMH, primary responsibility for efficacy, effectiveness, practice, and service systems research is assigned to

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
three branches within the Division of Services and Intervention Research. 

The Adult and Geriatric Treatment and Preventive Intervention Research Branch had a treatment research portfolio in

FY 1997 that included grants and clinical trials contracts supported for a total of $51.7 million, including research training

and career development (see Figure 2). The majority of the grants focus on treatment including mood disorders, principally

major depressive disorder ($22.3M), schizophrenia and other psychotic disorders ($11.1M), a broad spectrum of anxiety

disorders ($11.1M), and dysregulation disorders (principally eating and sleep disorders) account for the remainder

($4.8M). Pharmacology, psychotherapy, somatic treatments, and combination strategies are all represented. Treatment

grants in the Branch portfolio at present are largely short-term efficacy trials with highly selected populations and

outcomes focused on symptomatology. 

In the Child and Adolescent Treatment and Preventive Intervention Research Branch, the treatment research portfolio for

FY 1997 included grants and contracts for a total of $17.7 million, including research training and career development (see

Figure 3). This Branch focuses on clinical trials of treatments or preventive interventions for children and adolescents.

Slightly more than half of the Branch’s funding is devoted to preventive interventions (not represented in this figure).

Treatment areas receiving the most funding in FY 1997 were attention-deficit hyperactivity disorder for treatment

interventions ($6.6M), autism treatment ($1M), conduct disorder treatment ($2M), mood disorders treatment ($2.5M),

anxiety disorders treatment ($2.2M), and other treatment for $2.8M. 

The Services Research and Clinical Epidemiology Branch had a research portfolio in FY 1997, including research

training, for a total of $59.9 million (see Figure 4). The focus of this Branch’s funding is on access to care, structure and

organization of care, process of care, outcomes, and financing and costs of treatment. Supported research areas included

service systems and economics ($11.2M), primary care ($8.8M), children and adolescents ($10.7M), sociocultural

($8.6M), clinical epidemiology ($5.0M), outcomes and quality of care ($2.1M), research training ($6.6M), and the

remaining ($6.9M) included a small portfolio of methodological research and other portfolios. 

Figure 2

Adult and Geriatric Treatment and

Preventive Intervention Research Branch, DSIR

Treatment Grants and Contracts

Fiscal Year 1997 -- Total $51.7 million

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Figure 3

Child and Adolescent Treatment and

Preventive Intervention Research Branch, DSIR

Treatment Grants and Contracts

Fiscal Year 1997 -- Total $17.7 million

Figure 4

Services Research and Clinical Epidemiology Branch

Fiscal Year 1997

Services Grants* -- Total $59.9 million

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
DEFINING THE ISSUES

In addition to reviewing the Institute’s portfolio, the Workgroup sought input from a wide array of perspectives with which

it evaluated the Institute’s activities and future needs. Workgroup members conferred with staff of NIMH and other

Federal agencies. Patients/consumers and family groups, providers, public and private purchasers, and policymakers also

were consulted (see Appendix C for roster). 

The Workgroup took the innovative approach of inviting the participation of all interested parties and encouraging

discussion. For example, the Workgroup opened a web site on the NIMH home page to post summaries of its activities

and solicit comments from the public. Workgroup members held open discussions at various professional meetings to

engage the research field. Further, NIMH staff attended Workgroup meetings and were encouraged to implement, in a

timely manner, recommendations endorsed during the public Council updates rather than wait for the Workgroup’s final

report. The rapid adoption of the Workgroup’s interim suggestions attests to the speed with which NIMH and NAMHC

seek to provide more meaningful research findings. While other ideas require further discussion and a sustained

commitment on the part of NIMH, stylistically, the Workgroup hoped to set a new tone for the Institute in its planning

process. The need for open dialogue with users of research findings in an ongoing, intensive, priority-setting process at the

NIMH is a recurring theme in this report. 

Given the breadth of issues to be considered in developing an action plan, the Workgroup agreed upon four domains for

developing strategic recommendations. Subgroups were formed to focus on each: 

Increase the usefulness of NIMH research for individuals with mental illnesses, clinicians, purchasers, and

policymakers through informed priority setting. 

Selectively expand the NIMH portfolio in the domains of efficacy, effectiveness, practice, and service systems

research to foster integration across these fields and to expedite the implementation of research-based practices and

policies. 

Identify research innovations in design, methods, and measurement to facilitate the translation of new information

from bench to trial to practice. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Strengthen NIMH’s leadership and administrative activities to provide the infrastructure to achieve the goals stated

above in a timely manner. 

Each subgroup generated issues and recommendations, which were subsequently debated by the entire Workgroup. The

Workgroup then met to shape a report that would reflect the diversity of perspectives and constructive ideas provided by

the individuals and groups who had been consulted. The Workgroup believes that achieving these goals would position

NIMH to fund high quality research to address urgent public health concerns. The Workgroup’s recommendations to foster

each goal are described in the following chapters. 

| Home | Research | Top | 

CHAPTER 2 

ENHANCING THE PUBLIC HEALTH VALUE

OF MENTAL HEALTH RESEARCH

INTRODUCTION

A key measure of the value of research is its beneficial impact on society and, more precisely for NIMH-funded research,

on public mental health. 

This report defines public mental health as the number of mental health conditions of individual, clinical, and societal

significance in the general population. Mental health conditions are diagnosable mental illnesses or symptoms that reflect

disturbance in emotional, behavioral, or information processing regulation that causes significant suffering or impairment

for affected individuals. These conditions have observable clinical courses and outcomes. The major costs of these

conditions are borne by the individuals and their families, employers, and society. These conditions result in the disability,

pain, and suffering that also accompany other serious chronic health conditions (e.g., asthma, diabetes, and hypertension). 

Mental illness interventions are treatments and service delivery programs, and, in some instances, public policies designed

to reduce the pain and suffering, disability, and costs of mental illnesses or to improve the delivery of mental illness care.

Interventions occur in the broad context of the affected individual’s life, whether in the work, school, family, or

community setting.1

The products of research-new techniques and diagnostic tools and the promise of new treatments-are inherently exciting

and important to researchers and research administrators. As stated above, the act of generating new knowledge is not an

end in itself, however. Ultimately, the knowledge must lead to improved public mental health in measurable and

meaningful ways. 

This chapter reviews various issues that must be addressed to enhance the social and public health value and impact of

NIMH’s efficacy, effectiveness, practice, and service systems research. It offers three specific infrastructure goals

intended to make NIMH-supported research more relevant to the needs of diverse individuals and

organizations-stakeholders in the research enterprise: 

Prioritize: NIMH should create a planning process for intervention research that integrates public comment from

all key perspectives (e.g., patients/consumers, family members, clinicians, purchasers, policymakers, insurers, and

researchers) in setting priorities for research. 

Identify: NIMH should renew its role in providing a forum for focusing the key perspectives in the research

enterprise through its convening power. 

Monitor: NIMH should collaborate with other Federal agencies to establish an infrastructure to monitor public

mental health and study the impact of research and policy changes on it. 

These activities are highly interrelated. The success of the planning process will be contingent on the extent to which it

incorporates stakeholders’ perspectives and leads to research that positively affects the status of mental health and quality

of care. The public mental health status must be tracked to understand what has been accomplished and what remains to be

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
done. The planning process should blend an understanding of public health need with awareness of scientific opportunities

that will increase both the efficiency and value of NIMH’s research investment. 

KEY PERSPECTIVES AND PRIORITY SETTING

There are many contributors to and patients/consumers of research on mental illness. Although the diversity of opinions

within these groups cannot be fully captured in a brief paragraph or two, identification of these groups and their general

issues are mandatory for understanding the breadth and complexity of this arena. 

Individuals with Mental Illnesses and Their Families 

Individuals with mental illnesses2

 and their families have asserted themselves as active participants in developing

treatment and rehabilitation plans with their care providers. Grassroots groups such as the National Alliance for the

Mentally Ill, the National Mental Health Association, and the National Depressive and Manic Depressive

Association have formed a concerted effort to eliminate stigma against mental illness, promote access to care,

foster self-help, advocate for research targeted toward the elimination of mental illnesses, enhance the utility of

research from a patient/consumer perspective, and influence policymakers. Policy targets include fostering

research/treatment/services advances and enacting anti-discrimination laws. Such groups have wielded

considerable influence on policymakers by providing accounts of the devastation of mental illness, as well as

personal gains and triumphs achieved through research advances. 

These individuals want to be active participants in the priority-setting process to enhance the effectiveness and

availability of treatments. They seek out, for example, research on viable treatments that both provide symptom

relief and improve their functional status. They believe that better treatments with fewer side effects can be

developed. 

Clinicians 

Clinicians are the individuals who directly provide treatment or services to individuals. These may be specialists in

mental health care or primary care doctors and nurses. From a clinician’s perspective, there is a concern that many

treatment guidelines and procedures issued by the government, professional groups, or health care organizations

may not be in agreement with each other and may not be optimal for the individuals they serve. Further, there is

substantial uncertainty about the degree to which the evidence base of many guidelines relates to the kinds of

patients/consumers treated by community providers. Often only suggestive rather than definitive evidence exists to

inform some of the most important clinical decisions arising in daily practice (e.g., what to recommend should the

initial treatment fail). Clinicians need treatments that have utility and feasibility in a variety of practice settings

given available resources. They have adequate information, training, and support to use the most appropriate

treatments most efficiently. 

Providers also are concerned with patient/consumer receptivity to mental illness treatments and-especially in

primary care settings-availability, access to, and quality of specialty providers. Such conflicts and considerations

often create tensions among the goals of high quality care, compliance by providers with contracts and costcontainment goals, autonomy by providers and individuals with mental illnesses in clinical decision making, and

compliance with review standards and patient/consumer preferences. In sum, developing standards of care that

apply across provider groups and for a range of mental health conditions is challenging. 

Purchasers 

Treatment of mental illnesses takes place in a two-tiered system (employer and commercially paid insurance, or

public system). This complicated system impacts both the continuity and quality of care, as well as to what degree

scientifically based knowledge is applied. The continuum of settings for treatment of mental illnesses extends from

primary to specialty care and from private to public systems. Within each segment, there are additional discrete

subsystems for children and adolescents, the elderly, those with serious and disabling mental illnesses, and other

patient/consumer groups. Moreover, patterns of care change frequently as a function of managed care, public sector

reforms, availability of new medications, and other scientific, economic, political, and social factors. 

Employers and public agencies (i.e., local, State, and Federal programs) often are referred to as purchasers. They

finance the services and exercise choices over benefits offered to eligible providers or service locations, and

sometimes regulate prices of mental health services. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Employers. Private employers are looking increasingly at both the direct (treatment) and indirect (e.g., lack of

productivity while on the job or days missed from work) costs of health care. A major concern of employers is

whether employees return to work in a timely way and are productive. All employers, large and small, public and

private, cannot survive with mounting medical insurance costs, employee absenteeism, or workers incapacitated by

mental or general medical illness. 

Additional needs identified by private sector employers include: (1) better health system performance measures to

assess the quality of care and the impact of treatment on the individuals functioning; (2) an ongoing and

collaborative exchange between employers and health care researchers; (3) the conduct of population-based

services research in the public sector; (4) information on comorbid conditions (e.g., 60% of children with serious

mental illness have a co-occurring condition within 3 to 5 years); (5) issuance of local or national

treatment/performance guidelines to assist in writing specifications for benefit components; and (6) service systems

research on organized systems of care, costs of treatment in offsetting other costs such as medical visits, disability,

recurrence, etc. 

Public Purchasers. During the past several decades, the public mental health system has changed dramatically.

Changes initiated via the 1963 Community Mental Health Centers Act (Title II, Public Law 88-164) stimulated

creation of new community mental health organizations. The combination of new treatments and civil rights-related

reforms led to the policy of deinstitutionalization. 

The early years of deinstitutionalization largely involved hospital downsizing, without adequate creation of

community care alternatives. Stimulated by the 1977 Government Accounting Office (GAO) audit of

deinstitutionalization (U.S. GAO 1977), the report of the President’s Commission on Mental Health (1978), and the

Community Support Program initiated by the NIMH (Turner and TenHoor 1978), States made dramatic changes in

public mental health programs in the last 15 years. 

The number of inpatient residents at year end in State and county mental hospitals has dropped dramatically from

approximately 370,000 in 1969 to about 83,000 in 1992 (Redick et al. 1996). Similarly, the number of inpatient

beds in State and county mental hospitals decreased significantly from approximately 413,000 in 1970 to about

93,000 in 1992. In 1993, for the first time, State mental health agencies' expenditures for community programs

exceeded spending on State psychiatric hospital inpatient services (National Association of State Mental Health

Program Directors Research Institute, Inc. 1996). The expansion of Medicaid to cover "optional" clinic and

rehabilitation services helped to stimulate this significant redistribution. 

As the publicly funded mental health system continues to evolve, policymakers in State and local mental health

need more access to research results and evaluation methodologies. Key questions for public purchasers include:

(1) how to incorporate or promote the use of research-based treatments and treatment guidelines and quality

improvement processes in decentralized community care systems; (2) how to control cost increases (e.g., in

Medicaid) without incurring the worst consequences of badly managed care; (3) how to promote concepts of

recovery; (4) how best to engage and impact on mental health care in schools, prisons, and other non-mental health

settings; and (5) how to improve the quality of care for the very significant percentage of patients/consumers with

co-occurring illnesses. 

Insurers and MCOs 

Insurers and MCOs either serve as intermediaries between purchasers, providers, and patients/consumers, or they

organize and deliver services. Insurers and managed care companies are subject to legislative and court-ordered

regulations, as well as market demands for services delivery. Issues of particular contemporary concern include

understanding the ways payment systems impact access and the quality of care delivered. 

Particular interests of MCOs concern balancing cost and quality implications of treatment and service delivery

programs and ensuring that populations served in managed care (or covered through insurance) are represented

adequately in intervention. Studies of the effect of markets on delivery of mental health services in the public and

private sectors and research that tracks changes in market factors that potentially may affect care also provide

needed policy-relevant information. 

Effectively engaging the managed care industry in a research agenda, which will require addressing proprietary

concerns regarding data systems and management mechanisms, is as important as it is challenging given MCOs’

growing presence in the service system. Common ground may be found in the evaluation activities regarding the

impact of treatment programs and policies, such as utilization review, that influence the probability of

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
successful/effective treatment. 

Policymakers 

Policymakers formulate or interpret the rules, regulations, or laws that affect public health. In this context, mental

health often competes for resources with other types of health problems and, at times, with housing, education,

social programs, defense programs, etc., particularly at State and Federal levels. Court interpretation of legislation

and resolution of conflicts (e.g., suits) is another policymaking forum that affects the availability and delivery of

mental health interventions. 

Policymakers are concerned that the data on treatment and services all too often are not relevant to the constituent

population or are not provided in a timely manner. Policymaking occurs in real time. Research findings may lag

considerably behind policy decisions they were initially designed to inform. Thus, it is important both to increase

the relevance and timeliness of research endeavors to policy issues and to initiate a process of identifying and

evaluating trends in public policy as they are forming. 

Information on cost-effectiveness of mental health interventions is needed to formulate a rational mental health

benefit plan. Most policy debates are informed by available data on cost implications, but information on the likely

impact on public mental health also is needed. However, outcome data are more likely to be available at the

efficacy level, rather than at levels that address policy decision issues. 

Researchers 

Researchers conduct studies designed to answer questions posed by the various constituents. They are concerned

with ensuring that their studies provide valid answers to the concerns raised by these groups. To do so, they must

be able to formulate questions into a theoretical model that can be studied. Without an existing base of knowledge

in a particular area, it may be difficult to develop such models. In addition, researchers must be clear about what to

measure so they can answer a question. Even more importantly, they must have measures that are reliable and

valid. 

Many researchers operate in academic settings and must compete for private or Federal grants to support their

research efforts. They also strive to publish their findings in a timely manner. Large, complex studies also can be

difficult to conduct within one academic setting. Seeking out new collaborators in other locations can be costly and

difficult. Due to these constraints, researchers sometimes must simplify their research questions. Thus, many

investigators believe that the funding incentives have been weighted toward conducting more simplistic studies that

may not generalize well to larger populations of individuals seeking treatment or services. 

Figure 5 reflects the larger context in which these key stakeholders operate. These components and their

interconnections are where science and mental health policy converge to shape better care and service for

individuals with mental illnesses. 

Figure 5 

The Context of Public Mental Health

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
The Context of Public Mental Health 

Issue: Integrating Public Health Benefit and Scientific Merit to Establish a Research Agenda

Even though one can define science in value-free ways (e.g., theory building and testing based on replicable results),

values are clearly felt in the arena of research. Values determine which research idea, out of the many presented to NIMH,

receives taxpayer support. Yet, if the research agenda were to reflect predominantly the needs of only one of the many

stakeholders, it would be unbalanced and much could be lost. 

Research that may be of great benefit to individuals with mental illnesses may be less compelling to researchers. For

instance, a fairly simple treatment comparison trial may have high impact on developing evidence-based practice.

Comparison studies clarify which treatment a patient/consumer should try next or inform a State mental health

commissioner about whether to adopt a new treatment guideline. If the researchers alone were setting the agenda,

however, these trials might be insufficiently valued to recommend funding. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
As another example, NIMH must develop channels for input from policymakers regarding the biggest decisions they are

facing. For example, as State mental health authorities must decide how to allocate their fixed treatment dollars, the

NIMH research portfolio should help inform their decision making about which services should be provided, to whom, by

whom, for how long, and how to implement such changes in practice. Similarly, health maintenance organizations

(HMOs) and other managed care entities should be able to look to NIMH for research findings on how best to treat those

conditions that are most common in the people they serve and how to know whether such care is being delivered.

Providing timely, useful information to these purchasers and policymakers should also enhance their participation as

research partners. Dr. Steven Shon explains the issues from a policy perspective (see story box). 

Research and Policy Decisions: Obstacles to Effective Translation 

As someone in a public policy position, connecting research to practice and policy decisions is often extremely

difficult. The difficulties for me occur in the following areas: 

1. Obtaining knowledge about relevant research 

As each year goes by the volume of published research and descriptive articles increases. Sifting through these

publications in order to find research findings that may be relevant to public practice is time consuming and

difficult. There is no single arena or source of information which can synthesize the most important or relevant

findings. 

2. Lag time between research findings and publications 

Important research findings may take years before they are available in published form. This lag time may result

in missed windows of opportunity for public policy decisions. 

3. Focus of research in relation to public policy 

Most clinical research does not address the most relevant public policy considerations that would be useful for

mental health. Useful research available to public policy people is woefully lacking. The focus of much, if not

most, funded research is different than what most public policymakers would propose. 

4. Timeliness of research in relation to public policy 

In my experience, most research on important public policy issues in mental health occurs long after decisions

have been made. Very seldom are there relevant research findings available to influence policy decisions.

Again, this is probably due to #3. 

Many of the thorny issues that mental health public policy officials are struggling with today were identified as

issues on the horizon many years ago and still today there is little in the way of good research to guide

decisions. 

I think that the current situation could be significantly alleviated by creating a stronger and more collaborative

link between mental health public policymakers and research funding entities. Through such a process, forums

could be developed to help identify important future research directions, as well as design better mechanisms

for updates, interpretation of data, synthesis of important research, and the dissemination of relevant research

findings that have the potential for impacting public policy. 

Dr. Steven Shon is the Medical Director, Texas Department of Mental Health and Mental Retardation. 

The priority-setting process must involve all stakeholders (patients/consumers, providers, purchasers, and policymakers) to

enhance the ability of NIMH to anticipate the types of questions on the horizon. NIMH can then project trends that

influence mental health service delivery, benefits, or public mental health, and anticipate the need for relevant intervention

research data. Tracking developments in public policy is necessary to anticipate informational needs and to review existing

research and formulate new initiatives to fill critical gaps. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
The Workgroup encourages NIMH staff to explore models at other agencies or institutes where priority setting is

underway. At the National Institute of Allergy and Infectious Diseases (NIAID), for example, the research priority-setting

process is done annually and spans three fiscal years. The process opens with a summer policy retreat designed to address

research priorities and public health needs, including emerging scientific opportunities, policy issues, special problems,

and consideration of refinements to previously approved initiatives. The broad-based process, which may involve invited

participants from outside of the NIAID, integrates priorities for both the extramural and intramural programs. Feedback on

issues raised at the summer policy retreat are submitted to the fall meeting of the NIAID Advisory Council. During the

winter program review, current and future divisional priority activities are discussed. A general consensus is obtained on

the prioritization of these activities. Further communication then takes place among the divisions. These findings are

presented to Council and concepts are cleared for initiating program announcements (PAs), requests for applications

(RFAs), and requests for proposals (RFPs). In addition, issues of interest are posted on the Institute’s web site to inform

the extramural community of future priority areas and to provide an opportunity to comment. 

Other institutes and agencies have planful approaches, and NIMH staff should consider these and adapt or adopt attractive

elements into their priority-setting process. 

Recommendation 1

NIMH should establish an ongoing priority-setting process that integrates the perspectives of

patients/consumers, providers, purchasers, researchers, and policymakers in determining

long-term initiatives and responding to scientific opportunities to improve the relevance of

mental health intervention research. 

Issue: Research-Based Consensus Development 

Establishing a research agenda that is responsive to the needs and priorities of key stakeholders is likely to increase the

usefulness of research results. But such agenda setting requires both a forum and information from which consensus can be

achieved. Choices exercised by each of these stakeholders can affect whether treatments for illnesses are available, who

receives them, and the costs and quality of care. Not surprisingly, these decisions are difficult because the research

findings needed to inform a proper response are not always available. As a result, multiple parties are forced to make

unenviable choices: 

Patients/consumers are uncertain as to whether they should continue to take medication to alleviate symptoms but

increase risks of significant side effects. Should they pay out-of-pocket for new medication(s) with reportedly

fewer side effects or continue with the less expensive but more troublesome drug(s) offered through providers or

MCOs? What forms of psychotherapy will help? Would their time be better spent in a supported employment

program? 

Providers seek to give high quality care to individual patients/consumers, but also must comply with various

insurer requirements for cost-cutting measures. How should they allocate their time? 

Employers seek to optimize employee satisfaction and productivity as well as profitability. Will more extensive

coverage produce better care? Will treatment help productivity? 

Insurers and MCOs must compete for a better market share and for contracts, without sacrificing patient/consumer

satisfaction. Will paying for prevention or health activities have short-term payoffs or simply minimize cost for the

next insurer? 

Given the often conflicting goals within and across stakeholder groups, the field would benefit from having a forum to

identify, develop, and chart a shared path to achieve shared goals. Over the last 9 months, the Workgroup began to develop

such a forum by devoting significant time to hearing from representatives of these diverse sectors. 

The Workgroup urges NIMH to play an active role in establishing, maintaining, and using this critical forum. As an

overarching principle, coordination and cooperation with all of the Federal agencies addressing mental health and

substance abuse problems are encouraged. The Institute should review past, successful use of its “convening power.” A

recent example was that of an NIMH-sponsored meeting of all parties concerned with the lack of research to direct

pharmaceutical treatment in children, an urgent and important issue in light of the large number of off-label prescriptions

written for children in this Nation (Vitiello and Jensen 1997). Drs. Peter Jensen and Benedetto Vitiello, both of the NIMH,

invited representatives from parent groups, the pharmaceutical industry, ethicists, researchers, and clinicians to focus on

this issue and define what was needed and what they could contribute to this effort. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
The Workgroup encourages the NIMH to participate more actively in other institution’s consensus efforts. The NIMH has

had a liaison member on the National Advisory Council for the Center for Mental Health Services (CMHS) since its

inception, yet the CMHS Council has expressed concern that closer ties need to be created. The need for collaboration is

particularly clear given overlapping responsibilities in public information and evaluation of effective interventions. The

Workgroup shared this sentiment and extends it to private foundations and businesses. 

Recommendation 2

NIMH should renew its role in providing a forum for focusing the key perspectives in the mental

health research enterprise to develop clear and productive lines of bi-directional communication.

Bringing people together is not sufficient by itself. Each group requires easily accessible syntheses of research findings to

inform their positions. These syntheses should not be limited to NIMH-supported research. Although the NIMH plays a

unique role in sponsoring public research on mental health care delivery and treatments, other governmental agencies [e.g.,

Agency for Health Care Policy and Research (AHCPR) and CMHS], several large private foundations, pharmaceutical

companies, and health care delivery systems, such as large HMOs, sponsor research that has overlapping goals. 

The Internet is emerging as a primary source for organizing and manipulating information quickly. With this capacity, the

challenge for the mental health field is to synthesize research findings and present them in a way that researchers and other

important groups can evaluate the information. The work of the Cochrane Collaboration, an international group established

to conduct systematic reviews of the literature, offers an example of how large amounts of research data are being

collected and managed. It also provides an historical context to determine if the conduct of a future study would be

duplicative. Other synthesis processes exist as well, and can be brought to bear on the Institute’s scientific priorities. 

Recommendation 3

NIMH should support the synthesis of available information on mental illnesses, their treatment,

and service needs to enhance priority-setting meetings. 

Just as NIMH must renew its efforts to synthesize information, it also must take the lead in consensus development.

Consensus development conferences are effective in bringing together all stakeholders; thus, as findings emerge from

research and large-scale synthesis efforts, NIMH, in collaboration with other Federal agencies such as AHCPR and

CMHS, should actively initiate such conferences. The results of these conferences would then shape NIMH’s priorities

and be of guidance to the large purchasers of care such as State mental heath administrations and MCOs. 

Recommendation 4

NIMH should routinely organize consensus development conferences on specific mental illnesses

to synthesize the research findings.

Recommendation 5

NIMH should conduct these activities in conjunction with appropriate partners, such as CMHS

and AHCPR, to provide the necessary integration for the Federal effort.

Issue: Creating an Infrastructure for Monitoring Public Mental Health and Assessing the Impact of Research

Mental illnesses exact a tremendous toll on society as a whole. Public systems must reallocate increasingly tighter

resources and are being forced to make very difficult decisions regarding what services to provide. Private purchasers are

concerned about not only the costs of paying for mental health benefits, but also the costs of lost productivity associated

with illness. 

As sweeping changes in health care unfold, researchers and policymakers want to examine the effects of these changes on

the public’s mental health status. An enduring problem in assessing the value of a mental health intervention is that there

is no general index of the Nation’s mental health status. Although short-term epidemiological studies have provided

critically important “snapshots” of the country’s mental health status, it is difficult to ascertain whether the Nation’s

mental health has improved, since studies tend to be conducted too far apart in time and with sufficiently different

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
methodologies to allow evaluation of change. Similarly, national health surveys have periodically included mental health

items, but not in a consistent manner to allow this level of monitoring. 

Lacking an index, it is not possible to assess fully the large-scale changes in the mental health status of a State or the

Nation. Each study that addresses this question must do so in a vacuum. At present, the only method to assess changes is

to compare results of the few studies conducted at restricted points in time. Measures of mental health status used in these

studies must be created or adapted for that specific project, and staff must be trained to implement these measures. Such

costly, intensive efforts are prohibitive for an individual researcher, the State, or even a single institution such as NIMH.

For this reason, developing methodologies that can be included or incorporated into ongoing surveys in a routinized

manner is essential. Similarly, sampling methods have not been used effectively in assessing the patterns or needs for

mental health services. 

Improvements in public mental health are contingent on improvements in access to care and services delivery. Thus, it is

important to measure the status of the care delivered in actual practice so decisions about improving the quality of care can

be made. This is potentially a more demanding monitoring task, because quality of care may require studying providers

and service delivery systems, not just representative individuals in the household or patients/consumers of care and

services. 

No national databases of the diverse spectrum of mental health providers or service systems exist to enable either national

monitoring efforts or individual studies. Similarly, there are no national lists of health care delivery systems. The

Substance Abuse and Mental Health Services Administration regularly surveys larger public mental health providers

through its inventory of mental health care organizations, but individual providers, MCOs, and social rehabilitation

agencies are not included. Without augmenting the current infrastructure to tap into representative providers or mental

health care systems, it is prohibitively expensive for individual studies to improve their sampling at higher levels (systems,

providers). Current efforts will be limited and less representative over time, as is already reflected in the empirical

literature by a disproportion of case studies. 

A limited number of intensive, short-term studies of the Nation’s mental health status assessed continuously over time

offer an alternative infrastructure for measures of progress, change, and value. Indicators of particular interest should be

those that are potentially changeable through treatment interventions. In addition, researchers conducting local or regional

studies could incorporate these indicators, thereby testing their findings against the Nation at-large. As a research agency,

NIMH should “define the gold standard” for what should and practicably can be monitored (i.e., establish a process to

determine the priority areas). 

At the national level, for example, it would be useful to ascertain the number of patients/consumers with particular

diagnoses who are and are not receiving treatment, where treatment is being provided, and the nature of treatment

received. At the local level, an example is seen in the provision by school districts of information about the percentage of

children who require special education services and the percentage evaluated by guidance counselors or other mental

health professionals within the school setting. 

NIMH should provide leadership in developing the infrastructure to regularly and reliably track mental illnesses and the

impact of treatment breakthroughs or policy changes in the organization or financing of systems of care on public mental

health. A secondary goal of this monitoring is the provision of the next generation of research findings to inform the

priority-setting process. Key activities would include coordination with current Federal efforts to develop an economical,

written plan for presentation to the NAMHC. It is difficult to think of such a complex project and use the term

“economical.” Nonetheless, embedding the NIMH effort into ongoing studies, effectively using sampling strategies, and

collaborating with highly experienced Federal agencies should permit a relatively low-cost effort. This is especially

feasible given the expected payoff in the type of information policymakers need and patients/consumers want, as well as

stimulating more public health relevant research. Specific recommendations on the methods development required for this

effort are presented in Chapter 4. 

Recommendation 6

NIMH should collaborate with Federal, State, and private agencies to establish a mechanism to

monitor the value (as defined by an equation of quality, access, and cost) of mental health care

delivered across the Nation and the impact of policies on practice and service systems.

SUMMARY

NIMH should develop a process that integrates the input of diverse stakeholders regarding research priorities, data from

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
systematic monitoring of public mental health and quality of care, and information derived from research funded by NIMH

and other organizations. Each recommendation in this chapter represents one step toward development of an informed

priority-setting process for the research agenda. Priority-setting based on public mental health values and real impact data

will raise the Institute’s research portfolio to a higher level of scientific and public health excellence. This effort is devoted

to meeting the primary goal of research supported by NIMH: improving the condition of people with mental illnesses. 

| Home | Research | Top | 

CHAPTER 3 

MAKING CONNECTIONS

INTRODUCTION

Scientific research has generated information that has enhanced greatly the lives of many individuals with mental

illnesses. By way of example, recent developments that carry immense potential for improved clinical care are: 

A virtual explosion in the availability of sophisticated new treatments- Nine new antidepressant agents have

become available in the U.S. since 1988; three new atypical antipsychotic agents have been introduced within the

last 5 years; and at least three compounds (clozapine, lamotrigine, and gabapentin) currently are being studied for

efficacy in bipolar disorder; 

A shift from the primacy of “expert opinion” to that of scientific evidence as the basis for treatment decisions-As

mental illness diagnoses have been refined and shown to be as reliable as those used in general medicine (Spitzer,

Forman, and Nee 1979; Williams et al. 1992) and as treatment options have multiplied, evidence-based treatment

guidelines, specific disease management protocols, and medication algorithms are being formulated and tested

(Depression Guideline Panel 1993; Frances et al. 1998; International Psychopharmacology Algorithm Project

Participants 1995); 

Numerous research findings on innovative delivery system configurations-For example, findings on the

effectiveness of Assertive Community Treatment teams can inform policies as to preferred methods of delivering

care for persons with severe mental illnesses (p. 580).3

These and other advances have received high visibility in the professional literature. But, as in all other health fields,

publication in a journal is only the first step in the slow and arduous task of moving new information from the research to

a treatment or service setting. Dr. Anthony Lehman’s research captures the immensity of this gap for individuals with

schizophrenia and why making better connections between research and practice is so important (see story box,

Translating Research to Treatment: The Challenge). 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Translating Research to Treatment: The Challenge

Schizophrenia ranks high among the most disabling of all mental illnesses. Up to 2 million people are

treated for the illness each year, and up to 100,000 individuals with schizophrenia are in public mental

health hospitals on any given day. This serious brain disorder, consisting of delusions and hallucinations,

not only causes great personal devastation to the affected individuals and their families, but also has an

enormous impact on society in terms of lost talent and productivity. 

Tragically, fewer than half of the individuals in treatment for schizophrenia are receiving proper doses of

antipsychotic medications or appropriate psychosocial treatments. This is the major finding of the

Schizophrenia Patient Outcomes Research Team program co-sponsored by AHCPR and NIMH (Lehman et

al. 1998). We arrived at this finding by analyzing published research on schizophrenia and studying

whether and how evidence-based treatment is provided in current outpatient and inpatient care settings.

The 5-year scientific study found that the key to improving outcomes for these individuals is a

comprehensive strategy that combines the correct dose of an appropriate medication, patient and family

education and support, and community services in the case of high-risk patients. 

Dr. Anthony Lehman is Professor of Psychiatry and Director of the Center for Mental Health Services

Research at the University of Maryland. 

The large gap between knowing what works and finding these evidence-based interventions in a service setting points to

the acute need for: (1) evidence-based interventions that are easily applied in routine settings, and (2) research on how to

introduce these findings into practice and service settings to effect rapid adoption. The question for the Workgroup was

how best to generate these two types of research. The answer involved three steps. Step one is to establish the rich

priority-setting process that was specified in Chapters 1 and 2. Step two is to ensure that the four research domains of

efficacy, effectiveness, practice, and service systems research are adequately supported. The third is to make the necessary

connections between and across these four research areas that are critical junctures. For the Workgroup, these connections

are a matter of establishing overlapping methods and research questions to achieve a dynamic interplay that speeds the

discovery process. 

FOSTERING THE DYNAMIC NATURE OF PUBLIC HEALTH RESEARCH 

Chapter 1 described four discrete spheres of inquiry: efficacy, effectiveness, practice, and service systems research. While

such a parsing of research offers advantages, it also carries risks. The principal risk is one of compartmentalization, of

hampering the essential give-and-take among research arenas. Without this interplay, NIMH staff and its researchers

cannot see that questions generated in one arena may demand a search for answers in another. The challenge of moving a

potential new treatment from the point of development to its delivery by appropriate providers to the right

patients/consumers in a suitable context requires answers to a diligently informed cascade of questions. At any point,

answers to these public health questions may lead back to a reconsideration of earlier questions and stimulate new, needed

research. 

The structure of intervention development may be likened to an elaborate, multi-tiered fountain in which water splashes

downward from one tier to another, with a recirculation pipe at each level of the cascade. Efficacy research would be in

the uppermost basin of the fountain. Here, investigators strive to create research circumstances that will answer

fundamental questions as to the safety and efficacy of a given intervention. Experimental treatments that show no

significant effect, or are found to yield only a marginal improvement over extant treatments, should be siphoned out of the

cascade as an important feedback loop to the treatment development researchers. 

The second tier, effectiveness research, seeks to expand the scope of the discussion. Studies may relax exclusion criteria to

examine the effects of a treatment on a greater breadth of populations (e.g., children and adolescents, the elderly, or others;

on “atypical” presentations of the illness in question; or on the given illness when it co-occurs with another). Yet another

aim of effectiveness research may be to examine the effect on outcome of a range of procedures used to administer a

treatment. Here, before a proposed treatment cascades into further application, the evidence is weighed. For example, the

trial may indicate that the treatment is ineffective in adolescents or the elderly and is rerouted back to the level of efficacy

research, where more rigorous controls again may be employed to ascertain what characteristics of treatment,

patient/consumer population, or other variables explain the lack of effectiveness. Re- examination may suggest what

modifications might be warranted in the indications for the treatment in question, lest a treatment that is efficacious, albeit

for a narrow population of patients/consumers, be siphoned prematurely out of the system. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
With the imprimatur of the treatment community, the next step for a novel intervention is into the world of clinical

practice. At this juncture, new questions arise. Does a treatment produce the desired benefit at the cost predicted as it is

being used in routine care? Is it being properly used? Do clinicians need to be trained in its use? Do the inevitable practice

variations of independent practitioners increase or reduce the effectiveness of the treatment in representative populations?

Careful research monitoring, through practice research networks or other means, not only has the capability of contributing

answers to such questions, but also may identify practice patterns (e.g., off-label uses of drugs or innovative combined

treatment modalities) that may be suitable candidates for recirculation upward to any of the previously described research

arenas. 

Service systems research constitutes the fourth and final basin of the fountain. Here, interventions are subjected to

administrative, clinical, and policy questions. What does it cost to deliver the treatment? Are there savings and when do

they occur? Are the benefits worth the cost? How does the treatment compare to other treatments for a given illness across

systems of care? 

Although the tiers of the fountain are the visible structure, the vibrancy comes from the methodical recycling of the water.

The same can be said of research. It is through the constant feedback loop across the research fields that research can most

efficiently bring evidence-based interventions into practice. Opportunities to expedite development and testing are ripe, if

researchers in one domain anticipate the needs of the other domains given current findings. Might, for example, an

innovative treatment for a condition that occurs commonly in association with another illness be tested early in efficacy

trials under those conditions? Or, if efficacy trials strongly support further testing of a new treatment, should outcome

criteria be broadened early on to include, for example, its effects on functional outcomes in addition to its effect on

symptomatic outcomes ? 

This chapter recommends actions for NIMH to take to increase the dividends realized from its investment in each of the

four discrete research domains. For purposes of clarity, these domains are discussed separately and in sequence. But as our

hypothetical fountain illustrates, knowledge transfer is a complex, iterative process. Fortunately, an array of models for

synthesizing and disseminating clinically relevant information are available for use in the mental health field. 

Issue: Building the Bridges Among Efficacy, Effectiveness, Practice, and Service Systems Research

As presented in Figure 1 and discussed above, the interplay between research domains is an essential part of fostering

evidence-based interventions. Establishing such linkages is not a simple matter. As said before, NIMH must continue its

support of the four domains of research. In this section, the Workgroup will point to additional areas that require

development to provide an empirical link across these domains. Additionally, there will be a special emphasis on

expanding the Institute’s effort in practice research, a relatively new and underdeveloped area for mental health research.

Subsequent chapters will outline the infrastructure for developing the substantive areas described in this chapter. 

EFFICACY RESEARCH

Efficacy research typically emphasizes the reduction of symptoms, looks at a selected group of individuals with the

illness, and incorporates well-defined protocols for interventions. This type of research is essential for detecting what is

worth pursuing with a broader and more expensive array of outcomes and participants. NIMH’s investment in efficacy

work must be maintained. Although necessary, a robust efficacy trials portfolio alone is not sufficient. 

A case in point is the use of highly restrictive exclusion criteria in efficacy work. The need for this methodological device

was explained in Chapter 2. The problem with an over reliance on this type of trial is that as more and more individuals

are excluded, the applicability of the treatment findings is diminished. For instance, in an efficacy study of a treatment for

schizophrenia, typical exclusionary criteria would be substance abuse problems or suicidal thoughts. A common inclusion

criterion is that this be the participant’s first episode of schizophrenia because this provides a study population without

prior exposure to other treatments. With these three conditions alone, the treatment findings may be applicable to less than

20 percent of individuals with schizophrenia in the community. This narrow band of application or generalizability may be

why so many patients/consumers and providers find that research does not address their circumstances. 

Researchers and NIMH staff must be constantly watching for opportunities to invest wisely in expanding efficacy trials.

This may involve supplementing a promising efficacy trial midway through the grant to permit a broader range of

participants or to support a longer follow-up period to understand maintenance issues. 

Another opportunity is to selectively expand the domains of outcomes earlier in the development and evaluation of

treatments. Efficacy researchers speed the “discovery” of promising treatments by looking for shorter-term signs of

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
improvement. Again, this is an excellent strategy that can be improved by extending or expanding a successful trial to

include additional measures of improvement valued by individuals with mental illnesses, their families, and their

clinicians. Although all of these individuals value symptom reduction, outcomes in mental illnesses, as well as general

medical conditions, are far more complex than symptoms. Many patients/consumers respond to treatments with a

reduction in, but not a remission of, symptoms and an improvement in, but not a normalization of, function. These

“responders” still suffer greatly and are not fully recovered. This difficulty is captured in author Andrew Solomon’s story,

see Depression: A Patient's Perspective. Like diabetes, arthritis, and hypertension, mental illnesses are chronic and

recurring. Long- term strategies are necessary to prolong the period of wellness or symptom control and to reduce the risk

of relapse or recurrence. 

Depression: A Patient’s Perspective

In two years of writing about depression, I have interviewed hundreds of people with the illness and

dozens of researchers, and have now collated their experiences and my own depression. I believe, first and

foremost: the work of those who look to genetics, to histories of trauma, to cognition, to pharmacology, to

sleep and light research, and to other specializations must be integrated. At the moment, too many insights

are achieved too independently of one another. Without a unifying theory, we will never find a cure.

Second: existing drugs, though miraculous, are still woefully inadequate. The process of finding the right

one at the right dosage is punishing, and I have yet to meet anyone on medication who does not

accommodate substantial side effects. Third: people must stay on medications. Cycling on and off through

breakdown after breakdown, denying that depression is usually a chronic condition, hoping to be

“treatment-free,” patients endure unnecessary repetitive collapse. Fourth: depression interacts with

personality, and the understanding of depression will come only when we figure out how some people,

despite recurring periods of strong suicidal ideation and virtual catatonia, are able to have meaningful lives,

while others with comparatively mild symptoms are completely disabled. Last: there is a real human

difficulty in being depressed, a difficulty more profound even than depression’s symptoms. Medications

can return to the suffering patient the ability to get up in the morning and to experience pleasure, but they

do not solve the problem of reconceiving of oneself, of learning how to accommodate an illness, of coping

with the unstable relationship I and others I met had developed with our own minds. 

Andrew Solomon, a graduate of Yale and Cambridge Universities, is the author of The Irony Tower:

Soviet Artists in a Time of Glasnost and the novel A Stone Boat. He is currently writing a book about

depression, The Anatomy of Melancholy, to be published by Scribners in 2000. 

Due to the chronicity of many mental illnesses, efficacy research should also embrace rehabilitation as an important form

of intervention. The rationale for rehabilitation is the same as for physically challenged individuals-illness produces

impairments that affect basic life skills and the ability of patients/consumers to function as workers, parents, students, and

in other life roles. Rehabilitation provides basic life skills using methods that compensate for the impairments. Research

demonstrates rehabilitation is effective in helping individuals with severe mental illnesses to acquire new skills and apply

them in their own daily lives. 

Given the wide array of potentially useful interventions, a better understanding of how to use them alone, in combination,

or in sequence to increase the rates of recovery (both symptoms and function) is necessary. In short, NIMH must be

prepared to follow up on promising trials with more generalizable efficacy trials to smooth the transition to effectiveness

trials. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Recommendation 7

NIMH should augment its efficacy portfolio with research that assesses the generalizability of

interventions across diagnostic complexities (e.g., comorbidity or chronicity), as well as

individual, social, and demographic factors. 

Recommendation 8

NIMH should enrich its efficacy portfolio with trials that measure change in terms of both

symptoms and function over a meaningful period of time. 

Recommendation 9

NIMH should incorporate rehabilitation into its intervention portfolio. 

EFFECTIVENESS RESEARCH

The Workgroup recommends a number of approaches to revitalize the effectiveness domain of mental health treatment

research, thereby making it a stronger link to both the efficacy and practice research domains. 

First, the number of patients in effectiveness studies needs to be sufficient to extend the generalizability of efficacy

findings. Such studies should be large enough so that meaningful sub-analyses can be performed on patients often

excluded from efficacy trials (e.g., patients/consumers with comorbid conditions, minority patients, and other frequently

underrepresented groups). Variations in outcomes by subgroups might be an indication to re-examine the intervention in

the context of an efficacy trial for that particular subgroup. There are some excellent examples of large effectiveness trials

testing interventions in the cardiovascular and cancer areas. Mental health researchers should look closely at some of these

methodologies to inform mental health effectiveness study designs. 

Whenever possible, effectiveness research should incorporate multiple study sites, thus expanding organizational,

patient/consumer, and provider mix. Again, there should be sufficient numbers and sufficient data collected on these

aspects of the study to perform meaningful sub-analyses. Such findings can serve as excellent grist for generating practice

research hypotheses. For example, if a certain subgroup has a sub-optimal outcome, it may be possible to disentangle

certain patient/consumer, provider, or organizational characteristics that contribute to this finding. 

An additional need in the effectiveness area is to collect systematically data on the cost of treatment, so that

cost-effectiveness ratios can be determined. For this type of analysis, larger numbers of patients and longer timeframes

than are traditionally funded in randomized controlled trials at NIMH are needed. 

Other opportunities in effectiveness research are to generate studies of commonly practiced but not commonly studied

interventions. Hypotheses for such studies may be generated from practice research findings. Many psychosocial

treatments, rehabilitation interventions, and some off-label use of medications fall in this area. 

NIMH will need to choose wisely among all of the possible effectiveness trials that one would want to see conducted. This

selection will need to be informed by the priority- setting process, a shrewd understanding of what other funding sources

will wish to pay for partial or full support, and the scientific opportunities. Another determinant would be the anticipated

generalizability to adoption in either practice or service settings. The current research portfolio provides a glimpse at the

many settings in which promising mental health interventions should be offered. The service settings shown in Figure 5

include, but are not limited to, the interface between mental health and substance abuse treatment, the criminal justice

system, workplace, nursing homes, primary care practices, and schools. Ensuring a series of trials that addresses care in

each of these settings and the “revolving door” that too many individuals experience across these settings is vital. 

The Workgroup also points out that if effectiveness research is to be funded through NIMH, there must be an emphasis on

the importance of the findings for providing better interventions rather than an emphasis on perfection. There really is no

perfect design in effectiveness research. There will always be a setting left unexplored, a group of patients/consumers who

will not be included, or one more outcome measure that would be important to include. But at some point, the researcher

and the NIMH must strike a balance between what is ideal and what is feasible to derive what is important. The researcher

must do so in the one study, and NIMH staff must do so across the entire effectiveness portfolio. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Recommendation 10

Research should be encouraged that better characterizes promising interventions,

patient/consumer populations, ongoing treatments, and service settings. 

PRACTICE RESEARCH

The investigation, evaluation, dissemination, and implementation of optimal practices in contemporary treatment settings

should be an integral component of future mental health research. Inclusion of practice research into the NIMH portfolio is

relatively recent, and the Workgroup believes that a significant expansion in this area is warranted. Although new for

NIMH, this emerging field is built upon core areas with long histories in health care that have much to contribute to

understanding how mental health care is or can be provided. 

Clinical Epidemiology

There is a need for more resources to be placed into this area of inquiry. Data from observational studies on what happens

with people over time who are receiving treatment provide valuable information to inform clinical care. For example, it

would be helpful to mental health providers and patients/consumers in making treatment choices to have information on

the relapse and recurrence of illness in those who are treated in the community over long periods of time. Policymakers are

interested in knowing how treatment resources are used and, when a particular intervention is used across large numbers of

people, whether there are particular groups who may be at risk for problems. This is a special focus of

pharmacoepidemiology studies. Other types of questions that should be addressed include determining who provides care

for specific illnesses, where it is being provided, and at what stage in the illness. Attaching cost and quality indicators to

these data can be very helpful to those making decisions about allocation of resources. 

With the growth in new diagnostic and screening measures, it is important that they be studied to determine their utility in

various populations and service settings. For example, what is the likelihood that a given person who tests “positive” on a

particular screening instrument for depression will actually have a depressive disorder that requires a treatment

intervention? What is the utility of screening in promoting earlier treatment? 

In order to improve the outcomes of treatments for patients/consumers with mental illnesses, information is needed

concerning how treatments are typically provided in an array of provider settings (e.g., school settings for children and

adolescents, nursing homes for the elderly, jails for incarcerated patients/consumers, etc.) and to special populations (e.g.,

the elderly, children and adolescents, patients/consumers with a comorbid drug or alcohol dependence). 

NIMH should promote research that would help public and private sector policymakers (e.g., mental health commissioners

and corporate benefit managers) decide where to spend what clinical resources. Little is known about the cost or quality of

care, consequences of having the more highly trained clinicians relegated to the consultative, rather than direct care, and

roles they increasingly play in public and privately funded mental health care systems. 

Recommendation 11

NIMH should expand its effort in clinical epidemiology through workshops and through

developing research addressing the issues in the epidemiology of care. 

Quality of Care

Research on specifying the quality of mental health care is relatively new, with relatively few researchers now conducting

quality-of-care research for mental illnesses. This field has developed to a much greater extent in general medical services

research, and there are different conceptual and technical approaches that may be adapted to or implemented in mental

health research. Points of emphasis for NIMH are in the areas of treatment guidelines, outcome assessments, and in the

effects of service organization and financing on the quality of care. 

Treatment guidelines that define appropriate care for a variety of disease conditions and constructing monitoring tools and

systems to assess adherence to such guidelines are important for developing a capacity to monitor the quality of routine

care. Such monitoring helps identify areas where practice needs to be improved, appropriate questions for efficacy and

effectiveness research, and gaps between evidence-based treatment and prevailing practice. 

Research on treatment guidelines is needed to ensure that they are scientifically based and are sufficiently specific to the

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
providers for whom they are intended. Research also must assess the impact of treatment guidelines on clinicians'

practices and clinical outcomes. Surveys might help elucidate in what ways and why clinicians and institutions deviate

from treatment guidelines for both responsive and non-responsive patients/consumers. Another method for obtaining

practice parameter information is either to develop multidisciplinary practice networks or to engage existing practice

networks. Once established, these networks can facilitate the development of large practice parameter databases that can

be used to suggest areas where change is needed and to track changes in practice over time. 

The NIMH and other agencies (e.g., AHCPR) should collaborate in fostering studies of the processes by which to

implement and evaluate guidelines and scientifically evaluate the relevance of guidelines to the patients/consumers in a

variety of settings, as well as obstacles and incentives to follow guidelines. Also, attention should be paid to the increasing

number of guidelines issued for a specific illness. Summaries of guidelines should be provided to help clarify where there

are potential contradictions, gaps, and uncertainties in available treatment guidelines. Research is required to understand

the significance of these differences. 

For intervention research conducted from a public health perspective to be truly useful, investigators must use more than

the standard outcome measures of reduction in clinical symptoms. Researchers must consider and ascertain whether a

treatment or service helps an individual function better at home, work, or in the community. Evaluating improvement at

these levels is challenging. For researchers, a “good” measure is determined by psychometric properties such as reliability

and validity. That is, if the measurement is repeated, does one obtain the same result (reliability)? Does it actually

measure what it is intended to measure (validity)? A central issue that invariably arises is whether increments in the

measure correspond to meaningful changes in an individual’s life and whether some change that may be statistically

significant is actually clinically meaningful. Although the clinical utility of an intervention may be documented by

clinicians, family members, or friends, the most important arbiter is the individual receiving the treatment. Does he or she

actually feel better or see an improvement in fulfilling interpersonal, educational, or vocational goals? Only when the

needed measures are available and used to assess the effects of interventions will policymakers and purchasers have a

better idea of what services must be provided, which ones would be useful if additional funds were available, and which

ones are unnecessary. 

Quality of care research, a third key area, is the relationship of the structural, organizational, and financial arrangements

on the quality of care. As an example in structure and organization, investigators have studied the effects of

inter-organizational services coordination teams on the quality and outcomes of services to children in State custody.

Findings suggest that such teams improve access to services and enhance outcomes (Glisson and Hemmelgarn 1998). For

the general population of children and adolescents, treatment services often span the mental health, general health, school,

social welfare, and juvenile justice sectors, among others. Services are often fragmented, and research is needed to test

new strategies for integration and coordination. 

Other important questions include whether organizing mental health services in a carve- in versus a carve-out arrangement

impacts on the patterns and quality of the services delivered? Do various provider incentives to control costs actually

contain costs, and do they impact on the quality of care? Do administrative policies that aim to reduce cost actually result

in cost savings? 

Also of interest is identifying who should be providing what type of care (i.e., Does it matter if the care is provided by a

psychiatrist, nurse, psychologist, primary care physician, social worker, State-certified counselor, BA-level worker, or

anyone who can demonstrate competence in delivering Treatment X ?). This question of “Where should we spend what

clinical resources?” is tied closely to questions about how to get research into practice. It is one thing to say that

problem-oriented family therapy is an effective treatment for schizophrenia; it is another to say who should be trained in

such a treatment and how to structure the training and treatment environment such that the intervention actually enters

routine practice and produces the desired outcomes. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Recommendation 12

NIMH should expand its effort in generating research to inform and evaluate treatment

guidelines in conjunction with other Federal agencies. 

Recommendation 13

NIMH should review the current measures of functional status and support tests of which

measures or items are reliable, valid, and have clinical utility. 

Recommendation 14

NIMH should expand its research portfolio to include the interface between the architecture of

services (i.e., the structure, organization, and financing of services) and its effects on the quality

of care and clinical outcomes (symptoms and functioning).

Dissemination Research

The Workgroup felt strongly that “How to bring research findings into practice” is in itself a researchable issue and one

that warrants additional NIMH support. The reasons for the large gap between what is known and what is actually offered

can be numerous: 

Clinicians find that reported advances are difficult to implement within their settings. 

Science rarely has complete answers to common clinical and delivery system issues. 

Emerging research developments have outpaced the capacities of many practitioners to remain abreast of new

science. 

Individuals with mental illnesses and their family members do not have easy access to research findings to insist on

research-based interventions. 

Health care delivery systems cannot possibly implement each and every change, and the data necessary to

determine which innovations can most effectively be implemented at their particular site rarely exist. 

There are two thriving literatures that can speed this process. Both dissemination and behavioral science deal with the

processes involved in changing individual behavior and systems. Determining the best methods for improving knowledge

transfer from research findings to practical applications is a complex task involving a broad array of variables and their

interactions. Considerations must include the diverse psychological, social, and economic factors that influence the

behavior of individual practitioners and groups, as well as the organizational dynamics of large-scale service delivery

systems. Areas of relevance to mental health include: 

Behavior change literature to determine what factors seem to be associated with changes in behavior of individuals,

organizations, and systems. 

Patient/consumer and provider behavior change literature to determine similarities and differences in factors

between the mental health and general medical literatures. 

Complex decision-making literature to determine what is known about the acceptability and tolerability of a

treatment for both patients/consumers and providers. 

Community adoption or adaptation of innovations in treatment. 

Furthermore, dissemination must include care settings and care providers who are most commonly today’s treatment

providers. This will include not only clinicians, but case managers and rehabilitation workers who often make up the

greatest portion of personnel budgets for programs that treat people with the most serious mental illnesses, as well as

workers in nursing homes, schools, prisons, and MCOs. 

Given that practice is influenced by what is paid for, it will be important to include purchasers in developing such

research, in that they, as benefit managers, authorizers of care, and contractors of care, are critical players in creating

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
incentives for guideline-based practice. How do systems promote knowledge acquisition and knowledge application? For

example, do the systems provide incentives for staff to increase their knowledge about the effectiveness of established

treatments? Similarly, do they provide educational activities such as workshops, grand rounds, journal clubs, etc., all of

which are intended to increase knowledge? Do systems facilitate the use of the Internet, so that staff have access to current

reviews of treatment literatures and treatment guidelines? What other knowledge bases are available to staff? In addition,

information is needed about the way in which systems facilitate the application of knowledge regarding effective

treatments. Which methods of knowledge transfer improve outcomes? For instance, data indicate that giving Continuing

Medical Education (CME) credits for the first 15 hours of guideline-driven treatment of schizophrenia increases the use of

such guidelines by clinicians in need of CME. What other vehicles need to be developed to improve the practice of the

wide range of treatment providers for whom CME is irrelevant? 

At the system level, information needs to be obtained regarding obstacles to the facilitation of knowledge transfer and

application. Managed care and public sector delivery system policies (as well as selected Medicaid/Medicare

reimbursement policies) have set procedures by which individuals with mental illness are treated that may affect outcome

(e.g., capping the number of outpatient visits; recommending 15-minute medication “visits”; creating obstacles to

providing more expensive but often safer, more effective or better tolerated medications). None of these procedural

intrusions have undergone scientific evaluation. General medicine indicates that at some point, very infrequent visits result

in increased attrition, inadequate patient adherence, and poor outcomes. Similar findings are suggested by a few recent

studies with patients/consumers with mental illnesses (Katon et al. 1995). 

Recommendation 15

NIMH should support targeted research on how to synthesize and incorporate existing

knowledge into clinical practice better, with particular emphasis on understanding factors and

mechanisms involved in changing practice and systems of care. 

SERVICE SYSTEMS RESEARCH

Ample research evidence supports the general conclusion that the organization and financing of systems of care have an

impact on patients’/consumers’ access to needed services and on the quality of those services. There is a need to continue

much of the descriptive work in this domain. For example, it is still important to know the relative cost and distribution of

mental health services (e.g., inpatient versus community-based care in State mental health systems) and the changes over

time. 

However, it is critical to be able to link changes in the organization and financing of service systems to indicators of

quality of care and clinical outcomes in terms of both symptoms and function. For example, we need to understand the

impact of changes in service patterns and cost on quality of care and outcomes. How does the organization and financing

of systems affect treatment delivered in these systems? Do financial constraints limit the availability of known effective

treatments, duration of care, and access to specialists? Or, are some systems of care able to allocate resources

parsimoniously (and therefore more efficiently) without adversely affecting quality of care? How are scarce resources

allocated? What is the affect on outcomes? 

Also critical is the study of the impact of changes in one system on another. For example, when inpatient days of care are

reduced in the State mental health system, are there more days spent in forensic institutions? 

It is important to link service systems research with clinical epidemiology issues. For example, how do policy changes

instituted at the system level (such as mandatory screening of all patients/consumers) affect the overall health of the

covered population? 

Last, studies of procedures to improve access to care for underserved segments of our population (e.g., inner-city women

with children, rural, and frontier populations) are needed. Do these procedures improve outcomes? 

Recommendation 16

NIMH should encourage and sponsor more research that links service systems changes with

quality of care and other clinical indicators of patient/consumer status. 

The Workgroup identified several areas in service systems research where new partnerships can open unique research

opportunities given their importance to public health and/or their policy significance. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Increasingly, advocacy and accrediting organizations are looking for objective measures to assess the quality of the

clinical care provided in various systems of care. Entities now producing report cards or other performance rating scales

include the National Council on Quality Assurance, The Joint Commission on Accreditation of Healthcare Organizations,

the Veterans Health Administration, CMHS’s Mental Health Statistics Improvement Program, and the provider profiling

systems of various MCOs. 

Organizations are beginning to devote substantial resources to enhance performance on these report cards; hence, NIMH

should promote research on the validity of these measures and on enhancing their utility in improving the quality of care.

This information dissemination is occurring absent a substantial research base to support using these tools to inform

policy. Often, the scores on such report cards are from relative scales, and the reader is left wondering whether everyone

might be passing (or everyone might be failing). Similarly, it is unclear whether performance on one narrow domain can

predict performance more broadly. Studies identifying which processes of care are most predictive of good outcomes of

care and then examining whether increasing the performance of those processes results in improved outcomes could be

particularly informative. 

Recommendation 17

NIMH should sponsor validation studies of assessment tools designed to measure the quality of

health care across systems of care to assist policymakers and patients/consumers with decision

making.

Many private and public purchasers and MCOs are developing sophisticated information systems to pay claims, track

patient/consumer care, monitor service utilization, and assist in care management. These systems are the core technology

of carve-out MCOs and are being developed in other organizations as well. The Workgroup felt that there should be

increased attention to understanding and exploring the quality of the data in these evolving systems, since they have the

capacity to affect care substantially. Linking outcomes and diagnoses across such large samples would be significant, as

would linkages with type and duration of services provided. 

Recommendation 18

NIMH should encourage the use of existing databases for understanding service systems.

SUMMARY 

This Chapter advocates blurring the lines across research domains by creating areas of overlap between what are all too

often disparate areas of research. But make no mistake, progress cannot be made without the continuing use of

smaller-scale trials to efficiently detect promising leads. Equally critical are the service studies that work with existing

databases, whether clinical conditions are fully documented or not. What requires a boost are the areas of linkage that

include some elements of two or more areas of research, such as sharing diagnosis or outcome measures, that will provide

the common element for translating and building across research domains. One can think of these as hybrid designs or as

methodological grafts upon sturdy stock. The salient idea is that measures or methods that can function in multiple

domains should be developed and used. Specific recommendations on these methodological areas of overlap will appear in

Chapter 4. 

| Home | Research | Top | 

CHAPTER 4 

METHODS DEVELOPMENT AND INNOVATION

INTRODUCTION

Innovations in intervention research methodologies are essential to improving individual and public mental health through

enhanced treatments and service delivery systems. Among the necessary innovations are new research designs and

creative analytic approaches. In addition, the NIMH and other funding sources must employ these and other new tools to

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
maximum advantage by supporting sufficiently large studies that are necessary to generate data that can inform public

policy. This chapter describes basic methodological needs and offers several recommendations to promote methodological

development and innovation. Given the technical nature of many of these research problems, some portions of this chapter

require a fundamental understanding of research and evaluation methodology. 

Issue: Incorporating Measures of Costs Associated with Interventions

A priority in understanding public health relevance is the cost associated with providing an intervention. Costs can be

divided into direct costs (i.e., what it actually costs to deliver the services in a specific setting) and indirect costs (i.e.,

costs other than direct treatment, such as lost productivity). Research also may seek to calculate the longer-term cost

difference resulting from providing an intervention or not providing it. What is the cost to society as a whole, not simply to

the treatment system, of leaving a mental illness untreated? The answer to this question affords an economist's estimate of

the value of an intervention. 

At present, economic measures (e.g., cost-effectiveness and cost utility) are not widely applied in mental health

intervention research. Such measures should be considered for broader use. The NIMH will need to ensure that

well-targeted, large-scale opportunities for improved studies of the costs of treatments are developed carefully. In addition,

cost-effectiveness analyses should be incorporated as part of all large clinical treatment studies. 

There are numerous challenges to implementing good cost-effectiveness studies. Intervention research often examines only

direct treatment costs, but specific components of direct and indirect costs should be considered for different types of

studies. For example, in treatment studies for individuals with psychotic disorders, costs for lost work may not always be

applicable, but information on lost workdays for family members may be an important consideration. Implementing high

quality, cost-effectiveness studies necessitates improved cost measures, larger sample sizes, and incorporating economic

expertise into mental health intervention research. 

Recommendation 19

NIMH should improve measures and analysis of costs in intervention studies, with special

attention to successful examples from general medicine. 

Recommendation 20

Large intervention studies should include a cost-effectiveness component that uses the best

methodology available.

Issue: Tailoring Interventions to Communities

Greater attention is needed to ensure individual and community applicability of treatments that have been developed and

tested in more restricted settings and populations. External validity refers to the generalizability of research findings to a

broad variety of treatment settings. 

Treatments need to be tested for their application in community settings. Investigators may need to modify existing

treatment protocols for such studies or to develop new "applied" protocols that are independently tested for effectiveness.

The translation of treatment and service system interventions developed in the course of research studies into forms

suitable for community application may be a focus in its own right for research. 

Research also must examine how treatments and their impact vary with the characteristics of communities and health care

systems, and with the behaviors of patients/consumers and clinicians. A related task is to study the characteristics of

interventions used in community practice, and to develop classification schemes that permit key elements of these

interventions to be reproduced and tested in effectiveness studies. NIMH should encourage development of minimum

standards for describing treatment settings and should encourage researchers to report such information on treatment

context. Eventually, such information should prove helpful in models of external validity. 

Recommendation 21

Approaches to conceptualize and assess key characteristics of intervention settings should be

developed, as should models for understanding the effects of these characteristics on outcomes. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Understanding community settings requires incorporating usual care into the intervention protocol or, at least, to have

usual care as a control condition. For either purpose, the features of usual care must be defined explicitly. Usual care

settings may differ on a variety of characteristics, including types of patients/consumers, providers, financing,

organization, and constraints and barriers to care. Which characteristics of usual care are implemented in intervention

protocols should be specified. Since usual care can vary across settings, the comparison standard for a specific treatment

could vary across settings; this, in turn, imposes an interpretative challenge to the external and internal validity of study

results. Techniques to analyze these variations, such as explicit modeling of “process” variations in usual care and their

outcome implications, should be developed. Theoretical and empirical work is needed to determine which features of usual

care are relevant to the estimate of the true intervention effect. 

Recommendation 22

NIMH should encourage the development and evaluation of key research measures for assessing

“usual care” and develop analytic methods to adjust for variation in components of usual care.

Issue: Tailoring Interventions to Distinct Subgroups

External validity also refers to how well a finding may generalize to a specific individual with the target illness. This

means that the sample for the study should be selected to represent accurately the full range of those individuals suffering

with the illness. As previously discussed, intervention studies frequently exclude individuals with comorbidities, yet

comorbidities such as substance abuse or other health problems are the rule rather than the exception in community

settings. 

Some balance of separately studying comorbid populations, purposive sampling of key subgroups, and naturalistic

inclusion or representation of a range of comorbidities is needed to develop information on intervention effects in people

with comorbid problems. This requirement implies gathering participants from a broader range of treatment facilities,

including those which specialize in treating people with dual diagnoses, and expanded or modified protocols that

adequately address the treatment needs for people with comorbidities. 

Prioritizing among groups for inclusion in studies requires determining which cultural or demographic subgroups affect

true treatment responsiveness. Relatedly, it is important that cost-effectiveness research include groups for whom the costs

of care differ. This requirement implies need for theoretical and empirical research on the cultural and demographic

predictors of compliance and intervention response across a range of relevant outcomes. 

Broadly representative samples increase generalizability, but they pose many challenges to intervention studies. Samples

become quite large if the study is to be applied to many types of individuals, because a generalization must be based on

including a sufficient number of such individuals. When an intervention is to inform insurance coverage decisions, the task

becomes even more unwieldy for research findings ideally should be generalized to different types of employers and

insurers affected by the policy. One of the challenges for the future is developing representative sampling frames of large

systems. Another challenge will be engendering cooperation for recruiting representative systems. 

Recommendation 23

NIMH should promote development of innovative sampling strategies for inclusion of

underrepresented groups and sufficiently large intervention studies to incorporate

representative community populations.

Issue: Considering Treatment Preferences and Decision Making

Clinicians and patients/consumers often tailor treatment decisions (e.g., choice of medication or psychotherapy) on the

basis of provider or patient/consumer preferences and clinical characteristics. Outcomes may be monitored through

individualized goals and changes in symptoms that are troublesome to the individual patient/consumer. Little research has

been conducted on provider and patient/consumer treatment preferences, how to assess them, or how to incorporate them

explicitly into treatment protocols. Research is needed on the theory, constructs, and assessment of preferences, how these

factors are incorporated into usual clinical decision making, and their impact. It is important to understand, for example,

how clinicians utilize data from research findings to make treatment decisions. In addition, research is needed on

translating practice results to individual patient/consumer care decisions and on conducting research in a manner that more

closely mirrors individualized patient/consumer monitoring and decision making. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
As noted in Chapter 3, efficacious treatments are ineffective if they cannot be applied in a particular setting. For instance,

cognitive therapies have been shown to be efficacious with 12 to 20 sessions. In a setting where only 6 sessions are

permitted, what is a clinician to do? It is important to describe variations in both the clinician’s and the service setting’s

ability to adhere to recommended procedures. In addition, it is important to develop models of predictors of such

adherence and to develop methods that incorporate information on adherence into estimates of effectiveness. Adherence

rates may explain variation in effectiveness across systems or the conflicting results obtained from efficacy and

effectiveness studies, respectively. Research on adherence suggests the need for both measurement and theoretical

development. There has been much research in general medicine on clinician and patient adherence (e.g., treatment fidelity

and integrity). These findings should be extensively studied and applied in mental health intervention research. 

Recommendation 24

NIMH should encourage the development of methods to study and incorporate clinician and

patient/consumer decision-making processes into intervention research. 

Recommendation 25

NIMH should support research to identify common practices believed to be helpful and bring

them under research scrutiny, that is, ascertain what is going on in the practice community, and

determine how much of that is beneficial. 

Recommendation 26

NIMH should encourage the improvement of methods for both evaluating clinician

implementation and patient/consumer adherence to treatment recommendations and estimating

the consequences of these variations on the effectiveness of treatment.

Issue: Confidence in Drawing Conclusions from Findings

Decision analysis, econometric modeling, and other analytic techniques should be explored as tools to develop models of

effectiveness across systems with different characteristics, especially when observational data are used. Studying

treatment effects in the context of an observational study requires analytic techniques with reasonable assumptions about

measured and unmeasured differences in compared groups. The last two decades have brought considerable advances in

methods used in social policy evaluation, econometrics, evaluation research, and biostatistics to support drawing

inferences from observational data. The key technical issue is identifying and accounting for bias from unmeasured

variables. These techniques are controversial, can be difficult to apply, and are unfamiliar to most mental health

researchers. To address these issues adequately, the application of various new methods to mental health treatment

research should be explored vigorously. 

Since they are much less precise than analyses for experimental data, these methods could result in sample sizes that

become quite large. This increases the potential utility of large, naturalistic data sets that include outcomes and

quality-of-care data, but such data sets are rare. An alternative is carefully designed, often expensive studies that combine,

in innovative ways, experimental and observational features. 

The need to ensure confidence in findings also has implications for the types of data needed in a study. In economics, for

example, the instrumental variable technique is used to adjust for treatment biases that are due to observed or unobserved

factors that could bias the selection of subjects or their attrition from the study. It is well known that such instrumental

variables can be difficult to identify. Thus, methods development is needed to explore suitable instrumental variables for

mental health intervention research. Other techniques include causal modeling, propensity variables, and regression

discontinuity design and analysis. The strengths, limitations, and underlying assumptions of these techniques and their

suitability and feasibility for mental health intervention research should be explored. 

Recommendation 27

NIMH should explore new methods for analysis of data from studies that incorporate innovative

combinations of research designs.

Issue: Assessing External and Internal Validity

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Across medicine, numerous reviews of the quality of clinical trials have pointed to difficulties in design and

implementation of trials that limit the certainty of causal inferences. Such difficulties may include, for example, excluding

dropouts and individuals who refused treatment, which can limit interpretation of intent-to-treat samples. Contamination

across treatment groups in their exposure to treatment limits both generalizability and the estimate of the true treatment

effect. “As-treated” analyses break randomization and use the experiment as an observational study, which may lead to

many of the same problems encountered in correlational observational studies. However, the sample sizes in most mental

health clinical trials are often too small to evaluate selection effects adequately or to permit the use of observational

analysis techniques to analyze the potential effects of bias from unmeasured variables. Similar or even more troublesome

problems arise in experimental health services intervention studies, which typically do not have the advantages of masking

and other features to protect internal validity that are common to clinical efficacy trials. 

Studies that combine both experimental and observational designs or analytic features to address questions about treatment

in a community context will typically require studies of longer duration, larger sample sizes, and broader domains of

outcome, which, in turn, may require a reduction in the intensity (or depth) of data collection (e.g., about clinical factors)

for the study to be feasible. Thus, such alternative approaches can entail a trade- off between breadth and depth of

information collected and analyzed, on the one hand, and the ability of studies to inform either clinical or policy audiences

on the other. Some goals of these studies, such as having adequate case-mix adjustment methods to permit outcomes

comparisons, may be compromised by requirements for reductions in the depth of measures. Methodologists should

develop approaches to model those trade-offs in breadth and depth for given study purposes and to maintain reasonable

costs for the study. 

When studies involve multiple design decisions that favor external or internal validity, rarely are both improved

simultaneously. Decisions to improve community usual-care relevance, for example, probably will increase external and

decrease internal validity. Developing approaches to explicitly model the trade-off of internal and external validity, given

the research purposes, state of knowledge of the field, and research costs would be useful in informing these design

decisions. Such methods work may be feasible using data from early multiplex-designed intervention studies, existing

observational data, or from meta-analyses. These complex studies may require development of methods to estimate power

and conduct analyses in the context of hierarchical sampling designs and a mixture of experimental and nonexperimental

questions and analyses. 

Basically, once an intervention study indicates that a treatment works and can be tolerated by participants, the next series

of questions is to understand the characteristics of the study, to see how well the treatment can be generalized. Simply put,

the questions are who, what, how, and by whom? Who received the intervention and responded or did not respond? What

was the exact treatment? How was the treatment provided and by whom? Documenting these characteristics in the

efficacy and effectiveness stages is very important for building the next set of questions. 

While standard approaches exist to statistically adjust findings for research participants' demographics or clinical

characteristics, no comparable approach is available to adjust for characteristics of the treatment setting. It should also be

noted that as the treatment progresses through the cascade of tests, an efficacious intervention may be tried in a group of

patients/consumers where there is no significant improvement (e.g., the elderly). Only by knowing the specifics of the

previous efficacy and effectiveness work can such a result be interpreted and a plan to search for the cause of this lack of

effect established. The trade-offs between generalizability and feasibility are never ones that a researcher wants to make.

One simply cannot afford the time or costs that would be involved in recruiting a sufficient number of participants to

reflect each diagnostic complexity or age group. Equally difficult choices must be made regarding the number of service

settings that can be included in a study to afford generalizability. Because of the needs to make such trade-offs, it is

important to develop methods of explicitly considering the trade- offs. How should a researcher consider budget, sampling

plans, and public health relevance to determine when an increase in representativeness is worth the associated increase in

research costs? 

This lack of information limits the ability of clinicians or policymakers to consider whether their settings are sufficiently

similar to those in which the research was conducted to apply findings to a local or front-line situation. This is a critical

issue, since health care settings have diverse characteristics, such as physical setting, staff configuration, financing

characteristics, and treatment style. It is not clear which are salient to implementing a new intervention. 

Recommendation 28

NIMH should encourage development of methods to explicitly evaluate trade-offs in alternative

design features that differ in their implications for internal and external validity. 

Issue: Fostering Innovative Methods 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Pursuit of the goals of methodological development requires integration of clinical and social science, particularly

incorporation of statistical and econometric expertise. Expertise in qualitative analyses also is likely to be needed to

generate new approaches to studying systems, treatments, and adherence with treatment. Exposing mental health services

and treatment researchers to methods and methodologists from non-mental health and non-health fields may lead to

innovative solutions. The Workgroup considered several strategies to develop rapidly a body of knowledge and alternative

methodological strategies for implementation into mental health intervention studies. While emphasizing the need for

NIMH to support research studies, the Workgroup also urged the Institute to take some immediate steps to foster

development of methodological research. 

NIMH should host one or more methods workshops focused on several key aspects outlined in this chapter. These

workshops could advise the NIMH on likely alternative methods to develop improved standards for innovative, combined

research designs. Study section members should be invited to participate. 

Following these workshops, NIMH should host a broader conference on the key methodological problems facing

intervention research. This conference should be focused on data-based presentations from existing studies, and may even

mirror activities in the statistical genetics community by holding a “competition” for analysis of a particular data set.

Teams of researchers could present their approaches and findings, so that the robustness of the findings to the analytic

procedures becomes evident. 

Recommendation 29

NIMH should convene a methods workshop to identify options for advancing intervention and

service systems research. The results of this workshop should be debated broadly and options

tested in appropriate follow-up activities.

| Home | Research | Top | 

CHAPTER 5 

NIMH's LEADERSHIP ROLE IN PROGRAM DEVELOPMENT, REVIEW,

AND ADMINISTRATIVE ACTIVITIES

INTRODUCTION

To generate answers to the questions raised in the areas outlined in the previous sections, NIMH will need new approaches

to stimulate, review, and support the specified research, training, and dissemination activities. New approaches to develop

this infrastructure are required because of the nature of collaborations that must be developed and maintained, as well as

the numerous new service settings where research has not been routinely conducted. For example, collaborative, public

health-oriented projects may not fit easily into procedures that work well for basic science initiatives. Areas identified in

this report as being in need of emphasis typically require research staff networked across academic and provider settings.

Also, opportunities for research in diverse practice settings (e.g., primary care, workplace, schools, or public health

delivery systems) may arise and disappear quickly. NIMH must be able to make funding decisions rapidly to take full

advantage of these windows of opportunity. 

This chapter discusses administrative efforts that NIMH staff can undertake to provide the infrastructure for the

Workgroup's research, training, and dissemination recommendations. These recommendations principally urge tailoring

existing procedures and speeding administrative processes to meet emerging program objectives. 

Recommendations addressing the NIMH's role in developing new research areas--from the conceptualization of a research

or training initiative, through peer review, into the monitoring of funded projects--are particularly timely due to

Vice-President Gore's Reinvention Initiative. In response to this initiative, institutes across NIH are proposing novel

approaches to targeting research development, facilitating new alliances among research teams, and expediting

administrative processes. NIMH is encouraged to take advantage of this time of reinvention to build creatively and

efficiently the infrastructure for generating research to shape the public mental health. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
PROGRAM DEVELOPMENT: SELECTING OR CREATING AN APPROPRIATE FUNDING MECHANISM

The NIMH supports its extramural research activities through either grants or contracts.4 These two funding mechanisms

can take many different forms to support a surprisingly diverse range of activities responsive to scientific opportunities and

public health need. The Workgroup recommends that these same mechanisms be adapted as necessary to accomplish

newly identified program objectives: (1) creating research opportunities in nonacademic settings, and (2) training

researchers who can work in both academic and provider settings to yield more meaningful treatment and services

research. 

New areas slated for development, identified both in this report and through the evolving priority-setting process that the

Workgroup proposes in Recommendation 1 must be announced publicly. Interest in new grant activity is announced in the

NIH Guide,5

 which prints the full text of all RFAs6 and PAs. The Catalog of Federal Domestic Assistance7contains brief

synopses of each government agency's funding interests. Also, each NIH institute maintains a web site that lists all of its

PAs and RFAs.8 Similar tools are available to publicize RFPs for NIH-led contracts. Indeed, all Federal contracting

opportunities are announced in the Commerce Business Daily,9 and each institute typically posts these on its web site. 

Most announcements issued by the NIMH are calls for research grants. The Institute highlights general areas of research

interest and need, but asks independent investigators to develop applications within their area of interest. Many different

types of grants are used, with each solicited through an RFA or PA. 

Each RFA and PA lists programmatic objectives, a specific funding mechanism, review criteria, a description of the

review and award process, and the terms and conditions of the award process. The announcements are powerful tools for

shaping a research portfolio. Accordingly, recommendations in this section call for tailoring the specifics of future

issuances to begin shaping both the proposals submitted and the reviews that follow. The section discusses: (a) selection of

the funding mechanism; (b) project requirements; and (c) review criteria. 

In articulating program objectives, NIMH staff often find themselves constrained by existing mechanisms. Research

training, for example, typically occurs at single academic sites, supported by institutional training awards. This works well

for mature fields. In areas recommended for development in previous chapters, however, multiple academic and

non-academic settings may be required to give a well-rounded training experience for an individual. New or revised

research training support mechanisms are needed to enable research to be conducted in non-academic treatment and

service settings and to expedite the review and award process. 

Issue: Mechanisms for Training

New multi-site training models designed to provide developing researchers with well-rounded experiences should include

training in research and evaluation offered at academic institutions, along with hands-on experience in provider or policy

settings. More researchers also are required who focus on mental health and who are trained in economics. Because few

sites offer the required range of expertise and experiences, consortium arrangements between training sites in clinical and

services research are needed. 

Established investigators may be skilled in one area, but require additional training or experience to move into these new

areas of research. The Workgroup found that the new K awards in clinical research could be very helpful in achieving this

goal. Research grant supplements and support for faculty scholars programs also offer methods to support established

researchers' training at academic or non-academic areas of research excellence. Training sites might include current NIMH

Intervention or Services Research Centers, other academic sites, businesses, State or local governments, or private

foundations. Generalizing from successful examples of creative training activities should be encouraged. For example, the

Public Academic Fellowship Program at the National Association of State Mental Health Program Directors Research

Institute provides interested researchers with research skills in large-scale policy analysis on mental health issues. 

Additional possibilities for helping individuals hone their clinical and services research skills include summer institutes

(perhaps similar to those sponsored by the American Association of Geriatric Psychiatry), the establishment of a Faculty

Scholars Program in partnership with medical schools, and the creation of training programs without walls where an

individual would craft an individually tuned training program with components from a variety of relevant institutions (e.g.,

academic, purchaser, or governmental monitoring settings). These efforts should focus on substantive areas outlined in

Chapter 3, but also on the methodological areas in Chapter 4. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Recommendation 30

NIMH should develop additional training and career development programs that offer hands-on

experience in diverse research settings in order to provide researchers with an enriched training

experience.

Issue: Mechanisms for Creating Collaborative Research at Provider Sites

Working partnerships involving academic researchers and front-line service providers are the crux of efforts to develop the

types of research the Workgroup outlined to improve the quality of care and the public health impact of the research. The

diverse sources that the Workgroup consulted regarding research partnerships identified several common ingredients for

successful alliances: ensuring benefits to all parties; selecting topics of mutual concern; requiring that each participant

contributes a needed role or service; designing studies with timeframes appropriate to each partner; maintaining continuity

of relationships to promote ongoing access and follow-through; nurturing good relationships at all levels of the

partnership; and using partners' time efficiently. Such ingredients are challenging to assemble yet hold enormous promise. 

Creating and maintaining such partnerships will require flexible NIMH support over the long term. Although no single

recipe for partnerships exists, NIMH can provide a range of options for identifying and fostering possible new alliances

based on developing shared goals, joint problem solving, and resource sharing. 

Expanding Previously Successful Approaches 

Once established, support is necessary to maintain long-term relationships. A decade ago, the NIMH-sponsored

Public-Academic Liaison (PAL) Program created incentives for establishing such research partnerships requiring

collaboration with a State mental health administration. This criterion created incentives for academic researchers

to work with States and helped ensure the relevance and utility of their work. The program also helped to develop

the research capacity in several States. 

Recommendation 31

NIMH should revise and renew PAs in the spirit of the PAL Program to maintain and promote

existing partnerships between academic researchers and public care systems, health plans, both

carve-outs and HMOs, and employers providing health benefits and their representative groups.

In addition, NIMH should make use of its current investments wisely. The Workgroup found that a significant effort

should be made to enable the Intervention and Services Research Centers to develop ongoing partnerships with care

delivery systems. The States' public mental health systems are obvious partners, which Centers and other research groups

should cultivate more extensively. In addition, managed care entities such as large HMOs, Preferred Provider

Organizations, and managed care vendors are potential partners. The NIMH should judge the effectiveness of Centers, in

part, by their ability to establish and utilize productive partnerships; at present, however, although these Centers often

house needed research expertise, they may be constrained by their narrower missions and the funding levels. Permitting

the supplementation of these efforts would be quite valuable. Once adequately supported, the ability to establish and

maintain productive partnerships should be a core expectation for Centers. 

Recommendation 32

NIMH research centers, which demonstrate great potential to secure partnerships with service

delivery systems and to utilize these systems to conduct intervention research, should be

supplemented to develop, implement, and sustain such partnerships.

Purchasers–government health plans as well as corporate, HMOs, and other MCOs–are important research partners for

many reasons. Researchers increasingly are turning to purchaser plans as loci for research, especially service systems

research. These clinical settings typically have large administrative databases useful in describing patterns of care.

Research conducted in collaboration with database administrators could improve the availability, scope, and quality of data

obtained in the course of routine mental health care, thereby facilitating more accurate case-mix adjustment,

quality-of-care analyses, and outcomes analyses. 

Treatment research also may benefit from examining large-scale observational data, such as claims files. In presentations

to the Workgroup, individuals representing various practice settings described the promise of these data sets, but also the

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
difficulty in accessing and maintaining them. As outlined substantively in Recommendation 15, the Workgroup felt that a

more specific recommendation was required since this would be of immediate benefit to both sides of the research

partnership with minimal investment. Members pointed to the need for support to enable information specialists and

researchers to work on these data sets. 

Recommendation 33

NIMH should issue an RFA or PA to encourage secondary analyses of service systems data and

to establish accessible formats for such data to maximize the use of databases in practice

settings.

New Alliances 

In some cases, NIMH will need to foster initial alliances. Likely partners might include established academic sites

seeking to build research programs with State mental health administrations, MCOs, group practices, hospitals, or

businesses. For new partnerships, there are many aspects of planning and conducting research that must be

considered and reconsidered. Taking advantage of what has worked at other sites can be a significant starting point

in this trial and error process. 

For instance, the New Hampshire-Dartmouth Psychiatric Research Center originated in such an award, as did the

Research Division of the Connecticut Department of Mental Health. The story of the New Hampshire-Dartmouth

Psychiatric Research Center is provided in the side bar as an example of successful partnership, but also to capture

the flavor of the interactions in this partnership. 

The New Hampshire-Dartmouth Project

The New Hampshire-Dartmouth Psychiatric Research Center (PRC), a partnership of the Dartmouth Medical School

and the NH Division of Mental Health (DMH) is an oft-cited "success story" among public- academic research

alliances. Workgroup member Susan Essock spoke to DMH Director Paul Gorman and PRC Director Robert Drake

about what factors might contribute to the success.

Dr. Essock: Could you describe how the collaboration got going and features that helped make it successful? Bob, you

were there from the beginning.... 

Dr. Drake: It began as part of a larger contract between Dartmouth Medical School and the State to provide professional

services for the State hospital. Don Shumway, then director of the DMH and the Dean of the Medical School, asked if the

contract could also incorporate an evaluation component. All of us who were involved from the beginning were part of

Dartmouth, but also worked in community mental health centers and believed in what the State was doing and wanted to

be part of this. 

Dr. Essock: Paul, did you and Don Shumway have a vision of how you wanted Dartmouth to help? 

Dr. Gorman: The contract was envisioned originally as a means of helping elevate the level of care in the State hospital.

We were confident that if the attending physicians at the hospital were all to be faculty members, there would be a

distinctive difference in the clinical culture over time--and in fact that has happened. 

Simultaneously, we recognize that the lion's share of the people who depend on the public mental health system in

our State are in the community. So a major concern is with the community and with the clinical culture that exists in each

of the 10 community mental health centers--all independent organizations with their own culture. 

It is a challenge to think about how to get some common veins of best practices across that system. Our experience

has been that using the research agenda does that very effectively. 

Dr. Essock: What features did you build in that help to make sure this partnership is useful to each side? 

Dr. Gorman: The level of communication that we have is critical. Bob's office here in Concord is next door to mine and

we meet regularly. Also, from the very beginning, while the agenda was mental health research, that agenda corresponded

with a DMH initiative to develop a set of best treatment practices for people with dual diagnoses. Ever since, the PRC has

conducted its research on issues that the DMH is pushing forward. We moved from dual diagnosis and continuous

treatment teams, to vocational services, to trauma. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Dr. Drake: I also think this has worked quite well because the agenda really has come from DMH and the mental health

centers. We initially made a practice of going around to all of the mental health centers and talking with the clinicians and

administrators about their biggest problems and how we might be able to help. That is how we settled on the dual

diagnosis agenda--not because it was ours at the PRC. In 1990, DMH identified vocational services as a key agenda item

and we have been working with DMH and various centers on that ever since. 

Dr. Essock: Can you comment on that work? 

Dr. Drake: One research project involved a center that closed its day treatment program and converted to a supported

employment program. Another center agreed to serve as a comparison group. When the study was over, the second center

was shocked that the other center had moved so much ahead of them. 

Both clients and clinicians in the "control" center demanded more access to supported employment. They then set

up a trial wherein they did day treatment in the morning and supported employment in the afternoon. PRC staff helped

them learn how to do it and within six months, every client dropped out of the day treatment program in favor of the

supported employment program. The center then converted completely-- entirely as a result of sharing the research data

with them. The data drove the agenda.

So, our research showed that centers can close day centers, improve vocational services, and be more

cost-effective. On a statewide level, we have evidence that during any quarter the rate of competitive employment among

consumers who have participated has increased from 9 percent in 1990 to 35 percent in 1998. It goes up a little bit each

year, and we do not know where it will plateau. 

Dr. Essock: What advice would you have for States who might want to start such a project, recognizing that many who

have tried have been burned in the past? 

Dr. Drake: We always have had an internal rule that our first priority is to be part of the system, and to make sure that the

system benefits as much or more than the project benefits. Researchers should figure out what they can do, in terms of

training or other services, to make sure that they are good houseguests who will be wanted for the long run. We spend a lot

of time asking people in the centers how we can best help them and do a better job of things that are important to them. 

While this often leads to doing research on issues identified as important by the providers as opposed to

researchers, it usually is possible to identify topics on which our interests converge. 

Second, researchers should do what they can to help--give talks, review difficult cases, help set up training

seminars, make laboratory facilities available, assist service settings with their computer systems or statistical use of their

data. Centers usually need and appreciate help like that.

Third, it is very important to meet with the patient/consumer and family groups and try to assure an allegiance of

all of the stakeholders involved in a project. Everyone should learn something from it.

If you spend time educating patients/consumers, they will make good choices about the services they want and

demand. The same is true with mental health systems--if you include people and share your data, they see quickly that

things go well when researchers are around and will want to have them there more.

Recommendation 34

NIMH should stimulate new alliances by providing developmental funds to establish shared

research resources such as data banks, staff time, consultant time, etc. 

Recommendation 35

NIMH should commit resources to identify, describe, and disseminate models of successful

partnerships. 

Issue: Requirements for Meeting the Broader Scope of Public Health Research

PAs and RFAs afford NIMH program staff the opportunity to convey to the field the Institute's plans for augmenting its

portfolio. Therefore, it is critical that these announcements clearly and concisely articulate requirements for the research to

be supported; issuances must outline general areas of need, provide clear examples, and ensure that the requested research

is feasible and within budget allocations. These factors should be presented in the context of broader NIMH program

objectives. Specifically, announcements should call for public health-oriented research, enunciating the type of research

sought as suggested by this report and future priority setting. That is, make clear statements about the need for longer trials

or a focus on rehabilitation outcomes. This tailoring is required to promote such complex undertakings, and clarify what is

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
wanted for both the applicant and the reviewers. 

Sometimes, the RFA and PA approach is not specific enough to meet programmatic goals. Contractual support may be

preferable when a clearly articulated goal and plan of action is required. Emphasizing the targeted nature of the procured

service (in this case conducting the research or providing the infrastructure for the research) in the RFPs is essential. If the

goals and deliverables can be clearly specified to encompass priority intervention questions, a focus on contracting-out of

large-scale trials through a coordinating center can extend the Institute's ability to provide meaningful research findings.

NIMH should be prepared to support active external advisory groups to aid in the ongoing oversight of such large-scale

efforts. 

Recommendation 36

NIMH should consider contracts as a mechanism for supporting clearly articulated research

needs and deliverables.

Issue: Review Criteria

The NIH Director has established and the Center for Scientific Review (CSR) has implemented five specific criteria with

which to assess applications. These criteria include: 

Significance–Does this study address an important problem? If the aims of the application are achieved, how will

scientific knowledge be advanced? What will be the effect of these studies on the concepts or methods that drive

this field? 

Approach–Are the conceptual framework, design, methods, and analyses adequately developed, well integrated,

and appropriate to the aims of the project? Does the applicant acknowledge potential problem areas and consider

alternative tactics? 

Innovation–Does the project employ novel concepts, approaches, or methods? Are the aims original and

innovative? Does the project challenge existing paradigms or develop new methodologies or technologies? 

Investigator–Is the investigator appropriately trained and well-suited to carry out this work? Is the work proposed

appropriate to the experience level of the principal investigator and other researchers (if any)? 

Environment–Does the scientific environment in which the work will be done contribute to the probability of

success? Do the proposed experiments take advantage of unique features of the scientific environment or employ

useful collaborative arrangements? Is there evidence of institutional support? 

The Workgroup encourages NIMH to educate reviewers that the NIH criteria are highly relevant to evaluating scientific

and public health significance of grant applications. The point is that innovation need not be limited to research design.

The innovation can be in the aims, such that a new treatment evaluated by a standard clinical trials design can be quite

innovative. The "new" treatment could also lack innovation if the treatment was actually only a minor modification of a

previous treatment or an assessment of a drug that is highly similar chemically to another well-researched drug. Institute

staff are urged to insert in future RFAs and RFPs specific criteria that will prompt reviewers to consider public health

importance and policy significance when evaluating intervention applications. 

Recommendation 37

The importance of public health relevance should be specifically added to the review criteria in

new PAs, RFAs, and RFPs in the relevant areas to emphasize its significance for applicants and

reviewers.

Issue: Expediting Administrative Decisions

In addition to listing criteria for review and award, RFAs and PAs contain a timeframe for review and award processes.

Just as review criteria are important factors in tailoring responsive treatment and services research, the rapidity with which

the Institute can make and act on funding decisions is critical to building and shaping the research portfolio in such

fast-evolving areas as service systems research. 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Opportunities for research collaborations, particularly with purchasers, often come and go before a traditional review and

award process can be completed. For example, changes in welfare legislation that require wrap-around mental health

services, or the shift by States to Medicaid managed care contracts, offered valuable but fleeting opportunities for research

regarding the cost-effectiveness of alternative treatments and/or the impact of changes in payment mechanisms on people's

lives. Unfortunately, such initiatives typically are not fully evaluated because researchers cannot obtain research or

evaluation monies from NIMH quickly enough through the traditional review and grant award process. 

Developing an application for Federal support can take several months and an additional 9-12 months may elapse pending

a funding decision from NIMH. The more complicated and expensive the project, the more likely that a longer timeframe

will be required. Businesses or State mental health authorities cannot delay changes they want to or must implement until

researchers can obtain support to evaluate such changes. Even if they could, staffing and budgets can change significantly

over time. For these and other reasons, promises on the part of non-traditional research co-sponsors cannot be expected to

endure without timely follow-through. The same often is true of private foundations that launch research or evaluation

projects that would benefit–or require–supplemental funding from NIMH. 

The Workgroup hoped that some more timely mechanism could be developed. The members' shorthand for this was a "911

mechanism" for rapid review and consideration. Indeed, other institutes are exploring methods for shortening the 9-12

month framework. 

Recommendation 38

NIMH should develop a mechanism for responding to unique, but fleeting research

opportunities with great public mental health significance.

A related issue is the additional time and resources that might be required if an investigator assesses comorbid or other

conditions. In practice settings, patients/consumers often have comorbid substance abuse and other health problems that

complicate treatment and service plans. Ignoring these complexities reduces the clinical utility of information gained

through treatment and services research. Yet, each Institute receives its own allocation from Congress to address these

conditions separately [i.e., the National Institute on Alcohol Abuse and Alcoholism (NIAAA), the National Institute on

Drug Abuse (NIDA), and NIMH]. Investigators can find themselves negotiating with three NIH institutes for grants or

grant supplements, a process which they–not to mention business leaders and State mental health directors–describe as a

significant burden. Representatives of these groups urged that monies for cross-cutting research be established and that

cross-institute award processes be formalized and encouraged for research on comorbidity. 

Recommendation 39

NIMH is asked to work with NIAAA and NIDA to encourage timely joint funding decisions for

comorbid substance abuse and mental illness research. 

REVIEW CONSIDERATIONS

All research grant applications and contract proposals are subject to scientific peer review, wherein substantive or

methodological experts evaluate the merit of the research. This case-by-case evaluation is a primary factor in funding

decisions. The Workgroup emphasized the crucial role of peer review in fulfilling the NIMH's scientific and public health

mission. 

Achievement of the field development activities proposed in this report will increase the burden on review groups. The

number of applications requiring review will increase, as well as demands for a broader mix of expertise among reviewers

and review staff to evaluate the scientific merit and practical significance of proposed work. Applications responding to

calls for "boundary spanning" research will require reviewers who are able to assess the value of such research to diverse

stakeholders including patients/consumers, families, purchasers, governmental agencies, and service providers. 

Issue: Review Structure

The Workgroup conducted its work during a period of extensive change in peer review processes at the NIH. Following

the 1992 NIMH reunification with NIH, certain distinctions were maintained, such as separate peer review for the three

institutes that formerly comprised the Alcohol, Drug Abuse, and Mental Health Administration. In 1996, all NIH institutes

with substantial investments in neuroscience initiated efforts to merge the separate peer review systems into one. By the

time this Workgroup began its work, NIMH was completing the integration of its neuroscience and AIDS review

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
committees, and had begun the behavioral science review merger process. Recognizing the unusual opportunities for

change, the Workgroup requested that NIH reserve the treatment and service interventions for a later phase in the

integration process; a delay would afford additional time to examine review issues for the portfolio and generate

recommendations. Had these areas been merged, investigator-initiated applications to NIMH would be reviewed in newly

created standing panels at the CSR, whereas applications in response to NIMH-issued RFAs or special mechanisms would

undergo review by the Institute. 

The integration of peer review within the NIH structure has afforded NIMH the opportunity to restructure its

Institute-based review groups. Such restructuring is required in all areas, but the Workgroup limited its considerations to

the treatment and services applications, where both investigator-initiated and certain types of funding mechanisms will

continue to be reviewed by NIMH. 

Research that attempts to draw from both the treatment and the services research worlds may find such boundary

expansion difficult since the separate pools of reviewers rarely get to work together. Indeed, the Workgroup found that

review groups responsible for efficacy, effectiveness, practice, and service systems research applications typically have

had limited contact or crosstalk. As NIMH promotes the submission of boundary-spanning proposals for innovative

interventions research, it must ensure that reviews be conducted by experts in all relevant domains who have an

appreciation of the allied fields. 

After examining two rounds of applications, the Workgroup concluded that, in general, applications for efficacy,

effectiveness, and most of practice research should be reviewed in one committee. Service systems research and mental

health economics research would form a second standing committee. Despite this division, there should be areas of

overlap between the two committees in terms of expertise to handle applications at the cusp of this interface. As examples,

both groups will require expertise in cost-effectiveness and in quality of care since practice research is likely to fall into

both committees. Both groups will require expertise in substance abuse and in clinical epidemiology. Determination of

assignment should be based on the major question the proposed research seeks to answer. Both committees should span all

disorders and age groups. 

The question of merging NIMH treatment and services applications into the CSR remains open at present. NIMH staff are

participating in and learning from ongoing evaluations of the earlier mergers with CSR in neuroscience and AIDS. The

Workgroup believes that unintended benefits and problems may arise, and that it will be important to examine these before

determining what should stay in the NIMH review structure. NIMH should closely watch how such proposals fare within

the traditional review structure, making adjustments if necessary to ensure that field development efforts have a chance to

flower. 

The Workgroup anticipates that should NIAAA and NIDA place their treatment and services applications into centralized

CSR review, there could be a highly beneficial co-mingling of substance abuse and mental illnesses research. The

Workgroup emphasized the importance of exploring shared review across these institutes. Apart from the benefits of more

timely administrative decisions previously discussed, the Workgroup noted that considering shared review options would

offer a means of reducing demands on finite reserves of reviewer and staff expertise. Redundant review structures may

waste talent and time and hinder the review of applications looking at the complex interrelationships of co-occurring

illnesses. NIMH should explore preliminary areas of joint review as opportunities for small tests emerge. Also, lessons

from areas such as epidemiology and prevention, where such mergers will take place, may be a helpful predictor. 

Recommendation 40

Efficacy, effectiveness, practice, and service systems research applications should remain at the

Institute for review, and the clinical and services study sections reconstituted, including an

appropriate mix of expertise among reviewers. 

Recommendation 41

NIMH staff should attend carefully to planned evaluations of the CSR neurosciences and AIDS

review mergers, as well as the upcoming behavioral sciences merger, to enhance future

restructurings of NIMH review. 

Recommendation 42

NIMH should explore shared review opportunities with NIAAA and NIDA to enhance the

review of applications on comorbid substance abuse and mental illness applications.

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Issue: Ensuring Expertise in Peer Review

In the competition for research funding common to all medical science, what constitutes expert review? In rapidly

developing fields such as treatment and service systems research, generic challenges are exacerbated by special issues:

How to ensure the availability of appropriate and sufficient review expertise from within a relatively small pool of experts

and, given demands from all sectors of a field for the participation of those experts in multiple research activities, how to

apply and adhere to Federal conflict-of-interest policies and procedures. The Workgroup addressed these questions,

offering recommendations on issues under its purview and flagging others for the attention of the NIH leadership. 

Peer review traditionally has focused on scientific expertise in the substance of the topic under review. In order to ensure

appropriate and informed discussion, reasonable effort is made to have sufficient authorities in the subject matter of an

application as members of a peer review panel. A more recent enhancement of the process has been the inclusion, when

appropriate, of reviewers with expertise in the proposed methodology of a study. These reviewers may or may not have

expertise in a given content area; the purpose of their participation is to advise knowledgeably whether a project will be

able to address its stated aims using research and analytic techniques proposed by the researcher. 

Similarly, review committees for science seeking to affect care may require the expertise of individuals with a clear

understanding of the true public health need and the personal impact of mental illnesses. State mental health

commissioners, patients/consumers, and family members are particularly well qualified to assess the potential impact of

research within the care setting, to determine what would work in a care setting, and to help identify information most

useful for planning and policymaking. 

Recommendation 43

Review panels should have patients/consumers, providers, or policymakers as members to

ensure expertise on the utility of research findings in intervention research. 

Issue: Conflicts of Interest

A recurring issue in providing fair and expert review is the application of conflict-of-interest rules. With increasing

frequency, reviewers are surprised to learn that they (and often other experts in a particular area) must recuse themselves

from evaluating an application due to limited participation (e.g., consultation to a large multi-site study) on a related topic.

Such consultation means that the potential reviewer not only has a conflict with the particular multi-site project, but with

each of the investigators, consultants, and advisory group members on that project. As NIMH moves to promote large,

collaborative treatment trials, one can imagine instances where virtually all experts in a small, cutting-edge area of

investigation would be ineligible to serve as peer reviewers due to financial, personal, or scientific conflicts of interest.

Actually, there is already evidence that this problem is affecting established areas in mental health research as well. The

Workgroup generated no simple recommendation on this issue, but believes that the NIH leadership as a whole must

engage this issue that occurs throughout health research. 

Recommendation 44

NIMH staff should explore how other institutes deal with conflicts of interest and consult with

NIH on revising and implementing the conflict-of-interest rules for intervention research. 

Issue: Role of the Scientific Review Administrator (SRA)

The SRA is responsible for overseeing the review of a given application. This accomplished professional must ensure that

each application receives fair and expert review and that the review process adheres to all pertinent Federal regulations

and policies. Recent changes in review policy have shed light on a process that many applicants once viewed as shrouded

in mystery. For example, applicants now receive in their summary statements, compiled critiques rather than a single

integrated one; the former providing direct and comprehensive feedback, similar to that of a journal review. The strengths

and weaknesses of each application are measured against specific review criteria. If the application is judged to be in the

top half of those reviewed by a committee, the reviewers discuss their assessment of the application and this discussion,

the weighing of strengths and weaknesses, is summarized and provided to the applicant in a resume. 

Such steps to open the process notwithstanding, applicants who do not receive a competitive score often have lingering

concerns about the review process. In an effort to further demystify this process, the Workgroup considered it useful to list

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
the checks and balances that NIMH SRAs perform to ensure a fair and expert review. Workgroup members also suggested

that these points be presented at professional meetings to enhance all potential applicants' understanding of the review

process. 

SRAs discuss assignments with the chair and members of the review committee. 

SRAs discuss potential committee members with the chair, members of the review committee, and program staff. 

SRAs discuss potential conflicts of interest that may affect an applicant's review. All SRAs prefer to hear about

such conflicts before, rather than after, the review. 

SRAs advise all reviewers that extreme opinions are part of peer review, but must be discussed at the review

meeting. In the rare circumstance when a reviewer votes an extreme outlier score, without discussing it with the

other review members, the SRA follows uniform guidelines for considering the deletion of the reviewer's score in

calculating the final score. 

In addition to their own judgments regarding the need for augmenting the committee's expertise with an additional

reviewer or an outside opinion, SRAs ask the committee chair and members to advise them on this matter. 

An appeals process exists that provides a re-review of an application, if a flawed review takes place. This policy is

posted on the NIH web site.10

Reflecting upon the anticipated rise in applications at the interface of efficacy, effectiveness, practice, and service systems

research, the Workgroup wished to emphasize the importance of two additional options available to SRAs for enhancing

the review process. 

Recommendation 45

SRAs should strive to maintain a link between special emphasis panels and the regular review

committees through a subset of the regular reviewers and, whenever possible, the special

emphasis panels should be arranged to precede or follow immediately the regular review

meeting to facilitate the use of committee members and their scoring standards. 

Recommendation 46

SRAs and panel chairs should actively consider when scientific breakthroughs require a

substantive update for the review panel. The SRA and chair may convene a workshop to brief

the committee. 

ADMINISTRATION

Upon completion of review, NIMH program staff must decide which applications to fund. Principal sources of advice

include review committee determinations and the recommendation of the NAMHC. An important consideration in making

funding decisions is the balance between what exists in the Institute's portfolio in a given area and research that is needed

to address unanswered questions. For these reasons, program staff are encouraged to make decisions that are linked to, but

not limited by, exact review committee ratings. In some circumstances, NIMH may need to support a study on a topic not

rated as outstanding by the review committee but deemed useful in promoting field development. Likewise, on rare

occasions, NIMH may decline to fund an application that, while highly rated, is judged unlikely to generate significant

new information. Another factor requiring staff discretion is maintaining the vibrancy of the portfolio by investing not only

in "blue chip," lower risk research, but also having some "speculative," higher risk research ventures. Council advice is

invaluable when such judgment calls must be made. 

Recommendation 47

NIMH should make judicious use of its ability to make funding decisions out of percentile order

to achieve its programming objectives.

Once funding decisions are made, budgets need to be finalized and progress monitored. Investigators frequently face

budget reductions, regardless of the review committee's budgetary recommendations. While such reductions historically

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
have affected all facets of the NIMH portfolio, they pose a special challenge to treatment and services research. Unlike

research projects in which a few discrete studies can be dropped to meet cuts, treatment and services research offer only a

few non-determined cost categories: the length of the trial, the number or types of participants, or the variables assessed.

The Workgroup emphasized the need to ensure the integrity of all these elements or the generalizability of the findings

will be significantly diminished. This delicate balance was discussed more fully in Chapter 4. The Workgroup urges

NIMH staff and applicants to recognize that the importance of research lies in the utility of findings, and advises that it is

better to fund one such study adequately than to underfund two studies. 

Recommendation 48

Grant budgets must be based upon providing essential support as determined by review groups

and NIMH program staff for achieving research's scientific and public health aims.

After grants are awarded, staff should be vigilant in monitoring each project to determine if there are sudden opportunities

that arise and should be fostered through administrative or competitive supplements. Program staff should encourage

awareness in grantees of this option and notify investigators on the necessary procedures for requesting supplements. 

Recommendation 49

NIMH should increase its use of administrative and competitive supplements to provide

additional funds and time to researchers to test the generalizability of their findings or to

disseminate important research findings. 

| Home | Research | Top | 

REFERENCES

Attkisson, C., Cook, J., Karno, M., Lehman, A., McGlashan, T.H., Meltzer, H.Y., O'Connor, M., Richardson, D.,

Rosenblatt, A., Wells, K., Williams, J., and Hohmann, A.A. Clinical services research. Schizophrenia Bulletin,

18(4):561-626, 1992.

Cronin, K.A., Weed, D.L., Connor, R.J., and Prorok, P.C. Case-control studies of cancer screening: Theory and practice.

Journal of the National Cancer Institute, 90(7):498-504, 1998.

Depression Guideline Panel. Depression in Primary Care: Volume 1. Detection and Diagnosis. Clinical Practice

Guideline, Number 5. Rockville, MD. U.S. Department of Health and Human Services, Public Health Service, Agency for

Health Care Policy and Research. AHCPR Publication No. 93-0550, April 1993.

Frances, A.J., Kahn, D.A., Carpenter, D., Docherty, J.P., and Donovan, S.L. The Expert Consensus Guidelines for treating

depression in bipolar disorder. Journal of Clinical Psychiatry, 59(Suppl. 4):73-79, 1998.

Glisson, G. and Hemmelgarn, A. The effects of organizational climate and interorganizational coordination on the quality

and outcomes of children's service systems. Child Abuse & Neglect, 22(5):401-421, 1998.

Howard, P.A. and Duncan, P.W. Primary stroke prevention in nonvalvular atrial fibrillation: Implementing the clinical

trial findings. Annals of Pharmacotherapy, 31(10):1187-1196, 1997.

International Psychopharmacology Algorithm Project Report Participants. International Psychopharmacology Algorithm

Project Report. Psychopharmacology Bulletin, 31(3):457-507, 1995. 

Katon, W., Von Korff, M., Lin, E., Walker, E., Simon, G.E., Bush, T., Robinson, P., and Russo, J. Collaborative

management to achieve treatment guidelines. Impact on depression in primary care. Journal of the American Medical

Association, 273(13):1026-1031, 1995.

Lehman, A.F., Steinwachs, D.M., and the Survey Co-Investigators of the PORT Project. Patterns of Usual Care for

Schizophrenia: Initial Results from the Schizophrenia Patient Outcomes Research Team (PORT) Client Survey.

Schizophrenia Bulletin, 24(1):11-20, 1998.

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
McClellan, W.M., Knight, D.F., Karp, H., and Brown, W.W. Early detection and treatment of renal disease in hospitalized

diabetic and hypertensive patients: Important differences between practice and published guidelines. American Journal of

Kidney Diseases, 29(3):368-375, 1997.

National Advisory Mental Health Council. Caring for People with Severe Mental Disorders: A National Plan of

Research to Improve Services. Washington, DC: Superintendent of Documents, U.S. Government Printing Office, DHHS

Publication No. (ADM) 91-1762, 1991.

National Advisory Mental Health Council Workgroup on Mental Disorders Prevention Research. Priorities for Prevention

Research at NIMH. Bethesda, MD: National Institutes of Health/National Institute of Mental Health, NIH Publication No.

98-4321, 1998. 

National Association of State Mental Health Program Directors Research Institute, Inc. Closing and Reorganizing State

Psychiatric Hospitals: 1996. Report of the State Mental Health Agency Profiles System. Alexandria, VA: National

Association of State Mental Health Program Directors Research Institute, Inc., 1996. 

President's Commission on Mental Health. Report to the President From the U.S. President's Commission on Mental

Health. Vol. 1. Washington, DC: Superintendent of Documents, U.S. Government Printing Office, 1978.

Redick, R.W., Witkin, M.J., Atay, J.E., and Manderscheid, R.W. Highlights of organized mental health services in 1992

and major national and state trends. In: Manderscheid, R.W. and Sonnenschein, M.A., eds. Mental Health, United States

1996. Center for Mental Health Services, Substance Abuse and Mental Health Services Administration, DHHS

Publication No. (SMA)96-3098, 1996. pp. 90-137. 

Spitzer, R.L., Forman, J.B.W., and Nee, J. DSM-III field trials: I. Initial interrater diagnostic reliability. American Journal

of Psychiatry, 136(6):815-817, 1979.

Turner, J.C. and TenHoor, W.J. The NIMH Community Support Program: Pilot approach to a needed social reform.

Schizophrenia Bulletin, 4(3):319-348, 1978.

U.S. General Accounting Office. Returning the Mentally Disabled to the Community: Government Needs to Do More.

Washington, DC: Superintendent of Documents, U.S. Government Printing Office, 1977.

Vitiello, B. and Jensen, P. Medication development and testing in children and adolescents. Archives of General

Psychiatry, 54:871-876, 1997.

Williams, J.B., Gibbon, M., First, M.B., Spitzer, R.L., Davies, M., Borus, J., Howes, M.J., Kane, J., Pope, H.G., Jr.,

Rounsaville, B., and Wittchen, H-U. The Structured Clinical Interview for DSM-III-R (SCID).

II. Multisite test-retest reliability. Archives of General Psychiatry, 49:630-636, 1992. 

| Home | Research | Top | 

APPENDIX A:

National Advisory Mental Health Council

Chairperson

Steven E. Hyman, M.D.

Director

National Institute of Mental Health

Rockville, Maryland

Executive Secretary

Jane A. Steinberg, Ph.D.

Acting Director

Division of Extramural Activities

National Institute of Mental Health

Rockville, Maryland

Members

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Thomas J. Coates, Ph.D.

Professor 

Director, UCSF AIDS Research Institute and

 Center for AIDS Prevention Studies

University of California, San Francisco

San Francisco, California 

Kathy Cronkite 

Mental Health Advocate

Austin, Texas 

Mary Jane England, M.D. 

President

Washington Business Group on Health

Washington, DC 

Ellen Frank, Ph.D. 

Professor of Psychiatry and Psychology

Department of Psychiatry

School of Medicine

University of Pittsburgh

Pittsburgh, Pennsylvania 

Apostolos Georgopoulos, M.D., Ph.D. 

Professor, Department of Physiology, 

 Neurology and Psychiatry

University of Minnesota Medical School

Director, Brain Sciences Center

Veterans Administration Medical Center

Minneapolis, Minnesota 

Ann M. Graybiel, Ph.D. 

Walter A. Rosenblith Professor

Department of Brain and

 Cognitive Sciences

Massachusetts Institute of Technology

Cambridge, Massachusetts 

G. Richard Smith, Jr., M.D. 

Professor 

Director of Centers for

 Mental Healthcare Research

Department of Psychiatry

University of Arkansas for

 Medical Sciences

Little Rock, Arkansas

José Szapocznik, Ph.D. 

Professor and Director

Center for Family Studies

Department of Psychiatry and

 Behavioral Sciences

University of Miami 

 School of Medicine

Miami, Florida 

Michael F. Hogan, Ph.D. 

Director

Ohio Department of Mental Health

Columbus, Ohio 

Dale L. Johnson, Ph.D. 

Professor

Department of Psychology 

University of Houston

Houston, Texas 

Robert L. Johnson, M.D. 

Director of Adolescent Medicine

Department of Pediatrics

University of Medicine and Dentistry 

 of New Jersey

Newark, New Jersey 

Anne C. Petersen, Ph.D. 

Senior Vice President for Programs

W. K. Kellogg Foundation

Battle Creek, Michigan 

A. John Rush, M.D. 

Professor, Betty Jo Hay Distinguished Chair 

Department of Psychiatry

Southwestern Medical Center

University of Texas

Dallas, Texas 

Richard H. Scheller, Ph.D. 

Investigator,

 Howard Hughes Medical Institute

Professor, Department of Molecular and 

 Cellular Physiology

Stanford University School of Medicine

Stanford, California 

Joseph S. Takahashi, Ph.D.

Investigator,

 Howard Hughes Medical Institute

Walter and Mary E. Glass Professor

Department of Neurobiology and

 Physiology

Northwestern University

Evanston, Illinois 

James G. Townsel, Ph.D. 

Professor and Director 

Center for Molecular and Behavioral

 Neuroscience

School of Medicine

Meharry Medical College

Nashville, Tennessee 

Myrna M. Weissman, Ph.D. 

Professor

Department of Clinical and Genetic

 Epidemiology

New York State Psychiatric Institute 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Columbia University

New York, New York 

Ex Officio Members

Office of the Secretary, DHHS

Donna E. Shalala, Ph.D.

Secretary

Department of Health and

 Human Services

Washington, DC 

National Institutes of Health

Harold E. Varmus, M.D.

Director

National Institutes of Health

Bethesda, Maryland 

Liaison Representative

Center for Mental Health Services

Thomas H. Bornemann, Ed.D.

Deputy Director

Center for Mental Health Services

Substance Abuse and Mental 

 Health Services Administration

Rockville, Maryland 

Department of Defense

Robert A. Mays, Jr., Ph.D.

Colonel, U.S. Army

Office of the Inspector General

North Atlantic Regional 

 Medical Command

Walter Reed Army Medical Center

Washington, DC 

Department of Veterans Affairs

Thomas B. Horvath, M.D., F.R.A.C.P.

Chief Consultant for Mental Health

Department of Veterans Affairs

Veterans Health Administration

Washington, DC 

Special Consultants

Constance E. Lieber

President

National Alliance for Research on

 Schizophrenia and Depression

Great Neck, New York

Edward Scolnick, M.D.

President

Merck Research Laboratories

West Point, Pennsylvania 

| Home | Research | Top | 

APPENDIX B

NIMH Clinical Treatment and Services Research Workgroup

Members

A. John Rush, M.D.** (Chairman)

Professor, Betty Jo Hay Distinguished Chair

Department of Psychiatry

Southwestern Medical Center

University of Texas

Michael F. Hogan, Ph.D.**

Director

Ohio Department of Mental Health

Columbus, Ohio 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Dallas, Texas 

Howard Abikoff, Ph.D.

Professor of Clinical Psychiatry

Director of Research

New York University

 Child Study Center

New York University

 School of Medicine

New York, New York 

Kathy Cronkite**

Mental Health Advocate

Austin, Texas

Mary Jane England, M.D.**

President

Washington Business Group on Health

Washington, DC 

Susan Essock, Ph.D.

Director

Division of Health Services Research and

Professor 

Department of Psychiatry

Mount Sinai School of Medicine

New York, New York 

Ellen Frank, Ph.D.**

Professor of Psychiatry and Psychology

Department of Psychiatry

School of Medicine

University of Pittsburgh

Pittsburgh, Pennsylvania 

Junius Gonzales, M.D.

Deputy Chairman

Department of Psychiatry

Georgetown University Medical Center

Washington, DC 

Dale L. Johnson, Ph.D.**

Professor

Department of Psychology

University of Houston

Houston, Texas 

Robert L. Johnson, M.D.**

Director of Adolescent Medicine

Department of Pediatrics

University of Medicine and Dentistry

 of New Jersey

Newark, New Jersey 

John Kane, M.D.

Chairman 

Department of Psychiatry

Long Island Jewish Medical Center

Glen Oaks, New York 

Lon S. Schneider, M.D.

Professor

Department of Psychiatry

University of Southern California

 School of Medicine

Los Angeles, California 

G. Richard Smith, Jr., M.D.**

Professor

Director of Centers for

 Mental Healthcare Research 

Department of Psychiatry

University of Arkansas

 for Medical Sciences

Little Rock, Arkansas 

Kenneth B. Wells, M.D., M.P.H.

Director

Research Center in Managed Care

 for Psychiatric Disorders

Professor

Department of Psychiatry

UCLA Neuropsychiatric Institute

Los Angeles, California 

NIMH Staff to the Workgroup

Staff Director

Jane A. Steinberg, Ph.D.

Division of Extramural Activities 

Associate Staff Director

Susan Matthews

Division of Extramural Activities 

Science Writer

Paul J. Sirovatka, M.S.

Office of Science Policy

 and Program Planning 

Art and Graphics

Advisors

Barry D. Lebowitz, Ph.D. 

Division of Services and Intervention

 Research 

Kathryn M. Magruder, Ph.D., M.P.H.

Division of Services and Intervention

 Research 

Jean G. Noronha, Ph.D.

Division of Extramural Activities 

Grayson S. Norquist, M.D., M.S.P.H.

Division of Services and Intervention

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Catherine W. West

Office of Resource Management 

Editor

Joan A. Cole

Division of Extramural Activities 

Typist

Debra S. Dabney

Division of Extramural Activities 

 Research 

Benedetto Vitiello, M.D. 

Division of Services and Intervention

 Research 

| Home | Research | Top | 

APPENDIX C

Roster of Consultants*

Consultants

Robert Battjes, D.S.W.

Division of Clinical and

 Services Research

National Institute on Drug Abuse 

Soo Borson, M.D.

University of Washington

 School of Medicine 

James Burris, M.D.

Department of Veterans Affairs 

Gerald E. Calderone, Ph.D.

Division of Extramural Activities, NIMH 

Sarah Carr

Office of Policy Analysis

National Institute of Allergy

 and Infectious Diseases 

Isabel S. Davidoff

Division of Services and Intervention

 Research, NIMH 

Robert E. Drake, M.D., Ph.D.

New Hampshire-Dartmouth Psychiatric

 Research Center 

Michael J. English, J.D.

Center for Mental Health Services

Substance Abuse and Mental Health

 Services Administration 

Bennett Fletcher, Ph.D.

Division of Clinical and

 Services Research

National Institute on Drug Abuse 

Barbara Geller, M.D.

Henry J. Haigler, Ph.D.

Division of Extramural Activities, NIMH 

Ann Hohmann, Ph.D., M.P.H.

Division of Services and Intervention

 Research, NIMH 

Steven E. Hyman, M.D.

Director, NIMH 

Douglas B. Kamerow, M.D., M.P.H.

Center for Practice and Technology

 Assessment

Agency for Health Care Policy

 and Research 

Philip Lavori, Ph.D.

Palo Alto Veterans Affairs Health Care

 System 

Anthony F. Lehman, M.D.

University of Maryland

 School of Medicine 

Bruce Levin, Ph.D.

Columbia School of Public Health 

Teresa Levitin, Ph.D.

Office of Extramural Program Review

National Institute on Drug Abuse 

Lydia Lewis

National Depressive and Manic

 Depressive Association 

Clarissa Marques, Ph.D.

Green Spring Health Services 

Stanley Rosenberg, Ph.D.

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
Washington University

 School of Medicine 

Paul Gorman, Ed.D.

Division of Behavioral Health

Department of Health and

 Human Services

Concord, New Hampshire 

Robert A. Mays, Jr., Ph.D.***

Colonel, U.S. Army

Department of Defense 

Carolyn Mercer, Ph.D.

Dartmouth Medical School 

Richard K. Nakamura, Ph.D.

Office of the Director, NIMH 

Harold Perl, Ph.D.

Division of Clinical and

 Prevention Research

National Institute on Alcohol Abuse

 and Alcoholism 

Anne C. Petersen, Ph.D.**

W.K. Kellogg Foundation 

Dartmouth Medical School 

Agnes Rupp, Ph.D.

Division of Services and Intervention

 Research, NIMH 

Robert H. Stretch, Ph.D.

Division of Extramural Activities, NIMH 

Carolyn Strete, Ph.D.

Division of Mood Disorders,

 Behavioral Research and AIDS, NIMH 

Ronald H. Weich, J.D.

Zuckerman, Spaeder, Goldstein,

 Taylor and Kolker, L.L.P 

Washington Business Group on Health

Research Advisory Group Meeting

April 23, 1998

Angela G. Burgess

Washington Business Group on Health 

Wayne Burton, M.D.

First Chicago NBD 

Isabel S. Davidoff

National Institute of Mental Health 

Bruce Davidson, M.P.H.

Digital Equipment Corporation 

Lynne DeGrande, M.S.P.H.

General Motors Employee Assistance

 Program (Senior Consultant)

DeGrande and Associates 

Mary Jane England, M.D.**

Washington Business Group on Health 

Susan M. Essock, Ph.D.

Mount Sinai School of Medicine 

Veronica Goff

Washington Business Group on Health 

William Goldman, M.D.

United Behavioral Health 

James Harburger, M.D.

Harvard Pilgrim Health Care 

Paul Johnson, M.D.

US West 

Kathryn M. Magruder, Ph.D., M.P.H.

National Institute of Mental Health 

Grayson S. Norquist, M.D., M.S.P.H.

National Institute of Mental Health 

Jane A. Steinberg, Ph.D.

National Institute of Mental Health 

David Whitehouse, M.D.

MCC Behavioral Health Care 

Center for Mental Health Services Staff/National Advisory Council

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
May 5, 1998 Meeting

Bernard S. Arons, M.D.

Center for Mental Health Services 

Thomas Bornemann, Ed.D.

Center for Mental Health Services

Frank G. Burgmann

G. Pierce Wood Memorial Hospital 

Michael J. English, J.D.

Center for Mental Health Services 

Marshall Forstein, M.D.

Fenway Community Health Center 

Ruby J. Martinez, R.N., Ph.D.

University of Colorado Health

 Sciences Center 

Anne Mathews-Younes, Ed.D.

Center for Mental Health Services 

Donna Mayeaux

Louisiana Alliance for the Mentally Ill 

Andres J. Pumariega, M.D.

East Tennessee State University 

Ian A. Shaffer, M.D.

Value Behavioral Health 

Steven P. Shon, M.D.

Texas Department of Mental Health

 and Mental Retardation 

David Shore, M.D. (Liaison)

National Institute of Mental Health 

David K. Yamakawa, Jr., J.D.

Attorney-at-Law 

| Home | Research | Top | 

1 Interventions also refer to preventive strategies, which were addressed in a previous NAMHC Workgroup report entitled

Priorities for Prevention Research at NIMH: A Report by the National Advisory Mental Health Council Workgroup on

Mental Disorders Prevention Research. For purposes of this report, intervention refers to either treatment or services

delivery. 

2 Individuals with mental illnesses sometimes are referred to as patients or consumers. Each term has its own merits, and

to capture these strengths patients/consumers also will be used in this report. 

3 Attkisson, C. et al. Clinical services research. Schizophrenia Bulletin, 18(4):561-626, 1992. 

4 A grant is a financial assistance mechanism whereby money and/or direct assistance is provided to carry out approved

activities, and is to be used whenever the PHS awarding office anticipates no substantial programmatic involvement with

the recipient during performance of the financially assisted activities. A research and development contract project is a

circumscribed activity identified by the government agency that may involve a single contract or two or more similar,

related, and interdependent contracts encompassing the project concept and approach. 

5 Web site: http://www.nih.gov/grants/guide/index.html

6 An RFA is a formal statement that invites grant or cooperative agreement applications in a well-defined scientific area to

accomplish specific program objectives, and it indicates the estimated amount of funds set aside for the competition, the

estimated number of awards to be made, and the application receipt date(s). 

A PA is a formal statement about a new or re-emphasized, ongoing NIH extramural activity, and, except for a few

programs, applications in response to PAs may be submitted for any appropriate receipt date and are reviewed with all

other applications received at that time. 

7 Mailing address: Superintendent of Documents, Government Printing Office, Washington, DC 20402; web site:

http://www.access.gpo.gov/su_docs/dpos/topics/grants.html

8 NIMH web site: http://www.nimh.nih.gov

9 Web site: http:cbdnet.access.gpo.gov

10 Web site: http://www.nih.gov/grants/guide/1997/97.11.21/index.html

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material
* The workgroup appreciates the contribution of the consultants in developing this report; however, inclusion in the listing

does not necessarily indicate endorsement of the workgroup's recommendations. 

** Member of the National Advisory Mental Health Council 

*** Ex Officio Member of the National Advisory Mental Health Council 

Updated: January 14, 1998

| Home | Research | Top | 

NIMH Archive Material This document is no longer being updated.

For the latest information, please go to http://www.nimh.nih.gov

NIMH Archive Material