Report From The UNOCCAP Oversight Board to the 

National Advisory Mental Health Council 

Charting the Mental Health Status and 

Service Needs of Children: 

Recommendations from the UNOCCAP Oversight Board 

TABLE OF CONTENTS 

EXECUTIVE SUMMARY 

BACKGROUND 

IDENTIFYING AND ADDRESSING CRITICAL QUESTIONS 

DEVELOPING AN ACTION PLAN TO ADDRESS THESE QUESTIONS 

z Recommendation 1: Initiate an ongoing national survey to monitor children's symptoms, 

impairment, and service use 

z Recommendation 2: Initiate a national survey of the type of services used by children in 

treatment, the costs of these services, and how benefit coverage affects service use and costs 

z Recommendation 3: Develop activities to support the two large-scale surveys in 

Recommendations 1 and 2 

z Recommendation 4: Establish an initiative to enrich the conceptualization and measurement of 

childhood mental problems and impairment from infancy to 18 years of age 

z Recommendation 5: Establish an initiative to further the understanding of emotional functioning 

and mental health problems in young children 

z Recommendation 6: Establish a program of local or regional longitudinal studies in three 

separate areas: 1) paths into and out of disorder; 2) paths into and out of service use; and 3) 

effectiveness and quality of mental health services 

z Recommendation 7: Establish a program to foster innovative methods or analyses in child 

mental health 

THE JOURNEY: MOVING TOWARD A SHARED UNDERSTANDING OF THE ISSUES 

OVERVIEW 

LESSONS LEARNED 

PROCESS FOR IDENTIFYING AND PRIORITIZING RESEARCH QUESTIONS 

WHAT DOES READY MEAN? 

NATIONAL PREVALENCE RATES OF CHILDHOOD MENTAL HEALTH PROBLEMS: WHAT TO 

MEASURE? 

z Measuring Mental Health Status 

z Counting and Monitoring 

SERVICE USE 

SERVICE USE AND COSTS FOR CHILDREN IN TREATMENT 

PATHWAYS 

z Into and Out of Disorder 

z Into and Out of Services 

This report is a modified version of the Report of the UNOCCAP Oversight Board to the National 

Advisory Mental Health Council, presented on February 4, 1998

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
z Service Need, Effectiveness, and Quality 

z Need for Longitudinal, Local Designs 

STATE OF THE ART IN LARGE, POPULATION BASED, NATIONAL SURVEYS 

SUMMARY 

GETTING ANSWERS TO PRIORITY QUESTIONS 

OVERVIEW 

NATIONAL MONITORING SURVEY IN PREVALENCE AND SERVICE USE 

NATIONAL SURVEY ON USE AND COSTS AMONG CHILDREN IN TREATMENT 

GENERATIVITY AND AVAILABILITY OF THE DATA 

ANSWERING QUESTIONS ON DIAGNOSIS, PATHWAYS, AND OUTCOMES 

CEPTUALIZATION AND MEASUREMENT 

YOUNG CHILD 

LONGITUDINAL RESEARCH INITIATIVES 

INNOVATIONS IN METHODS AND ANALYSES 

INTEGRATIVE AND GENERATIVE NATURE OF THE PLAN 

MOVING FORWARD: NIMH'S ROLE 

PLANNING 

z Selecting a Feasible Funding Mechanism 

z Establishing a Time Line> 

z Budgeting for Large-scale Initiatives 

z Identifying Co-sponsors for Large-scale Projects 

GOVERNANCE AND STAFFING NEEDS 

ACCOUNTABILITY 

z Peer Review 

z Advisory Board 

CONCLUDING COMMENTS 

APPENDICIES 

EXECUTIVE SUMMARY 

Background 

The Oversight Board for the multi-site study of Mental Health Service Use, Needs, Outcomes, and Costs in 

Child and Adolescent Populations (UNOCCAP) had a dual charge (A roster for the Board is presented in 

Appendix A). The first was to provide scientific guidance and administrative advice to the UNOCCAP 

investigators as they finished pilot and feasibility work (See Appendix B). This task is now completed, and the 

necessary work is underway and important products are anticipated. As the UNOCCAP pilot work draws to a 

close, the National Institute of Mental Health (NIMH) needs to reassess its action plan for charting the mental 

health status and service needs of children. 

Drafting an action plan was the Board's second charge and the impetus for this report to NIMH and the 

National Advisory Mental Health Council (NAMHC). Specifically, Dr. Hyman, Director of NIMH and 

Chairperson of the NAMHC, asked the Board to assess the research needs regarding the prevalence and 

etiology of children's mental health problems and the use, need, quality, and costs of service associated with 

these problems. 

It is the Board's hope that this report will spark a spirited discussion within the research and advocacy 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
communities. These discussions should identify complementary, practical, and productive approaches to 

addressing important and urgent research questions concerning the responsiveness of this Nation's mental 

health service system to the needs of children. 

Consensus must be developed quickly since the science is ready to address many of these critical research 

questions. A sustained program of research, such as that proposed in this report, carried out in conjunction 

with the Institute's research portfolio in the treatment and prevention of childhood mental disorders, remains a 

timely investment that promises great benefit for children, for their families, and for the future of the Nation. 

Identifying and Addressing Critical Questions 

Efficiently advancing this research agenda will require a fresh start based on identifying important questions 

that the field can address rigorously. The Board chose to focus on significant questions facing policymakers 

as the extensive changes across this Nation's mental health, education, and justice systems unfold. 

Interestingly, these policy-driven questions are completely consistent with scientifically important questions: 

1. What are the frequencies of symptoms and impairment in U.S. children and adolescents? 

{ How do these frequencies vary over time and with changing policies? 

{ How do these frequencies vary across demographic factors such as age, socioeconomic status, 

location, gender, and ethnicity? 

{ Which symptoms cluster to form patterns or diagnostic groups? 

2. What is the service use associated with various symptoms and impairment? 

{ How does service use change over time? 

{ How does service use differ across demographic groups? 

3. A. What are the costs of mental health services to youth in treatment? 

{ How do costs vary by demographic groups? 

{ How do costs vary by mental health problem? 

B. How does the structure of insurance benefits influence the use of services and the costs of care? 

{ What are the effects of managed care on service use and costs? 

{ What benefit packages are associated with variation in service use and cost? 

{ How do parity laws affect the cost of treating children? 

4. What are the paths into and out of disorder and into and out of service use? 

{ How do these paths vary across different demographic groups? 

{ Are the paths different for different disorders? 

5. Which services are effective for which children and adolescents? 

{ How does effectiveness vary over demographic groups? 

{ Who are the children who need, but do not receive, effective services? 

{ What are the costs of effective services? 

{ How does quality vary across settings and provider characteristics? 

Developing an Action Plan to Address These Questions 

The path for answering these questions will be difficult. First, no one study can address each of these 

questions well simultaneously. There is no way to combine these questions into one method or design without 

sacrificing the quality of data. Second, the many disciplines that are allied in children's mental health research 

are not yet ready to answer each of these questions. Some areas require additional conceptualization, better 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
instrumentation, or more empirical work to generate or test hypotheses. This critical basic research must be 

conducted to lay the groundwork for answering all of these questions. 

The Board's recommendations, therefore, were shaped by both the urgency with which these questions must 

be addressed and by the realities of today's scientific abilities. The guideline was a simple maxim: if it's ready, 

use it; if it's not ready, develop it. This guideline resulted in a program of research, each component of which 

is ready for initiation today. The program, with new, complementary initiatives spanning large-scale surveys 

and fine-grain developmental assessments of psychopathology, establishes: 

z A mechanism for answering selected priority questions with the best available tools 

z The vehicle for constructing the needed tools and methods for answering all of the remaining priority 

questions 

z A rich data source to speed the process of scientific discovery and policy-making in child and 

adolescent mental health 

The Board recommendations call for a program of research having multiple components. Some of the 

components are designed to stimulate new concepts and measures crucial to elaborating our understanding 

of children's mental health. Some of the components are designed to provide a monitoring of prevalence of 

mental health impairments and related service use, as a national indicator of the seriousness of the 

phenomenon. Some of the components are designed to inform scientific inquiries concerning pathways into 

mental health impairments. Some of the components are designed to measure the costs of services to those 

using them. Thus, the recommendations attempt to enhance the science side of children's mental health, 

supporting research addressing causal hypothesis, at the same time as they serve basic informational needs 

of the society about the magnitude of the problems faced within the child mental health domain. 

NIMH Role 

NIMH must provide qualified staff, access to expert and timely peer and administrative review, adequate 

budgets, and competition through appropriate funding mechanisms. The staff must ensure that these 

activities are supported in economical, yet feasible ways. In particular, the Board encourages NIMH to begin 

looking for a cosponsor for these activities, either Federal or private. There could be significant cost savings 

on large-scale efforts by imbedding the NIMH questions within on-going surveys. Further, a suitable partner 

could provide NIMH with much needed expertise in survey methodology. 

Recommendations to Establish a Program of Research 

z Recommendation 1: Initiate an ongoing national survey to monitor children's symptoms, 

impairment, and service use. 

The Board concluded that questions on prevalence and service use are best answered, and are ready to be 

answered, at a national level through a large-scale survey. A rigorous study is required to describe symptoms, 

symptom patterns, impairments, and service use in a sample large enough to reveal important sources of 

demographic variation in the United States. It is of critical importance to launch this study soon, since valuable 

time has already been lost for informing the policies and strategies that are changing service provision and 

financing. The Board also places great emphasis on NIMH creating the necessary infrastructure to initiate this 

activity immediately. 

The Board recommends a national survey, employing cross-sectional samples to monitor the frequencies of 

symptoms and impairments, as well as service use. Each assessment wave would have a new sample of 

children administered a minimum core set of measures (i.e., symptom, impairment, service use, and 

demographic variables) and a developmental module of test questions, estimated to take 20-30 minutes of 

the respondent's time. The first year, and perhaps every five years or so, a benchmark assessment would be 

conducted that would contain the core battery and an augmented set of assessment measures (estimated to 

be 90 minutes for the full symptom and impairment checklists, service use measure, and demographic 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
variables). The benchmark assessments would provide more detailed information and allow for the 

recalibration of the core measures with better questions from the developmental modules. In addition, topical 

modules could be included, to assess critical scientific or policy issues. 

This ongoing monitoring design would provide a cost-efficient platform for: 1) monitoring symptoms, 

impairment, and service use nationally and revealing variation over time and demographic groups; 2) 

generating unanticipated inductive hypotheses about symptom patterns and diagnostic categories; 3) 

providing a time series of indicators to help determine the effects of policy changes; and 4) testing and 

refining new measures as they mature (e.g., symptom patterns or diagnostic categories). 

z Recommendation 2: Initiate a national survey of the type of services used by children in 

treatment, the costs of these services, and how benefit coverage affects service use and costs. 

The proposed monitoring survey in Recommendation 1 will not capture a significant number of children who 

use mental health services and therefore will not be able to examine the factors associated with more costly 

services. Therefore, the Board recommends conducting a national survey of children who use mental health 

services to determine the overall use of these services, the costs of these services, and the factors 

associated with the use and costs of services. This separate survey should be initiated immediately. 

z Recommendation 3: Develop infrastructure activities to support the two large-scale surveys in 

Recommendations 1 and 2.

{ Once collected, the national survey data should be released to the field, after ensuring 

the participants' confidentiality. 

A national survey's value is heightened by the rapid availability of public use data files. The 

individuals who collect the data must make the data available to the field as soon as it can be 

checked for accuracy and documented. If the grant or co-operative agreement process is not 

consistent with this requirement, then contracts should be considered for this effort. Available 

means that the data set should be fully documented and maintained, as well as having possible 

workshops on its use. 

{ A program announcement should be issued for secondary analysis of the survey data.

The value of the large-scale surveys will be enhanced by supporting the reanalysis of the data 

by many investigators. NIMH should provide resources through a program announcement to 

encourage and support secondary analyses. 

z Recommendation 4: Establish an initiative to enrich the conceptualization and measurement of 

childhood mental problems and impairment from infancy to 18 years of age. 

Conceptualizing and measuring children's mental health problems require a sustained effort by NIMH. 

Although the Institute has funded work in this area, it is now time to foster multiple approaches to this issue 

through consensus activities, theoretical work, development and testing of measures, and comparisons 

between measures and theoretical models. 

z Recommendation 5: Establish an initiative to further the understanding of emotional 

functioning and mental health problems in young children. 

As difficult as the area of child and adolescent psychopathology is, of particular concern within this area is the 

study of the young child, where efforts are just beginning. Measures of normative functioning are being 

developed and tested, but what they mean for children's mental health and later problems is unknown. NIMH 

is encouraged to develop an RFA for stimulating research in this key area, which is ready to move ahead 

now. 

z Recommendation 6: Establish a program of local or regional longitudinal studies in three 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
separate areas: 1) paths into and out of disorder; 2) paths into and out of service use; and 3) 

effectiveness and quality of mental health services. 

Longitudinal research has been consistently supported through the Institute, but the portfolio should be 

reviewed for what is known and moved toward synthesizing and testing these findings regarding risk factors 

for disorders and for service use. The Institute must consider and direct its efforts toward understanding these 

developmental processes. 

Understanding how the provision of service improves outcome is an area of research that also requires 

longitudinal study. Children in services must be tracked to see if they do get better or if further impairment is 

prevented. This work must begin immediately because it is a prerequisite for understanding and indexing the 

service need and service quality (i.e., how effective services are provided in a particular service sector or in a 

sector site). 

Because of the need for deeply textured studies and for tracking participants over time, these studies are 

proposed for local or regional work rather than large-scale survey approaches. 

z Recommendation 7: Establish a program to foster innovative methods or analyses in child 

mental health. 

Questions of continuity and change remain intriguing for all of the life sciences and for policy development. 

Innovative methods for conceptualizing, measuring, and analyzing these processes should be fostered at 

NIMH. Current efforts should be augmented with a new and immediate emphasis on supporting the 

infrastructure for methodological, statistical, and software development. 

THE JOURNEY: MOVING TOWARD A SHARED UNDERSTANDING OF 

THE ISSUES 

Overview 

This chapter reviews the Board's process of identifying the critical issues in children's mental health 

assessment and services research. The Board's discussions were informed by the UNOCCAP investigators, 

the NIMH and Administration on Children Youth and Families staff, UNOCCAP workshops, and the NAMHC 

reports. Additional consultation was received from Drs. Ronald Kessler and Richard Frank on the purpose 

and value of national data (see Appendix C & D for workshop titles and participants). The Board also 

reviewed additional background material on the intent of the UNOCCAP program, the National Plan for 

Research on Child and Adolescent Mental Disorders, and the empirical literature. 

Lessons Learned 

The Board had a rich context for identifying the lessons learned from UNOCCAP. The reader should note that 

the lessons are being reported with the benefit of hindsight. Even though the Board is serving as an "armchair quarterback" in this capacity, the lessons should be shared. They were learned through the pursuit of an 

important goal: NIMH was working to conduct large-scale science. Projects of this scope are relatively rare for 

NIMH. NIMH must build its capacity to develop and conduct such large-scale projects to generate the type of 

data that can inform clinical practice and service systems. 

The overarching lessons for NIMH are: 

z NIMH leadership must recognize its relative inexperience in large-scale ventures and seek consultation 

on future efforts from other NIH Institutes and Federal agencies that routinely conduct such science. 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
z NIMH staff must have the necessary experience with and expertise in large-scale survey methodology 

and program management. This knowledge must be available among the staff participating in scientific 

or in administrative roles. Among this group of experts, there should be one NIMH leader. 

z The team of grantees or contractors must also have one leader to sustain the scientific vision for the 

project and make administrative decisions. 

z Multiple vehicles to ensure accountability must be established for both the NIMH and the extramural 

leader, such as the NAMHC, a project advisory board, and peer review. 

z One study cannot address all of the questions in children's diagnostic and service issues. Even 

national surveys are bounded by feasibility issues and by limits on statistical inference. 

z Each research question must be answered through appropriate methods that reflect the best that 

science has to offer. 

z Some areas do not have the necessary tools to address rigorously the questions posed, and resources 

must be devoted to developing and refining these tools. 

z Mounting a national, one-time survey, where data sharing is postponed for years, places undue 

pressure on the participating investigators, the field, and the Institute. 

The last point requires clarification. If NIMH plans one national survey as the only such large-scale initiative 

that the field will see for decades, then this puts a huge burden on the eventual design to be all things to all 

people. There is a great temptation to address too many research questions. 

More important, science is not ready for such an approach. Quite simply, the necessary conceptualization, 

tools, and designs are not uniformly available in all substantive areas. If resources were diverted into 

prematurely conducting such an immense and elaborate effort, the Board is concerned that it would generate 

data that policymakers should not use and that a substantial portion of the researchers would not accept as 

credible. 

Process for Identifying and Prioritizing Research Questions 

Navigating through the substantive areas to specify recommendations was far more difficult than recognizing 

the lessons of UNOCCAP. Each domain in child and adolescent mental health needs large, well-documented 

data sets. But as said before, each area is not equally ready to embark on such a costly mission. Some areas 

are better served by devoting resources to conceptual development, creating or refining measurements, or 

collecting richly textured data. 

The guideposts of utility, readiness, and building the science shaped all of the Board's considerations. The 

remainder of this chapter is devoted to exploring these issues in each of the target areas specified by Dr. 

Hyman in his charge to the Board. The following portion provides a description of the steps the Board 

followed in generating and prioritizing the policy questions that can be answered in a scientifically meaningful 

way. 

In each area, the reader will see how the Board followed certain steps, similar to a task analysis, to generate 

their conclusions. The Board offers these steps as a helpful template to assess the feasibility and readiness of 

the future large-scale studies at NIMH. The Board has also outlined the products that should be expected to 

serve as briefing documents for the Council, NIMH leadership, and as the historical record. 

STEP 

1:

Identify Policy and Science Questions 

Product: A Short List of Critical Policy and Science Questions Written In Clear Language 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
What Does Ready Mean? 

The steps outlined above ask for repeated determinations about readiness, yet defining this term is not 

simple. Ready does not mean perfect or even finished. Ready means serviceable. For a large-scale survey, 

the value rests with the application of standardized measurement to vast numbers of persons. Inferential 

power to the U.S. population and important subpopulations is the chief asset of surveys. With this in mind, 

and acknowledging the large costs of a national survey, most fields choose to use instruments that have 

passed the test of time. This does not mean that the measure is flawless, but that its psychometric properties 

are known. 

Cutting-edge measurement is not often tried in large-scale efforts because this approach risks large sums of 

funding and may potentially misguide a field for years. Typically, smaller-scale studies are used to test 

cutting-edge methods, leading to the understanding of the measure's psychometric properties. Then, 

researchers can decide the value of the instrument for inclusion in a later large-scale or national survey. But 

how does one balance the excitement of the newest measures or technologies with the need to commit to a 

level of technology for initiation of a project? 

The space program provides a clarifying example. When Neil Armstrong walked on the moon in l969, it was 

not the science of 1969 that put him there. A decision was made years earlier that the science was good 

enough to send an astronaut to the moon and allow a safe return. With this decision made, the technology 

was "frozen" so that the ship and its many systems could be built. Even though science marched on and new 

technologies were incorporated into subsequent moon walks, a commitment to a satisfactory level of 

technology for a particular project was made so that planning, testing, refining, and retesting could occur. 

In summary, this technological "freeze" is necessary for initiating a large-scale project. Work for better 

methods continues along its own course and will be utilized in a timely manner, but the large-scale project is 

not delayed for the purpose of including these ever improving measures. 

The remainder of the chapter contains a section on each of the substantive areas Dr. Hyman asked the Board 

to consider and a brief review of survey methodologies. 

National Prevalence Rates of Childhood Mental Health Problems: What to Measure? 

The Board repeatedly debated the need for national rates of children's mental health problems. The outcome 

of the debate hinged upon two interrelated questions: what measures of mental health status were ready to 

be used, and then what would the value of these data be at a national level? 

STEP 

2:

Identify Extant Data to Address the Questions 

Product: Synthesis of Extant Literature 

STEP 

3:

Identify New Data Required to Address the Questions

Product: List of Research Needs 

STEP 

4:

Determine Field's Readiness to Collect these Data

Product: List of Policy and Scientific Research to Undertake Now With Feasible Designs Outlined 

Product: List of Developmental Research to Undertake Now To Build the Necessary Conceptual 

Frameworks and Measurement Approaches 

STEP 

5:

Determine the Value to Science and Policymakers for the Questions that Can Be Successfully 

Addressed with the Current Science

Product: Consensus Document on the Value and Limitations of the Data To Be Collected in a 

Specific Design 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
z Measuring Mental Health Status 

Some definitions would be helpful first. Mental health problems can be measured in many ways. As previously 

discussed, when a child's mental heath or behavioral problem can be classified as a disorder or diagnosis 

remains unsettled. 

Here are the key concepts in measuring mental health problems: 

Symptom. A symptom is the physical or mental expression of an illness. In mental disorders, symptoms can 

be abnormalities of emotions, thinking, behavior, or relationships. In a few instances, a single symptom is 

sufficient to define an illness or a diagnosis (e.g., elective mutism), but most illnesses are recognized by the 

expression of a reproducible cluster of symptoms. 

Symptom patterns. Symptom patterns refer to clusters of symptoms that are highly correlated with each other. 

Symptom patterns should be reproducible over time within the same person and between persons believed to 

have the same illness in terms of frequency, intensity, and duration. 

Impairment. An impairment is the reduction or deterioration in the capacity to perform a valued role. 

Impairments can be judged using three criteria: educational, social or vocational restriction; interference with 

development; and/or negative effect on others. Impairments occur in work or social functioning associated 

with being a family member, an employee, a student, or a companion. In short, impairment is a measure of 

the extent to which symptoms interfere with full functioning. 

Diagnosis. A specific diagnosis should consist of a symptom pattern and level of impairment that warrants 

therapeutic concern due to the frequency, intensity, or duration of the symptoms or impairments. Attempts 

have been made in community surveys to classify children into different categories of disorder, and some 

diagnostic categories appear to be of utility for both the researcher and the clinician. 

The challenges to measuring the prevalence of childhood problems within an acceptable range of precision 

and accuracy are considerable. Instruments conceived from a categorical perspective may be most 

appropriate for problems that lend themselves to such nosological distinctions, while instruments such as 

symptom checklists may be most appropriate when the task is to discriminate deviancy from a population 

norm. In considering this diagnosis-versus-symptoms quandary, several other realities such as the use of 

multiple informants and the need for documented reliability and validity must be recognized. 

The Board determined that, as discussed above, too many of the childhood diagnoses remain underconceptualized or not easily measured. The Board therefore turned to the use of symptoms and impairment

as ready and valid measures for indexing children's mental health problems. The Board found that using 

current symptom and impairment measures can provide an urgently needed starting point. Once collected, 

the "building blocks" of symptoms and impairment can be examined to look for symptom patterns and to 

establish or test possible diagnostic categorization. 

Counting and Monitoring 

The Board also determined that generating national frequencies of symptoms and impairments, or 

prevalences, was of value. Nonetheless, what is even more important is the ability to provide policymakers 

and researchers with indicators of change in the population. Tracking these changes in the population may 

reveal how new policies affect children and their care. In addition, differences in problems among 

demographic groups are of concern. Such variation in the frequencies of mental health problems across age, 

geographic, cultural, ethnic, socioeconomic, and gender subgroupings will interest policymakers and 

scientists. 

The Board identified the critical questions on prevalence as: 

What are the frequencies of symptoms and impairment in U.S. children and adolescents? 

z How do these frequencies vary over time? 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
z How do these frequencies vary across demographic factors such as age, socioeconomic status, 

location, gender, and ethnicity? 

z Do these symptoms cluster to form patterns or diagnostic groups? 

The Board referred to this need to follow children's mental health status through measurable indicators as the 

monitoring function. As with all social indicators, such tracking must be done through repeated surveys, but it 

need not be done on the same children over time. 

Service Use 

The past decade has witnessed a remarkable transition in the financing, organization, and delivery of mental 

health services in the United States, as public and private payers have sought to decrease costs and improve 

accountability in mental health care. These include the development of comprehensive provider networks, 

discounted fees or other risk sharing payment systems, and intensive utilization management which together 

make up the key components of the heterogeneous plans known as managed care organizations. Although 

most policymakers believe costs are no longer increasing at double-digit rates, it is unclear if the quality of 

care has improved or deteriorated. Similar privatization efforts are starting in other sectors serving children 

and adolescents. Utilization management and capitation arrangements are being organized for child welfare, 

juvenile justice, and educational programs for children and adolescents with mental disorders. 

These forces will change the way services are delivered for years to come. The implications of such 

continuing change for any assessments of psychopathology and services use in a national study are clear. 

Study designs that do not measure changes in the population over time with sufficiently detailed community 

and regional data, are unlikely to lead to meaningful findings on policy issues for services research. Moreover, 

designs that do not allow repeated assessments over longer periods of time on populations may not capture 

larger trends in access, utilization, and cost that emerge as new systems become entrenched. 

The Board identified the following priority questions for service use: 

What is the service use associated with specific symptoms and impairments? 

z How does service use change over time? 

z How does service use differ across demographic groups? 

Although the field of service research in children is relatively new, the measurement issues for service use 

seem to be straightforward for certain service vectors. In particular, the UNOCCAP investigators should take 

great pride in their work on the Services Assessment for Children and Adolescents (SACA), since it stands 

out as the measure of choice for capturing the architecture of children's services. Further, the imbedded 

questions involving the respondent's insurance will provide a limited, but important, index of access to 

services. 

Conclusions 

The Board concluded that researches trying to understand child diagnostic issues are wrangling 

with a complex and enduring problem. The debate over the classification of disorders and 

problems, as well as their measurement, is healthy and necessary. 

Measures of symptoms and impairment are ready for use in a national survey. 

Knowing frequencies of mental health problems is of value, but a method for assessing changes 

in the frequencies and identifying subgroup variation is particularly valuable.

Conclusion 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
Service Use and Costs for Children in Treatment 

Generating an overall cost estimate for the Nation of children's mental health services is difficult since 

services may be received from multiple providers (e.g., schools, welfare, health care, or justice). Such an 

estimate may not be particularly accurate since tracking a child through these multiple providers is 

problematic. Further, the utility of such total cost estimates, with this level of accuracy, for policymakers, 

payers, or researchers is not clear. However, the study of some children and adolescents receiving services 

may provide critical information on important policy questions related to system design, insurance issues, or 

organizational factors. 

The Board identified the following priority questions in costs: 

A. What are the costs of providing mental health services to children and adolescents? 

z How do costs vary by demographic groups? 

z How do costs vary by mental health problem? 

B. How does the structure of insurance benefits influence the use of services and the costs of care? 

z What benefit packages are associated with variation in service use and cost? 

z What are the effects of managed care on service use and costs? 

z How do parity laws affect the cost of treating children? 

From a methodological standpoint, the problem that must be addressed is that a small number of individuals--

those using residential or inpatient care or partial hospitalization--account for a majority of the aggregate costs 

of mental health services. Most cross-sectional samples of the U.S. population, even with a national sampling 

frame, will identify too few service users to provide stable cost estimates. For this reason, studies on the cost 

of mental health services for children in treatment will require directly sampling service providers and the 

children receiving care. 

Pathways 

z Into and Out of Disorder

One of the research issues in the National Plan is charting the pathways into and out of disorder. This 

captures the notion of development, and how children with various risk and protective factors change. Many 

childhood conditions can be traced to complex interactions among psychological, social, and biological 

phenomena that unfold over time. To address questions about the pathways into and out of disorders, 

longitudinal studies should reflect this complexity through multiple developmental assessments over time. In 

this way, the progressive unfolding of the child's abilities in relation to changing environmental demands can 

be captured. Without this developmental perspective, assessment of the course of children's development 

The field is ready for a national survey of mental health service use in the general child 

population. 

Conclusion 

The field is ready to undertake a national survey of children receiving mental health services to 

assess the use and costs of services and the influence of benefit structure on use and costs. 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
may miss important markers of their evolving biological, psychological, and social capacities. 

The factors that predict which individuals with moderate symptoms go on to develop a problem or avoid a 

problem need to be verified. Additionally, since many individuals have multiple disorders (comorbidity) or have 

many symptoms (co-occurrence), it is important to chart the timing and patterning of multiple conditions and 

symptoms. Related issues include an examination of the persistence versus transience of various disorders 

and the strength of the association between moderate symptoms at one point of time and the likelihood of 

developing a disorder later. 

The Board identified these questions: 

What are the paths into and out of disorder? 

z Are these paths similar for different demographic groups? 

z Are these paths similar for different disorders? 

z Into and Out of Services 

What are the factors associated with the use of services? In addition to sociodemographic and health status 

factors, the following factors are of particular policy and scientific interest: the effect of the presence or 

absence of insurance, the type of insurance, the structure of the cost-sharing provisions, the extent of 

managed care, etc. Also of interest is the question of how the use of services is influenced by the availability 

of service providers and differences in the cost of the service. 

Examining prevalence and service use questions at one point in time does not account for the interplay of 

disorders and symptoms, or the changes in risk and protective factors over time. Children's lives and 

circumstances change over time and such changes are associated with changes in patterns of mental health 

and service use. If rich, developmentally sensitive, longitudinal studies are not initiated, these changes cannot 

be modeled. Concomitantly, nothing can be learned about the course of mental problems in children. 

The Board identified the following questions: 

What are the paths into and out of service use? 

Are these paths similar for different demographic groups? 

Are these paths similar for different disorders? 

z Service Need, Effectiveness, and Quality 

Eventually, planning for mental health service delivery systems and their evaluation will require careful 

assessment of the needs of individuals and populations for mental health services. Subsequently, providers, 

patients, and payers will require assessments of the quality of service delivery in the mental health sector. 

Unfortunately, the Board judged that considerable work is necessary in mental health research before such 

careful analyses can take place. Because so little is known about which treatments work in which settings and 

for which individuals, it is not possible to speculate on the level of need for any particular service. In the past, 

need has been estimated for populations through the use of such gross indicators as diagnoses or more 

recently, impairment. However, current treatment models relying on such indicators have not yet proven 

accurate for children and adolescents. 

The Board identified the following priority questions related to service need and quality: 

Which services are effective for which children and adolescents? 

z How does effectiveness vary over demographic groups? 

z Who are the children who need, but do not receive, effective services? 

z What are the costs of these services? 

z How does quality vary across settings and provider characteristics? 

z Need for Longitudinal, Local Designs 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
Only richly textured, longitudinal studies provide answers to these questions. Given the depth and breadth of 

the data required, studies cannot be reasonably done at this time through a large-scale national survey 

technique. Nonetheless, large samples, albeit geographically confined, are needed to test the interplay 

between multiple factors. 

Longitudinal surveys permit the comparison of a person's attributes over time, allowing one to study the 

precursors and sequelae of mental health impairments. Longitudinal surveys offer abilities to study pathways 

that no other design can rival. 

However, longitudinal surveys on fixed samples of individuals become less representative of the full 

population over time, as attrition in the sample and cumulative effects of emigration and immigration occur. 

Rotating panel surveys are designs that measure the same sample of persons for multiple times and then 

remove them from the measurements, replacing them with a new sample of the current population. They 

assemble shorter time series of data on each sample person but keep the sample representative of the 

current, full population. 

All longitudinal surveys face problems of tracking "movers" over time and of contaminating effects of repeated 

measurements on the same topics. Movers are survey participants who enter the survey, but leave the 

geographic area before the next assessment. The mover problem leads to attrition and to higher costs from 

trying to trace these individuals. Further, young families tend to move at higher rates than others do. Also, the 

repeated measurement of individuals using the same instruments can lead to behaviors that reduce the 

participant's interview burden (e.g., not reporting symptoms because such reports are known to lead to further 

burdensome questions). Finally, longitudinal surveys quickly lose representation of the youngest ages and 

need more frequent refreshing to remain informative about young children. In short, longitudinal surveys also 

have limitations. 

An advantage of local longitudinal surveys over national longitudinal surveys is the relative ease of appending 

richly textured community and service system attributes to the survey data. Doing this in a national survey is 

cumbersome and expensive. 

In sum, longitudinal surveys are a key tool for the study of pathways, but must be viewed as complementary 

to other designs (e.g., national, repeated, cross-sections) that more efficiently measure prevalence 

characteristics. 

State of the Art in Large, Population Based, National Surveys 

Because large-scale surveys are multivariate in nature, they allow statistical analyses to measure the relative 

Conclusion 

Studies of risk and protective factors related to mental health problems require rich, longitudinal 

designs, but not in a national survey design. Considerable work must be undertaken to ready the 

field for such an effort. 

Studies of which children enter and leave which service settings are highly complex and also 

require significant longitudinal work. Methods for linking children with multiple services sites are 

just now beginning. 

Research on the effectiveness, need for, and quality of services will also require such 

preliminary approaches.

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
frequency of a selected phenomenon on small subclasses of the national population. This allows 

policymakers to have faith that, for example, no unobserved regional factors are influencing the data. Further, 

such surveys can examine whether measurable attributes combine in unexpected ways to produce unusual 

susceptibility to problems. For example, gender differences in exposure to risk factors, types of symptoms, 

and their severity may vary across subgroups resulting in particular patterns of vulnerability (e.g., stronger or 

weaker links to impairment or perhaps shorter or longer duration of suffering). 

It is essential to review the current status of survey research technology, as background to the Board's 

recommendations on how this technology can be applied to children's mental health research. Also, this 

methodology is vital to incorporate if the national studies are to be scientifically rigorous, economical, and 

generative over time. 

The state of the art in the development of survey measurements involves: 

z Qualitative investigations that study the linguistic and conceptual variation within the population 

regarding the phenomena; 

z Semi-structured group and one-on-one interviews that explore the ease of comprehension of 

terminology to be used in survey questions; 

z "Cognitive interviewing" that asks laboratory-based respondents to report their cognitive processes in 

constructing an answer to a survey question; 

z Household-based pre-testing of fully structured instruments, sometimes with reinterviews or validitycheck components; 

z Development of training protocols for professional interviewers to use the instrumentation; 

z "Dress rehearsals" to study the performance of the instrumentation in the context of professional 

survey interviewers; 

z Implantation of design features within the survey to provide statistical estimates of accuracy and 

precision (reliability and validity) of survey measurements, as they were actually implemented in the 

survey. 

This last step deserves elaboration because it is critical to the Board's vision, and yet contrary to the culture of 

traditional psychometric practice of mounting reliability and validity studies separate from the study in which 

they are used. The literature on surveys has uncovered repeated evidence that surveys are sensitive to 

biases from many characteristics of the interviewing environment that one cannot anticipate in the laboratory. 

Survey designers attempt to eliminate these sources of systematic error in measurement (e.g., training 

interviewers to deliver questions and probe answers in a non-directive manner). Many of these influences, 

however, cannot be completely removed from the survey measurements. 

Instead of relying on estimates of reliability and validity from prior small scale-studies (often using different 

interviewing staffs, different sites of data collection, different protocols), survey researchers increasingly 

attempt to imbed side-studies that provide reliability and validity estimates on the survey data as collected in 

vivo. These studies can use probability subsamples of the full sample, with reinterviews involving clinical 

examinations or more in-depth questioning or replication of the original measurements. The value of 

imbedding such studies in the actual survey is that multivariate models can be used to estimate the data 

quality for improving the inferential statements derived from the statistics. 

Conclusion 

Recent advances in survey methodology should be incorporated into NIMH's large-scale survey 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
Summary 

The Board is recommending immediately moving forward on monitoring the levels of symptoms and 

impairment in children and adolescents and their service use because this research would be important for 

children and because the field is ready to do so. Also, a large-scale national survey on the patterns of service 

use, the costs of services, and the factors that influence both use and costs could be conducted now. 

Priority questions regarding paths into and out of both disorders and service use, and understanding the need 

and effectiveness of services should be supported through local and regional longitudinal studies of sufficient 

size and with sufficient breadth of variables. NIMH has a strong history of supporting these efforts which 

should provide this new initiative with the necessary groundwork. 

Determining readiness and setting priorities are always difficult processes. The Board expects that each of its 

assumptions and determinations will be subjected to criticism, and this criticism should be viewed as part of 

the necessary dialectic process for initiating a major research effort. 

GETTING ANSWERS TO PRIORITY QUESTIONS 

Overview 

Addressing the questions identified by the Board will require a sustained program of research on children's 

mental health status, services, and the factors that influence each. This plan for a sustained program of 

research contains a subset of questions that require national data and are ready to be addressed. In addition, 

the plan calls for initiatives to foster better conceptualization, new efforts in measurement development, and 

hypothesis-generating studies to build the groundwork for answering the remaining questions. 

As depicted in Figure 1 below, the components of the program of research are actually freestanding. 

However, the components are also seen as synergistic. For instance, the national survey work will provide a 

testing ground for the new conceptual work and serve as a reference-base for the local and regional studies. 

Further, when the data from surveys are "democratized" (i.e., given promptly and freely to the field), 

researchers can examine whether the data are consistent with competing hypotheses from various 

perspectives. The remainder of this section will outline the following seven components of the proposed 

program. 

Figure 1 

PROGRAM OF CHILD AND ADOLESCENT RESEARCH1

projects.

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
 

National Monitoring Survey in Prevalence and Service Use 

There are many possible designs for conducting a national assessment of children's mental health problems 

and their service use. These designs vary in the precision, timeliness, and quality of estimates, as well as 

costs. In considering the merits of alternative designs, the Board used five criteria: 

a) Ability to estimate change in prevalence of symptoms, impairments, and use of services over 

time; 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
b) That the specific question is best addressed (and ready to be addressed) through a national 

design; 

c) Ability to foster the development and evaluation of new measurement tools, taking advantage 

of advances in the field; 

d) Extent of openness to the entire field, permitting participation of large numbers of 

researchers; 

e) Possibility of partnerships with ongoing data collection systems that will reduce costs to 

NIMH. 

As an aside, the Board wants to recognize a noteworthy asset of the monitoring study, even though this 

comment is beyond its purview. Mental health service researchers are faced with many problems when 

looking to see the effects of a policy shift on the public's mental health or service use. One of the most 

frustrating must be the delay between an announced policy change and obtaining funds to evaluate the 

policy's impact. Having an ongoing monitoring study, also called an indicators study by service researchers, 

with public data files provides the immediacy researchers and policymakers need. This asset is not limited to 

children's mental health services and should be considered for adults as well. 

Figure 2 depicts the monitoring program, beginning with a benchmark assessment (about 90 minutes of 

interviewing time) in the initial year. The Board recommends that the first benchmark assessment consists of 

a selected number of demographic variables, a symptom checklist, an impairment checklist, and the SACA. 

Future benchmark surveys, conducted on a planned schedule every 4 to 6 years, might offer enhanced 

diagnostic measures, multiple reporters of the child's attributes, insurance and provider follow-backs, or 

psychiatric diagnostic interviews on subsamples. 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
 

The Board believes that the intensive benchmark measures will not be needed in each assessment wave. 

Rather, most assessment waves would consist of a very brief set of core indicators and a short 

developmental module (about 15-20 minutes total interview time). The core indicators would be a small 

subset of the benchmark measures. The indicators would be brief, sensitive measures of mental health status 

and service use that could be used to estimate change between core survey years or between core survey 

years and the benchmark survey year. 

The developmental module of each core survey would be the "laboratory" for innovation in children's mental 

health measurement, fulfilling criterion c) above. NIMH would announce a competition for these modules. 

Proposals could involve new measurements on subsets of children (e.g., the young child), measures on 

specific disorders that have previously eluded accurate measurement, or urgent policy questions. The 

proposals might involve small-scale preliminary studies prior to the introduction into the next available 

developmental module. Each edition of the developmental module would be analyzed to assess its success in 

providing unprecedented insights into the mental health of children. Individual measures or questions would 

be candidates to replace the benchmark measures or core assessments. 

The repeated benchmark assessments would offer the field a tool to recalibrate its core indicators based on 

changes in the population or service characteristics or based on inventions of improved measurements. Each 

core assessment and benchmark assessment is envisioned as a new cross-sectional sample of U.S. children. 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
In summary, the proposed monitoring survey can be seen as similar to the National Aeronautics and Space 

Administration's (NASA) shuttle program. NASA invests in running the shuttle program and a stream of 

ongoing research for developing shuttle technology. Additionally, NASA reserves funds, airtime, and staff for 

conducting add-on projects to the ongoing procedures. That is, NASA will solicit research on a given topic, 

and the competitive winners are provided time during the space flight for their studies. NIMH can adopt this 

model to promote timely and optimal application of scientific advances, as well as coordination in its largescale initiatives. 

National Survey on Use and Costs among Children in Treatment 

The monitoring system of prevalence, insurance coverage, and service utilization is a powerful method for 

identifying overall trends on use of services across the population. The study will not, however, provide much 

information about the nature of treatment received by children. The main reason for this is that the national 

design will not identify large numbers of children who are receiving treatment in specialty mental health care 

services. (It should have the power to identify children who are less seriously ill and using outpatient services 

and frequently prescribed medications.) Consequently, it will not provide precise information on many of the 

questions that NIMH is asked about the treatment of seriously ill children and how it is affected by the 

financing and organization of care. 

Some of these questions can be addressed by regional and single-site studies, but regional and single-site 

studies will not be able to answer questions on the impact of parity laws, differences in the way that managed 

care carve-outs or health plans are organized, and geographic and ethnic differences in the use of intensive 

services. The Board therefore recommends that a national study of children in treatment be conducted to 

examine processes of service access, use and financing. 

The Board is not making specific design recommendations about how to execute such a study, but offers 

general advice. The study can be initiated promptly since the instruments necessary for the conduct of such 

an effort are already available through the efforts of NIMH grantees and the UNOCCAP investigators. In 

addition, the qualities outlined above in the survey section must be reflected in this study as well. 

Considerable thought should be given to the particular sampling of mental health service settings, geographic 

areas, and populations to take advantage of the natural variation in the spread of managed care, provider 

types, and public laws related to the delivery of mental health care. Natural variation across states in policy 

changes or the provision of services should influence the sampling frame. For example, states or regions 

undergoing early welfare reform should be contrasted with those not undergoing reform at the same pace. 

The sample and assessments should probably include youth in various types of restrictive services to include 

partial hospitalization, intensive outpatient therapy, and in-home crisis services. 

The proposed study of service use for children in treatment is not designed to provide a national total cost 

estimate. The field is not yet ready to tap into all of the streams of costs for children receiving services in nonspecialty settings (e.g., health, welfare, justice and education). Making these links across service settings will 

be a vital area for developing the field and improving the current cost estimates. The Board believes, 

however, that the proposed study on the architecture of service use and costs would be highly informative for 

both services researchers, payers, and for policymakers. 

The Board concluded that questions on prevalence and service use are best answered, and are ready to be 

answered, at a national level through a large-scale survey. A rigorous study is required to describe symptoms, 

Recommendation 1 

Initiate an ongoing national survey to monitor children's symptoms, impairment, and service use. 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
symptom patterns, impairments, and service use in a sample large enough to reveal important sources of 

demographic variation in the United States. It is of critical importance to launch this study soon, since valuable 

time has already been lost for informing the policies and strategies that are changing service provision and 

financing. The Board also places great emphasis on NIMH creating the necessary infrastructure to initiate this 

activity immediately. 

Generativity and Availability of the Data 

The survey data would be publicly available as soon as possible after data collection. This rapid availability is 

standard procedure for surveys conducted through other governmental agencies, such as the National Health 

Interview Survey and the National Ambulatory Medical Care Survey. 

Because the data sets developed by the proposed project would be public use data as with many other 

federal data sets, resources to support their analyses and use are essential to maximize return. A variety of 

options are possible including targeted contracts for specific analyses or a grant program for analyses. Of 

course, the data set will require documentation and seminars on access and use of the data to maximize its 

utility. These activities may be undertaken by NIMH staff or contracted. 

The Board believes that if data were publicly available and analysis supported by NIMH, would be generative 

of new sets of hypotheses to stimulate the field. These might arise from unexpected differences among 

subgroups of children in prevalence or service use patterns. They might arise from evidence of changes over 

time in prevalence that suggest causal mechanisms undetected in clinic samples or homogeneous groups 

studied previously. They might also arise from newer scientists or scientists not bound by the dominant 

paradigm, testing novel hypotheses with the observational data and finding support. In short, the national, 

local, and regional studies should complement and reinforce one another. 

Answering Questions on Diagnosis, Pathways, and Outcomes 

As described in this chapter, NIMH should provide initiatives for building the areas that are seen as critical to 

achieving the goals in the National Plan for Research on Child and Adolescent Mental Disorders. Issues of 

measurement and diagnosis are critical to advance research in children's mental health problems. Better 

understanding of disorders and their course will enhance treatment research. Service programs cannot be 

Recommendation 2 

Initiate a national survey of the type of services used by children in treatment, the costs of these 

services, and how benefit coverage affects service use and costs. 

Recommendation 3 

Develop infrastructure activities to support the two large-scale surveys in Recommendations 1 

and 2. 

z Once collected, the national survey data should be released to the field, after ensuring the 

participants' confidentiality. 

z A program announcement should be issued for secondary analysis of the survey data. 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
targeted to needy children until research documents what works for whom. The hand-in-glove nature of 

research and policy, as well as basic and applied research, must be recognized and fostered. Certain 

preliminary, basic work must be accomplished before the questions on diagnosis, pathways, and outcomes 

can be posed and feasibly answered. In addition to continued support of investigator-initiated projects, 

Program Announcements, contracts, or RFAs should be issued as quickly as possible to stimulate research in 

the areas proposed in the remainder of this section. 

Conceptualization and Measurement 

The Board concluded that researchers trying to understand child diagnostic issues are wrangling with a 

complex and enduring conceptual problem. The debate over the classification of disorders and problems, as 

well as their measurement, is healthy and necessary. Inductive as well as deductive approaches must be 

advanced and tested. For example, large-scale data collection on the natural distribution of symptoms and 

impairments in children may provide a new structure of how these problems coalesce into disorders. The 

Institute is encouraged to support many alternative models to assessment in children. Further, comparisons 

between and among methods for validity and utility are strongly supported. However, methods for certain 

subgroups of children are not as ready as others for assessment. The Board recommends particular attention 

to developing these areas. 

Young Child 

The assessment of mental health in older children and adolescents is far more developed than procedures for 

determining mental health in younger children. For example, it is not clear that the Diagnostic Interview 

Schedule for Children (DISC), as currently designed, taps the symptoms of various disorders very well for 

children under the age of 7 to 8 years of age. In addition, not only are there unanswered questions vis-à-vis 

the validity of parental reports of young children's symptoms as mapping onto disorders, but reliability 

questions remain. The Board is heartened by the efforts of the researchers working on the DISC and the 

UNOCCAP Young Child project in addressing these issues. 

Alternative techniques for assessing the mental health of young children are being developed. Several of 

these laboratory-based measures appear to be very promising, but more research is needed to see if these 

measures can be modified for use in the field. Questions also remain on the usefulness of some of the 

techniques for children with low verbal comprehension and expression. 

The Board encourages the further development of such assessment techniques in field settings and with 

various groups of children (i.e., different ethnic groups, bilingual children, immigrant children). It also 

encourages the assessment of young children using multimethod approaches (i.e., parental report using the 

DISC, observation of parents and children, emotional regulation tasks). Finally, the Board encourages shortterm longitudinal follow-up studies in order to see if these techniques have predictive value. 

This necessary work should not hold up the national monitoring survey. Depending on the psychometric 

properties of the selected symptom and impairment checklists for young children, they may either be included 

or not included in the initial sampling waves. If the selected checklist is reliable and valid for this age group, 

then younger children should be included. If the selected measures are not yet validated for young children, 

this testing and refinement should be undertaken. But such testing should not delay the initiation of the 

Recommendation 4 

Establish an initiative to enrich the conceptualization and measurement of childhood mental 

problems and impairment from infancy to 18 years of age. 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
monitoring survey. In this latter case, full incorporation of the younger age group into the national monitoring 

survey can be a future goal, and the young child initiative will be an important catalyst for the field in reaching 

this goal. 

Longitudinal Research Initiatives 

Pathways. NIMH has a long history of supporting risk factor research. Providing synthesis for this research 

and refining important testable questions is an important activity for the Institute to sustain. Small-scale 

investigator-initiated projects may not be sufficient for this activity. The Institute is encouraged to work with 

other funding agencies and ongoing surveys to collect data on samples that are large enough and sufficiently 

representative to examine the factors and circumstances that predispose some children to develop mental 

disorders. Of particular importance is the need to identify the factors that predispose some children to short 

bouts of problems while other have more persistent problems. The monitoring study will not allow for an 

examination of how some children, who live in difficult circumstances, avoid problem behavior or mental 

disorders over time. 

Applying the best of services research, epidemiology, developmental psychopathology, and social sciences 

will be necessary to understand the paths into and out of services. The complexities are profound since 

children receive services in so many sectors and neither they nor their parents are good reporters of the 

services received. The Board's best advice is to start small and focus on mental health services first, working 

toward building linkages with the educational, health, welfare, and justice systems. Brief measures of mental 

health status and service use will be invaluable in making these ties. 

Outcomes and Quality. The conceptualization and assessment of treatment outcomes for children have 

progressed markedly in the past decade in conjunction with a growing recognition of the importance of 

functional status/impairment and family perspectives. However, much additional work needs to be done. In 

particular, better characterization of child mental health status remains elusive. Similarly, investigators have 

yet to elucidate the key components of treatment necessary to link outcomes with services in the examination 

of effectiveness. The Board heartily supports NIMH's efforts to develop methods for assessing child and 

family outcomes and for better describing components of mental health services. 

Further complicating the inclusion of treatment outcomes and quality of care assessments in a national study 

is the lack of data on effective treatments for many childhood disorders. It is essential that local or regional 

studies examining the translation of efficacious treatments into community settings be conducted as soon as 

possible, especially in the face of numerous studies documenting limited effectiveness for routine mental 

health services. The Board encourages all reasonable efforts to develop effective treatments for a wider 

range of children and adolescents. The assessment of the quality of care for a national study will progress 

rapidly once such treatments are available. 

Recommendation 5 

Establish an initiative to further the understanding of emotional functioning and mental health 

problems in young children.

Recommendation 6 

Establish a program of local or regional longitudinal studies in three separate areas: 

z paths into and out of disorder 

z paths into and out of service use 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
>Innovations in Methods and Analyses 

Many fields (e.g., astronomy) have made advances in theoretical and applied knowledge by directly funding 

the development of new measurement tools. These investments have multiplier effects because inventions in 

measurement can be used to improve the insights from hundreds of scientists. The Board believes that a 

revitalized program is needed in this area to ensure that methodological and analytic opportunities rapidly 

advance the quality of research in mental health. Some agencies devote a certain percentage of its data 

collection budget to such an infrastructure building initiative. 

Research in this program might involve addressing issues of new standardized measurement of very young 

children. It might involve studies of the impact of linguistic constructions on question comprehension in selfreports concerning mental impairments. It might involve development of statistical methods to reduce the 

harmful effects of participant attrition. It may involve survey methods to combine rapidly and accurately 

administrative data from health systems with self-report data from survey participants. It might involve the 

development of computer-assisted measurement protocols that decrease the effects of social desirability on 

self-report of mental health problems and risk behaviors. It may involve the development of statistical analytic 

techniques to provide insights into the role of the family or schools in the course of mental disorders. 

In general, mental health researchers cannot be expected to define the cutting edge of developments in 

survey design, data collection, and analysis as well as be on the forefront of mental health. The field of survey 

methodology is a fast-moving one, involving technological advancements in both data collection and 

estimation. To sponsor an innovative grant program in methods, NIMH would want to be open to applications 

from design statisticians, analytic statisticians, computer scientists, survey methodologists, cognitive scientists 

and others, who are building the key theories and applications in research methods. NIMH should encourage 

collaborations with mental health researchers, but these methodologists should be supported as principal 

investigators as well. 

The review of these applications should involve a significant representation of specialists in measurement. 

That is, a majority of the reviewers should be basic scientists actively contributing to the field of measurement, 

as well as mental health researchers who are working on measurement issues. 

Integrative and Generative Nature of the Plan 

These seven recommendations are seen as an integrated package, even though each initiative can be 

freestanding. The synergistic value of the components will be evident over time as new findings are integrated 

into or tested through the monitoring survey. As an example of envisioned interplay, the national services use 

and costs survey for children receiving services can begin now. From this short-term survey, important 

hypotheses or measures will be generated that can be included in the national monitoring survey either as a 

developmental or topical module in four to five years. 

The effectiveness and path issues need more time for basic research and conceptualization, but initiating 

these activities now will speed the field in answering the important policy questions related to developmental 

z effectiveness and quality of mental health services 

Recommendation 7 

Establish a program to foster innovative methods or analyses in child mental health. 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
processes. By supporting longitudinal studies and methods development today, the Board foresees future 

possibilities for topical modules in the monitoring study. These modular add-ons should be competitive and 

may also include solicitation of research to address pressing policy or scientific questions as they arise. 

MOVING FORWARD: NIMH'S ROLE 

The initiation and management of large-scale efforts require both accountability and scientific vision. NIMH 

staff must balance administrative necessities with achieving the scientific and public health goals of any largescale effort. The Board offers a series of suggestions on the planning, staffing, and oversight of such efforts. 

The Board also encourages the staff to talk to colleagues at other agencies and institutes to identify other 

possibilities for enabling large-scale projects. Some of the ideas mentioned here were initially raised at the 

NAMHC's May l997 policy meeting. 

Planning 

The Board has noted the urgency of mounting the proposed program of research as quickly as possible. 

Nonetheless, the planning phase cannot be short-changed. The Board reviews below some of the critical 

components of planning a large-scale effort, and encourages a period of open planning that engages the field 

and other Federal agencies and institutes. 

z Selecting a Feasible Funding Mechanism 

Deciding among contracts, cooperative agreements, and grants is an administrative decision, based on the 

type of activity. However, the execution of each of these mechanisms can be done in many ways as 

demonstrated throughout NIH. The NIMH should consider the following in initiating these programs of 

research. 

Cooperative agreements are solicited through RFAs. Individuals or teams of individuals compete by 

submitting their research plans and protocols to a review committee. The review committee carefully 

considers these applications against the review criteria. NIMH then awards individual projects. Typically, 

NIMH then convenes the successful applicants, who then design a shared protocol. There is the potential for 

many problems in this approach, and NIMH should consider improving this process so that: 

z Teams of researchers respond to the RFA. Awards can then be made to a self-composed team that 

already has or can easily build a workable alliance. 

z In the extramural team, there must be one designated leader. The leader's role and how this role is 

assigned must be specified in the solicitation. 

z The review process could be segmented. There could be a minimum application based on a capacity 

review. The successful applicants would then submit a more involved application including the 

proposed design. 

When the best way to achieve the programming goal requires scientific expertise from investigators, a 

cooperative agreement is initiated. However, when the NIMH staff knows what needs to be done and how it 

should be accomplished, a contract should be considered. One issue regarding the use of contracts is that a 

contract requires OMB clearance for a survey. This can be difficult to achieve in a timely manner, but should 

not dissuade NIMH staff from using the contract mechanism. 

In drafting the RFA or RFP, staff should focus on a few, feasible scientific and/or public health issues that 

must be addressed. Having such focus will enrich the process from drafting the RFA to making design 

decisions, to the final analysis of the data. 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
z Establishing a Time Line 

Planning involves drafting a feasible time line. The Board stresses feasibility, even though the needs of 

children are so urgent. As an example, the Board projected a time line for initiating the proposed monitoring 

survey. If the Institute wants to be in the field with the national monitoring survey in September of 2000, the 

time line for developing, competing, reviewing, and awarding the initiative is already behind schedule: 

z Budgeting for Large-scale Initiatives 

Budgets must be fully anticipated for flagship studies such as national surveys. Staff must reconnoiter the 

estimated costs with the questions to be addressed. If the budget does not permit the answering of these 

questions, changes must be made. 

In addition, NIMH should plan on complex projects having problems during implementation. Some of these 

problems can be fixed with additional financing, and the budget should have a reserve fund for this purpose. 

Efforts to equalize payments to large-scale projects over funding years should never jeopardize the science or 

financial accountability. That is, although budgeting may run more smoothly if yearly costs are equivalent, 

large-scale studies rarely have identical yearly costs. NIMH should adopt a pay-as-you-go policy, with 

sufficient monitoring. 

Once the applications/proposals have been received and reviewed, the budget estimates should be 

reassessed. Staff should examine whether initial estimates will permit the projected work to be completed as 

required. 

z Identifying Co-sponsors for Large-scale Projects 

In a world of constrained research funding, a legitimate question for NIMH to discuss is how much of its 

resource base should be devoted to large-scale observational studies and surveys versus small-scale 

experimentation. This is a question that faces all funders of scientific activities. There appears to be at least 

four alternative approaches to this issue: a) fund the survey research using agency funds, b) recruit a set of 

partners who will pay for their own compatible modules on the NIMH survey, c) seek out another survey of 

compatible design as a vehicle for the NIMH questions, and d) use existing survey data to provide proxy 

indicators relevant to the agency's needs. 

For the monitoring survey, the Board urges NIMH to begin contacting other Federal agencies, charged with 

tracking the health and welfare of the Nation?s children, to determine whether partnerships of the sort in b) 

and c) above might be possible. Much of the survey data collected by the Federal government on the 

household population comes from the U.S. Census Bureau. The collection of mental health-related 

Tasks Contract (RFP) or Grant (RFA) Process 

Estimated Target Dates

In the Field September 2000

Full Pilot January 2000

Award September 1999

Review June 1999

Submission Deadline March 1999

Request Announced November 1998

Request Cleared October 1998

Request Written August 1998

Scope Finalized July l998

Discussion on Scope January 1998 - June l998

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
measurements on children would be a departure from normal practice of that agency, and the Board is 

doubtful that such a partnership would be feasible. Candidates for partners include the follow-ups to the 

National Health Interview Survey of the National Center for Health Statistics, the Behavioral Risk Factor 

Surveys of the Centers for Disease Control and Prevention, the Current Population Survey of the Bureau of 

Labor Statistics, the National Household Education Survey of the National Center for Education Statistics, the 

National Household Survey on Drug Abuse of the Substance Abuse and Mental Health Services 

Administration, the National Immunization Survey of the Centers for Disease Control, among others. 

For instance, in the monitoring survey, the core survey years could involve the addition of the NIMH measures 

to an existing ongoing survey. The size of the question set would be determined by the amount that the 

partner survey could incorporate. For discussion purposes, it would be less than 20 minutes of questions 

focused on one selected child in the sample household. 

NIMH would enter into an agreement with the sponsor of the survey to "purchase" some amount of time with 

sample respondents to ask a set of questions and transfer funds to pay for the increase in survey costs for 

that core. By taking advantage of the infrastructure of the base survey, NIMH could reduce costs relative to 

mounting its own survey (addressing point d) above). 

An NIMH-led survey could also be initiated along with a partner. NIMH might wish to consider a co-funder with 

interests in collecting similar data. Alternatively, NIMH may want to "rent" a service, such as the sampling of 

households from an agency using desirable sampling procedures, but seeking different respondents (e.g., 

National Institute on Aging (NIA) and NIMH sharing respondents over 65 or under 19 in identified residences). 

Yet another possibility for a partnership would be the ability of another agency to provide survey expertise or 

other needed staff. If no partnership can be established that would be mutually attractive, the work should 

continue nonetheless. 

Governance and Staffing Needs 

The ongoing nature of the proposed program combined with the need for accountability suggest that an ad 

hoc governance structure for each project will not be adequate and that a permanent, or at least long-term, 

administrative component should be established. The budgetary and oversight responsibility for the project 

should rest within NIMH. Survey contract and collaborative functions could reside within NIMH or within a cosponsoring agency like the National Center for Health Statistics that has similar relationships with other 

agencies. 

To ensure that the problems are identified quickly and handled appropriately and fairly, NIMH must have a 

separate unit of scientifically appropriate staff to serve both as administrators for these projects. These 

include survey statisticians and methodological experts as well as experts in mental health measurement. 

Such staff members are a requirement for the duration of the project. Administrative staff must be able to 

assess when scientific progress is being made and its adequacy. They must have credibility with the field, the 

NIMH leadership, and the participating researchers. 

Among these staff members, there must be one individual with the scientific credentials to serve as the NIMH 

leader for this effort. The organizational location of this individual and reporting requirements must be 

formalized and maintained. The Board suggests looking across NIH for possible models, considering Nationa 

Instiute on Drug Abuse's Drug Abuse Warning Network program or certain National CancerInstitute projects. 

Accountability 

z Peer Review 

The role of peer review should not be reserved for concept clearance and review of applications. For 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
instance, typically RFAs and RFPs have been reviewed for their concept only. That is, an external group of 

scientists is convened to review the concept, but not the specifics of the anticipated request. Frequently, the 

concept is of undeniable appeal, but the feasibility cannot be addressed because too few details are provided. 

The advantage of this approach is that the concept reviewers, because they do not know the specifics of the 

request, may still compete for the award. The down side is that when the applications go to the peer review 

committee, reviewers are asked to assess the applications relative to the criteria in the RFA or RFP, even if 

the request is flawed. 

As previously mentioned, the current NIMH approach to cooperative agreements is to select participants 

based on their single-site applications. Then these peer-reviewed designs are dropped from consideration to 

form the collaborative design. Not only is this a problem for establishing productive collaborations, but it may 

result in a final design that has not been peer-reviewed prior to fielding. This can happen in contracts as well, 

when designing the study is an initial step in the scope of work. Regardless, the final protocol should be 

subjected to peer review (in a timely manner) to ensure the design's ability to test the proposed hypotheses 

and its staffing, budget and time frame. A peer review would also be helpful in the event of significant staff or 

budget changes. 

Separately, but equally as important, some of the specific initiatives will require special review groups or 

standing review panels to evaluate applications for added modules, instrument development, or the young 

child area. These review activities would likely focus on two primary aspects, scientific content and policy 

implication. For the former, expertise in survey design, statistics and sampling, psychometrics, and content 

areas would be required. Policy analysis expertise would also be required to assess the relative contributions 

of various studies to the national debates around child mental health. 

z Advisory Boards 

Ongoing, timely oversight, may best be provided by establishing an advisory board at the initiation of these 

large-scale efforts. The quality of the advisory board may even be part of the RFA's or RFP's review criteria, 

asking each team to nominate their own advisors and letting the peer-review group assess the adequacy of 

the board. This is consistent with NIH policy and will prevent time lags in constructing multiple ad hoc review 

groups and the lack of continuity in approach as well. 

In addition to the project's advisory board, large-scale projects would require significant oversight by the 

NAMHC. A program of research, such as that proposed in this report, will need oversight that only the Council 

or a subcommittee, with its perspective of NIMH's entire portfolio, can provide. 

CONCLUDING COMMENTS 

The Oversight Board welcomed the opportunity to integrate this information and to advise NIMH and its 

Council; however, the recommendations in this report will require broad debate and scrutiny to best shape the 

future initiatives. Certainly NIMH should reach out to the research community and to professional groups, but 

the Board encourages even broader consultation. Key players in the debate include the service providers and 

payers who can enhance the utility of the data to be collected in any new initiative. Parents of children with 

mental disorders should also be made aware of this effort so that they can provide NIMH with the realities of 

referring their children for diagnosis and treatment, identifying services, and financing these services. 

The closing words must be dedicated to the children of this Nation. On behalf of those children living with 

mental problems, the Board reminds NIMH of the critical need to conduct research so that these children can 

be identified and effective services provided. Achieving this goal will not be easy, but the necessary initial 

steps can be taken immediately. Do not wait for the perfect moment or the perfect measure. The Board 

advises NIMH to adopt its maxim for moving forward--if it's ready, use it now; if it is not ready, develop it. 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
APPENDICIES 

APPENDIX A 

UNOCCAP OVERSIGHT BOARD

APPENDIX B 

UNOCCAP PRINCIPAL INVESTIGATORS

CHAIRPERSON

David R. Offord, M.D. 

Professor 

Department of Psychiatry 

McMaster University 

Hamilton, Ontario 

Canada 

NIMH STAFF

Jane A. Steinberg, Ph.D. 

Associate Director 

Office of Special Projects 

National Institute of Mental Health 

Rockville, Maryland 

MEMBERS

Jeanne Brooks-Gunn, Ph.D. 

Virginia and Leonard Marx Professor 

Child Development and Education 

Teachers College, Columbia University 

New York, New York 

Felton Earls, M.D. 

Professor of Human Behavior and 

Development 

Department of Maternal and Child 

Health 

School of Public Health 

Harvard University 

Boston, Massachusetts 

Robert M. Groves, Ph.D. 

Professor and Research Scientist 

University of Michigan 

Director, Joint Program in Survey 

Methodology 

University of Maryland 

College Park, Maryland 

Kelly Kelleher, M.D. 

Professor 

Child Services Research 

University of Pittsburgh 

Medical Center 

Pittsburgh, Pennsylvania 

Judith Lave, B.A., Ph.D. 

Professor of Health Economics 

Graduate School of Public Health 

University of Pittsburgh 

Pittsburgh, Pennsylvania 

Linda B. Cottler, Ph.D. 

Associate Professor of Epidemiology 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
APPENDIX C 

CASENESS AND ADOLESCENT 

PSYCHIATRIC EPIDEMIOLOGY 

WORKSHOP ROSTER

in Psychiatry 

Department of Psychiatry 

Washington University 

St. Louis, Missouri 

Benjamin B. Lahey, Ph.D. 

Professor of Psychiatry and Chief 

of Psychology 

Department of Psychiatry 

University of Chicago 

Chicago, Illinois 

Philip J. Leaf, Ph.D. 

Professor and Director 

Hopkins Prevention Research Center 

Department of Mental Hygiene 

School of Hygiene and Public Health 

Johns Hopkins University 

Baltimore, Maryland 

Mary Jane Rotheram-Borus, Ph.D. 

Professor, Division of Social 

and Community Psychiatry 

Department of Psychiatry 

University of California, Los Angeles 

Director, Clinical Research Center 

UCLA Neuropsychiatric Institute 

and Hospital 

Los Angeles, California 

Benjamin Lahey, Ph.D., Chairperson 

University of Chicago 

Chicago, Illinois 

Hector Bird, M.D. 

Columbia University 

New York, New York 

Michael Boyle, Ph.D. 

McMaster University 

Hamilton, Ontario 

Canada 

Peter Jensen, M.D. 

National Institute of Mental Health 

Rockville, Maryland 

David Offord, M.D. 

McMaster University 

Hamilton, Ontario 

Canada 

Mary Schwab-Stone, M.D. 

Yale Child Study Center 

New Haven, Connecticut 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
APPENDIX D 

ANALYSIS OF LONGITUDINAL DATA 

WORKSHOP ROSTER

Naomi Breslau, Ph.D. 

Henry Ford Hospital 

Detroit, Michigan 

Glorisa Canino, Ph.D. 

University of Puerto Rico 

San Juan, Puerto Rico 

Michelle Cooley-Quille, Ph.D. 

Johns Hopkins University 

Baltimore, Maryland 

Elizabeth J. Costello, Ph.D. 

Duke University Medical Center 

Durham, North Carolina 

Byron Egeland, Ph.D. 

University of Minnesota 

Minneapolis, Minnesota 

Felicisima Serafica 

Ohio State University 

Columbus, Ohio 

David Shaffer, M.D. 

Columbia University 

New York, New York 

Roger Tourangeau, Ph.D. 

NORC Washington 

Washington, DC 

Irwin Waldman, Ph.D. 

Emory University 

Atlanta, Georgia 

Carolyn Zahn-Waxler, Ph.D. 

National Institute of Mental Health 

Bethesda, Maryland 

Philip J. Leaf, Ph.D., Chairperson 

John Hopkins University 

Baltimore, Maryland 

Adrian Angold, M.R.C. Psych. 

Duke University Medical Center 

Durham, North Carolina 

Barbara Bailar, Ph.D. 

National Opinion Research Center 

Chicago, Illinois 

Patricia Cohen, Ph.D. 

New York State Psychiatric Institute 

New York, New York 

Michele Cooley-Quille, Ph.D. 

Johns Hopkins University 

Baltimore, Maryland 

E. Jane Costello, Ph.D. 

Duke University Medical Center 

Sally Horwitz, Ph.D. 

Yale University 

New Haven, Connecticut 

Richard Kulka, Ph.D. 

Research Triangle Institute 

Research Triangle Park, North Carolina 

Benjamin Lahey, Ph.D. 

University of Chicago 

Chicago, Illinois 

Rolf Loeber, Ph.D. 

Western Psychiatry Institute and Clinic 

Pittsburgh, Pennsylvania 

Kristin Moore, Ph.D. 

Child Trends Incorporated 

Washington, DC 

Editha Nottelmann, Ph.D. 

National Institute of Mental Health 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material
Durham, North Carolina 

Naihua Duan, Ph.D. 

The RAND Corporation 

Santa Monica, California 

Byron Egeland, Ph.D. 

University of Minnesota 

Minneapolis, Minnesota 

Kimberly Hoagwood, Ph.D. 

National Institute of Mental Health 

Rockville, Maryland 

Rockville, Maryland 

David R. Offord, M.D. 

McMaster University 

Hamilton,Ontario, 

Canada 

Anne W. Riley, Ph.D. 

Johns Hopkins University 

Baltimore, Maryland 

Scott Zeger, Ph.D. 

Johns Hopkins University 

Baltimore, Maryland 

13/4/9848 

NIMH Archive Material This document is no longer being updated.

For the latest information please go to http://www.nimh.nih.gov

NIMH Archive Material