Department of Health and Human Services 

OFFICE OF 

INSPECTOR GENERAL 

FEBRUARY 2001 

OEI-05-00-00240 

STATE CHILDREN’S HEALTH 

INSURANCE PROGRAM 

Assessment of State Evaluations Reports 
OFFICE OF INSPECTOR GENERAL 

The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, as 

amended by Public Law 100-504, is to protect the integrity of the Department of Health and 

Human Services programs as well as the health and welfare of beneficiaries served by them. This 

statutory mission is carried out through a nationwide program of audits, investigations, 

inspections, sanctions, and fraud alerts. The Inspector General informs the Secretary of program 

and management problems and recommends legislative, regulatory, and operational approaches to 

correct them. 

Office of Evaluation and Inspections 

The Office of Evaluation and Inspections (OEI) is one of several components of the Office of 

Inspector General. It conducts short-term management and program evaluations (called 

inspections) that focus on issues of concern to the Department, the Congress, and the public. The 

inspection reports provide findings and recommendations on the efficiency, vulnerability, and 

effectiveness of departmental programs. 

OEI's Chicago regional office prepared this report under the direction of William Moran, Regional 

Inspector General and Natalie Coen, Deputy Regional Inspector General. Principal OEI staff 

included: 

REGION HEADQUARTERS 

Deborah Walden, Team Leader Joan Richardson, Program Specialist

Nora Leibowitz, Project Leader

Michael Craig, Program Analyst

To obtain copies of this report, please call the Chicago Regional Office at (312) 353-9867. 

Reports are also available on the World Wide Web at our home page address: 

http://www.hhs.gov/oig/oei/ 
EXECUTIVE EXECUTIVE SUMMARY SUMMARY 

PURPOSE 

To assess States’ evaluations of their performance goals, particularly those focused on 

reducing the number of uninsured children. 

BACKGROUND 

Many low-income children receive health care coverage through Medicaid and the State 

Children’s Health Insurance Program (SCHIP). SCHIP is designed to reduce the number 

of uninsured children by offering health care coverage to previously uninsured children. 

The program targets children with family income too high for Medicaid and too low to 

afford private insurance. 

States annually report the results of SCHIP assessments to the Secretary of the 

Department of Health and Human Services. States also submitted evaluations of their 

programs and progress meeting goals to the Secretary by March 31, 2000. 

The Balanced Budget Refinement Act of 1999 requires that the Office of Inspector 

General (OIG) sample States in order to: (1) evaluate whether States are correctly 

enrolling children in SCHIP or Medicaid, and (2) assess States’ progress reducing the 

number of uninsured low-income children. The Act further requires the OIG to evaluate 

the progress sampled States have made to achieve their strategic objectives and 

performance goals. Sampled States must be selected from States only using SCHIP funds 

to operate SCHIPs separate from their Medicaid programs and cannot be selected from 

States using SCHIP funds to expand their Medicaid programs. 

This is one of two reports on five States administering separate child health programs. 

This report focuses on five States’ evaluations of their performance goals. In the 

companion report, we address State efforts to ensure that Medicaid eligibles are not 

enrolled in SCHIP. 

FINDINGS 

Questionable evaluations undermine the reliability of State reports of success 

Sampled States report reductions in the number of uninsured children and mixed success 

meeting other goals. SCHIP reports rely heavily on enrollment data and tend to use 

descriptive information in lieu of evaluation. These problems may render State reports of 

reductions in the number of uninsured unreliable. 

SCHIP - Evaluation i OEI-05-00-00240 
Evaluations have conceptual and technical weaknesses 

States report, and our evaluations confirmed, that problems with data impair State 

evaluations. States often set progress goals without thoroughly conceptualizing how 

evaluations would be conducted. Program evaluation is generally a low priority for 

SCHIP administrators, and program staff often lack evaluation skills and training. State 

reporting also is impaired by unreliable data often not comparable among populations and 

based on small populations that move frequently in and out of the programs. State data is 

often incomplete. 

RECOMMENDATIONS 

The Health Care Financing Administration (HCFA) should develop a more specific 

framework for the content and structure of the reports States are required to 

submit. 

The HCFA should identify a core set of evaluation measures that will enable all SCHIP 

States to provide useful information. HCFA should also develop a more specific 

framework for the content and structure of State evaluations. 

HCFA and the Health Resources and Services Administration (HRSA) should 

provide guidance and assistance to States in conducting useful evaluations of 

their programs. 

To conduct more useful evaluations, SCHIP staff would benefit from assistance and/or 

training regarding data collections and evaluation, and resource determination. 

The HCFA and HRSA should assist States in building their capacity to collect required 

data, and work together to provide States with training and guidance. States would 

benefit from assistance regarding what data to collect, how to obtain specific data, how to 

determine if data is reliable, how to determine what the data is yielding, and how to 

evaluate the data. In addition, HCFA and HRSA should work with States to identify 

external sources of data and, as appropriate and necessary, work with these external 

sources to improve the reliability of data. 

Currently, poor coordination exists between SCHIP administration and the evaluation of 

SCHIPs. Program administrators indicated that their current focus is on enrolling children 

into the SCHIPs and, consequently, less attention is paid to evaluation. They also 

indicated that Federal reporting requirements are burdensome and funds for robust 

evaluations are lacking. The HCFA and HRSA should provide training to administrators 

and staff to help reduce the poor coordination between administration and evaluation, and 

make normal staff activities part of the evaluation process. The HCFA and HRSA should 

also work with States to identify, where needed, additional evaluation resources. 

SCHIP - Evaluation ii OEI-05-00-00240 
AGENCY COMMENTS 

The Health Care Financing Administration and the Health Resources and Services 

Administration provided formal comments to the draft report. Both concur with the 

findings and recommendations. The full text of their comments are included in 

Appendices A and B. 

SCHIP - Evaluation iii OEI-05-00-00240 
TABLE OF CONTENTS 

PAGE

EXECUTIVE SUMMARY ................................................ i

INTRODUCTION ..................................................... 1

FINDINGS

Questionable evaluations undermine State reports of success ............... 6

Evaluations have conceptual and technical weaknesses ................... 11

RECOMMENDATIONS ............................................... 15

APPENDICES

A: HCFA Comments ........................................... 18

B: HRSA Comments ........................................... 21

SCHIP - Evaluation OEI-05-00-00240 
INTRODUCTION INTRODUCTION 

PURPOSE 

To assess States’ evaluations of their performance goals, particularly those focused on 

reducing the number of uninsured children. 

BACKGROUND 

Many low-income children receive health care coverage through Medicaid and the State 

Children’s Health Insurance Program (SCHIP). Both Medicaid and SCHIP are joint 

State-Federal programs that allow States administrative discretion within guidelines 

established by statute and the U.S. Department of Health and Human Services. 

Medicaid, which was established in 1965, is the largest program providing health care 

services to America's poorest people. Within broad national guidelines set by the Federal 

Government, each State administers its own program, sets its own eligibility standards, 

determines the type, amount, duration and scope of services, and sets payment rates. 

Total non-administrative expenditures for the Medicaid program were $180.9 billion in 

1999 ($102.5 billion in Federal and $78.4 billion in State funds). The Health Care 

Financing Administration (HCFA) reports that in 1998, 51 percent of all Medicaid 

recipients were children. In 1998, Medicaid paid an average of $1,150 per child for the 

20.6 million children who received services through the program. 

The State Children’s Health Insurance Program and uninsured children 

Over 10 million children in the United States are uninsured. SCHIP is designed to help 

reduce the number of uninsured low-income children by providing them with health care 

coverage. Title XXI of the Social Security Act (SCHIP’s enabling legislation) was 

created by the Balanced Budget Act of 1997 (P.L. 105-33). The law appropriated nearly 

$40 billion over 10 years to help States expand health insurance to children whose families 

earn too much to qualify for Medicaid, yet not enough to afford private insurance. 

Like Medicaid, SCHIP is a State and Federal partnership, but the Federal match for 

SCHIP expenses is greater than the match for Medicaid. States have three options for 

covering uninsured children under Title XXI: design a separate SCHIP program, expand 

Medicaid eligibility, or a combination of the two strategies. As of October 1, 2000, 

SCHIP - Evaluation 1 OEI-05-00-00240 
15 States are administering separate SCHIPs, 23 run expanded Medicaid, and 18 have a 

combination of programs.1 This report focuses on 5 of the 15 States operating separate 

SCHIP programs. These States are Arizona, Colorado, Delaware, Georgia, Kansas, 

Montana, Nevada, North Carolina, Oregon, Pennsylvania, Utah, Vermont, Virginia, 

Washington, and Wyoming. 

Under Title XXI, a State must have an approved State plan for a fiscal year in order to 

receive an allotment that year. Each State submitted a SCHIP plan to HCFA, outlining 

program structures, use of funds, benefits, strategic objectives, and performance goals. 

Close to two million children who would otherwise be without health insurance coverage 

were enrolled in SCHIP in fiscal year 1999. This is double the number reported enrolled 

in 1998, the first full year of the program. Of the 56 State and territorial children's health 

insurance programs to date, 53 were operational during fiscal year 1999. Of the nearly 

two million children covered as of September 30, 1999, over 1.2 million children were 

enrolled in separate State-designed children's health insurance programs and almost 

700,000 were enrolled in Medicaid expansion plans. 

Ensuring correct enrollment in SCHIP and Medicaid 

To encourage States to expand child health insurance eligibility, the Federal match for 

States’ Title XXI expenditures is greater than the match for Medicaid. The average 

federal match for Title XXI is 71 percent, while the average Federal match for Medicaid is 

57 percent. Medicaid is an entitlement program, and States are required to enroll all 

eligible children. SCHIP is not an entitlement program and States have the discretion to 

cap enrollment for SCHIP eligibles or create waiting lists. So that States do not try to 

maximize SCHIP reimbursements by enrolling Medicaid eligibles in SCHIP, 

Title XXI requires States to screen SCHIP applicants for Medicaid eligibility. Children 

found eligible for Medicaid must be enrolled in Medicaid rather than in SCHIP. States 

receive only the Medicaid-level match for any child enrolled in Medicaid based on the 

eligibility standards that existed before Title XXI was enacted. Due to their effort to 

enroll children in new Title XXI funded programs, some States have noticed the so-called 

“woodwork effect”; by conducting outreach for SCHIP, they encourage greater numbers 

of Medicaid eligibles to apply for health care services as well. 

Statutory Requirement for States to Evaluate SCHIP 

Section 2107 of the SCHIP enabling legislation outlines the requirements for the 

programs’ strategic objectives and performance goals. However, the language is fairly 

broad, requiring only that each SCHIP State plan include a description of the program’s 

strategic objectives and performance goals. The strategic objectives must relate to 

increasing health coverage for low-income children. The State must establish at least one 

performance goal for each strategic objective. 

1 Currently, 50 States, 5 Territories and the District of Columbia administer Title XXI programs. 

SCHIP - Evaluation 2 OEI-05-00-00240
The SCHIP plans must also include the program performance measures that the State 

plans to use to assess the provision of health insurance to low-income children and other 

efforts to maximize health benefit coverage for low-income and other children. 

Performance measures must be measurable through objective, independently verifiable 

means. To determine State program performance, the measures are to be assessed against 

State goals described in the approved SCHIP plan. 

Section 2108(b) of the Social Security Act requires each State to evaluate its 

Title XXI program. Each January 1, States must provide an annual report to the

Secretary of the Department of Health and Human Services on the results of State

assessments. In addition, each State was required to submit an evaluation to the

Secretary by March 31, 2000. Information that the States provided to the Secretary in

their March 2000 reports will be used in the Secretary’s Report to Congress in 2001. A

contractor is currently analyzing State evaluations and will report to the Secretary on

States’ efforts. 

SCHIP Evaluation 

The annual SCHIP report and the March 2000 evaluation report cover some of the same 

information. For this reason, HCFA allowed States to submit the most recent annual 

report as part of the March 2000 report. All States utilized a single framework for their 

State evaluation reports. The National Academy for State Health Policy developed the 

framework for the report, along with a group of State officials and representatives from 

HCFA. Congressional staff, State program administrators, and child advocates reviewed 

and commented on drafts of the report framework. 

Evaluation reports submitted to HCFA in March 2000 were designed to provide 

information on the progress of State efforts to meet the strategic objectives they outlined 

in their SCHIP plans. The framework included a section for a summary description of key 

accomplishments in decreasing the number of uninsured low-income children. States were 

asked to provide information about how they developed estimates of insurance coverage, 

as well as how reliable they think their estimates are. 

In order to indicate progress made to achieve a State’s strategic objectives, the report 

framework includes a structure for States to report the data and methodology used to 

measure progress toward objectives and goals, and an assessment of any progress made. 

The report also includes significant background information on the SCHIP as well as local 

environmental factors and trends affecting the State program. States are asked to assess 

their experiences with enrollment, disenrollment, expenditures, access to care, and quality 

of care. The final section provides a chance for the State to reflect on successes and 

challenges facing the program. 

Strategic objectives. Title XXI allows States to establish their SCHIP strategic 

objectives. As outlined in their SCHIP State plans, the States in our sample established 

strategic objectives focused on reducing the number of uninsured children and improving 

access to care for children. 

SCHIP - Evaluation 3 OEI-05-00-00240 
State strategic objectives also relate to:

C increasing awareness of health care programs and options;

C conducting outreach and increasing public awareness of SCHIP;

C monitoring utilization of services; 

C improving health care services or outcomes; and

C encouraging service use by program participants.

Performance goals. Most States identified a single goal for each of their strategic

objectives. One sampled State went further, providing three to five goals for several of its

objectives. The goals were developed as part of the SCHIP plans and were often

developed by a group comprised of representatives from multiple State agencies. Two

States specifically set up SCHIP, and thus their goals for the program, to match their

existing Medicaid programs. 

Legislative Requirement for OIG Studies 

Section 703 of the Balanced Budget Refinement Act of 1999 required that the Office of 

Inspector General (OIG) sample States in order to: (1) evaluate whether States are 

correctly enrolling children in SCHIP or Medicaid as appropriate; and (2) assess the 

progress made by States to reduce the number of uninsured low-income children. The Act 

further required that the OIG evaluate the progress sampled States have made to achieve 

their strategic objectives and performance goals. Sampled States must be selected from 

States not using any SCHIP funds to expand Medicaid. 

This is one of two reports focusing on a sample of five States administering separate 

SCHIP. This report focuses on States’ evaluations of their performance goals. In a 

companion report entitled “State Children’s Health Insurance Plan - Ensuring Medicaid 

Eligibles are not Enrolled in SCHIP,” OEI-05-00-00241, we address State efforts to 

ensure Medicaid eligibles are not enrolled in SCHIP. 

METHODOLOGY 

Based on the requirements outlined in the Balanced Budget Refinement Act of 1999, we 

limited our universe to the 15 States utilizing their Title XXI allotments to operate 

separate child health programs. Two States, Wyoming and Washington, were excluded 

from the population because they began administering SCHIP in December 1999 and 

February 2000, respectively, and thus had no enrollees in fiscal year 1999. 

We used a 2-stage stratified-cluster sampling plan to select 5 States from the 13 remaining 

States. The remaining 13 States were divided among 2 strata. Pennsylvania was a selfrepresenting State as illustrated in Strata 1, based on the significantly large number of 

children enrolled in their SCHIP. From the remaining 12 States, we randomly selected 4 

States, North Carolina, Oregon, Utah, and Vermont, for Strata 2. The chart on the 

following page outlines the number of children enrolled in SCHIP in each of the Strata and 

sample States in Fiscal Year 1999. 

SCHIP - Evaluation 4 OEI-05-00-00240 
Selected Sample States 

STRATA 1 

States Selected Number of Children Enrolled in 

SCHIP in FY 1999 

Number of Cases Selected For 

Review 

Pennsylvania 81,758 100 

Totals 81,758 100 

STRATA 2

States Selected Number of Children Enrolled in 

SCHIP in FY 1999 

Number of Cases Selected For 

Review 

North Carolina 57,300 100 

Oregon 27,285 100 

Utah 13,040 100 

Vermont 2,055 100 

Totals 99,680 400 

We reviewed the 5 States’ Title XXI plans and the SCHIP evaluation reports submitted to 

HCFA in March 2000. We conducted on-site visits and met with staff at the agency 

administering the SCHIP in each of the five States. 

We conducted case file reviews and discussed enrollment issues and goal attainment with 

State staff. We reviewed 100 randomly selected active SCHIP case files in each State in 

order to evaluate whether ineligible children were enrolled in SCHIP. We provided each 

State with randomly selected case numbers and staff pulled these cases in preparation for 

our visit. The results of our case review were analyzed using a statistical analysis package 

that takes into account complex sample designs. 

One part of our discussions with SCHIP staff focused on State efforts to appropriately 

enroll children and safeguards used to ensure proper enrollment. We also discussed 

SCHIP evaluations. We collected documentation of State eligibility and enrollment 

practices and of evaluation efforts. 

We conducted our review in accordance with the Quality Standards for Inspections issued 

by the President’s Council on Integrity and Efficiency. 

SCHIP - Evaluation 5 OEI-05-00-00240 
FINDINGS FINDINGS 

Questionable evaluations undermine the reliability of State reports of 

success 

The five sampled States report success reducing the number of uninsured children, but 

they did not evaluate all of their goals. Further, the evaluations they conducted are 

impaired by their reliance on descriptive explanations, a lack of objective measurement, 

and problems with the data used in analysis. 

State reports of reductions in number of uninsured may be unreliable 

States report reductions in the number of uninsured children. Our analysis of States’ data 

and methods used leads us to question the reliability of State figures. 

Reports of reductions. The evaluation report in one of the sampled States notes that it is 

difficult to prove a causal relationship between increased SCHIP enrollment and a 

reduction in the number of uninsured. Evaluation reports in the other four States provide 

information on decreases in the number of uninsured children within the State and 

estimated pre-SCHIP baseline numbers of uninsured children. Two States estimate 

changes in the number of uninsured by comparing current (post-SCHIP implementation) 

estimates of uninsured to their baselines. The other States estimate reductions in the 

number of uninsured children by subtracting SCHIP enrollees from their original baseline 

estimates. The fifth State provides estimates of the total number of uninsured children and 

the number of uninsured SCHIP eligibles, and offers information about reductions in the 

number of uninsured children over the 1990's, but does not evaluate the impact the 

program has had on size of the uninsured population. 

Census and State data. Three States report baseline and current estimates of uninsured 

children based on Census data. One State gets its data from rolling averages of Census 

data. The State indicates that its degree of confidence in the estimate is tempered by 

Census data limitations that may impact data reliability. Another State utilizes SCHIP and 

Medicaid enrollment numbers and State insurance data (based in part on Census numbers) 

to estimate a reduction in the size of the uninsured population. The State credits the 

change to increases in SCHIP rolls because between the 1997 baseline and the 1999 

estimate, the percentage of low-income children on Medicaid remained constant, the 

percentage of children with other health insurance dropped, and the SCHIP program 

enrolled children for the first time. 

Three States, including one that also uses some Census data, base their estimates on Statecollected data. In one State, the Department of Health conducts a survey every 

5 years which includes information that allows estimates to be made of the uninsured 

population. Other States rely on a State health insurance survey, a State population 

survey, and work done by a local university. 

SCHIP - Evaluation 6 OEI-05-00-00240 
Size of change varies. State estimates of reductions in the number of uninsured children 

vary between the sampled States. It is difficult to compare the size of changes across 

States due to the differences between States in initial rates of uninsurance, overall State 

populations, and the size of SCHIP eligible populations. In the four States that offered 

both pre- and post-SCHIP implementation estimates, the change in the number of 

uninsured ranges from 5.8 to 21 percent. The State with the largest percent change in 

uninsured children had very low uninsurance rates even prior to SCHIP implementation. 

This State has both a small population and low initial and current number of uninsured, 

which makes the percent change in insurance so large. In addition, one State estimates the 

number of uninsured children under 200 percent of the Federal poverty level, while the 

others estimate the total number of uninsured children in their States. 

Estimating different numbers. The four States that assessed change in uninsurance 

rates each used data derived from different sources to show progress toward different 

target numbers. One State focused its assessment of enrollment and “crowd out” on the 

number of uninsured children in families with incomes below 200 percent of the Federal 

poverty level. A second State, using 3 year rolling averages of U.S. Census Survey data, 

reported the change in the total number of uninsured children in the State. 

Two other States each provided a baseline number of uninsured children from prior to 

implementation of SCHIP and subtracted the number of children enrolled in the program 

to show a decline in the percentage of uninsured children. Using this method, the State 

surpassed its projected decline in the number of children without insurance. The report 

did not indicate whether any children currently enrolled in SCHIP were enrolled in 

Medicaid or private insurance at the time the baseline uninsurance estimate was made. 

Both the State that did not provide information on changes in the number of uninsured 

children and one of the States that made a projection indicated they had difficulties making 

estimates because current information is not available. One State relies on insurance 

information from the State Department of Health that is updated every 5 years. This State 

reports that its increase in SCHIP enrollment was not matched by losses in private 

coverage, suggesting that children now enrolled in SCHIP were represented in the State’s 

earlier count of uninsured children. 

While States report success in enrolling children into SCHIP, not all States compared their 

progress to an established “goal” regarding enrollment or insurance coverage. One 

sampled State reports that it did not establish enrollment goals because it wants to enroll 

all eligible children. The State set benchmarks that it wanted to meet by certain dates but 

indicated that staff did not see these numbers as final enrollment goals. Another State 

indicated that it had not established an enrollment goal for the number of previously 

uninsured children but did use estimates of uninsured children and State resources to 

establish an initial enrollment goal. 

States report mixed success meeting other goals 

Just as States were allowed to design and assess their own strategic objectives and 

performance goals, they also had free reign to describe their progress meeting them. 

SCHIP - Evaluation 7 OEI-05-00-00240 
The 5 States we sampled provided evaluations or descriptions of 57 percent of the 

42 goals established among them. While State evaluations showed that some evaluated

more of their goals than others, looking only at the percentage of goals assessed does not

provide full information. The State that provided progress reports on all of its goals had

poorly developed goals and did not provide a full evaluation of those goals. The State

with the most unevaluated goals has more overall goals and, to a much greater extent than

the other States, has rigorous goals that can be evaluated quantitatively.

States report success meeting between 67 and 100 percent of the goals on which they

provided information. States reported the most success meeting goals that were assessed

with simple quantitative measures -- particularly enrollment rates. 

Enrollment rates. All five State evaluation reports provided information about increases

in SCHIP enrollment. As States differ in their population size, initial rate of uninsured

children, eligibility rules, and other program specifics, it is impossible to compare States’

level of success enrolling children into SCHIP. Further, each State defined its own goals,

so enrollment targets are not comparable among States. Three States met their selfdefined enrollment goals, while a fourth did not. The fourth State’s evaluation report

explains that, although the program enrolled more than the number of children specified in

its enrollment goal, a larger than expected number of children disenrolled during the

program’s existence. The fifth State did not set an enrollment goal, but instead used its

enrollment numbers to show progress toward its goal of reducing the number of uninsured

low-income children in the State. 

One State in the sample established goals to increase access and improve service

coordination by enrolling SCHIP children into managed care. The State was unable to

meet these goals because the contracted managed care organizations shut down operations

in the State. This forced the State to create its own managed care plan. During

development, the State halted new managed care enrollments. The State did not evaluate

how enrollment in managed care impacted access or service coordination. 

Outreach. Three sampled States analyzed goals aimed at developing and implementing

outreach plans for SCHIP and other health care programs for low-income children. These

States used their reports to describe, rather than evaluate, their outreach programs. This

may be in part because evaluating the success of outreach efforts is difficult and inherently

somewhat subjective. These States focus more on their successful implementation of

planned efforts than on whether the outreach programs encouraged enrollment in SCHIP

and Medicaid. 

Another State’s outreach goal stressed the role State agencies should play in executing

outreach strategies. While agency participation was not analyzed, the State commissioned

a survey to determine whether outreach activities increased awareness 

of SCHIP and Medicaid among targeted families. The survey found public awareness of 

SCHIP rose considerably from before the media campaign to 6 months after the

campaign’s initiation. 

SCHIP - Evaluation 8 OEI-05-00-00240 
Utilization and access. Two States tried to evaluate service utilization and all five States 

attempted to measure access to care. Both types of effort met with mixed success. 

One State’s goals included monitoring enrollees’ utilization patterns, including their use of 

primary care and well child visits. The evaluation report indicated that the level of primary 

care visits was comparable with national standards, but the rate of well care visits was 

lower than State-wide goals for all children. In addition, the State did not evaluate its 

other utilization goals, such as immunization rates, dental care, and client satisfaction. 

Another State surveyed SCHIP families about whether enrolled children had a primary 

source of care, whether children were able to get needed care, and to what extent families 

were satisfied with their children’s health care. The survey indicated that the State is 

meeting its goals in this area, as most children had a primary source of care, few children 

were unable to access health care, and an overwhelming majority of respondents were 

satisfied with their children’s health care. The State indicates that as part of its next 

SCHIP survey, the State will administer the tool it developed to measure “crowd-out.” 

Other progress. All five State evaluations provided information about progress toward 

their goals in several other areas. One State assessed its attempts to improve enrollment 

practices by measuring what percentage of SCHIP applications they received through the 

mail or non-traditional sources. While the State did not meet its 50 percent goal, this 

assessment provided information about which applicants apply using the different 

methods. Another State described its efforts to expand its eligibility rules, restructure its 

data and operating systems, develop a unique coding system for SCHIP enrollees, and 

train staff to run the program. None of these descriptions are strictly evaluative, but they 

do provide updates on the activities taken by the SCHIP program administrators. 

State reports are not very evaluative 

SCHIP programs are required by statute to evaluate their programs. Section 2107 of 

Title XXI of the Social Security Act requires that SCHIP plans include strategic objectives 

and performance goals, and that measures be established to evaluate program 

performance. Performance measures must be “(A) measured through objective, 

independently verifiable means, and (B) compared against performance goals, in order to 

determine the State’s performance.” 

Section 2108 of the Act outlines that each SCHIP must submit an annual assessment of 

“the operation of the State plan . . . including the progress made in reducing the number of 

uncovered low-income children.” Evaluations must assess “the effectiveness of the plan in 

increasing the number of children with creditable health coverage.” Additionally, each 

State must describe and analyze the effectiveness of elements of the State plan. 

Much of the information provided in State evaluations is qualitative and subjective. Four 

of the five States extensively utilize qualitative descriptions in their evaluation reports to 

HCFA. Descriptive responses generally provide information about State efforts without 

assessing what impact efforts had on attaining specific goals. Further, any evaluation of 

such a program would be subjective and could not be independently verified. None of the 

sampled States attempted any evaluation of their outreach programs or offered an 

SCHIP - Evaluation 9 OEI-05-00-00240 
explanation of how such programs impacted their measurable progress in enrollment or 

the level of uninsured children. 

Several States provided information about what happened but were not able to explain 

why it happened. The sampled States have goals stating their outreach and other efforts 

will increase access to care, raise enrollment, or decrease the number of uninsured children 

in the State. State staff were unable to express how they chose their goals. 

Despite goals that suggest State efforts and outcomes will be measured, several States 

provide information about SCHIP and Medicaid enrollment increases without offering an 

explanation of how enrollment is impacted by their outreach or other efforts. For 

example, one State goal outlines the State’s intention to increase State government 

participation in outreach activities in order to increase access to health insurance. The 

evaluation provides information about the overall increase in SCHIP enrollment in the 

State but does not explain whether the goal (increasing government outreach efforts) was 

met or even what efforts were attempted. Further, the evaluation report does not indicate 

how such State efforts could be seen as causing the increased enrollment rates the State 

offers in its progress summary. 

State evaluations rely heavily on enrollment data and descriptive information 

Sampled States utilize a variety of data but rely most on enrollment and descriptive data. 

All five States used enrollment data to assess enrollment increases, decreases in 

uninsurance levels, and increases in access to coverage. On the whole, States were able to

assess the goals that rely on enrollment data. 

Some States also utilized survey and health care utilization data to evaluate their SCHIP

progress. Two States utilized participant surveys to evaluate their goals, while another

plans a survey, but has not implemented it yet. State surveys were used to identify

improved access to care and increased public awareness of SCHIP and Medicaid. Planned

surveys will measure “crowd out,” use of preventative care, and client satisfaction. 

One State compared Medicaid and SCHIP enrollees’ utilization patterns for primary care

and well child visits to National Committee for Quality Assurance standards. Three other

States planned to evaluate utilization data, but did not do this for their 

March 2000 evaluation reports, in part because data was not available or did not exist in a

format that allowed its analysis or comparison to other health information. 

State reports tend to be descriptive rather than evaluative 

While three States provide descriptions of SCHIP efforts, the information does not include 

assessments of the impact of State efforts on goals. Descriptions of expanded program 

rules, outreach efforts, and anti-“crowd-out” policies do not indicate how these efforts 

impact access to care or children’s health status. One State uses its evaluation report to 

describe the changes to State program rules and procedures that established 

SCHIP - Evaluation 10 OEI-05-00-00240 
SCHIP. As a summary of its progress in improving service provision through managed 

care enrollment, another State explains its use of an automated referral process. 

States’ descriptive progress summaries do not evaluate State efforts. States tend to 

employ such summaries when they do not have the skills or the available data to analyze 

the work they have done or show progress toward their goals. While they provide some 

useful information about SCHIP, they are not a viable alternative to concrete evaluations 

of the impact of State efforts on program goals. 

Evaluations have conceptual and technical weaknesses 

As described in the previous finding, State evaluations are flawed; some goals were not 

evaluated, and evaluations of others lacked objective measures of success. The reports 

were impacted by issues related to goal setting, staff skills and resources, program 

priorities, and data problems. States are aware of some of the flaws in their assessments 

and report barriers to conducting better evaluations. 

Some goals were set without evaluation in mind 

State goals are not always easy to evaluate, in part because the staff who crafted the goals 

often did not consider what it would take to evaluate them. Most States established their 

goals, as required, in their State plans. The State plans were often written by groups made 

up of representatives from different agencies, each with their own priorities for children’s 

health. While this can help the State craft a more robust program, it also may mean that 

individuals not responsible for program implementation or evaluation establish goals 

without thought to how they will later be assessed. Sample States reported that their 

goals were designed at the time the State plan was written, but an explanation of how 

evaluation would occur was not developed. One State specifically noted that they 

established their plan without thinking about goal assessment. 

Not considering how goals would be evaluated led to the creation of unmeasurable goals. 

One State reported they used a goal taken directly from the State plan but are now not 

able to measure it because information is not available, and definitions that would allow 

comparisons across populations do not exist. One respondent reported that he is not sure 

how to measure some of the State’s goals and wondered how he should go about 

measuring coordination. Further, as the goal does not define what type of coordination 

should be assessed, he was not sure with what other agencies and organizations the 

program should be measuring the level of coordination. 

A third State admitted that its goals may not be entirely measurable. For example, one 

goal is to improve access to care by increasing State government participation in the 

program. The analysis shows increased enrollment but does not explain what government 

participation has occurred or how that participation impacted enrollment or access. When 

asked whether the program intended to evaluate this connection, respondents reported 

that they did not think they could quantify this. 

SCHIP - Evaluation 11 OEI-05-00-00240 
They further reported that they have no indicators to evaluate efforts and no interest in 

developing any. 

Program administrators do not make evaluation a priority 

The ultimate objective for SCHIP is to increase enrollment in health insurance programs to 

ensure that children receive appropriate health care services that maintain or improve 

health. Evaluation is important to ensure that State programs are meeting identified needs 

and are spending their allotments effectively, but ultimately program administrators are 

much more focused on running the programs than on assessing their success. Resources 

for evaluation are limited, particularly for smaller States which often feel constricted by 

the 10 percent cap on administrative activities. 

Staff in one State reported they are not sure when they will find time to decide what 

indicators they will use to assess quality. They indicated that they will probably default to 

using established measures because they have limited staff and are more focused on 

running the program than on measuring quality. 

Another sampled State reported doing a good job meeting its goals but is not doing a 

good job measuring its success. In part this is because States that believe they are 

successful may not be strongly motivated to evaluate themselves. Respondents in this 

State indicated that they are bringing a lot of children into the program and staff do not 

feel the need to assess what outreach efforts are working. They think that as long as they 

continue to enroll children, they will not worry about changing things. 

Even when States are not as convinced that they are succeeding, evaluation is generally a 

low priority for SCHIP. When asked about whether they planned to evaluate their stated 

goals more fully than they had for the March 2000 evaluation report, staff at one State 

told us that they did not plan to further assess their reported goals. They did not think it 

would be worth the required effort to track down the connections between their actions 

and the outcomes they are seeing in the program. Another State that is more focused on 

evaluation reported that although they believe their goals are important, it is expensive to 

evaluate a large number of goals. The cost and time required to evaluate multiple 

progress goals is slowing down evaluation efforts. 

Program staff lack evaluation skills and training 

When State respondents were asked about evaluation issues such as how goals were 

developed or how assessments were conducted, they were often unable to provide 

meaningful information. Program staff were much more comfortable discussing the 

content of their goals than they were talking about how they decided to establish the goals 

or how they assessed progress. In several discussions, State staff were unable to explain 

why they thought an evaluation showed what it was intended to show. 

For the most part, SCHIP staff are not trained evaluators. Their skills lie where they are 

needed on a daily basis -- in program management and ensuring the effective and efficient 

administration of their programs. The language of evaluation is also foreign to 

SCHIP - Evaluation 12 OEI-05-00-00240 
many program administrators. Without a more thorough understanding of evaluation 

concepts and practices, program staff cannot be expected to turn out detailed assessments 

of program activities and outcomes. 

The lack of trained evaluation staff may be a particular problem for States in which 

Medicaid and SCHIP are administered separately. SCHIP staffs are often small, and 

program administrators have difficulty hiring evaluation staff when program staff are 

needed. 

Problems with data impair evaluations 

Insufficient or flawed data caused problems for the sample States. States faced difficulties 

regarding unreliable data, missing data, non-comparable data sets, and small population 

sizes. 

Unreliable data. Several States indicated that they rely on Census data to make estimates 

of uninsured children and eligible children in the State. The use of Census data is widely 

criticized, particularly for States with smaller populations. One State reported that it’s 

estimate of total SCHIP eligibles comes from Census Current Population Survey (CPS) 

data on children below 200 percent of the Federal poverty level. This State told us that 

staff fear the CPS numbers are “no good.” This concern is in large part based on the poor 

record of CPS estimates of low-income children in the State. The CPS data indicated that 

there were 150,000 children under 150 percent of poverty in the State, despite the fact 

that more than twice this many children are currently enrolled in Medicaid. The State 

changed their process for estimating the number of uninsured children, but their new 

formula still relies on the CPS for some parts of the calculation. They know the data is 

flawed but have no other data on which to base their estimates. 

Another State relying on Census data reported frustration with the quality of Census 

information for small populations. State staff do not trust the Census data and have found 

that other population-based studies are not reliable for small States. This State reports its 

faith in the estimate of SCHIP eligible children in the State as “5” on a 1 (least reliable) to 

10 (most reliable) scale. 

Data collected by the States themselves also may be deficient. A State that relies on a 

health status survey administered by the State Department of Health expressed frustration 

with the age of the data. The survey, conducted every 5 years, provides baseline 

information on the number of uninsured children. Program administrators worry that the 

data is unreliable because it is old by the time they get it. Respondents from this State 

reported that on a scale from 1 to 10, they rated this data between 5 and 6. 

The SCHIP administrator used State-collected data to calculate a new uninsured rate by 

subtracting the total number of SCHIP enrollees from an estimate of uninsured derived 

from survey data. The State administrator admits that this methodology assumes that all 

SCHIP enrollees were previously uninsured and would be uninsured without the program, 

but reports that the State does not have better numbers or methods on which to base 

calculations. 

SCHIP - Evaluation 13 OEI-05-00-00240 
Missing data. Obtaining accurate and meaningful utilization data is a challenge for 

States, which means that any subsequent analysis may be incomplete. States often rely on 

the contracting managed care organizations for data on use of services. Although States 

contractually require participating managed care organizations to provide data, the receipt 

of this data may not be timely. One State reported that encounter data they used to 

calculate use of preventative services is 25 percent incomplete. Data from the managed 

care contractors is missing, despite the State’s continued efforts to get complete data from 

the plans. 

States may also have problems getting data from other agencies within their own 

department. One SCHIP program adopted a goal for immunization rates at the request of 

the Department of Health, which coordinates a State-wide immunization campaign. 

Another agency within the Department that administers SCHIP collects the immunization 

information, but its data collection system is not very mature. 

Non-comparability. Comparing managed care and fee-for-service data was a problem 

for one State. The State intended to assess whether children received appropriate medical 

screenings but found that comparing children enrolled in different service types would 

require reorganizing the data, as utilization data from the SCHIP fee-for-service system is 

presented differently from their Medicaid managed care data. This State noted that 

Medicaid’s managed care program is oriented toward prevention. This led to the 

establishment of specific data elements for health screening and other prevention services. 

The State set up its SCHIP program on a fee-for-service model that provides less specific 

information about the delivery of prevention services to children. The SCHIP does not 

have a single code for health screening similar to the one used in the Medicaid program. 

The administrator indicated that her staff is working on making the fee-for-service and 

managed care data consistent, but that they might have to abandon the goal if they cannot 

compare the two groups. 

Small population. SCHIP is new, and even with strong enrollment, there may not be 

sufficient children in a State program to perform robust analysis. One State’s utilization 

analysis were conducted for all children enrolled in State-sponsored health care. The State 

felt that there were too few children in SCHIP alone to allow for reliable evaluations, 

especially when information on specific age or other categories is required. 

Another State report was based on 1 year of data. Few children were enrolled at the start 

of the year, and enrollment increased dramatically during that first year. The State 

administrator reports that because they experienced such a dramatic change in enrollment 

over a short time period, they cannot accurately determine the number of children who 

should make up the denominator for their health screening goal. Without this information, 

the State cannot assess whether the target number was met. 

SCHIP - Evaluation 14 OEI-05-00-00240 
RECOMMENDATIONS 

Over the past 3 years, States have been concentrating their efforts on getting the SCHIP 

program organized and operating by focusing on outreach and enrollment. There is now a 

need for States to shift their focus to evaluating the effectiveness of program 

administration and progress made towards the achievement of each State’s goals and 

objectives, as outlined in their State Plans. While State administrators believe they are 

successfully enrolling eligible children and reducing the overall number of uninsured 

children, they admit that they do not evaluate their efforts well. 

Improving States’ ability to conduct useful evaluations may require a multi-stage process. 

Initial stages should focus on improving State program staffs’ general understanding of 

evaluations, including the distinction between simply describing a process and true 

evaluation. Later stages should focus on staff knowledge of different types of evaluation 

and how these evaluation types can be used. In addition, SCHIP staff should become 

knowledgeable in preparing data for collection, determining the types of data that should 

be collected for various analysis, and evaluating data once it is collected. 

Federal regulations and policy provide little guidance to States on what constitutes an 

acceptable evaluation. Without more specific guidance about what SCHIP evaluations 

should contain and assistance conducting assessments, the information States provide does 

little to ensure that Federal Title XXI dollars are used to the greatest possible advantage. 

To increase the usefulness of SCHIP evaluations and facilitate the ability of SCHIP staff to 

conduct such improved evaluations: 

HCFA should identify a core set of evaluation measures and develop a more 

specific framework for the content and structure of the reports States are 

required to submit. 

SCHIP staff need direction from HCFA on what kind of information it is seeking from 

States. The HCFA should identify a core set of evaluation measures that will enable all 

SCHIP States to provide useful information, independent of the way each State operates 

their SCHIP. The HCFA can do this by determining what information it needs evaluations 

to yield, how detailed this information must be, and what purpose the evaluations will 

serve. Providing this guidance will enable States to submit information best suited to 

address HCFA’s issues and concerns. 

The HCFA should develop a more specific framework for the content and structure of 

State evaluations. The HCFA should continue working with the National Academy for 

State Health Policy to create a template that provides States direction on how to address 

issues uniformly, leaving little room for ambiguity. 

SCHIP - Evaluation 15 OEI-05-00-00240 
HCFA and the Health Resources and Services Administration (HRSA) should 

provide guidance and assistance to States in conducting useful evaluations 

of their programs. 

To conduct more useful evaluations, SCHIP staff would benefit from assistance and/or 

training regarding data collection and evaluation, and resource determination. 

Data Collection and Evaluation - The HCFA and HRSA should assist States in building 

their capacity to collect required data. The need for quality data is intensifying as Federal, 

State, and local governments rely more heavily on program evaluations to make important 

programmatic decisions. However, States are finding those evaluations difficult due to 

limited data capacity and a lack of reliable and useful data. 

The HCFA and HRSA should work together to provide States training and guidance in 

the following areas: 1) what data to collect, 2) how to obtain specific data, 3) how to 

determine if the data is reliable, 4) how to determine what the data is yielding, and 5) how 

to evaluate the data. In addition, HCFA and HRSA should work with States to identify 

external sources of data and, as appropriate and necessary, work with these external 

sources to improve the reliability of data. For example, HCFA and other Federal agencies 

are working with the Census Bureau to increase the overall quality of CPS data. Because 

of States’ reliance on CPS data to conduct evaluations, this and similar efforts would 

increase States access to quality data without requiring large resource commitments from 

States. 

Resource Determination - Currently, poor coordination exists between SCHIP 

administration and the evaluation of SCHIPs. Program administrators indicated that their 

current focus is on enrolling children into the SCHIPs and, consequently, less attention is 

paid to evaluation. They also indicated that Federal reporting requirements are 

burdensome and funds for robust evaluations are lacking. The HCFA and HRSA should 

train administrators and staff to help reduce the poor coordination between administration 

and evaluation and make normal staff activities part of the evaluation process. 

Existing staff resources may not allow some States to fully address their needs to conduct 

useful and meaningful evaluations. Therefore, HCFA and HRSA should work with States 

to identify, where needed, additional evaluation resources. Depending on the extent of 

this need, HCFA and HRSA may want to explore various options, including offering 

States matching funds, creating set-aside funds within SCHIP for evaluations, increasing 

the amount of SCHIP funds that can be used for administration (i.e., evaluation), or 

tapping into available Departmental evaluation funds. 

Agency Comments 

The Health Care Financing Administration and the Health Resources and Services 

Administration provided formal comments to the draft report. Both concur with the 

findings and recommendations. The full text of the their comments are included in 

Appendices A and B. 

SCHIP - Evaluation 16 OEI-05-00-00240 
The HCFA provided information regarding their work with States. The HCFA also noted 

they will continue to work in partnership with the National Academy for State Health 

Policy to develop a framework for Title XXI annual reports. The HRSA provided 

comments to clarify portions of the report. We made appropriate changes based on the 

comments we received. 

SCHIP - Evaluation 17 OEI-05-00-00240 
APPENDIX A 

SCHIP - Evaluation 18 OEI-05-00-00240 
SCHIP - Evaluation 19 OEI-05-00-00240 
SCHIP - Evaluation 20 OEI-05-00-00240 
APPENDIX B 

SCHIP - Evaluation 21 OEI-05-00-00240 
SCHIP - Evaluation 22 OEI-05-00-00240