Department of Health and Human Services 
OFFICE OF 
INSPECTOR GENERAL 
STATE CHILDRENS HEALTH 
INSURANCE PROGRAM 
Assessment of State Evaluations Reports 
FEBRUARY 2001 
OEI-05-00-00240  OFFICE OF INSPECTOR GENERAL 
The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, as 
amended by Public Law 100-504, is to protect the integrity of the Department of Health and 
Human Services programs as well as the health and welfare of beneficiaries served by them.  This 
statutory mission is carried out through a nationwide program of audits, investigations, 
inspections, sanctions, and fraud alerts.  The Inspector General informs the Secretary of program 
and management problems and recommends legislative, regulatory, and operational approaches to 
correct them. 
Office of Evaluation and Inspections 
The Office of Evaluation and Inspections (OEI) is one of several components of the Office of 
Inspector General.  It conducts short-term management and program evaluations (called 
inspections) that focus on issues of concern to the Department, the Congress, and the public.  The 
inspection reports provide findings and recommendations on the efficiency, vulnerability, and 
effectiveness of departmental programs. 
OEI's Chicago regional office prepared this report under the direction of William Moran, Regional 
Inspector General and Natalie Coen, Deputy Regional Inspector General.  Principal OEI staff 
included: 
REGION  HEADQUARTERS 
Deborah Walden, Team Leader  Joan Richardson, Program Specialist

Nora Leibowitz, Project Leader

Michael Craig, Program Analyst

To obtain copies of this report, please call the Chicago Regional Office at (312) 353-9867. 
Reports are also available on the World Wide Web at our home page address: 
http://www.hhs.gov/oig/oei/  EE XX EE CC UU TT II VV EE     SS UU MM MM AA RR YY  
PURPOSE 
To assess States evaluations of their performance goals, particularly those focused on 
reducing the number of uninsured children. 
BACKGROUND 
Many low-income children receive health care coverage through Medicaid and the State 
Childrens Health Insurance Program (SCHIP).  SCHIP is designed to reduce the number 
of uninsured children by offering health care coverage to previously uninsured children. 
The program targets children with family income too high for Medicaid and too low to 
afford private insurance. 
States annually report the results of SCHIP assessments to the Secretary of the 
Department of Health and Human Services.  States also submitted evaluations of their 
programs and progress meeting goals to the Secretary by March 31, 2000. 
The Balanced Budget Refinement Act of 1999 requires that the Office of Inspector 
General (OIG) sample States in order to: (1) evaluate whether States are correctly 
enrolling children in SCHIP or Medicaid, and (2) assess States progress reducing the 
number of uninsured low-income children.  The Act further requires the OIG to evaluate 
the progress sampled States have made to achieve their strategic objectives and 
performance goals.  Sampled States must be selected from States only using SCHIP funds 
to operate SCHIPs separate from their Medicaid programs and cannot be selected from 
States using SCHIP funds to expand their Medicaid programs. 
This is one of two reports on five States administering separate child health programs. 
This report focuses on five States evaluations of their performance goals.  In the 
companion report, we address State efforts to ensure that Medicaid eligibles are not 
enrolled in SCHIP. 
FINDINGS 
Questionable evaluations undermine the reliability of State reports of success 
Sampled States report reductions in the number of uninsured children and mixed success 
meeting other goals.  SCHIP reports rely heavily on enrollment data and tend to use 
descriptive information in lieu of evaluation.  These problems may render State reports of 
reductions in the number of uninsured unreliable. 
i 
SCHIP - Evaluation  OEI-05-00-00240  Evaluations have conceptual and technical weaknesses 
States report, and our evaluations confirmed, that problems with data impair State 
evaluations.  States often set progress goals without thoroughly conceptualizing how 
evaluations would be conducted.  Program evaluation is generally a low priority for 
SCHIP administrators, and program staff often lack evaluation skills and training.  State 
reporting also is impaired by unreliable data often not comparable among populations and 
based on small populations that move frequently in and out of the programs.  State data is 
often incomplete. 
RECOMMENDATIONS 
The Health Care Financing Administration (HCFA) should develop a more specific 
framework for the content and structure of the reports States are required to 
submit. 
The HCFA should identify a core set of evaluation measures that will enable all SCHIP 
States to provide useful information.  HCFA should also develop a more specific 
framework for the content and structure of State evaluations. 
HCFA and the Health Resources and Services Administration (HRSA) should 
provide guidance and assistance to States in conducting useful evaluations of 
their programs. 
To conduct more useful evaluations, SCHIP staff would benefit from assistance and/or 
training regarding data collections and evaluation, and resource determination. 
The HCFA and HRSA should assist States in building their capacity to collect required 
data, and work together to provide States with training and guidance.  States would 
benefit from assistance regarding what data to collect, how to obtain specific data, how to 
determine if data is reliable, how to determine what the data is yielding, and how to 
evaluate the data.  In addition, HCFA and HRSA should work with States to identify 
external sources of data and, as appropriate and necessary, work with these external 
sources to improve the reliability of data. 
Currently, poor coordination exists between SCHIP administration and the evaluation of 
SCHIPs.  Program administrators indicated that their current focus is on enrolling children 
into the SCHIPs and, consequently, less attention is paid to evaluation.  They also 
indicated that Federal reporting requirements are burdensome and funds for robust 
evaluations are lacking.  The HCFA and HRSA should provide training to administrators 
and staff to help reduce the poor coordination between administration and evaluation, and 
make normal staff activities part of the evaluation process.  The HCFA and HRSA should 
also work with States to identify, where needed, additional evaluation resources. 
ii 
SCHIP - Evaluation  OEI-05-00-00240  AGENCY COMMENTS 
The Health Care Financing Administration and the Health Resources and Services 
Administration  provided formal comments to the draft report.  Both concur with the 
findings and recommendations.  The full text of their comments are included in 
Appendices A and B. 
iii 
SCHIP - Evaluation  OEI-05-00-00240  T A B L E   O F   C O N T E N T S  
PAGE

EXECUTIVE SUMMARY  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  i

INTRODUCTION  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  1

FINDINGS

Questionable evaluations undermine State reports of success  . . . . . . . . . . . . . . .  6

Evaluations have conceptual and technical weaknesses . . . . . . . . . . . . . . . . . . .  11

RECOMMENDATIONS  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  15

APPENDICES

A: HCFA Comments  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  18

B: HRSA Comments  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  21

SCHIP - Evaluation  OEI-05-00-00240  II NN TT RR OO DD UU CC TT II OO NN  
PURPOSE 
To assess States evaluations of their performance goals, particularly those focused on 
reducing the number of uninsured children. 
BACKGROUND 
Many low-income children receive health care coverage through Medicaid and the State 
Childrens Health Insurance Program (SCHIP).  Both Medicaid and SCHIP are joint 
State-Federal programs that allow States administrative discretion within guidelines 
established by statute and the U.S. Department of Health and Human Services. 
Medicaid, which was established in 1965, is the largest program providing health care 
services to America's poorest people.  Within broad national guidelines set by the Federal 
Government, each State administers its own program, sets its own eligibility standards, 
determines the type, amount, duration and scope of services, and sets payment rates. 
Total non-administrative expenditures for the Medicaid program were $180.9 billion in 
1999 ($102.5 billion in Federal and $78.4 billion in State funds).  The Health Care 
Financing Administration (HCFA) reports that in 1998, 51 percent of all Medicaid 
recipients were children.  In 1998, Medicaid paid an average of $1,150 per child for the 
20.6 million children who received services through the program. 
The State Childrens Health Insurance Program and uninsured children 
Over 10 million children in the United States are uninsured.  SCHIP is designed to help 
reduce the number of uninsured low-income children by providing them with health care 
coverage.  Title XXI of the Social Security Act (SCHIPs enabling legislation) was 
created by the Balanced Budget Act of 1997 (P.L. 105-33).  The law appropriated nearly 
$40 billion over 10 years to help States expand health insurance to children whose families 
earn too much to qualify for Medicaid, yet not enough to afford private insurance. 
Like Medicaid, SCHIP is a State and Federal partnership, but the Federal match for 
SCHIP expenses is greater than the match for Medicaid.  States have three options for 
covering uninsured children under Title XXI:  design a separate SCHIP program, expand 
Medicaid eligibility, or a combination of the two strategies.  As of October 1, 2000, 
1 
SCHIP - Evaluation  OEI-05-00-00240  15 States are administering separate SCHIPs, 23 run expanded Medicaid, and 18 have a 
combination of programs.1  This report focuses on 5 of the 15 States operating separate 
SCHIP programs.  These States are Arizona, Colorado, Delaware, Georgia, Kansas, 
Montana, Nevada, North Carolina, Oregon, Pennsylvania, Utah, Vermont, Virginia, 
Washington, and Wyoming. 
Under Title XXI, a State must have an approved State plan for a fiscal year in order to 
receive an allotment that year.  Each State submitted a SCHIP plan to HCFA, outlining 
program structures, use of funds, benefits, strategic objectives, and performance goals. 
Close to two million children who would otherwise be without health insurance coverage 
were enrolled in SCHIP in fiscal year 1999.  This is double the number reported enrolled 
in 1998, the first full year of the program.  Of the 56 State and territorial children's health 
insurance programs to date, 53 were operational during fiscal year 1999.  Of the nearly 
two million children covered as of September 30, 1999, over 1.2 million children were 
enrolled in separate State-designed children's health insurance programs and almost 
700,000 were enrolled in Medicaid expansion plans. 
Ensuring correct enrollment in SCHIP and Medicaid 
To encourage States to expand child health insurance eligibility, the Federal match for 
States Title XXI expenditures is greater than the match for Medicaid.  The average 
federal match for Title XXI is 71 percent, while the average Federal match for Medicaid is 
57 percent.  Medicaid is an entitlement program, and States are required to enroll all 
eligible children.  SCHIP is not an entitlement program and States have the discretion to 
cap enrollment for SCHIP eligibles or create waiting lists.  So that States do not try to 
maximize SCHIP reimbursements by enrolling Medicaid eligibles in SCHIP, 
Title XXI requires States to screen SCHIP applicants for Medicaid eligibility.  Children 
found eligible for Medicaid must be enrolled in Medicaid rather than in SCHIP.  States 
receive only the Medicaid-level match for any child enrolled in Medicaid based on the 
eligibility standards that existed before Title XXI was enacted.  Due to their effort to 
enroll children in new Title XXI funded programs, some States have noticed the so-called 
woodwork effect; by conducting outreach for SCHIP, they encourage greater numbers 
of Medicaid eligibles to apply for health care services as well. 
Statutory Requirement for States to Evaluate SCHIP 
Section 2107 of the SCHIP enabling legislation outlines the requirements for the 
programs strategic objectives and performance goals.  However, the language is fairly 
broad,  requiring only that each SCHIP State plan include a description of the programs 
strategic objectives and performance goals.  The strategic objectives must relate to 
increasing health coverage for low-income children.  The State must establish at least one 
performance goal for each strategic objective. 
1 
Currently, 50 States, 5 Territories and the District of Columbia administer Title XXI programs. 
2 
SCHIP - Evaluation  OEI-05-00-00240 The SCHIP plans must also include the program performance measures that the State 
plans to use to assess the provision of health insurance to low-income children and other 
efforts to maximize health benefit coverage for low-income and other children. 
Performance measures must be measurable through objective, independently verifiable 
means.  To determine State program performance, the measures are to be assessed against 
State goals described in the approved SCHIP plan. 
Section 2108(b) of the Social Security Act requires each State to evaluate its 

Title XXI program.  Each January 1, States must provide an annual report to the

Secretary of the Department of Health and Human Services on the results of State

assessments.  In addition, each State was required to submit an evaluation to the

Secretary by March 31, 2000.  Information that the States provided to the Secretary in

their March 2000 reports will be used in the Secretarys Report to Congress in 2001.  A

contractor is currently analyzing State evaluations and will report to the Secretary on

States efforts. 

SCHIP Evaluation 
The annual SCHIP report and the March 2000 evaluation report cover some of the same 
information.  For this reason, HCFA allowed States to submit the most recent annual 
report as part of the March 2000 report.  All States utilized a single framework for their 
State evaluation reports.  The National Academy for State Health Policy developed the 
framework for the report, along with a group of State officials and representatives from 
HCFA.  Congressional staff, State program administrators, and child advocates reviewed 
and commented on drafts of the report framework. 
Evaluation reports submitted to HCFA in March 2000 were designed to provide 
information on the progress of State efforts to meet the strategic objectives they outlined 
in their SCHIP plans.  The framework included a section for a summary description of key 
accomplishments in decreasing the number of uninsured low-income children.  States were 
asked to provide information about how they developed estimates of insurance coverage, 
as well as how reliable they think their estimates are. 
In order to indicate progress made to achieve a States strategic objectives, the report 
framework includes a structure for States to report the data and methodology used to 
measure progress toward objectives and goals, and an assessment of any progress made. 
The report also includes significant background information on the SCHIP as well as local 
environmental factors and trends affecting the State program.  States are asked to assess 
their experiences with enrollment, disenrollment, expenditures, access to care, and quality 
of care.  The final section provides a chance for the State to reflect on successes and 
challenges facing the program. 
Strategic objectives.  Title XXI allows States to establish their SCHIP strategic 
objectives.  As outlined in their SCHIP State plans, the States in our sample established 
strategic objectives focused on reducing the number of uninsured children and improving 
access to care for children. 
3 
SCHIP - Evaluation  OEI-05-00-00240  State strategic objectives also relate to:

C  increasing awareness of health care programs and options;

C  conducting outreach and increasing public awareness of SCHIP;

C  monitoring utilization of services; 

C  improving health care services or outcomes; and

C  encouraging service use by program participants.

Performance goals.  Most States identified a single goal for each of their strategic

objectives.  One sampled State went further, providing three to five goals for several of its

objectives.  The goals were developed as part of the SCHIP plans and were often

developed by a group comprised of representatives from multiple State agencies.  Two

States specifically set up SCHIP, and thus their goals for the program, to match their

existing Medicaid programs. 

Legislative Requirement for OIG Studies 
Section 703 of the Balanced Budget Refinement Act of 1999 required that the Office of 
Inspector General (OIG) sample States in order to: (1) evaluate whether States are 
correctly enrolling children in SCHIP or Medicaid as appropriate; and (2) assess the 
progress made by States to reduce the number of uninsured low-income children.  The Act 
further required that the OIG evaluate the progress sampled States have made to achieve 
their strategic objectives and performance goals.  Sampled States must be selected from 
States not using any SCHIP funds to expand Medicaid. 
This is one of two reports focusing on a sample of five States administering separate 
SCHIP.  This report focuses on States evaluations of their performance goals.  In a 
companion report entitled State Childrens Health Insurance Plan - Ensuring Medicaid 
Eligibles are not Enrolled in SCHIP, OEI-05-00-00241, we address State efforts to 
ensure Medicaid eligibles are not enrolled in SCHIP. 
METHODOLOGY 
Based on the requirements outlined in the Balanced Budget Refinement Act of 1999, we 
limited our universe to the 15 States utilizing their Title XXI allotments to operate 
separate child health programs.  Two States, Wyoming and Washington, were excluded 
from the population because they began administering SCHIP in December 1999 and 
February 2000, respectively, and thus had no enrollees in fiscal year 1999. 
We used a 2-stage stratified-cluster sampling plan to select 5 States from the 13 remaining 
States. The remaining 13 States were divided among 2 strata.  Pennsylvania was a self-
representing State as illustrated in Strata 1, based on the significantly large number of 
children enrolled in their SCHIP.  From the remaining 12 States, we randomly selected 4 
States, North Carolina, Oregon, Utah, and Vermont, for Strata 2.  The chart on the 
following page outlines the number of children enrolled in SCHIP in each of the Strata and 
sample States in Fiscal Year 1999. 
4 
SCHIP - Evaluation  OEI-05-00-00240  Selected Sample States 
STRATA 1 
States Selected  Number of Children Enrolled in  Number of Cases Selected For 
SCHIP in FY 1999  Review 
Pennsylvania  81,758  100 
Totals  81,758  100 
STRATA 2

States Selected  Number of Children Enrolled in  Number of Cases Selected For 
SCHIP in FY 1999  Review 
North Carolina  57,300  100 
Oregon  27,285  100 
Utah  13,040  100 
Vermont  2,055  100 
Totals  99,680  400 
We reviewed the 5 States Title XXI plans and the SCHIP evaluation reports submitted to 
HCFA in March 2000.  We conducted on-site visits and met with staff at the agency 
administering the SCHIP in each of the five States. 
We conducted case file reviews and discussed enrollment issues and goal attainment with 
State staff.  We reviewed 100 randomly selected active SCHIP case files in each State in 
order to evaluate whether ineligible children were enrolled in SCHIP.  We provided each 
State with randomly selected case numbers and staff pulled these cases in preparation for 
our visit.  The results of our case review were analyzed using a statistical analysis package 
that takes into account complex sample designs. 
One part of our discussions with SCHIP staff focused on State efforts to appropriately 
enroll children and safeguards used to ensure proper enrollment.  We also discussed 
SCHIP evaluations.  We collected documentation of State eligibility and enrollment 
practices and of evaluation efforts. 
We conducted our review in accordance with the Quality Standards for Inspections issued 
by the Presidents Council on Integrity and Efficiency. 
5 
SCHIP - Evaluation  OEI-05-00-00240  FF II NN DD II NN GG SS  
Questionable evaluations undermine the reliability of State reports of 
success 
The five sampled States report success reducing the number of uninsured children, but 
they did not evaluate all of their goals.  Further, the evaluations they conducted are 
impaired by their reliance on descriptive explanations, a lack of objective measurement, 
and problems with the data used in analysis. 
State reports of reductions in number of uninsured may be unreliable 
States report reductions in the number of uninsured children.  Our analysis of States data 
and methods used leads us to question the reliability of State figures. 
Reports of reductions.  The evaluation report in one of the sampled States notes that it is 
difficult to prove a causal relationship between increased SCHIP enrollment and a 
reduction in the number of uninsured.  Evaluation reports in the other four States provide 
information on decreases in the number of uninsured children within the State and 
estimated pre-SCHIP baseline numbers of uninsured children.  Two States estimate 
changes in the number of uninsured by comparing current (post-SCHIP implementation) 
estimates of uninsured to their baselines.  The other States estimate reductions in the 
number of uninsured children by subtracting SCHIP enrollees from their original baseline 
estimates.  The fifth State provides estimates of the total number of uninsured children and 
the number of uninsured SCHIP eligibles, and offers information about reductions in the 
number of uninsured children over the 1990's, but does not evaluate the impact the 
program has had on size of the uninsured population. 
Census and State data.  Three States report baseline and current estimates of uninsured 
children based on Census data.  One State gets its data from rolling averages of Census 
data.  The State indicates that its degree of confidence in the estimate is tempered by 
Census data limitations that may impact data reliability.  Another State utilizes SCHIP and 
Medicaid enrollment numbers and State insurance data (based in part on Census numbers) 
to estimate a reduction in the size of the uninsured population.  The State credits the 
change to increases in SCHIP rolls because between the 1997 baseline and the 1999 
estimate, the percentage of low-income children on Medicaid remained constant, the 
percentage of children with other health insurance dropped, and the SCHIP program 
enrolled children for the first time. 
Three States, including one that also uses some Census data, base their estimates on State-
collected data.  In one State, the Department of Health conducts a survey every 
5 years which includes information that allows estimates to be made of the uninsured 
population.  Other States rely on a State health insurance survey, a State population 
survey, and work done by a local university. 
6 
SCHIP - Evaluation  OEI-05-00-00240  Size of change varies.  State estimates of reductions in the number of uninsured children 
vary between the sampled States.  It is difficult to compare the size of changes across 
States due to the differences between States in initial rates of uninsurance, overall State 
populations, and the size of SCHIP eligible populations.  In the four States that offered 
both pre- and post-SCHIP implementation estimates, the change in the number of 
uninsured ranges from 5.8 to 21 percent.  The State with the largest percent change in 
uninsured children had very low uninsurance rates even prior to SCHIP implementation. 
This State has both a small population and low initial and current number of uninsured, 
which makes the percent change in insurance so large.  In addition, one State estimates the 
number of uninsured children under 200 percent of the Federal poverty level, while the 
others estimate the total number of uninsured children in their States. 
Estimating different numbers.  The four States that assessed change in uninsurance 
rates each used data derived from different sources to show progress toward different 
target numbers.  One State focused its assessment of enrollment and crowd out on the 
number of uninsured children in families with incomes below 200 percent of the Federal 
poverty level.  A second State, using 3 year rolling averages of U.S. Census Survey data, 
reported the change in the total number of uninsured children in the State. 
Two other States each provided a baseline number of uninsured children from prior to 
implementation of SCHIP and subtracted the number of children enrolled in the program 
to show a decline in the percentage of uninsured children.  Using this method, the State 
surpassed its projected decline in the number of children without insurance.  The report 
did not indicate whether any children currently enrolled in SCHIP were enrolled in 
Medicaid or private insurance at the time the baseline uninsurance estimate was made. 
Both the State that did not provide information on changes in the number of uninsured 
children and one of the States that made a projection indicated they had difficulties making 
estimates because current information is not available.  One State relies on insurance 
information from the State Department of Health that is updated every 5 years.  This State 
reports that its increase in SCHIP enrollment was not matched by losses in private 
coverage, suggesting that children now enrolled in SCHIP were represented in the States 
earlier count of uninsured children. 
While States report success in enrolling children into SCHIP, not all States compared their 
progress to an established goal regarding enrollment or insurance coverage.  One 
sampled State reports that it did not establish enrollment goals because it wants to enroll 
all eligible children.  The State set benchmarks that it wanted to meet by certain dates but 
indicated that staff did not see these numbers as final enrollment goals.  Another State 
indicated that it had not established an enrollment goal for the number of previously 
uninsured children but did use estimates of uninsured children and State resources to 
establish an initial enrollment goal. 
States report mixed success meeting other goals 
Just as States were allowed to design and assess their own strategic objectives and 
performance goals, they also had free reign to describe their progress meeting them. 
7 
SCHIP - Evaluation  OEI-05-00-00240  The 5 States we sampled provided evaluations or descriptions of 57 percent of the 

42 goals established among them.  While State evaluations showed that some evaluated

more of their goals than others, looking only at the percentage of goals assessed does not

provide full information.  The State that provided progress reports on all of its goals had

poorly developed goals and did not provide a full evaluation of those goals.  The State

with the most unevaluated goals has more overall goals and, to a much greater extent than

the other States, has rigorous goals that can be evaluated quantitatively.

States report success meeting between 67 and 100 percent of the goals on which they

provided information.  States reported the most success meeting goals that were assessed

with simple quantitative measures -- particularly enrollment rates. 

Enrollment rates.  All five State evaluation reports provided information about increases

in SCHIP enrollment.  As States differ in their population size, initial rate of uninsured

children, eligibility rules, and other program specifics, it is impossible to compare States

level of success enrolling children into SCHIP.  Further, each State defined its own goals,

so enrollment targets are not comparable among States.  Three States met their self-

defined enrollment goals, while a fourth did not.  The fourth States evaluation report

explains that, although the program enrolled more than the number of children specified in

its enrollment goal, a larger than expected number of children disenrolled during the

programs existence.  The fifth State did not set an enrollment goal, but instead used its

enrollment numbers to show progress toward its goal of reducing the number of uninsured

low-income children in the State. 

One State in the sample established goals to increase access and improve service

coordination by enrolling SCHIP children into managed care.  The State was unable to

meet these goals because the contracted managed care organizations shut down operations

in the State.  This forced the State to create its own managed care plan.  During

development, the State halted new managed care enrollments.  The State did not evaluate

how enrollment in managed care impacted access or service coordination. 

Outreach.  Three sampled States analyzed goals aimed at developing and implementing

outreach plans for SCHIP and other health care programs for low-income children.  These

States used their reports to describe, rather than evaluate, their outreach programs.  This

may be in part because evaluating the success of outreach efforts is difficult and inherently

somewhat subjective.  These States focus more on their successful implementation of

planned efforts than on whether the outreach programs encouraged enrollment in SCHIP

and Medicaid. 

Another States outreach goal stressed the role State agencies should play in executing

outreach strategies.  While agency participation was not analyzed, the State commissioned

a survey to determine whether outreach activities increased awareness 

of SCHIP and Medicaid among targeted families.  The survey found public awareness of 

SCHIP rose considerably from before the media campaign to 6 months after the

campaigns initiation. 

8 
SCHIP - Evaluation  OEI-05-00-00240  Utilization and access.  Two States tried to evaluate service utilization and all five States 
attempted to measure access to care.  Both types of effort met with mixed success. 
One States goals included monitoring enrollees utilization patterns, including their use of 
primary care and well child visits.  The evaluation report indicated that the level of primary 
care visits was comparable with national standards, but the rate of well care visits was 
lower than State-wide goals for all children.  In addition, the State did not evaluate its 
other utilization goals, such as immunization rates, dental care, and client satisfaction. 
Another State surveyed SCHIP families about whether enrolled children had a primary 
source of care, whether children were able to get needed care, and to what extent families 
were satisfied with their childrens health care.  The survey indicated that the State is 
meeting its goals in this area, as most children had a primary source of care, few children 
were unable to access health care, and an overwhelming majority of respondents were 
satisfied with their childrens health care.  The State indicates that as part of its next 
SCHIP survey, the State will administer the tool it developed to measure crowd-out. 
Other progress.  All five State evaluations provided information about progress toward 
their goals in several other areas.  One State assessed its attempts to improve enrollment 
practices by measuring what percentage of SCHIP applications they received through the 
mail or non-traditional sources.  While the State did not meet its 50 percent goal, this 
assessment provided information about which applicants apply using the different 
methods.  Another State described its efforts to expand its eligibility rules, restructure its 
data and operating systems, develop a unique coding system for SCHIP enrollees, and 
train staff to run the program.  None of these descriptions are strictly evaluative, but they 
do provide updates on the activities taken by the SCHIP program administrators. 
State reports are not very evaluative 
SCHIP programs are required by statute to evaluate their programs.  Section 2107 of 
Title XXI of the Social Security Act requires that SCHIP plans include strategic objectives 
and performance goals, and that measures be established to evaluate program 
performance.  Performance measures must be (A) measured through objective, 
independently verifiable means, and (B) compared against performance goals, in order to 
determine the States performance. 
Section 2108 of the Act outlines that each SCHIP must submit an annual assessment of 
the operation of the State plan . . . including the progress made in reducing the number of 
uncovered low-income children.  Evaluations must assess the effectiveness of the plan in 
increasing the number of children with creditable health coverage. Additionally, each 
State must describe and analyze the effectiveness of elements of the State plan. 
Much of the information provided in State evaluations is qualitative and subjective.  Four 
of the five States extensively utilize qualitative descriptions in their evaluation reports to 
HCFA.  Descriptive responses generally provide information about State efforts without 
assessing what impact efforts had on attaining specific goals.  Further, any evaluation of 
such a program would be subjective and could not be independently verified.  None of the 
sampled States attempted any evaluation of their outreach programs or offered an 
9 
SCHIP - Evaluation  OEI-05-00-00240  explanation of how such programs impacted their measurable progress in enrollment or 
the level of uninsured children. 
Several States provided information about what happened but were not able to explain 
why it happened.  The sampled States have goals stating their outreach and other efforts 
will increase access to care, raise enrollment, or decrease the number of uninsured children 
in the State.  State staff were unable to express how they chose their goals. 
Despite goals that suggest State efforts and outcomes will be measured, several States 
provide information about SCHIP and Medicaid enrollment increases without offering an 
explanation of how enrollment is impacted by their outreach or other efforts.  For 
example, one State goal outlines the States intention to increase State government 
participation in outreach activities in order to increase access to health insurance.  The 
evaluation provides information about the overall increase in SCHIP enrollment in the 
State but does not explain whether the goal (increasing government outreach efforts) was 
met or even what efforts were attempted.  Further, the evaluation report does not indicate 
how such State efforts could be seen as causing the increased enrollment rates the State 
offers in its progress summary. 
State evaluations rely heavily on enrollment data and descriptive information 
Sampled States utilize a variety of data but rely most on enrollment and descriptive data. 

All five States used enrollment data to assess enrollment increases, decreases in 

uninsurance levels, and increases in access to coverage.  On the whole, States were able to

assess the goals that rely on enrollment data. 

Some States also utilized survey and health care utilization data to evaluate their SCHIP

progress.  Two States utilized participant surveys to evaluate their goals, while another

plans a survey, but has not implemented it yet.  State surveys were used to identify

improved access to care and increased public awareness of SCHIP and Medicaid.  Planned

surveys will measure crowd out, use of preventative care, and client satisfaction. 

One State compared Medicaid and SCHIP enrollees utilization patterns for primary care

and well child visits to National Committee for Quality Assurance standards.  Three other

States planned to evaluate utilization data, but did not do this for their 

March 2000 evaluation reports, in part because data was not available or did not exist in a

format that allowed its analysis or comparison to other health information. 

State reports tend to be descriptive rather than evaluative 
While three States provide descriptions of SCHIP efforts, the information does not include 
assessments of the impact of State efforts on goals.  Descriptions of expanded program 
rules, outreach efforts, and anti-crowd-out policies do not indicate how these efforts 
impact access to care or childrens health status.  One State uses its evaluation report to 
describe the changes to State program rules and procedures that established 
10 
SCHIP - Evaluation  OEI-05-00-00240  SCHIP.  As a summary of its progress in improving service provision through managed 
care enrollment, another State explains its use of an automated referral process. 
States descriptive progress summaries do not evaluate State efforts.  States tend to 
employ such summaries when they do not have the skills or the available data to analyze 
the work they have done or show progress toward their goals.  While they provide some 
useful information about SCHIP, they are not a viable alternative to concrete evaluations 
of the impact of State efforts on program goals. 
Evaluations have conceptual and technical weaknesses 
As described in the previous finding, State evaluations are flawed; some goals were not 
evaluated, and evaluations of others lacked objective measures of success.  The reports 
were impacted by issues related to goal setting, staff skills and resources, program 
priorities, and data problems.  States are aware of some of the flaws in their assessments 
and report barriers to conducting better evaluations. 
Some goals were set without evaluation in mind 
State goals are not always easy to evaluate, in part because the staff who crafted the goals 
often did not consider what it would take to evaluate them.  Most States established their 
goals, as required, in their State plans.  The State plans were often written by groups made 
up of representatives from different agencies, each with their own priorities for childrens 
health.  While this can help the State craft a more robust program, it also may mean that 
individuals not responsible for program implementation or evaluation establish goals 
without thought to how they will later be assessed.  Sample States reported that their 
goals were designed at the time the State plan was written, but an explanation of how 
evaluation would occur was not developed.  One State specifically noted that they 
established their plan without thinking about goal assessment. 
Not considering how goals would be evaluated led to the creation of unmeasurable goals. 
One State reported they used a goal taken directly from the State plan but are now not 
able to measure it because information is not available, and definitions that would allow 
comparisons across populations do not exist.  One respondent reported that he is not sure 
how to measure some of the States goals and wondered how he should go about 
measuring coordination.  Further, as the goal does not define what type of coordination 
should be assessed, he was not sure with what other agencies and organizations the 
program should be measuring the level of coordination. 
A third State admitted that its goals may not be entirely measurable.  For example, one 
goal is to improve access to care by increasing State government participation in the 
program.  The analysis shows increased enrollment but does not explain what government 
participation has occurred or how that participation impacted enrollment or access.  When 
asked whether the program intended to evaluate this connection, respondents reported 
that they did not think they could quantify this. 
11 
SCHIP - Evaluation  OEI-05-00-00240  They further reported that they have no indicators to evaluate efforts and no interest in 
developing any. 
Program administrators do not make evaluation a priority 
The ultimate objective for SCHIP is to increase enrollment in health insurance programs to 
ensure that children receive appropriate health care services that maintain or improve 
health.  Evaluation is important to ensure that State programs are meeting identified needs 
and are spending their allotments effectively, but ultimately program administrators are 
much more focused on running the programs than on assessing their success.  Resources 
for evaluation are limited, particularly for smaller States which often feel constricted by 
the 10 percent cap on administrative activities. 
Staff in one State reported they are not sure when they will find time to decide what 
indicators they will use to assess quality.  They indicated that they will probably default to 
using established measures because they have limited staff and are more focused on 
running the program than on measuring quality. 
Another sampled State reported doing a good job meeting its goals but is not doing a 
good job measuring its success.  In part this is because States that believe they are 
successful may not be strongly motivated to evaluate themselves.  Respondents in this 
State indicated that they are bringing a lot of children into the program and staff do not 
feel the need to assess what outreach efforts are working.  They think that as long as they 
continue to enroll children, they will not worry about changing things. 
Even when States are not as convinced that they are succeeding, evaluation is generally a 
low priority for SCHIP.  When asked about whether they planned to evaluate their stated 
goals more fully than they had for the March 2000 evaluation report, staff at one State 
told us that they did not plan to further assess their reported goals.  They did not think it 
would be worth the required effort to track down the connections between their actions 
and the outcomes they are seeing in the program.  Another State that is more focused on 
evaluation reported that although they believe their goals are important, it is expensive to 
evaluate a large number of goals.  The cost and time required to evaluate multiple 
progress goals is slowing down evaluation efforts. 
Program staff lack evaluation skills and training 
When State respondents were asked about evaluation issues such as how goals were 
developed or how assessments were conducted, they were often unable to provide 
meaningful information.  Program staff were much more comfortable discussing the 
content of their goals than they were talking about how they decided to establish the goals 
or how they assessed progress.  In several discussions, State staff were unable to explain 
why they thought an evaluation showed what it was intended to show. 
For the most part, SCHIP staff are not trained evaluators.  Their skills lie where they are 
needed on a daily basis -- in program management and ensuring the effective and efficient 
administration of their programs.  The language of evaluation is also foreign to 
12 
SCHIP - Evaluation  OEI-05-00-00240  many program administrators.  Without a more thorough understanding of evaluation 
concepts and practices, program staff cannot be expected to turn out detailed assessments 
of program activities and outcomes. 
The lack of trained evaluation staff may be a particular problem for States in which 
Medicaid and SCHIP are administered separately.  SCHIP staffs are often small, and 
program administrators have difficulty hiring evaluation staff when program staff are 
needed. 
Problems with data impair evaluations 
Insufficient or flawed data caused problems for the sample States.  States faced difficulties 
regarding unreliable data, missing data, non-comparable data sets, and small population 
sizes. 
Unreliable data.  Several States indicated that they rely on Census data to make estimates 
of uninsured children and eligible children in the State.  The use of Census data is widely 
criticized, particularly for States with smaller populations.  One State reported that its 
estimate of total SCHIP eligibles comes from Census Current Population Survey (CPS) 
data on children below 200 percent of the Federal poverty level.  This State told us that 
staff fear the CPS numbers are no good.  This concern is in large part based on the poor 
record of CPS estimates of low-income children in the State.  The CPS data indicated that 
there were 150,000 children under 150 percent of poverty in the State, despite the fact 
that more than twice this many children are currently enrolled in Medicaid.  The State 
changed their process for estimating the number of uninsured children, but their new 
formula still relies on the CPS for some parts of the calculation.  They know the data is 
flawed but have no other data on which to base their estimates. 
Another State relying on Census data reported frustration with the quality of Census 
information for small populations.  State staff do not trust the Census data and have found 
that other population-based studies are not reliable for small States.  This State reports its 
faith in the estimate of SCHIP eligible children in the State as 5 on a 1 (least reliable) to 
10 (most reliable) scale. 
Data collected by the States themselves also may be deficient.  A State that relies on a 
health status survey administered by the State Department of Health expressed frustration 
with the age of the data.  The survey, conducted every 5 years, provides baseline 
information on the number of uninsured children.  Program administrators worry that the 
data is unreliable because it is old by the time they get it.  Respondents from this State 
reported that on a scale from 1 to 10, they rated this data between 5 and 6. 
The SCHIP administrator used State-collected data to calculate a new uninsured rate by 
subtracting the total number of SCHIP enrollees from an estimate of uninsured derived 
from survey data.  The State administrator admits that this methodology assumes that all 
SCHIP enrollees were previously uninsured and would be uninsured without the program, 
but reports that the State does not have better numbers or methods on which to base 
calculations. 
13 
SCHIP - Evaluation  OEI-05-00-00240  Missing data.  Obtaining accurate and meaningful utilization data is a challenge for 
States, which means that any subsequent analysis may be incomplete.  States often rely on 
the contracting managed care organizations for data on use of services.  Although States 
contractually require participating managed care organizations to provide data, the receipt 
of this data may not be timely.  One State reported that encounter data they used to 
calculate use of preventative services is 25 percent incomplete.  Data from the managed 
care contractors is missing, despite the States continued efforts to get complete data from 
the plans. 
States may also have problems getting data from other agencies within their own 
department.  One SCHIP program adopted a goal for immunization rates at the request of 
the Department of Health, which coordinates a State-wide immunization campaign. 
Another agency within the Department that administers SCHIP collects the immunization 
information, but its data collection system is not very mature. 
Non-comparability.  Comparing managed care and fee-for-service data was a problem 
for one State.  The State intended to assess whether children received appropriate medical 
screenings but found that comparing children enrolled in different service types would 
require reorganizing the data, as utilization data from the SCHIP fee-for-service system is 
presented differently from their Medicaid managed care data.  This State noted that 
Medicaids managed care program is oriented toward prevention.  This led to the 
establishment of specific data elements for health screening and other prevention services. 
The State set up its SCHIP program on a fee-for-service model that provides less specific 
information about the delivery of prevention services to children.  The SCHIP does not 
have a single code for health screening similar to the one used in the Medicaid program. 
The administrator indicated that her staff is working on making the fee-for-service and 
managed care data consistent, but that they might have to abandon the goal if they cannot 
compare the two groups. 
Small population.  SCHIP is new, and even with strong enrollment, there may not be 
sufficient children in a State program to perform robust analysis.  One States utilization 
analysis were conducted for all children enrolled in State-sponsored health care.  The State 
felt that there were too few children in SCHIP alone to allow for reliable evaluations, 
especially when information on specific age or other categories is required. 
Another State report was based on 1 year of data.  Few children were enrolled at the start 
of the year, and enrollment increased dramatically during that first year.  The State 
administrator reports that because they experienced such a dramatic change in enrollment 
over a short time period, they cannot accurately determine the number of children who 
should make up the denominator for their health screening goal.  Without this information, 
the State cannot assess whether the target number was met. 
14 
SCHIP - Evaluation  OEI-05-00-00240  R E C O M M E N D A T I O N S  
Over the past 3 years, States have been concentrating their efforts on getting the SCHIP 
program organized and operating by focusing on outreach and enrollment.  There is now a 
need for States to shift their focus to evaluating the effectiveness of program 
administration and progress made towards the achievement of each States goals and 
objectives, as outlined in their State Plans.  While State administrators believe they are 
successfully enrolling eligible children and reducing the overall number of uninsured 
children, they admit that they do not evaluate their efforts well. 
Improving States ability to conduct useful evaluations may require a multi-stage process. 
Initial stages should focus on improving State program staffs general understanding of 
evaluations, including the distinction between simply describing a process and true 
evaluation.  Later stages should focus on staff knowledge of different types of evaluation 
and how these evaluation types can be used.  In addition, SCHIP staff should become 
knowledgeable in preparing data for collection, determining the types of data that should 
be collected for various analysis, and evaluating data once it is collected. 
Federal regulations and policy provide little guidance to States on what constitutes an 
acceptable evaluation.  Without more specific guidance about what SCHIP evaluations 
should contain and assistance conducting assessments, the information States provide does 
little to ensure that Federal Title XXI dollars are used to the greatest possible advantage. 
To increase the usefulness of SCHIP evaluations and facilitate the ability of SCHIP staff to 
conduct such improved evaluations: 
HCFA should identify a core set of evaluation measures and develop a more 
specific framework for the content and structure of the reports States are 
required to submit. 
SCHIP staff need direction from HCFA on what kind of information it is seeking from 
States.  The HCFA should identify a core set of evaluation measures that will enable all 
SCHIP States to provide useful information, independent of the way each State operates 
their SCHIP.  The HCFA can do this by determining what information it needs evaluations 
to yield, how detailed this information must be, and what purpose the evaluations will 
serve.  Providing this guidance will enable States to submit information best suited to 
address HCFAs issues and concerns. 
The HCFA should develop a more specific framework for the content and structure of 
State evaluations.  The HCFA should continue working with the National Academy for 
State Health Policy to create a template that provides States direction on how to address 
issues uniformly, leaving little room for ambiguity. 
15 
SCHIP - Evaluation  OEI-05-00-00240  HCFA and the Health Resources and Services Administration (HRSA) should 
provide guidance and assistance to States in conducting useful evaluations 
of their programs. 
To conduct more useful evaluations, SCHIP staff would benefit from assistance and/or 
training regarding data collection and evaluation, and resource determination. 
Data Collection and Evaluation - The HCFA and HRSA should assist States in building 
their capacity to collect required data.  The need for quality data is intensifying as Federal, 
State, and local governments rely more heavily on program evaluations to make important 
programmatic decisions.  However, States are finding those evaluations difficult due to 
limited data capacity and a lack of reliable and useful data. 
The HCFA and HRSA should work together to provide States training and guidance in 
the following areas:  1) what data to collect, 2) how to obtain specific data, 3) how to 
determine if the data is reliable, 4) how to determine what the data is yielding, and 5) how 
to evaluate the data.  In addition, HCFA and HRSA should work with States to identify 
external sources of data and, as appropriate and necessary, work with these external 
sources to improve the reliability of data.  For example, HCFA and other Federal agencies 
are working with the Census Bureau to increase the overall quality of CPS data.  Because 
of States reliance on CPS data to conduct evaluations, this and similar efforts would 
increase States access to quality data without requiring large resource commitments from 
States. 
Resource Determination - Currently, poor coordination exists between SCHIP 
administration and the evaluation of SCHIPs.  Program administrators indicated that their 
current focus is on enrolling children into the SCHIPs and, consequently, less attention is 
paid to evaluation.  They also indicated that Federal reporting requirements are 
burdensome and funds for robust evaluations are lacking.  The HCFA and HRSA should 
train administrators and staff to help reduce the poor coordination between administration 
and evaluation and make normal staff activities part of the evaluation process. 
Existing staff resources may not allow some States to fully address their needs to conduct 
useful and meaningful evaluations.  Therefore, HCFA and HRSA should work with States 
to identify, where needed, additional evaluation resources.  Depending on the extent of 
this need, HCFA and HRSA may want to explore various options, including offering 
States matching funds, creating set-aside funds within SCHIP for evaluations, increasing 
the amount of SCHIP funds that can be used for administration (i.e., evaluation), or 
tapping into available Departmental evaluation funds. 
Agency Comments 
The Health Care Financing Administration and the Health Resources and Services 
Administration provided formal comments to the draft report.  Both concur with the 
findings and recommendations.  The full text of the their comments are included in 
Appendices A and B. 
16 
SCHIP - Evaluation  OEI-05-00-00240  The HCFA provided information regarding their work with States.  The HCFA also noted 
they will continue to work in partnership with the National Academy for State Health 
Policy to develop a framework for Title XXI annual reports.  The HRSA provided 
comments to clarify portions of the report.  We made appropriate changes based on the 
comments we received. 
17 
SCHIP - Evaluation  OEI-05-00-00240  APPENDIX A 
18 
SCHIP - Evaluation  OEI-05-00-00240  19 
SCHIP - Evaluation  OEI-05-00-00240  20 
SCHIP - Evaluation  OEI-05-00-00240  APPENDIX B 
21 
SCHIP - Evaluation  OEI-05-00-00240  22 
SCHIP - Evaluation  OEI-05-00-00240