Medicaid Demonstrations: Evaluations Yielded Limited Results, Underscoring Need for Changes to Federal Policies and Procedures
Released: 2018-02-20T00:00:00

Letter

Background

Medicaid Section 1115 Demonstrations

Nearly three-quarters of states (37 as of November 2016) have CMS- approved Medicaid section 1115 demonstrations, which allow states to test new approaches to coverage and to improve quality and access or generate savings or efficiencies. CMS has approved demonstrations for a wide variety of purposes. For example, under demonstrations, states have extended coverage to populations or for services not otherwise eligible for Medicaid, made payments to providers to incentivize delivery system improvements, and, more recently, expanded Medicaid to certain low-income adults by using Medicaid funds to purchase private health insurance coverage. While state demonstrations vary in size and scope, many are comprehensive in nature, affecting multiple aspects of states’ Medicaid programs simultaneously. For example, Kansas’s demonstration, approved in 2012, significantly expands the use of managed care to deliver physical, behavioral, and long-term care services to almost all the state’s Medicaid populations, care that for some populations was previously provided on a fee-for-service basis. The demonstration also established a funding pool of up to $344 million to provide payments to hospitals to finance uncompensated care.

Kansas’s demonstration expenditures accounted for about 94 percent of the state’s total Medicaid expenditures in fiscal year 2015.

In fiscal year 2015, federal spending under demonstrations represented a third of all Medicaid spending nationwide. In 10 states, federal spending on demonstrations represented 75 percent or more of all federal spending on Medicaid. (See fig. 1.)

Demonstrations are typically approved by CMS for an initial 5-year period (referred to as a demonstration cycle), but some states have operated portions of their Medicaid programs under a demonstration for decades. This can be achieved through a series of renewals approved by CMS, generally occurring every 3 to 5 years. What a state is testing and implementing under its demonstration can change from one cycle to the next. States often make changes to their demonstrations, either through the renewal process or by requesting an amendment during the demonstration cycle. These changes can be relatively small or can be significant and can represent testing of a new approach for the state. For example, at renewal a state could request approval to expand coverage to a new population or add requirements that beneficiaries share in the cost of care by paying a monthly premium.

CMS Oversight of State- Led Evaluations

CMS has long required states to conduct evaluations of section 1115 demonstrations. CMS oversees the evaluations and can influence them at several key points during the demonstration process.

Application review and approval: When a state applies for a demonstration, CMS reviews the state’s application, which describes the goals and objectives of the demonstration and what the demonstration will test, among other things. As part of the review and approval process, CMS negotiates with the state on the STCs, including evaluation requirements. These requirements might include, for example, reporting timeframes and broad standards for the evaluation, such as standards around the independence of the evaluator and acceptable evaluation methods.

Evaluation design phase: After a demonstration is approved, states are required to submit an evaluation design to CMS for review and approval. The evaluation design must discuss, among other things, the hypotheses that will be tested, the data that will be used, and how the effects of the demonstration will be isolated from other changes occurring in the state. During review of the design, CMS can seek adjustments such as requiring the state to address certain objectives or using particular performance measures.

Demonstration renewal: In the event that a state wishes to renew its demonstration, it must generally submit an application to CMS at least 1 year before the demonstration is scheduled to expire. The application must include, among other things, a report presenting the evaluation’s findings to date, referred to as an interim evaluation report. CMS can use the information from the interim evaluation report to negotiate changes in the STCs for the evaluation of the next demonstration cycle. If CMS renews the demonstration, the evaluation process starts over with the state submitting a new evaluation design that reflects changes in what is being tested in the new cycle.

Demonstration end: CMS requires states to submit a final evaluation report for review and approval generally after the end of the demonstration, at which time the agency can work with the state to, for example, add clarity and disclose the limitations of the evaluation before the final evaluation report is made public.

Within the framework that CMS has established for state-led evaluations, states design evaluations to the specifics of their demonstrations. As the size and scope of demonstrations varies considerably across states, so, too can evaluations vary in their breadth and complexity. State-led evaluations may assess the effects of several different policies, each with its own set of hypotheses—predictions of the effects of the policy—and methods. For example, a state could evaluate the effects of moving to a managed care delivery model for providing managed long-term services and supports (referred to as MLTSS), implementing provider payment pools aimed at delivery system reform, and expanding coverage to a new population all within the same demonstration. Each of those three elements would have its own hypotheses and methods and may have varying timeframes for the number of years of experience needed to be able to effectively measure the effects of what is being tested.

Federal Evaluations

CMS has the authority to initiate its own federal evaluations of section 1115 demonstrations, and states must fully cooperate with any such evaluations. Between 2014 and 2016, CMS initiated three federal evaluations that were ongoing as of November 2017. The first evaluation, initiated in 2014, is a large, multi-state evaluation examining four broad demonstration types in several states. (See table 1.) According to CMS, it selected these demonstration types—which together account for tens of billions of dollars in federal and state Medicaid spending—because they included policies that the agency considered priority areas for evaluation. CMS awarded a contract to an evaluation organization to implement the 5-year study. According to CMS, the estimated total cost of this evaluation for the 5-year life of the contract is $8.3 million. The evaluation was designed to produce three sets of results: a series of reports providing contextual information about the demonstrations being evaluated, referred to as rapid cycle reports; interim evaluation reports featuring early results of more in-depth analysis; and final evaluation reports.

CMS contracted with another evaluation organization to conduct two federal evaluations examining demonstrations in single states—Indiana and Montana—over 4 years. As of September 2017, the estimated cost of this contract, inclusive of all options, was $8.2 million. In total, spending for Indiana’s and Montana’s demonstrations was about $2 billion in fiscal year 2015, including $1.6 billion in federal spending.

Indiana: CMS initiated this evaluation in 2015. CMS officials told us they started this evaluation to better understand how policies in Indiana’s demonstration, many of which were unprecedented, were affecting beneficiaries. These policies included, for example, charging monthly contributions for most newly eligible adults with incomes from 0 to 138 percent of the federal poverty level; imposing a lock-out period of 6 months for nonpayment of premiums for most people with incomes above the federal poverty level; and charging co-payments above statutory levels for non-urgent use of emergency room services. The federal evaluation is aimed at estimating the effects of Indiana’s demonstration on health insurance coverage and access to and use of care, and documenting beneficiary understanding of enrollment, disenrollment, and copayment policies, among other things.

Montana: CMS initiated this evaluation in 2016. CMS officials told us they started this evaluation to provide a point of comparison to Indiana’s demonstration, as Montana was implementing similar policies to Indiana but with some variations. For example, under Montana’s demonstration, the state charges premiums to most newly eligible adults with incomes between 51 and 138 percent of the federal poverty level; and disenrolls beneficiaries with incomes above the federal poverty level for nonpayment of premiums, with reenrollment when overdue premiums are paid. Similar to the federal evaluation of Indiana’s demonstration, the evaluation of Montana’s demonstration is aimed at estimating the effects of the demonstration on insurance coverage, access to and use of care, and documenting beneficiary understanding of and experience with premiums, copayments, enrollment, and disenrollment, among other things.

Limitations in State- Led Evaluations Hindered Their Usefulness and May Not Be Fully Addressed by CMS Improvements

State-led evaluations of demonstrations in selected states often had significant methodological weaknesses and gaps in results that affected their usefulness for federal decision-making. Though CMS has been taking steps since 2014 to improve the quality of these evaluations, the agency has not established written procedures to help implement some of these improvements.

State-Led Evaluations in Selected States Often Had Significant Limitations That Affected Their Usefulness in Informing Federal Decision-Making

The state-led evaluations we reviewed in our selected states often had methodological limitations that affected what could be concluded about the demonstration’s effects. CMS hired a contractor to review state evaluation designs and reports, and that contractor identified a number of methodological concerns with the evaluations in our selected states. For example, CMS’s contractor raised concerns about the comparison groups, or lack thereof, used to isolate and measure the effects of the demonstrations in the Arkansas, California, Indiana, and Maryland evaluations. The contractor also raised concerns with the sufficiency of sample sizes and survey response rates for beneficiary surveys in Indiana. These surveys were key methods for assessing the effect of demonstrations on access, beneficiary understanding, and perceptions on affordability. Finally, the contractor raised concerns with the analysis of the effects of the demonstration on cost in Arkansas, California, and Maryland. Officials in several states told us that some of the methodological limitations in their evaluations were difficult to control. For example, officials in two states told us that isolating the effects of the demonstration was difficult given other changes happening in the state’s health care system at the same time. Some state officials also noted that state resources, including both funding and staff capacity, present challenges in completing robust evaluations. program, with approved funding up to about $690 million. Under the demonstration STCs, the state was required to evaluate whether the seven hospitals participating in the DSRIP were able to show improvements on certain outcome measures related to improving quality of care, improving population health and access to care, and reducing the per capita costs of health care. However, the evaluation report, submitted by the state 5 years after approval of the DSRIP program, provided only descriptive or summary information about the number and types of projects implemented by the hospitals receiving payments and did not provide any data to measure or conclusions on the effects of those payments.

Arkansas: Under its demonstration, the state was testing the effects of using Medicaid funds to provide premium assistance for the more than 200,000 beneficiaries newly eligible under PPACA to purchase private insurance offered through the state’s health insurance exchange. The state’s evaluation was designed to assess whether beneficiaries would have equal or better access to care and equal or better outcomes than they would have had in the Medicaid fee-for- service system. The evaluation was also aimed at examining continuity of coverage for beneficiaries, as the expansion population was anticipated to have frequent income fluctuations leading to changes in eligibility and gaps in coverage. However, evaluation results submitted over two and a half years into the demonstration— the only results submitted for the state’s first cycle—were limited to data only from the first year of the demonstration and did not provide data on continuity of coverage. Achieving continuity of coverage was part of the state’s rationale for using an alternative approach to Medicaid expansion.

Arizona: Among other things, Arizona’s demonstration includes MLTSS, including for the particularly complex populations of adults who have intellectual and developmental disabilities and for children with disabilities. As part of its evaluation, the state was assessing whether the quality of and access to care, as well as quality of life, would improve during the demonstration period for long-term care beneficiaries enrolled in MLTSS. However, evaluation results submitted in October 2016—the only results submitted for the state’s most recently completed demonstration cycle—lacked data on key measures of access, such as hospital readmission rates, and on quality of life, such as beneficiaries’ satisfaction with their health plan, provider, and case manager.

A key contributor to the gaps in the information included in the state-led evaluations we reviewed was that CMS historically had not required the states to submit final, comprehensive evaluation results at the end of each demonstration cycle. As a result, for our selected states, including those discussed above, CMS had received only interim evaluation reports that were generally based on more limited data from the early years of the demonstration cycle and did not include all of the analyses planned. Though CMS had required final evaluation reports in the demonstration STCs, the due dates for those reports were tied to the expiration of the demonstrations or, in one case, CMS did not enforce the specified due date. Under such conditions, due dates for final evaluation reports were effectively pushed out when the demonstrations were renewed. Evaluation due dates could be pushed out for multiple cycles. CMS officials acknowledged that the lack of data in the interim evaluation reports from the more mature years of the demonstration affected the conclusions that could be drawn from them.

We found that due dates for final evaluation reports were pushed out upon renewal in all seven of our states that had completed a demonstration cycle, leading to a gap in evaluation reporting of up to 6 or 7 years for several states. In Maryland, for example, CMS approved the demonstration to run from 2013 to 2016 with a final evaluation report due 120 days after the expiration of the demonstration. In 2016, CMS extended the demonstration, pushing the deadline for the final evaluation report to 18 months following the end of the new cycle, or June 2023. At that time, it will be 7 years since the interim evaluation report was submitted. See figure 2.

The limitations in state-led evaluations—including methodological weaknesses and gaps in results—have, in part, hindered CMS’s use of them to inform its policy decisions. CMS officials told us that, historically, state-led evaluations have generally provided descriptive information but lacked evidence on outcomes and impacts. As a result, officials noted that they consider the data reported in the evaluations but, generally, state-led evaluations have not been particularly informative to their policy decisions. CMS officials told us that there have been cases where data, but not the conclusions, from state-led evaluations have informed their thinking on certain policy changes. For example, CMS officials said that data reported in early evaluations of DSRIP programs helped them in considering whether and how the agency should modify the basic policy structure of these programs. State officials had mixed perspectives on whether state-led evaluations influenced CMS decision-making around renewing their demonstrations. Officials in one state told us that while CMS reviewed their interim evaluation results, the results did not appear to influence the negotiations around the demonstration renewal. In contrast, officials from another state told us that discussion of interim evaluation results and limitations was a significant part of negotiations in 2016 regarding whether CMS would be willing to reauthorize funding for certain programs, including a new DSRIP investment and broader delivery system reforms the state was trying to implement. Officials in several states told us that there was value to state-led evaluations and in the federal-state partnership in designing the evaluations.

CMS Is Taking Steps to Improve the Quality of State-Led Evaluations, but Lacks Written Procedures to Ensure That All Evaluations Will Be Subject to New Requirements

CMS has implemented several procedures since 2014 aimed at improving the quality of state-led evaluations. CMS officials told us that these changes were part of CMS placing increased focus on monitoring and evaluation, which also resulted in CMS establishing a new office in 2015 that is responsible for these activities. One of the key changes CMS began implementing in 2014 was to set more explicit requirements for evaluations in the STCs, including requirements to improve the evaluation methodologies. According to CMS officials, the agency realized that one reason why state-led evaluations had generally lacked rigor and been of limited usefulness was that CMS had not been setting clear expectations for evaluations in the STCs. The officials said that CMS began strengthening evaluation requirements starting in 2014 with demonstrations implementing approaches in CMS’s high priority policy areas.

In our review of the STCs for current demonstration cycles in our seven selected states that had completed a demonstration cycle, all of which were approved in 2014 or later, we found evidence of CMS’s efforts. Specifically, we found an increased focus on the use of independent evaluators and more explicit expectations for rigor in the design and conduct of evaluations:

Consistent requirements for independent evaluators. The STCs for the most recently approved cycle of demonstrations in all seven states required the state to use an independent evaluator to conduct the evaluation. In some cases, the STCs also required that the evaluation design discuss the process to acquire the independent evaluator, including describing the contractor’s qualifications and how the state will assure no conflict of interest. These requirements were new in most states.

More explicit expectations for rigor. In four of the seven states we reviewed, the STCs for the most recently approved cycle of states’ demonstrations included new, explicit language requiring state evaluations to meet the prevailing standards of scientific and academic rigor. These included standards for the evaluation design and conduct as well as the interpretation and reporting of findings. Some states’ STCs further specified the characteristics of rigor that CMS expected, including using the best available data, discussing the generalizability of results, and using controls and adjustments for and reporting the limitations of data and their effects on results. According to CMS, in the past, states have not always discussed methodological limitations in their evaluation reports.

In addition to strengthening evaluation requirements, CMS has also taken steps since 2014 to enhance its oversight during the design and early stages of state-led evaluations, and, according to officials, some of these steps are likely to improve the usefulness of evaluations. Specifically, CMS has provided technical assistance to help states design their evaluations, sometimes leveraging expertise from other parts of HHS, including the HHS Office of the Assistant Secretary for Planning and Evaluation and the Center for Medicare & Medicaid Innovation as well as outside contractors. For example, officials stated that the agency assists states in developing relevant and standardized measures and provides assistance to help address states’ data limitations. Officials said this has resulted in more robust evaluation designs with increased potential to isolate outcomes and impacts.

CMS has also used contractors to help in its review of state evaluation designs, including sampling designs, and evaluation reports. Since 2014, one contractor has provided over 30 assessments of evaluation designs and findings in at least 11 states. According to officials, this has increased CMS’s capacity to identify methodological weaknesses and negotiate changes with states to improve the usefulness of evaluations. For example, CMS’s contractor reviewed four draft survey instruments that Indiana planned to use in its evaluation, providing comments on the sampling frames and the structure and organization of survey questions. In response to the contractor’s feedback, Indiana made changes to the surveys to gather more reliable information and improve their readability.

Finally, CMS has begun making changes to how it sets due dates for final evaluation reports. CMS officials told us that in spring 2017, CMS began requiring states to submit a comprehensive evaluation report for demonstrations in its high priority policy areas for evaluation at the end of each demonstration cycle, rather than after the expiration of the demonstration. CMS’s recent demonstration renewals in Florida and Missouri—approved in August and September of 2017, respectively— required a final, summative evaluation report at the end of the demonstration cycle, consistent with the policy. In October 2017, CMS officials stated that the agency was expanding this policy and was now planning to require final reports at the end of each cycle for all demonstrations, as they are approved or renewed. However, CMS had not established written procedures for implementing this new policy.

It is too soon to assess the effectiveness of CMS’s recent efforts to strengthen state-led evaluations. CMS has been implementing the strategies on a rolling basis as states apply for demonstration renewals and new demonstrations. If implemented and enforced consistently, CMS’s efforts to improve the quality of state-led evaluations have the potential to result in more conclusive evaluations. Further, CMS’s efforts to improve the quality of state-led evaluations and its plan to require final reports after each demonstration cycle are consistent with evaluation guidance from the American Evaluation Association that recommends that federal agencies conduct evaluations of public programs and policies throughout the programs’ life cycles, not just at their end, and that agencies use evaluations to improve programs and assess their effectiveness. Federal internal control standards also state that management should implement control activities through policies. However, CMS does not have written procedures for implementing its planned policy, for example, for ensuring that the requirement is included in the STCs for all demonstrations, despite unique negotiations with each state, and that those requirements are consistently enforced. As a result, some state-led evaluations could continue to produce only more limited, interim findings that leave critical questions about the effects of the these demonstrations on beneficiaries and costs unanswered.

CMS oversight of state-led evaluations may see further changes, as CMS officials told us that their oversight procedures are still evolving. For example, CMS officials told us that as of October 2017 the agency plans to begin to make distinctions in the level of evaluation required across demonstrations. They said that they are considering, for example, whether longstanding and largely unchanged components of a demonstration, and approaches previously tested by a number of other states without concern, require the same level of evaluation as testing a new approach to Medicaid expansion. Officials said that they plan to include language in demonstration STCs, as the agency did in the recent renewals for Florida and Missouri, instructing the state to consider those factors as the state designs its evaluation. Specifically, in the evaluation design submitted for CMS approval, the state should include in the discussion of limitations whether the demonstration is long-standing, noncomplex, has previously been rigorously evaluated and found to be successful, or is also considered to be successful without issues or concerns. CMS officials said that the expected level of rigor for the evaluation could be balanced against such factors.

The implications of limiting evaluation requirements for certain types of demonstration approaches would depend on CMS’s definitions of what is, for example, noncomplex or has previously been rigorously evaluated. As of October 2017, CMS had not established specific criteria for determining when a demonstration component would require less rigorous evaluation. Agency officials told us they were planning to develop such criteria after concluding a pilot of alternative criteria and expectations in certain demonstrations related to providing services for family planning and former foster care children. They said that when these pilots have concluded they will evaluate the results. It is unclear how these narrowly scoped demonstrations—scoped for a particular type of service or population—can be used to inform criteria for comprehensive demonstrations that can affect a state’s entire Medicaid population and all services. Further, though CMS has begun indicating to states, including those with comprehensive demonstrations, that the agency may allow less rigorous evaluations for certain types of demonstration approaches, CMS has not established timeframes for issuing the criteria defining those conditions.

Federal standards for internal control stress that management should implement control activities through policy and should internally and externally communicate necessary information to achieve the agency’s objectives. If CMS does not establish clear criteria for components of demonstrations that require limited evaluation, characteristics such as “long-standing” or “noncomplex” could be broadly interpreted. This could result in demonstrations that receive significant amounts of federal funds and affect many beneficiaries not being thoroughly evaluated. Written criteria could also reduce the potential for inconsistencies in the level of evaluation required across demonstrations.

Ongoing Federal Evaluations Led by CMS Have Been Limited by Data Challenges and It Is Uncertain When Results Will Be Available

Data and other challenges have significantly limited the scope and progress of CMS’s large, multi-state evaluation and the agency’s evaluation of Indiana’s demonstration. Further, CMS has not released available evaluation results from the multi-state evaluation nor set timeframes for making these and future federal evaluation findings public.

Data Challenges Have Limited the Scope and Progress of Federal Evaluations

CMS encountered numerous data challenges in its multi-state evaluation that significantly reduced the scope of the analyses planned. These data challenges included limitations in the quality of CMS data and delays obtaining data directly from states. These limitations caused CMS to narrow the evaluation’s scope, often by reducing the number of state demonstrations evaluated or limiting what was being examined. All four demonstration types targeted in the multi-state evaluation—which reflect CMS’s high priority policy areas—were affected by these challenges. In the most extreme case, data limitations reduced the scope of the MLTSS evaluation to two states out of the more than 20 states operating such programs. As a result, the evaluation findings will not be generalizable to all MLTSS programs. (See table 2.) The data challenges were in addition to other challenges that affected the evaluation. For example, there were difficulties in trying to isolate demonstration effects in the context of rapidly changing health systems, or recent demonstrations had not been in operation long enough to allow CMS to appropriately assess longer- term effects.

Many of the data challenges CMS encountered in the multi-state evaluation reflect long-standing concerns with the lack of accurate, complete, and timely Medicaid data. Specifically, we and others have found that data states are required to submit to CMS have, at times, been incomplete or have not been reported at all, particularly managed care encounter data. Complicating the availability of these data is CMS’s ongoing transition to a new data system, the Transformed Medicaid Statistical Information System (T-MSIS), which is CMS’s primary effort to improve Medicaid expenditure and utilization data. States’ transitions to T-MSIS, however, have introduced substantial delays in state data submissions. For example, by 2015, a large number of states had stopped submitting data through the legacy information system until they established T-MSIS submissions, which meant CMS had to obtain data directly from individual states for the multi-state evaluation. New data challenges have also emerged as states under demonstrations have enrolled newly eligible beneficiaries in health insurance exchange coverage. Lack of accessible data on beneficiaries enrolled in plans offered through the exchange resulted in the delays in obtaining data for Arkansas for the multi-state evaluation. In the past, we have made recommendations to CMS to take action to improve the data available for Medicaid program oversight, including to T-MSIS.

As with the multi-state evaluation, data challenges, particularly obtaining needed data from the state, also proved to be a significant hurdle in CMS’s evaluation of Indiana’s demonstration. CMS initiated its federal evaluation of Indiana’s demonstration in 2015 to understand how the approaches being tested in Indiana’s demonstration affected beneficiaries (see sidebar). However, in 2016, Indiana raised concerns about sharing enrollee data with CMS’s evaluation contractors. Specifically, in a letter to CMS, the state cited concerns about the controls that CMS had in place to ensure that its contractors would protect enrollee information consistent with state and federal privacy protections. Despite assurances by CMS, CMS’s contractor and the state were not able to execute a data use agreement. This effectively halted the evaluation’s progress. The data use agreement was necessary for the contractor to access state enrollment data that drove a number of planned evaluation activities, including a key beneficiary survey. In October 2017, CMS officials told us that they were continuing to work with the state and anticipated that a data use agreement would be executed and the federal evaluation of Indiana’s demonstration would proceed. They did not have timeframes for when the agreement would be reached.

Despite the data challenges and delays, CMS’s evaluations of Medicaid demonstrations, as planned, are likely to provide new information on the effects of demonstrations in different states to inform policy decisions. The multi-state evaluation, for example, is expected to provide information on whether living in a state that collects monthly contributions from beneficiaries affects the likelihood of beneficiaries enrolling in Medicaid and how per-beneficiary spending differs between premium assistance demonstration states and states that have implemented more traditional Medicaid expansions. CMS officials emphasized that federal evaluations allow for cross-state evaluations that can be used to validate the findings of related studies and also to identify which findings are generalizable to other states and populations.

CMS Has Not Released Rapid Cycle Reports and It Is Uncertain When Final Evaluation Results Will Be Available

CMS has yet to make initial reports from the multi-state evaluation publicly available, limiting the potential use of those findings by states and other federal policymakers. As of October 2017, CMS’s contractor had produced 15 rapid cycle reports on states’ progress in implementing demonstrations in the high priority policy areas. These reports provide information on states’ implementation of their demonstrations and variations in design and provide details that can help with the interpretation of evaluation results, inform federal policymaking, and provide lessons learned to states and other stakeholders. The reports also describe policy and other challenges states encountered in implementing their programs, which could be useful to other states interested in replicating these models. (See table 3.)

However, despite having received some of these reports from its contractor in 2015, CMS had not released these findings as of October 2017. CMS officials said that the reports were still under agency review and acknowledged that since some of the rapid cycle reports were almost 2 years old, CMS’s contractor was reviewing and updating the information in them. CMS officials noted that the rapid cycle reports had provided useful information and had influenced ongoing work with states designing related demonstrations. For example, according to officials, findings from the rapid cycle reports played a part in how the agency structured the latest DSRIP demonstrations. They also said that rapid cycle reports on beneficiary engagement have shed light on the effectiveness of different beneficiary education strategies, such as what approaches are more successful in capturing beneficiaries’ attention and what strategies are easiest for states to implement. In October 2017, CMS officials stated that they had recently decided to make the rapid cycle reports public, although the agency’s clearance process for the reports was still being decided and the officials did not have timeframes for the reports’ release.

It is also uncertain when CMS will make interim and final evaluation reports from the multi-state evaluation public. By September 2017, CMS’s contractor for the multi-state evaluation produced three interim evaluation reports covering the four demonstration types. CMS officials regard these as draft interim evaluation reports, and, as of October 2017, said they were under agency review and would not be publicly released. CMS expects the contractor to submit final interim evaluation reports, which are anticipated to include some additional information beyond the draft reports, by September 2018, about 1 year later than when the final interim evaluation reports were originally due. CMS officials said that the agency planned to release the final interim evaluation reports, although there was no specific timetable for this. Timeframes for the completion and release of final evaluation results are even more uncertain, both because of the delays in the evaluation progress and because CMS has no standard policy for timeframes for releasing evaluation results.

It is also uncertain when evaluation results will be available and made public for CMS’s evaluations of the Indiana and Montana demonstrations. Two years after the approval of the contract for the Indiana evaluation, CMS’s contractor has produced an evaluation design but no evaluation findings. CMS had not posted the evaluation design on its website until November 2017, according to officials, about 1 year after it was originally submitted. As discussed above, the lack of findings is due to the contractor and state not having negotiated a data use agreement. To the extent that Indiana’s evaluation moves forward and evaluation reports are produced, CMS officials said the agency plans to release the final evaluation report but did not indicate whether interim findings, available a year earlier, would be released. With regard to the Montana evaluation, CMS expects to receive the interim evaluation report by September 2018 and the final evaluation report by September 2019. How soon these findings would be publicly available, however, is difficult to estimate, as CMS officials told us the agency must review these before making them publically available and does not have timeframes for this review.

The lack of a standard policy for the public release of findings from federal evaluations of Medicaid demonstrations is inconsistent with recommendations of the American Evaluation Association. The Association recommends that evaluation findings related to public accountability be disseminated to the public, and that evaluation results be made available in a timely manner and be easily accessible through the internet. For state-led evaluations, CMS must post on its website, or provide a link to the state’s website, all evaluation materials, including research and data collection, for the purposes of sharing findings with the public within 30 days of receiving the materials. CMS has not established a comparable policy for the release of findings from federal evaluations of demonstrations. CMS officials stated that federal evaluations provide a unique cross-state perspective that states typically do not have the capacity to provide in their own state-led evaluations; however, if these reports are not made public in a timely fashion, opportunities may be missed to inform federal and state policymakers and other stakeholders on the effects of Medicaid demonstrations.

Conclusions

Section 1115 demonstrations have long been an important tool for providing states with the flexibility to test new approaches to providing and financing Medicaid coverage. Given the potential effects on millions of beneficiaries and significant federal investment in these demonstrations—over $100 billion in 2015—it is critical that they be evaluated. Evaluating Medicaid demonstrations is complex, both within a single state and across states. These programs are dynamic, and there are many factors affecting outcomes, making it challenging to isolate the effects of policy changes implemented under a demonstration. Further, persistent challenges with Medicaid data that we have highlighted over the years add to the complexity of evaluating demonstrations. Despite these challenges, targeted and well-designed evaluations offer the potential to identify policies that improve outcomes for beneficiaries and reduce costs to Medicaid. With the growing complexity of Medicaid programs and limited resources, that information could prove key in helping to sustain the program.

CMS’s approach to overseeing state-led evaluations in the past has resulted in limited information about the effects of demonstrations, leaving gaps in evidence about policies that might improve state Medicaid programs. CMS’s efforts since 2014 to improve the usefulness of evaluations in informing state and federal Medicaid policy decisions have promise. If CMS consistently sets and enforces clear expectations and provides support for rigorous and timely state-led evaluations for all demonstrations as planned, those evaluations could yield more useful information within the next several years. However, CMS has not established written procedures for requiring final, comprehensive evaluation reports at the end of each cycle for all demonstrations, a key step in improving the usefulness of state-led evaluations. Further, CMS is planning to allow less rigorous evaluations for some demonstrations but has not yet established specific criteria for doing so.

Federal evaluations led by CMS also show promise. The evaluations currently underway—despite challenges that caused delays and reduced scope—are likely to provide a cross-state look at the effects of policies that are of great interest to CMS, Congress, and other states. However, CMS has not yet made potentially useful rapid cycle reports public and has no established policy for making future evaluation reports public. By not making the results of the federal evaluations public in a timely manner, CMS is missing an opportunity to inform important policy discussions happening at the state and federal levels.

Recommendations for Executive Action

We are making the following three recommendations to CMS: The Administrator of CMS should establish written procedures for implementing the agency’s policy that requires all states to submit a final evaluation report after the end of each demonstration cycle, regardless of renewal status. (Recommendation 1)

The Administrator of CMS should issue written criteria for when CMS will allow limited evaluation of a demonstration or a portion of a demonstration, including defining conditions, such as what it means for a demonstration to be longstanding or noncomplex, as applicable. (Recommendation 2)

The Administrator of CMS should establish and implement a policy for publicly releasing findings from federal evaluations of demonstrations, including findings from rapid cycle, interim, and final reports; and this policy should include standards for timely release. (Recommendation 3)

Agency Comments and Our Evaluation

We provided a draft of this report to HHS for review and comment. HHS concurred with all three recommendations. Regarding our first recommendation that CMS establish written procedures for implementing its policy requiring states to submit final evaluation reports after the end of each demonstration cycle, HHS said that it is in the process of developing such written procedures. HHS said that it is currently making this a requirement through the STCs for each demonstration as demonstrations are approved or renewed. Regarding our second recommendation that CMS issue written criteria for when the agency will allow states to limit evaluations of their demonstrations, HHS said it is in the process of testing such criteria, and that once it has experience with the criteria, it will develop written guidance. Regarding our third recommendation that CMS establish and implement a policy for publicly releasing findings from federal evaluations of demonstrations, HHS said that CMS is in the process of establishing such a policy. HHS added that CMS plans to have all finalized federal rapid cycle reports and final interim evaluation reports publicly available in the near future.

HHS also provided technical comments, which we incorporated as appropriate. HHS’s comments are reproduced in appendix II.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies of this report to the Secretary of Health and Human Services, appropriate congressional committees, and other interested parties. The report will also be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff members have any questions about this report, please contact me at (202) 512-7114 or iritanik@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Characteristics of Selected States’ 1115 Demonstrations

The Medicaid section 1115 demonstrations (referred to as demonstrations) in our eight selected states varied in terms of the number of years the demonstrations had been in effect and cost, among other things. For example, three of the more mature demonstrations—those in Maryland, Massachusetts, and New York—had been in place for two decades. Demonstrations in Arkansas and Kansas represented more recent approvals, both approved in 2013. (See table 4.) With regard to cost, all of the selected states were among the top 15 states in terms of amount of spending under demonstrations. Together, spending under demonstrations in our selected states accounted for about 47 percent of all spending under demonstrations in fiscal year 2015.

Appendix II: Comments from the Department of Health and Human Services

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Susan Barnidge (Assistant Director), Linda McIver (Analyst-in-Charge), John Lalomio, Hannah Locke, and Corissa Kiyan-Fukumoto made key contributions to this report. Also contributing were Laurie Pachter and Emily Wilson.