Comparative Effectiveness Research: HHS Needs to Strengthen Dissemination and Data-Capacity-Building Efforts
Published: 2015-03-03T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: PPACA imposed new requirements on HHS related to CER—research that evaluates and compares health outcomes and the clinical effectiveness, risks, and benefits of two or more medical treatments or services. Among other things, PPACA required AHRQ to broadly disseminate findings from federally funded CER and the Secretary of HHS (who, by delegation, charged ASPE) to coordinate federal programs to build data capacity for CER. PPACA also mandated that GAO review HHS's CER activities. This report examines (1) AHRQ's activities to disseminate the results of federally funded CER and (2) ASPE's activities to coordinate federal programs to support CER by building the capacity to collect, link, and analyze data, among other objectives. GAO reviewed relevant legal requirements and HHS documentation; interviewed HHS officials; and obtained information from five stakeholder groups that AHRQ targeted to receive disseminated information or were otherwise involved in AHRQ's dissemination efforts.
section_title: What GAO Found
paragraphs: The Agency for Healthcare Research and Quality (AHRQ), an agency within the Department of Health and Human Services (HHS), has taken some steps to disseminate comparative clinical effectiveness research (CER), as required under the Patient Protection and Affordable Care Act (PPACA), but has not taken other actions to help it fully address its dissemination requirements. The steps it has taken include the creation of tools that organize and disseminate research findings to certain targeted stakeholder groups and the development of plans for a publicly available database that includes CER. For example, AHRQ's marketing plans—customized plans to help convey key messages about AHRQ's research—include various informational tools to disseminate CER, such as research summaries that communicate research findings to clinicians, consumers, caregivers, and policymakers. However, the agency has not clearly defined how to disseminate information to certain stakeholder groups specified in the law, nor has it identified and documented time frames to implement the marketing plans and distribute the associated informational tools, as would be consistent with federal internal control standards, which state that significant events need to be clearly documented to ensure management goals are carried out. Additionally, in order to implement PPACA's requirement for developing a publicly available database that contains CER evidence, AHRQ officials told GAO that they plan to create a web page to list and provide users with links to existing publicly available databases that could be used to search for CER, but they have not documented a specific implementation plan that includes time frames and strategies to address known potential limitations, such as difficulties that certain users may face in searching the databases for CER results.
HHS's Assistant Secretary for Planning and Evaluation (ASPE) has coordinated among various agencies to fund projects intended to build data capacity for CER, but its approach lacks key elements needed to ensure its effectiveness. For example, these projects include an effort to better standardize data that could be used in multiple research projects. However, HHS's approach to building data capacity for CER lacks key elements, such as defined objectives, milestones, and time frames, that are necessary to ensure effectiveness. ASPE officials worked with the Office of the National Coordinator for Health Information Technology to develop a strategic road map to guide both the identification and selection of ASPE's projects beginning in fiscal year 2014 through fiscal year 2019. Although the February 2014 strategic framework for the road map highlighted several priority objectives, such as enabling the collection of standardized clinical data, these objectives were broad and not clearly defined. For example, although ASPE identified and considered related, ongoing federal and non-federal data infrastructure projects in an attempt to identify needs or gaps, among other things, its strategic road map is unclear on the timing and level of coordination that would be necessary for its projects to work together with these related projects to improve data capacity. Standard practices for project management call for agencies to conceptualize, define, and document specific goals and objectives in the planning process, along with the appropriate steps, milestones, time frames, and resources needed to achieve those results.
section_title: What GAO Recommends
paragraphs: GAO recommends that HHS direct (1) AHRQ to take several actions related to its dissemination efforts, including identifying and documenting time frames for the implementation and distribution of marketing plans and informational tools, and (2) ASPE to include clearly defined objectives, milestones, and time frames, or other indicators of performance, in its strategic road map used to identify its CER-funded projects. HHS concurred with the recommendations.

section_title: Letter
paragraphs: Comparative clinical effectiveness research (CER)—research that  evaluates and compares health outcomes and the clinical effectiveness,  risks, and benefits of two or more medical treatments, services, or items  such as health care interventions—can help provide both patients and  clinicians with more complete information to make health care decisions.  CER includes comparisons of drugs, medical devices, tests, surgeries,  and ways to deliver health care. Specifically, it can help determine which  intervention may be most effective or beneficial for a given patient.  However, the availability of this information is often limited, resulting in  more than half of medical treatments that may be delivered without clear  evidence of their effectiveness, according to the Institute of Medicine.  This uncertainty contributes to high variability in managing clinical  problems, with costs and outcomes differing markedly across the country.  Furthermore, even when research is conducted, applying research  findings to improve health care practice can be a challenging and lengthy  process.
In addition, the Patient Protection and Affordable Care Act  1.  broadly disseminate findings from federally funded CER, including  findings published by the Patient-Centered Outcomes Research  Institute (PCORI)—a non-profit corporation established under PPACA  to improve the quality and relevance of CER. HHS is required to  implement this requirement through AHRQ, in consultation with NIH.  2.  establish a grant program to train researchers on CER methodological  approaches. HHS is required to implement this requirement through  AHRQ, in consultation with NIH.  3.  coordinate relevant federal health programs to build data capacity for  CER, in order to develop and maintain a comprehensive,  interoperable data network that collects, links, and analyzes CER  data. HHS is implementing this requirement through the Office of the  Assistant Secretary for Planning and Evaluation (ASPE).
PPACA also mandated GAO to report on, among other things, HHS’s  implementation of CER-related activities every 5 years, with the first  report due in March 2015. This report examines (1) AHRQ’s activities to  disseminate the results of federally funded CER; (2) AHRQ’s activities to  support training related to CER; and (3) ASPE’s activities to coordinate  federal programs to support CER by building the capacity to collect, link,  and analyze data.
GAO-12-332. among other things. To identify stakeholder groups, we took several  steps. First, we selected four CER studies as examples from among 101  studies for which information was disseminated to targeted stakeholder  groups from June 2012 to June 2014. The four studies we selected were  systematic reviews—syntheses of existing comparative clinical  effectiveness research—and each aligned with one of AHRQ’s 14 priority  conditions and with one of the top four chronic diseases and conditions  for adults identified by the Centers for Disease Control and Prevention.  We then requested the marketing plans for these four studies and  compared the organizations listed in those plans to the targeted groups  specified in PPACA. We selected and obtained information from four  stakeholder groups—a physicians’ group; a health care providers’ group;  a professional association; and a patient advocacy group—that appeared  most frequently in the marketing plans, from among 56 stakeholder  organizations mentioned in the plans two or more times. For stakeholder  groups that were specified in PPACA but not included in the marketing  plans for the four studies, we requested additional documentation from  AHRQ to determine the extent to which these stakeholder groups may  have appeared in other marketing plans. We also interviewed an official  from an additional stakeholder group—a health plan—because AHRQ  staff indicated that they partnered with this organization to disseminate  information, although it was not included in the marketing plans. Findings  based on information we obtained from these stakeholders cannot be  generalized to all targeted groups. In addition, we reviewed relevant  internal control standards and key collaboration practices and compared  them against AHRQ’s dissemination activities.
To examine AHRQ’s activities to support training related to CER, we  reviewed the relevant legal requirements related to training. We also  reviewed AHRQ documentation, such as its spending plans for fiscal  years 2011 through 2013 and funding announcements for the grants  associated with its training grant program for CER. These funding  announcements describe the purpose of the grants and funding amounts.  Additionally, we interviewed relevant AHRQ officials responsible for  managing the training grant program and inquired about their consultation  with NIH officials.
To examine ASPE’s activities to coordinate federal programs to support  CER by building the capacity to collect, link, and analyze data, we  reviewed the relevant legal requirements, as well as relevant agency  documentation such as ASPE’s spending plans for fiscal years 2011  through 2013, which describe the office’s proposed projects for its CER- related data capacity activities. We also reviewed other documentation,  such as meeting minutes with HHS senior leadership and interagency  agreements with other HHS agencies that describe specific CER projects  ASPE has supported. We interviewed relevant ASPE officials responsible  for implementing these efforts, inquiring about how they monitor projects,  and their coordination with other HHS officials on the CER projects that   We  ASPE supports, as well as with PCORI on its PCORnet initiative.also reviewed relevant standard practices for project management, in  addition to internal control standards, regarding information used by  agencies to compare them against ASPE’s CER activities.
To assess the reliability of AHRQ and ASPE cost data for dissemination,  training, and building data capacity, we collected information from AHRQ  and ASPE officials regarding the accuracy of data entry and the systems  that contain the data. We determined the data were sufficiently reliable for  the purposes of our review.
We conducted this performance audit from April 2014 through March  2015 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform our work to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: PPACA requires HHS to perform several duties related to CER, including  disseminating, training, and building data capacity for research. (See  table 1.)
Although PPACA did not direct HHS to complete these duties by a  specified deadline, it appropriated funds to the Patient-Centered  Outcomes Research Trust Fund (PCORTF) through fiscal year 2019 to  enable HHS and PCORI to implement their respective requirements.  PPACA specified that 20 percent of the amounts appropriated or credited  to PCORTF be transferred to the Secretary of HHS in each of fiscal years  2011 through 2019. In total, HHS estimates that about $731 million will be  transferred to AHRQ (16 percent of the PCORTF) and about $190 million  will be transferred to ASPE (4 percent of the PCORTF). With the  exception of the amounts transferred to HHS, PPACA designates the  remaining PCORTF funds for PCORI’s CER work—an estimated   $3.5 billion from fiscal year 2010 through fiscal year 2019.
section_title: AHRQ Has Taken Some Steps to Disseminate Comparative Clinical Effectiveness Research, but Has Not Taken Other Actions to Help It Fully Address Its Dissemination Requirements
paragraphs: AHRQ has taken some steps to disseminate CER as required under  PPACA, including the creation of systematic reviews to develop CER  findings, tools to disseminate CER, plans for a website to list and provide  links to research databases that include CER, and plans for receiving  feedback from stakeholders to whom information is disseminated.  However, AHRQ has yet to take other actions that would help it address  all PPACA dissemination requirements.
subsections: section_title: AHRQ Has Taken Some Steps to Implement Key Dissemination Requirements
paragraphs: AHRQ has taken some steps to implement the law’s key requirements for  disseminating federally funded CER, that is to (1) broadly disseminate— develop and distribute—CER in consultation with NIH, (2) create tools  that organize and disseminate research findings to certain targeted  stakeholder groups, (3) develop a publicly available database, and   (4) establish a process for receiving feedback from entities to which  information is disseminated. From fiscal year 2012 through 2013, AHRQ  has obligated about $37 million of the estimated $731 million it expects to  receive through 2019 from the PCORTF on its dissemination activities.
Development and distribution of CER findings. AHRQ contributes to  the dissemination of CER in various ways, including through the  development of systematic reviews, technical briefs, and research  summaries that explore the benefits and harms of treatments. In  particular, a key method to disseminate CER is through systematic  reviews—syntheses of existing research that compare the effectiveness  and harms of different healthcare interventions. A systematic review is an  assessment and evaluation of all research studies that address a  particular clinical issue. Researchers use an organized method of  locating, assembling, and evaluating a body of literature on a particular  topic. Systematic reviews typically include a description of the findings  from the research studies.
AHRQ identifies topics for systematic review of CER, such as  cardiovascular disease and arthritis, by evaluating topics nominated by  individuals or groups against program selection criteria, in order to  determine if the topic is appropriate or not appropriate for review. In  addition to using its own criteria to identify CER topics for systematic  reviews and dissemination, AHRQ documentation states that the agency  will consult with experts, such as those from NIH, and review literature to  determine whether any similar systematic reviews of relevant studies  have already been conducted by other agencies or research  organizations in order to reduce potential duplication. Topics selected  for a systematic review are further refined with input from key stakeholder  groups, technical experts, and patients to develop focused research  questions. According to AHRQ officials, research funded by PCORI is  not yet included in these systematic reviews because PCORI research is  not yet complete.
For each systematic review AHRQ synthesizes CER findings from  existing research, and the agency disseminates these findings to various  targeted stakeholder groups. From June 2012 to June 2014, AHRQ  synthesized CER findings through 74 systematic reviews. (See appendix I  for a listing of the 74 systematic reviews for which AHRQ disseminated  CER findings.) Once a systematic review is complete, AHRQ follows  procedures included in its dissemination guidance materials to develop a  marketing plan that identifies key messages and targeted stakeholder  groups, as well as the types of dissemination mechanisms it will use to  conduct outreach. AHRQ officials told us they distribute CER results  generally by using the same mechanisms as we previously reported in  2012. These mechanisms include social media, as well as AHRQ’s  website and AHRQ’s Effective Healthcare Program website. According  to AHRQ officials, the agency determines which specific mechanisms will  be used to disseminate CER results by considering the unique  characteristics of the research, such as its type, potential impact, and  stakeholder groups most likely to use its findings. For example, CER  identified as being of particular interest to specific specialties may be  disseminated to certain clinical professional associations.
Tools to organize and disseminate CER. AHRQ’s marketing plans  include various informational tools to disseminate CER. Informational  tools include (1) patient decision aids that walk patients through options  and choices that patients should consider in working with their clinicians  to make informed health care decisions; (2) continuing education and  medical education modules to help clinicians understand and use CER  findings; (3) slide sets to assist clinicians, researchers and other health  professionals with education and training needs; and (4) short, plain- language research summaries that communicate research findings to  clinicians, consumers, caregivers, and policymakers. For example, the  marketing plan for the systematic review titled, Childhood Exposure to  Trauma: Comparative Effectiveness of Interventions Addressing  Maltreatment, was developed for a systematic review that examines  evidence about interventions for maltreated children. The marketing plan  included the specific informational tools to be used to disseminate this  project’s findings, such as research summaries for clinicians, a summary  of treatments for parents and caregivers, a continuing education module  for health care providers, and a slide presentation on the topic.
They  on all CER that has been conducted. National Library of Medicine officials  told us that they have informally consulted with AHRQ on its plans and  agree with this approach. In November 2014, AHRQ officials told us that  they were sharing their planned approach with senior HHS officials for  review and approval.
Feedback and evaluation process. As required by PPACA, AHRQ  officials told us they receive feedback on dissemination efforts and  materials from stakeholders, both formally and informally. For example,  officials said that for some of their projects, AHRQ convenes focus groups  and advisory panels to assess the needs of stakeholder groups and  determine how best to disseminate materials. Some stakeholders we  spoke to told us that they have provided feedback to AHRQ on materials  the agency has disseminated; however, they were uncertain about the  extent to which their feedback was incorporated into AHRQ’s  dissemination efforts. AHRQ conducted a feedback assessment and  issued a March 2012 feedback report that highlighted stakeholders  perspectives about the agency’s disseminated materials. In this report,  AHRQ noted that although there is a growing awareness about its  disseminated materials, clinicians raised concerns about the timeliness of  the information included in the materials, among other things. Officials  told us that the agency may conduct future feedback assessments, but  they do not know when these will occur and which targeted stakeholder  groups will be included.
AHRQ also has funded an evaluation to assess its CER dissemination  activities and materials supported by the Recovery Act. In September  2013, IMPAQ International—the contractor that conducted the  evaluation—issued presentation slides as its final report. The evaluation  indicated that stakeholders’ exposure to AHRQ’s CER information, such  as the number of website visits and dissemination materials requested,  increased over time with AHRQ’s dissemination efforts. The final report  also included feedback from certain stakeholder groups through focus  groups and surveys. For example, clinicians who participated in focus  groups indicated that they typically had little to no experience with the  CER information that AHRQ disseminates to clinicians, and suggested  that AHRQ more visibly promote the benefits and credibility of this  information and then integrate the results and products into existing,  easy-to-access sources of medical information focused on point-of-care  decision-making. AHRQ officials told us that they plan to award a contract  to evaluate the CER dissemination mechanisms—along with the  materials they use to share CER findings—that they continued under  PPACA. This evaluation project, according to officials, is under  development as staff and senior leadership determine the objectives and  methods for the study. Although AHRQ staff have not documented their  plans as of November 2014, they told us that the evaluation is likely to  measure progress on process and intermediate outcome goals of  dissemination activities—similar to the last CER evaluation conducted for  Recovery Act investments where the agency assessed the level of  awareness, understanding, use, and perceived benefits of CER. Officials  said the evaluation will also address longer term goals, such as improving  health care practice.
section_title: AHRQ Has Not Taken Other Actions to Help It Fully Address Its Dissemination Requirements
paragraphs: AHRQ has not taken other actions to help it fully address requirements for  disseminating CER in PPACA. Specifically, AHRQ has not taken actions  to help it fully address (1) the time frames for disseminating CER, (2) how  it will disseminate to all targeted stakeholder groups, (3) its  implementation plans for the publicly available database, and (4) how it  will coordinate with NIH.
Time frames for certain aspects of the dissemination process have  not been identified and documented. Although AHRQ has outlined its  dissemination process in various documents, it has not clearly identified  and documented time frames for one of its key dissemination activities— to implement marketing plans and distribute associated informational  tools. According to GAO’s Standards for Internal Control in the Federal  Government, significant events need to be clearly documented to ensure  management goals are carried out.which together describe the key activities of its dissemination process,  including the steps the agency takes to identify key CER findings from  systematic reviews, draft and finalize its marketing plans, and distribute  its informational tools to the public. While certain AHRQ documents  highlight time frames associated with key dissemination activities, we did  not identify any documents that specify time frames for when the  marketing plans are to be implemented and associated informational tools  are to be distributed to stakeholder groups. Once the marketing plans are  finalized, the informational tools are to be distributed to targeted  stakeholder groups after results of the research have been posted online,  such as publication in a major journal. AHRQ officials said they would  expect to distribute the informational tools as soon as the results of the   AHRQ has several documents  research have been posted; however, the dissemination guidance  materials we reviewed did not specify time frames for the completion of  the implementation of the marketing plans and distribution of  informational tools. Without identifying and documenting time frames for  these key activities, AHRQ cannot ensure that CER findings are  disseminated in a timely manner or that the dissemination process is  consistently implemented by all parties. Setting time frames is especially  important for dissemination given the length of time and uncertainty  inherent in applying CER findings; the large volume of CER research  expected from PCORI in the near future, which will increase AHRQ’s  dissemination responsibilities; and the need to maximize the investment  of PCORTF appropriations made through fiscal year 2019.
Dissemination plan for some stakeholders identified in PPACA has  not been clearly defined. Additionally, AHRQ has not determined how it  will disseminate information to certain stakeholder groups identified in  law, and its dissemination to some of these groups has been limited.  While AHRQ’s marketing plans include informational tools aimed at most  of the targeted stakeholder groups— physicians, health care providers,  patients, and appropriate professional associations—federal and private  health plans, and vendors of health information technology focused on  clinical decision support are not included.dissemination to all of the targeted stakeholder groups, AHRQ may be  missing opportunities to reach the key stakeholder groups identified in the  law. Although as of October 2014 there were no specific marketing plans  that identified private or federal health plans to receive disseminated CER  information, AHRQ officials told us they have conducted outreach to  these groups. For example, we spoke to a representative at a private  health plan who confirmed receipt and use of AHRQ disseminated CER  materials. For federal health plans, AHRQ officials said that they worked  with the Office of Personnel Management, which manages the Federal  Employees Health Benefits Program, and this program encouraged  health plans to use an AHRQ report on the comparative effectiveness of  autism treatments when determining coverage decisions. AHRQ officials  noted that some health plans told them that CER information without a  corresponding cost analysis is insufficient in informing coverage   Without a defined plan for  decisions. Officials also told us that AHRQ found challenges translating  CER findings into clinical decision support applications; plans are  underway to determine next steps.
Implementation plans for addressing the requirement to create a  publicly available database have not been documented. As of  November 2014, AHRQ officials also have not developed and  documented a specific implementation plan to create a publicly available  database for CER. GAO’s Standards for Internal Control in the Federal  Government state that management should compare actual performance  to plans, and as previously noted, should document significant events.  The agency formerly acknowledged its plan to address the PPACA  requirement to build a publicly available database during our prior work in  2012,existing databases has not been documented and is in the process of  being fully vetted with senior leadership. Additionally, while AHRQ  officials told us that their instructions on how to search databases for CER  will be aimed at the general public, they have not yet determined how  effective these tactics will be to meet the needs of various user groups,  such as non-researchers who may be unfamiliar with research databases.  For example, officials have not determined if or how they may seek  feedback from potential users or test the instructions or search terms to  see if they meet potential users’ needs. Additionally, AHRQ officials told  us they have not determined how to address potential limitations with this  new approach. Without taking steps to develop and document an  implementation approach that includes time frames and strategies to  address potential limitations and AHRQ’s plans to assess whether its  tactics meet the needs of various users, the agency does not have  reasonable assurance that it will implement the PPACA requirement in a  timely or effective manner.   but AHRQ has since modified this plan, and the new plan to use  NIH’s consultation role regarding AHRQ’s dissemination efforts is  unclear. AHRQ is required by law to consult with NIH regarding  dissemination efforts, and agency officials told us they meet informally  with NIH staff. NIH officials concurred. AHRQ officials said that they have  had interactions with NIH on specific dissemination projects of interest to  specific NIH institutes or centers, such as the National Cancer Institute.  AHRQ and NIH have not determined what role NIH should take in the  dissemination process, or which NIH officials should be involved.  Previous GAO work has identified key practices that can help federal  agencies collaborate effectively when they work together to achieve  goals. This work highlighted, for example, the importance of agreeing on  roles and responsibilities and establishing compatible policies,  procedures, and other means to operate across organizational  boundaries. While coordination between the two entities has been  informal and limited to specific NIH institutes or centers at this time,  AHRQ officials told us that there is a designated AHRQ official that serves  as a liaison to NIH to work on this effort. Additionally, AHRQ officials told  us that the agency’s senior management is currently working with NIH to  determine how best to more formally coordinate on AHRQ’s  dissemination activities, but the officials did not state when this effort will  be complete. Without specific plans on how it will collaborate, AHRQ  officials lack reasonable assurance that they have buy in from NIH  regarding dissemination activities or that their independent efforts are not  unnecessarily duplicative.
section_title: AHRQ Has Implemented a Comparative Clinical Effectiveness Research Grant Program to Support Researcher Training
paragraphs: As required by PPACA, AHRQ has implemented a training program  aimed at individual researchers and academic institutions that is designed  to increase the supply and expertise of CER investigators. Through this  program, AHRQ awards grants to support graduate training on CER,  career enhancement of beginning and midcareer investigators who utilize  CER methods, and institutional CER teaching programs. (See table 2.)  AHRQ provides grants to individuals it selects and also to institutions that  can select a number of individuals to train on CER. During the planning  stages for AHRQ’s training program, AHRQ officials told us they  consulted with NIH staff members with expertise on the design and  management of training grants.
An AHRQ official told us that funding will continue for the existing grants  awarded to date through 2018. For example, there are currently some  training awards that AHRQ will continue to fund through 2018. However,  because AHRQ’s allocation from the PCORTF is scheduled to end in  2019, AHRQ officials told us that they do not expect to create or initiate  additional individual grants. Additionally, AHRQ does not expect that  additional funding announcements will be made for the institutional  grants, since these grants are on a 5-year cycle with current grants  running through 2018. For any grant on a 2-year cycle, there will likely be  new awards made, but only up until 2018.
In order to monitor the various training grant awards funded since 2012,  AHRQ collects progress reports from training grantees on an annual  basis. AHRQ officials told us that participants learn about CER methods  and apply what they learn to conduct research projects as part of their  training. AHRQ requires that grantees annually submit progress reports to  assess their performance on these activities. These reports include  performance information, such as (1) a description of career development  and research-related activities undertaken; (2) a list of accomplishments  including publications, scientific presentations, dissemination activities,  new collaborations, inventions, or project-generated resources made;   (3) any methodological changes implemented; (4) key preliminary findings  from research; and (5) an annual evaluation statement of the award  recipient’s progress by the mentor.
AHRQ officials told us that they are considering an interim evaluation of  the training grant program for fiscal year 2016 and an overall evaluation  after the program is complete in fiscal year 2019. Officials stated that they  expect to document specific details about their plans before the  evaluations occur, which would be consistent with findings in our prior  work that a plan for data collection and evaluation is a key attribute of  effective training and development programs and can guide an agency in  a systematic approach to assessing effectiveness and efficiency. AHRQ  officials emphasized that the training program is ongoing and grantees  are not yet expected to have outcomes. For these evaluations, they have  collected baseline data from progress reports and they plan to collect  additional data once the grant program ends to help inform their  evaluations, such as a recipient’s promotion and tenure status to measure  academic progress.
section_title: ASPE Has Coordinated and Funded Projects to Build Data Capacity for Research, but Its Approach Lacks Key Elements Needed to Ensure Its Effectiveness
paragraphs: ASPE has coordinated among various agencies to fund projects intended  to build data capacity for CER. However, its approach to building data  capacity for CER lacks key elements, such as defined objectives,  milestones, and time frames, that are necessary to ensure effectiveness.
subsections: section_title: ASPE Has Coordinated Activities and Funded Projects Intended to Build Data Capacity for Comparative Clinical Effectiveness Research
paragraphs: ASPE officials have coordinated and funded projects that they say will  help build data capacity for CER. According to ASPE officials, building  CER data capacity involves improving data infrastructure, such as  facilitating the creation of new health data sets or the sharing of existing  health data via the creation of needed standards, services, policies,  federal data, and governance structures. ASPE officials say the agency  intends these projects to enable interoperable data networks that could  support the efficient collection, linkage, and analysis of data for CER from  multiple sources. ASPE officials told us that the agency’s goal is to  identify a number of investment opportunities through fiscal year 2019 for  enabling the development of a CER data infrastructure using funds from  the PCORTF.
Beginning in fiscal year 2013, ASPE officials worked with the Office of the  National Coordinator for Health Information Technology (ONC) to develop  a strategic road map to guide both the identification and selection of  ASPE’s PCORTF projects beginning in fiscal year 2014 through fiscal  year 2019. The strategic framework for the road map, completed in  January 2014, specified five component types—standards, services,  policies, federal data, and governance structures—necessary to build  CER data capacity. As of October 2014, ASPE has funded a total of 10  projects. (See appendix II for descriptions and funding amounts for the 10  ASPE projects.) ASPE has obligated about $23 million of the total  estimated $190 million it expects to receive through FY 2019 from the  PCORTF.
Prior to the development of the road map, ASPE worked with HHS’s  Leadership Council, responsible for overseeing ASPE’s PCORTF  investment process, to identify and fund new projects that utilized the  expertise of an HHS agency. Some projects extended the work of  existing Recovery Act projects, with the initial projects beginning in  2011. These projects focused on developing new or enhancing existing  data resources, such as expanding administrative and clinical data sets  for CER and establishing health information technology standards to  leverage electronic health records for CER. For example, ASPE funded a  new project conducted by ONC known as the Structured Data Capture  initiative. For this project, ONC identifies standards for common data  elements that consist of structured data definitions and electronic case  report forms, to capture patient data from electronic health records for  CER studies.
section_title: ASPE’s Approach to Building CER Data Capacity Lacks Key Elements Needed to Ensure Its Effectiveness
paragraphs: ASPE’s approach to building data capacity for CER through investments  in data infrastructure lacks key elements necessary to ensure its  effectiveness. Specifically, ASPE updated the strategic framework for the  road map in February 2014, but did not define specific objectives linked  with performance metrics or establish milestones and time frames that  could be used to gauge its progress toward the goal of coordinating  relevant federal health programs to build data capacity, as required by  PPACA. Without these key elements, ASPE may be unable to gauge its  progress towards meeting the requirements of the law. Standard  practices for project management call for agencies to conceptualize,  define, and document specific goals and objectives in the planning  process, along with the appropriate steps, milestones, time frames, and  resources needed to achieve those results.
Although the updated February 2014 strategic framework for the road  map highlighted a purpose—to identify a set of investment opportunities  for developing CER data infrastructure to build CER data capacity—and  included guiding principles and objectives, it did not clearly define those  objectives, nor did it include other elements such as milestones or time  frames that would help allow for monitoring and reporting on progress.  Specifically, ASPE identified several guiding principles, such as ensuring  that data infrastructure projects are “non-duplicative of other related  federal and non-federal investments” and “achieve synergy with PCORI  and AHRQ.” It also included priority objectives, such as further enabling  the collection of standardized clinical data, but many of the objectives  were broad and not clearly defined—and did not specify milestones or  time frames—as would be consistent with effective project management.
Although ASPE identified and considered related, ongoing federal and  non-federal data infrastructure investments in an attempt to identify needs  or gaps, opportunities where contributions could be made, and ways to  avoid duplication, its strategic road map was unclear on the timing and  level of coordination necessary for its investments to work together with  existing projects—such as PCORI’s PCORnet initiative—to improve data  capacity. For example, ASPE officials were not clear on how precisely the  standards for common data elements resulting from the ONC Standard  Data Capture initiative could be incorporated into PCORnet or other  existing publicly funded data networks, although ASPE does plan to make  them available for use, and officials told us that they will work with other  HHS agencies and PCORI to determine adoption strategies.  Furthermore, the ONC Standard Data Capture initiative is not expected to  be completed until 2016, which occurs after PCORI’s common data  model for the PCORnet initiative is expected to be used for conducting  research, beginning in September 2015. Having more clearly defined  objectives and establishing milestones and time frames can also help  ASPE assess how it expects the results of its CER investments to build  data capacity, and how they will be coordinated in a timeline with many  other entities’ existing and planned efforts. Moreover, this information can  help ASPE officials understand the extent to which their efforts are not  duplicative and align with other federal efforts.
ASPE officials told us that as of October 2014, they are planning to award  a contract for developing an evaluation framework that will be used to  assess the effectiveness of their CER data infrastructure projects. They  also told us that they monitor and assess the 10 individual projects by  collecting quarterly reports and assessing progress against the  statements of work that were developed for each project. However, it is  unclear from ASPE’s strategic road map whether these efforts will be  sufficiently timely and coordinated with other federal and non-federal  efforts to result in improvements to CER data capacity.
section_title: Conclusions
paragraphs: Comparative clinical effectiveness research can give health care  providers information to help decide which treatments may be most  beneficial for a given patient, and it also can inform decisions by patients  and caregivers. However, this information is often incomplete or  unavailable. While HHS has multiple, ongoing efforts to meet its  requirements under PPACA related to CER, it has not determined how it  will fully address some of these requirements, particularly those related to  dissemination and data capacity building.
Disseminating CER in a timely manner is particularly challenging given  the length of time and uncertainty inherent in applying research findings  to help improve health care practice. AHRQ, for instance, has taken steps  to disseminate CER and documented these processes, including time  frames for some, but not all, of its key dissemination activities. Such time  frames may become especially important due to the large volume of CER  research expected from PCORI in the near future, which will increase  AHRQ’s dissemination responsibilities, and the need to maximize the  investment of PCORTF appropriations made through fiscal year 2019.  Additionally, effective dissemination of research findings involves multiple  stakeholders, some of which are specified in PPACA. Without clear plans  to target each of these stakeholder groups, including federal and private  health plans and vendors of health information technology focused on  clinical decision support, it is unclear whether pertinent CER findings are  being directed to key targeted stakeholders identified in PPACA and  presented in a meaningful way to those groups. Other aspects of AHRQ’s  dissemination process, such as its plans for a publicly available database  of CER—including whether AHRQ’s instructions and CER search terms  will be effective to meet the needs of various potential users in the  general public—and its collaboration with NIH on dissemination activities,  have not been fully defined.
HHS’s plan to build data capacity involves identifying projects that would  enhance existing data resources for CER. While HHS has a strategic  road map with information on projects that it is funding to build the  capacity for CER data, the road map does not include key elements, such  as clearly defined objectives, milestones, and time frames needed to  assess the agency’s progress toward the goal of building data capacity  for CER, as would be consistent with practices for effective project  management. Without defining these key elements, for example, it is  unclear to what extent ASPE’s projects will build on or contribute to other  similar federal or non-federal activities, rather than being duplicative.  ASPE officials, for instance, could use more defined objectives and time  frames to help them better assess the extent to which the CER projects  they choose to fund will be useful and timely for other relevant federal and  non-federal work, such as PCORI’s PCORnet initiative.
section_title: Recommendations for Executive Action
paragraphs: To help ensure that HHS fully addresses its dissemination requirements  under PPACA, we recommend that the Secretary of Health and Human  Services direct AHRQ to take the following four actions:  1.  identify and document time frames for the implementation and  distribution of marketing plans and informational tools;  2.  expand dissemination efforts to federal and private health plans and  vendors of health information technology focused on clinical decision  support;  3.  document and complete plans to develop a publicly available  database, including plans to meet the needs of various potential users  in the general public; and  4.  develop specific plans on how it will collaborate with NIH on its  dissemination activities.
In addition, to ensure that HHS fully addresses the PPACA requirements  to build data capacity for CER, the Secretary should direct ASPE to  include clearly defined objectives, milestones, and time frames, or other  indicators of performance, in its strategic road map that is used to identify  its PCORTF projects.
section_title: Agency Comments
paragraphs: We provided a draft of this report to HHS, and HHS provided written  comments, which are reprinted in appendix III. HHS concurred with all  five of our recommendations and provided additional information about its  work to build data capacity for CER. Additionally, HHS provided technical  comments, which we incorporated as appropriate.
Specifically, for the first four recommendations, HHS—including AHRQ— stated that it would ensure that starting and ending time frames for the implementation  and distribution of patient-centered outcomes research findings are  clearly specified and documented.   continue and expand dissemination activities that target federal and  private health insurance plans, as well as vendors of health  information technology focused on clinical decision support. HHS  stated that it recently issued a funding opportunity announcement  focused on the use of clinical decision support to disseminate and  implement patient-centered outcomes research findings. document and complete its plans to ensure that multiple potential  users, including the general public, have access to patient-centered  outcomes research studies and their findings. As noted in our  findings, these plans include creating a web page to list and provide  users with links to existing publicly available databases that could be  used to search for these studies. Complete plans would include time  frames, strategies to address potential limitations, and whether the  needs of various users are being met.   continue to collaborate with NIH institutes and centers, and develop  and document specific collaborations around patient-centered  outcomes research dissemination activities. HHS stated that AHRQ  has begun regular meetings with NIH—through its Office of Science  Policy and the NIH Deputy Director for Science, Outreach, and  Policy—to discuss how NIH’s and AHRQ’s activities can best  complement one another.
Regarding our last recommendation, HHS stated that it intends, through  ASPE, to further develop the road map by specifying milestones with  corresponding time frames. HHS will also develop specific performance  indicators for its portfolio of data capacity investments. Consistent with  our findings and conclusions, HHS’s comments also stated that its data  capacity investments need to coincide with other key HHS policy  initiatives and be responsive to the needs of CER data networks,  including PCORI’s PCORnet.
We are sending copies of this report to the Secretary of Health and  Human Services, the Director of AHRQ, the Assistant Secretary for  ASPE, and other interested parties. In addition, the report is also  available at no charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-7114 or kohnl@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs are on the last page of this  report. GAO staff who made major contributions to this report are listed in  appendix IV.
section_title: Appendix I: AHRQ’s 74 CER Systematic Reviews Disseminated between June 2012 and June 2014
paragraphs: According to the Agency for Healthcare Research and Quality (AHRQ), it  conducted 74 systematic reviews—syntheses of existing research—that  were related to comparative clinical effectiveness research (CER) and  resulted in findings disseminated between June 2012 and June 2014.  Table 3 lists each systematic review with dates for each processing step  leading up to posting the results of the review on AHRQ’s website for the  public. Based on GAO’s analysis of the 74 systematic reviews, the time  frame from when a systematic review began to when the findings were  disseminated, including posting via AHRQ’s website, ranged from 1 year  to more than 4 years.
section_title: Appendix II: ASPE’s Comparative Effectiveness Infrastructure Projects (FY 2011-2014)
paragraphs: Appendix II: ASPE’s Comparative  Effectiveness Infrastructure Projects   (FY 2011-2014)
subsections: section_title: Project title CER Inventory
paragraphs: Amount obligated  (dollars in millions)  $1.7  a centralized inventory of CER studies to serve as the foundation for a  publicly accessible database of current publicly and privately funded CER  projects, and related published policy and scientific literature.  algorithms to accurately identify and classify CER studies.  an improved web-based tool to provide a better understanding of the  landscape of current CER activity to users.  a mechanism and plan to pilot test the tool prior to making it publicly  accessible.
Enhancements to the existing database will  combine claims data from public and private sources, matching patient  information as appropriate, as is necessary for cross-payer and longitudinal  analysis.  pursue options to test the value of secure distributed data networks for  research applications like CER.
Ongoing pre-existing project to support CER through a research database  that provides researchers with Medicare and Medicaid beneficiary claims and  assessment data linked by beneficiary across the continuum of care.  Funded enhancements include expanding the amount of Medicaid data  available and security enhancements.
Collaboration between the Office of the National Coordinator for Health  Information Technology and the National Library of Medicine to integrate  clinical information and research information within a “template” that can be  utilized by researchers.
Expand the amount of data collected by a nationwide network of 19  community health centers and five research organizations in 10 states, which  together collect CER-related data about patients in underserved  communities.
Build upon previous Centers for Disease Control and Prevention efforts by  augmenting a publicly available dataset for CER with additional longitudinal  follow-up data on disease recurrence and vital status for colon, rectum, and  breast cancer cases.  Enhance software tools and methodology for management and consolidation  of electronic data reported on a real-time basis from electronic health records  to registries.
Identify concrete, strategic opportunities to contribute long term to building  data infrastructure for CER, and help maximize the impact of the Patient- Centered Outcomes Research Trust Fund investments.  Assess the current landscape of data infrastructure for CER, identify gaps,  and opportunities.
Develop, select, and validate standards for common data elements for use in  CER and a template to collect data from electronic health records for  research purposes.  Allow providers to access data in their own electronic health records in a  standardized way to support CER.  Allow researchers outside of the organization who have remote access  authorization to access an organization’s electronic health record data for the  purpose of CER.
These projects have two initiatives under the same project description.
section_title: Appendix III: Comments from the Department of Health and Human Services
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Will Simerl, Assistant Director;  Jennie Apter; La Sherri Bush; Christine Davis; Ashley Dixon;   Colbie Holderness; Andrea Richardson; and Jennifer Whitworth   made key contributions to this report.