Comparative Effectiveness Research: Activities Funded by the Patient-Centered Outcomes Research Trust Fund
Released: 2018-03-23T00:00:00

Letter

Background

In 2010 PPACA authorized the establishment of PCORI to improve CER quality and relevance. PPACA also established requirements for HHS to, among other things, disseminate findings from federally funded CER, including findings published by PCORI, and coordinate with relevant federal health programs to build data capacity for research. PPACA established a Trust Fund to fund these CER activities by PCORI and HHS through fiscal year 2019.

PCORI Activities Required by PPACA

PPACA authorized the establishment of PCORI as a federally funded, nonprofit corporation aimed at advancing the quality and relevance of evidence through research to help patients, clinicians, purchasers, and policy-makers to make informed health care decisions. PCORI is required to identify research priorities, establish a research project agenda, fund research consistent with its research agenda, and disseminate research findings, among other responsibilities. In 2015 we reported that PCORI had conducted activities consistent with its legislative requirements. For example, we reported that since its inception in 2010, PCORI established and implemented priorities for funding CER and related activities, developed plans to disseminate funded research and track its utilization, and took steps to make its research more centered on outcomes prioritized by patients. Further, PCORI developed PCORnet as a distributed research network initiative that enables electronic health-related data from multiple sources to be available for research.

HHS CER-related Activities Required by PPACA

PPACA requires HHS to perform several requirements related to CER, which it has implemented through AHRQ and ASPE. Specifically, AHRQ is required to disseminate and support the incorporation of CER funded by PCORI and other federal entities, as well as to foster capacity for conducting CER by supporting training in the methods used to conduct such research. ASPE, in turn, is required to build data capacity for conducting CER. In 2015, we reported that AHRQ had taken some steps to disseminate research findings, but had not taken other actions to help it fully address its dissemination requirements. Furthermore, we reported that ASPE coordinated among various agencies to fund projects intended to build data capacity for research, but that its approach lacked key elements—such as defined objectives, milestones, and time frames—that are necessary to ensure effectiveness. In our 2015 report, we made five recommendations to HHS to direct AHRQ and ASPE to address these issues, as appropriate. HHS concurred with these recommendations and specified actions it would take to address them. Four of the recommendations have since been implemented.

PPACA Funding for CER

PPACA established the Trust Fund through which PCORI and HHS receive funds for CER activities. The law provides that for fiscal years 2010 through 2019, the Trust Fund will receive appropriations from the general fund of the Treasury, transfers from the Medicare trust funds, and fees collected by the Department of the Treasury (Treasury) from private insurance and self-insured health plans. Eighty percent of the amounts in the Trust Fund must be made available to PCORI in fiscal years 2011 through 2019, and Treasury must transfer the remaining 20 percent to the Secretary of HHS in each of those years. Under current law, appropriations and transfers to the Trust Fund will end in fiscal year 2019. The law also provides that no amounts shall be available for expenditure from the Trust Fund after September 30, 2019, and specifies that any amounts remaining in the Trust Fund after that time will be transferred to the general fund of the Treasury. (See fig. 1 for an overview of transfers to the Trust Fund and distribution of funds to PCORI and HHS).

PPACA limits the use of CER in certain ways; for example, the law prohibits PCORI from developing or using a dollars-per-quality adjusted life-year to establish what type of health care is cost effective or recommended, and prohibits the Secretary of HHS from using such measures as a threshold to determine coverage, reimbursement, or incentive programs under Medicare. HHS may use CER findings to help inform Medicare coverage decisions, but PPACA does not allow Medicare coverage to be denied solely on the basis of CER findings.

PCORI Committed Funds Primarily to Research and Data Capacity Efforts; Awards for Dissemination and Implementation of Findings Were Limited as Most Research Was Still Underway

In fiscal years 2010 through 2017, PCORI committed about $1.6 billion (or 79 percent of its total award commitments of $2.0 billion) to awards for conducting CER and $325 million (or 16 percent) to awards for building data capacity for research. In addition, PCORI committed $93 million for engagement and workforce awards to involve stakeholders in the research process and expand the research workforce, and committed $12 million for awards to disseminate and implement its research findings. Awards for the dissemination and implementation of its research findings were limited as of the end of fiscal year 2017, as most of this research was still underway. (See table 1 for PCORI’s award commitments for fiscal years 2010 through 2017.)

By the end of fiscal year 2024, PCORI projects to spend a total of almost $3.3 billion, which reflects its projected Trust Fund revenue through fiscal year 2019 plus interest income. This total amount encompasses the commitments PCORI has made for awards through fiscal year 2017, as well as $514 million in projected additional research award commitments to be made by the end of fiscal year 2019 and $207 million for other award commitments to be made by the end of fiscal year 2021. In addition to awards, the total includes PCORI’s expenditures for program and administrative support services in fiscal years 2010 through 2017, as well as projected expenditures for these services through fiscal year 2024. (See fig. 2 for PCORI’s actual and projected commitments and expenditures and see app. I for an overview of PCORI’s awards.)

The following information provides details on PCORI’s awards related to research, building data capacity, engagement and workforce activities, and the dissemination and implementation of its research findings.

Research Awards

PCORI committed $1.6 billion, or 79 percent of its total award commitments, for research in fiscal years 2010 through 2017. In fiscal years 2018 and 2019, PCORI projects to commit an additional $514 million for research awards. PCORI research awards have increasingly focused on conditions that impose a substantial health or financial burden on patients and the healthcare system. (See table 2 for information on the health conditions that received the highest research award funding.)

Similar to certain types of CER that may take many years, the entire research award process for PCORI-funded CER may span multiple years from the funding announcement to the dissemination of completed research. Specifically, the process PCORI established can take as many as 6 years, which includes requesting and reviewing proposals, awarding contracts, recruiting participants or obtaining data, conducting and reviewing research, and disseminating findings and typically involves awards that span multiple years. For example, PCORI estimates that the typical timeframe for announcing funding and selecting applications to receive research awards can take 8 to 11 months as PCORI brings scientists, patients, payers, and other stakeholders together to prioritize proposals based on the impact of the condition, potential to improve health, technical merit, patient-centeredness, and engagement. (See fig. 3.)

Most of PCORI’s research projects, awarded through fiscal year 2017, were still underway. Only 53 of its 543 research projects had been completed as of the end of fiscal year 2017—in part because PCORI’s research award process typically takes 2 to 6.5 years to complete, and because almost two-thirds of the funds committed for research projects were awarded in fiscal years 2015 through 2017. While most PCORI- funded research is underway, a larger number of research studies are projected to be completed by of the end of each year from 2018 to 2022, with all of the remaining studies to be completed by 2024. (See fig. 4.) PCORI officials told us that the institute attempts to manage its funds to ensure that its research awards are funded and managed through completion, including peer review and the distribution of research findings, in recognition of the time needed to conduct this research as well as the uncertainty regarding the total amount of funding available.

Officials from all but one of the stakeholder organizations we interviewed—public and private payers, health care providers, and patient advocacy organizations that represented potential users of CER— generally supported PCORI’s research award priorities. Most of the stakeholders we interviewed stressed the importance of research conducted by unbiased organizations, such as the federally funded research funded by PCORI and HHS. In addition, most stakeholders also told us that PCORI’s efforts to engage patients in the research process have changed the way research is conducted for the better, such as prioritizing research outcomes that are most meaningful to patients. However, officials from an organization representing payers (and from an individual health plan) told us that PCORI’s priorities did not fully align with their needs, such as their needs for CER on certain high-cost conditions, medications or treatments.

Building Data Capacity Awards

PCORI committed the second largest portion of award funding—$325 million through fiscal year 2017—for awards to build data capacity for research through the development of PCORnet. PCORI officials told us that the institute supported the development of the PCORnet initiative in order to use existing medical records and claims data and to transform much of that data into a common data model to be used for clinical research, until such time when such data will have been standardized in electronic health records so that they can easily be used for research. As of December 2017, PCORnet included 36 partner networks agreeing to link their electronic claims and health data. PCORI officials told us that this distributed data network already comprises a nationally representative sample of approximately 128 million individuals whose data can be used in randomized clinical trials, large observational studies, and other research. In fiscal years 2018 and 2019, PCORI projects to commit an additional $70 million for these awards to continue building this data capacity.

PCORnet research is managed through its Coordinating Center, which oversees the translation of certain categories of the partner networks’ data into the common data model and forges agreements with each of the partners to share results of queries using their data with researchers.

This research process generally starts when a researcher requests to query data on a specific population, after which PCORnet may approve the request and invite network partners to participate. Participating network partners then run queries on their data following established parameters and submit the results to a secure portal that the researcher can access in order to analyze the results for research. (See fig. 5.) PCORI officials told us that there were 32 research projects using PCORnet that received funding through PCORI’s research award process as of the end of December 2017, as well as 45 research projects funded by other parties, including federal agencies and private industry.

Further, as part of its building data capacity awards, in fiscal year 2017 PCORI committed $25 million to the People-Centered Research Foundation, a nonprofit foundation formed in March 2017 to support the network partners and other entities conducting research using PCORnet. This funding was provided to support this foundation’s development of a business plan, as well as its governance structure, to ensure the continuity of the PCORnet network partnership efforts after PCORI funding for PCORnet ends. PCORI has indicated it may provide additional funding to the foundation, provided that the foundation and the networks make progress toward self-sustainability.

Officials from most stakeholder organizations we interviewed generally agreed that PCORnet offers value by improving the data available to conduct CER. Officials from two organizations told us that PCORnet has made it possible to use network partners’ aggregated data to make conducting research more efficient than in the past.

Engagement and Workforce Awards

Through fiscal year 2017, PCORI also committed $93 million for engagement and workforce awards. For example, PCORI committed a total of $63 million for engagement awards, intended to involve a variety of stakeholders in the research process and to improve the methodology for carrying out CER. Engagement awards include “Eugene Washington Engagement Awards” that are intended to bring patients, caregivers, clinicians, and other healthcare stakeholders into the research process and to disseminate study results. In addition, “Pipeline to Proposal Awards” are intended to bring together stakeholders with strong interests in a specific health issue to develop research proposals to address their needs. Officials from the two patient advocacy organizations we interviewed told us that PCORI’s engagement awards have helped to support patient involvement in the research process. For example, one official noted that, while it has not been easy to find patients willing to participate, these awards have been important to train and support patients in the research process.

PCORI also committed $30 million to workforce training awards for clinicians and researchers. For example, one of PCORI’s career development programs, conducted in partnership with AHRQ, is designed to train clinician and research scientists to conduct patient-centered outcomes research and to actively engage stakeholders in efforts to improve the quality and safety of care.

Dissemination and Implementation Awards

Dissemination and implementation awards for PCORI-funded research findings thus far have been limited as most of the research was still underway, but, according to PCORI officials, awards for this work will substantially increase as research is completed. Specifically, through fiscal year 2017, PCORI committed a total of $12 million for awards to disseminate and implement PCORI-funded research by helping researchers and other stakeholders to publicize findings and by supporting patients and providers to utilize findings. PCORI projects to commit an additional $91 million for these awards in fiscal years 2018 through 2021.

Dissemination and implementation awards are intended to encourage PCORI awardees that have completed research and their patient and stakeholder partners to pursue strategic activities to disseminate and implement their findings. For example, PCORI awarded about $0.4 million to increase awareness and promote the use of research findings on using technology to deliver virtual care home visits for those with Parkinson’s disease. According to PCORI, these funds will be used to train neurologists and other health professionals to provide virtual care for patients in their homes. In addition, as part of its efforts to summarize research findings, PCORI also awarded funds to the American Institutes for Research to establish a Translation Center that develops two summaries of each of PCORI’s research findings: a public abstract for general audiences that is also translated into Spanish and a professional abstract for clinicians.

In addition to awards, PCORI has fostered the dissemination and implementation of its research findings in other ways, including through its website, publications, and roundtable briefings. For example, according to PCORI, it posts research findings on its website within 90 days of receiving final peer-reviewed research results so that patients and providers have access to the information to make healthcare decisions. In addition, according to PCORI, it pays journals’ open access fees to allow free public access to selected research and plans to support research awardees to place accepted journal manuscripts in the PubMed Central database. PCORI also facilitates roundtable briefings that bring together clinicians, patients, and others with interests in recent findings in order to build support for immediate use of the findings. PCORI also coordinates its dissemination efforts with AHRQ.

PCORI considers the implementation of its research methods and findings to be an integral part of its dissemination efforts and a culmination of its work and so has begun efforts to track implementation, such as the number of its findings published in peer-reviewed journals, and the use of its findings in clinical care. For example, PCORI officials told us that that there were 891 publications in peer-reviewed journals that resulted from studies fully or partially funded by PCORI through October 2017. According to PCORI, two PCORI-funded studies on prostate cancer, one study on oral versus intravenous antibiotics for certain children, and one study on self-monitoring of blood glucose were included in medical resource software that is used by nearly 90 percent of academic medical centers in the United States.

Most of the stakeholder officials we interviewed noted the importance of disseminating research findings quickly and in ways that are readily available and understandable to both experts and the general public to raise awareness about the findings. While officials representing two payers noted limitations to the usefulness of PCORI’s research findings because they do not take treatment costs into account, most stakeholder officials noted the importance of the PCORI-funded research underway and looked forward to utilizing the research findings once they become available. In particular, officials representing provider and patient advocacy organizations told us that they were interested in ensuring that the most important research findings would be quickly implemented by patients and clinicians.

HHS Obligated Funds Primarily for the Dissemination and Implementation of Research

Between fiscal years 2011 and 2017, HHS’s AHRQ obligated about $260 million (or 58 percent of HHS’s $448 million in total obligations) for the dissemination and implementation of CER findings. According to AHRQ officials, because most PCORI-funded research had not been completed by the end of fiscal year 2017, these efforts were primarily focused on the dissemination and implementation of research funded by other entities, including NIH and the Centers for Disease Control and Prevention (CDC). Additionally, AHRQ obligated $94 million for efforts to train researchers on conducting CER, and ASPE obligated $85 million for efforts to build data capacity. AHRQ and ASPE have obligated a total of $9 million for administrative activities during those years. Table 3 provides an overview of HHS’s obligations in each fiscal year.

AHRQ and ASPE plan to obligate an additional $120 million for dissemination and implementation, training, building data capacity, and administrative activities during fiscal years 2018 through 2020. They expect to have $245 million available to fund ongoing and future CER activities, based on expected transfers from the Trust Fund in fiscal years 2018 and 2019. (See fig. 6.)

The following information provides details on HHS-funded projects related to dissemination and implementation, training on conducting CER, and building data capacity.

Dissemination and Implementation

During fiscal years 2011 through 2017, AHRQ obligated a total of $260 million for CER dissemination and implementation initiatives and plans to obligate an additional $93 million for these initiatives in fiscal years 2018 through 2020. According to officials, AHRQ plans to fund additional dissemination and implementation initiatives in fiscal years 2018 and 2019 but had not finalized those plans as of January 2018. (See app. II for an overview of all of AHRQ’s dissemination and implementation initiatives.) AHRQ’s dissemination and implementation initiatives comprise efforts to synthesize CER findings, translate and communicate research findings to potential users, and implement them:

Synthesis of CER findings: According to AHRQ officials, AHRQ’s Evidence-Based Practice Centers developed 48 systematic reviews of CER findings based on completed research. As of the end of fiscal year 2017, 40 of these reviews had been published, while 8 were still in progress. Officials told us that these systematic reviews have likely not included PCORI-funded research, as most of that research had not been completed by the end of fiscal year 2017.

Translation and communication of CER findings: AHRQ funded initiatives, which—according to the agency—are aimed at making CER findings accessible and understandable to health care professionals, patients, and others. For example, AHRQ developed a “Library of Patient-Centered Outcomes Research Resources” website with links to CER databases maintained by other entities including NIH and PCORI. Another example is AHRQ’s “John M. Eisenberg Center for Clinical Decisions and Communications Science,” which translates research findings into information that can be used by consumers, health care providers, and policymakers.

Implementation of CER findings: AHRQ funded four key initiatives to implement CER findings. According to AHRQ officials, one of the four initiatives includes PCORI-funded research, while the other three have thus far focused on implementing existing CER funded by other entities:

The “Dissemination and Implementation Initiative” was designed to disseminate and implement government-funded CER findings— including PCORI-funded findings—relevant to physicians, healthcare providers, patients, and others. This initiative consists of a multi-step approach for identifying several areas of CER each year that—according to AHRQ officials—have the greatest potential for impact and are feasible to implement. (See figure 7 for an overview of this process.) According to AHRQ officials, as of December 2017, 37 findings have been nominated for consideration under AHRQ’s Dissemination and Implementation Initiative, including 5 findings nominated by PCORI. According to these officials, 1 of the findings PCORI has nominated is under consideration for implementation. Two were rejected—1 because of insufficient impact and the other because of challenges in implementation feasibility. (Two are still under review.)

The “Evidence Now” initiative disseminates CER evidence directly to primary care practices and supports them in implementing clinical and organizational evidence in practice through regional cooperatives.

The “Comparative Health System Performance Initiative” established three centers of excellence and a coordinating center to identify, classify, track, and compare health systems. AHRQ’s goal is to understand the factors that affect health systems’ use of CER and to identify best practices in disseminating and using CER.

The “Clinical Decision Support (CDS) Initiative” is designed to use CDS to promote the timely incorporation of CER findings into clinical practice.

Some of AHRQ’s dissemination and implementation initiatives—such as “Evidence Now” and “CDS Initiative”—include an evaluation component, as described in app. II. According to AHRQ officials, as of January 2018 results from these evaluations were not yet available.

Training on Conducting CER

Between fiscal years 2011 and 2017, AHRQ obligated a total of $94 million for awards supporting training in the methods used to conduct CER. AHRQ plans to obligate an additional $14 million for training on conducting CER by fiscal year 2020. AHRQ has funded eight categories of awards for individual researchers or research institutions. For example, AHRQ’s “Infrastructure Development Program in Patient-Centered Outcomes Research” award supports institutions in the development of their capacity to conduct and implement CER. Its “Institutional Mentored Career Development Award Program in Patient-Centered Outcomes Research” award supports the development of researchers in academic and applied settings. (See app. III for an overview of these awards.) Starting in fiscal year 2018, AHRQ plans to fund an additional training award category in conjunction with PCORI. AHRQ developed a plan to evaluate its training activities and, according to AHRQ officials, the evaluation is expected to be funded in fiscal year 2018.

Building Data Capacity

Between fiscal years 2012 and 2017, ASPE obligated a total of $85 million for 30 projects designed to build data capacity for conducting CER and plans to obligate an additional $6 million to existing projects and 1 new project through fiscal year 2019. Officials told us that ASPE plans to fund additional projects to build data capacity in fiscal years 2018 and 2019, based on HHS leaders’ priorities, but had not finalized those plans as of January 2018. ASPE manages these projects, which are largely carried out by other HHS agencies through interagency agreements and are intended to develop and maintain a comprehensive, interoperable data network to collect, link, and analyze data on outcomes and effectiveness from multiple sources for CER. (See app. IV for an overview of these activities.)

In response to a recommendation in our 2015 report on HHS’s CER activities, ASPE implemented a monitoring system to track progress toward its milestones and deliverables for these projects. ASPE also contracted to evaluate its projects to build data capacity for CER. The evaluation, completed in December 2017, found that ASPE made progress managing these projects towards the core functionalities outlined in its strategic framework. However, among other things, the evaluation found that additional efforts are needed to explore how to enhance data privacy and security, ensure data quality, and operationalize related standards. According to ASPE officials, the evaluation will inform the development and implementation of future ASPE projects to build data capacity for conducting CER.

Agency Comments

We provided a draft of this report to PCORI and HHS for review and comment. PCORI and HHS provided technical comments, which we incorporated as appropriate.

We are sending copies of this report to the appropriate congressional committees, the Executive Director of PCORI, the Secretary of Health and Human Services, the Director of AHRQ, the Assistant Secretary for ASPE, and other interested parties. In addition, the report will be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-7114 or dickenj@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in app. V.

Appendix I: Patient-Centered Outcomes Research Institute (PCORI) Award Commitments Made During Fiscal Years 2010 through 2017

Appendix I: Patient-Centered Outcomes Research Institute (PCORI) Award Commitments Made During Fiscal Years 2010 through 2017 Award category description These awards generally fund research studies in priority areas for conditions that impose a substantial burden on patients and the healthcare system. Information about individual research awards can be found at: https://www.pcori.org/research-results?f%255B0%255D=field_project_type%3A298&f%5B0%5D=fiel d_project_type%3A298#search-results.

These awards fund infrastructure projects to build data capacity through the development of PCORnet and support for clinical and patient-powered data research networks. Information about individual building data capacity awards can be found at: https://www.pcori.org/research-results?f%5B0%5D=field_project_type%3A441#search-results.

Engagement awards fund projects to improve the methodology for carrying out research by involving patients, caregivers, clinicians, and other healthcare stakeholders into the research process. Workforce training awards provide accredited continuing education opportunities, in coordination with the Agency for Healthcare Research and Quality, for researchers and clinicians. Information about individual engagement awards and workforce awards can be found at: https://www.pcori.org/research-results?f%5B0%5D=field_project_type%3A299#search-results and https://www.pcori.org/research-results/2017/k12-institutional-mentored-career-development-program.

Dissemination and implementation awards are intended to help researchers and other stakeholders to publicize findings, and support the utilization of findings for patients and providers. Information about individual dissemination and implementation awards can be found at: https://www.pcori.org/research-results?f%5B0%5D=field_project_type%3A308#search-results.

Appendix II: Agency for Healthcare Research and Quality’s (AHRQ) Dissemination and Implementation Initiatives, as of September 30, 2017

Appendix II: Agency for Healthcare Research and Quality’s (AHRQ) Dissemination and Implementation Initiatives, as of September 30, 2017 Description Disseminates evidence to primary care practices and supports them in implementing clinical and organizational evidence in practice through regional cooperatives. AHRQ awarded a separate grant to establish an independent, external evaluation to study improvements in the delivery of the ABCs.

Develops three Centers of Excellence on “Comparative Health System Performance in Accelerating PCOR Dissemination.” According to AHRQ, the Centers of Excellence will identify and classify characteristics of health care systems over 5 years. They will also identify ways to assess the quality and cost of such systems, including their use of PCOR, understand the characteristics of high performing systems, and identify what system characteristics are associated with more rapid adoption and diffusion of PCOR- recommended practices throughout a system.

Evaluates and synthesizes research findings to aid decision-making for patients, providers, and payers, among others.

Translates PCOR findings into tools, such as research summaries and decision aids, designed to help patients and consumers, clinicians, and policymakers make informed and evidence-based health care decisions.

Expands an existing initiative or creates a new initiative that supports multi-site, multi-region, multi-stakeholder dissemination and implementation of evidence.

Develops and tests methods for translating and disseminating PCOR findings to hard-to- reach audiences, including patients with low health literacy, disadvantaged populations, isolated clinicians and policy makers, and other decision makers who may not have had the benefit of more traditional translation and dissemination efforts.

Integrates PCOR into clinical practice using various methods shown to improve the uptake of scientific evidence in clinical decision making. Grantees were asked to consider both educational theory and the relevance of “new media” as they designed their programs.

Searches for emerging interventions, prioritizes those most likely to have a large impact in the near future, and disseminates the information to the public. According to AHRQ, the Horizon Scanning System screened more than 22,000 potential intervention leads and tracked over 2,300 intervention topics.

Provides targeted audiences—such as providers and payers—with an accessible tool for obtaining objective, detailed information on evidence-based clinical practice guidelines to further their dissemination, implementation, and use.

Promotes collaboration, reduces redundancy, and improves transparency in patient registries.

Description Created informational tools to support the dissemination and implementation of PCOR findings, including best practices and new knowledge about the use of electronic health record data for research and quality improvement.

Promotes the timely incorporation of PCOR findings into clinical practice—which encompasses a variety of tools to enhance clinical decision-making.

Collaborated with 176 national organizations to disseminate materials for the Effective Health Care Program.

Multi-media campaign to educate health care consumers about the value of reviewing medical evidence when weighing treatment options.

Educating the Educators Conducted and disseminated research to develop a process for shared decision making that includes exploring and comparing the benefits, harms, and risks of each option through meaningful dialogue about what matters most to the patient.

Collects patient-generated health data, integrates patient-generated health data with PCOR evidence, and disseminates PCOR findings using mobile health technology.

Identifies ways to reduce health care differences across diverse populations with a particular focus on minority populations in under-resourced healthcare settings.

Established five regional offices, responsible for developing and cultivating dissemination partnerships within each region.

This repository houses study data extracted from primary research publications during the course of conducting systematic reviews. It is designed to increase the transparency of comparative effectiveness reviews, improve the ability to update systematic reviews, improve the quality of abstracted data, and enhance the efficiency and reduce the costs of conducting reviews.

Gathers input from patients on a complex topic related to the implementation of evidence- based health-care decision making.

Increases the relevance of AHRQ systematic reviews for patients, clinicians, and policymakers by examining and addressing challenging topic areas that may affect the credibility and utility of the review for end users and that are areas of inconsistency or variation among AHRQ systematic reviews.

Conducted three projects to improve the development of registries, a major activity of AHRQ’s Effective Health Care Program.

Description Provides online continuing education materials that inform physicians and other health care providers about PCOR from the Effective Health Care Program.

Worked with health professional student associations to evaluate students’ understanding of the importance and clinical applicability of PCOR and shared decision-making to their practice and evaluated students’ educational needs and preferences related to integrating PCOR findings into their training curricula.

Created a decision-modeling methods center that reviewed the existing research and guidance published on modeling methods with input from a multidisciplinary group of experts.

Provides for maintenance and updating of existing data resources to conduct future CER through a grant competition. The grants fund three to four 1-year pilot projects aimed at enabling a future, larger competition to enhance the data infrastructure and move the resources to self-sustaining models.

Disseminates CER findings published by the Patient-Centered Outcomes Research Institute (PCORI) and other government entities to providers, patients, payers, and others. This initiative consists of a seven-step approach for identifying research findings that have the greatest potential for implementation.

Provided an understanding of how AHRQ could effectively disseminate and promote PCOR findings and tools in the development and maintenance of clinical decision support systems. The project included a market analysis and an assessment of potential stakeholders and audiences, including vendors of health information technology focused on clinical decision support. Information gathered from this project directly informed the concept for the PCOR clinical decision support initiative that was launched in 2016.

Promoted PCOR through public service announcements nationwide.

Created a new page on AHRQ’s website that highlights the agency’s own resources, as well as directs researchers, health professionals, patients, caregivers, and families to additional databases that collect information on CER. These databases provide summaries of findings from a wide range of CER findings and research that is in progress.

PCOR is a form of CER.

Appendix III: Agency for Healthcare Research and Quality’s (AHRQ) Training Awards, as of September 30, 2017

Appendix III: Agency for Healthcare Research and Quality’s (AHRQ) Training Awards, as of September 30, 2017 Description Funds a 5-year, renewable effort to support the development of PCOR capacity among institutions that have basic health services research capacity but need to develop capacity to conduct and implement PCOR. The program would potentially include institutions located in geographic areas that lack capacity, and institutions that serve predominantly minority populations.

Supports the development of researchers in academic and applied settings. The program combines didactic and experiential opportunities, focusing on the generation, adoption, and spread of new scientific evidence. The goal is to improve population-specific health outcomes by developing and disseminating evidence-based information to patients, clinicians, and other decision-makers, responding to their expressed needs, about which interventions are most effective for which patients under specific circumstances.

Provides basic, advanced, and experiential training on the methods to conduct PCOR, particularly prospective observational research, registries, and clinical trials. The program was open to researchers employed in both the public and private sectors, particularly those who serve minorities, economically or medically disadvantaged populations.

Facilitates the transition of postdoctoral candidates from mentored to independent research positions, accelerating research independence for PCOR researchers.

Provides support for intensive, research career development for individual investigators in academic or applied settings, leading to research independence in the field of PCOR and the generation and translation of new scientific evidence and analytic tools.

Provides career development awards for established investigators to further develop their research expertise in PCOR methodologies. This concept seeks to accelerate the development of the research workforce capable of conducting PCOR.

Provides 2-year fellowships for training in PCOR. A focus for these fellowships is recruitment of trainees from diverse disciplines, including social and behavioral sciences, business, and engineering. The expected output of these fellowships is trained PCOR researchers.

Establishes an expert panel, comprised of 7 to 10 leaders in the fields of learning healthcare system, health services research, and PCOR, to assess the current state of health services research and PCOR training and recommend ways to improve core competencies/curriculum to meet the needs of the health system. Develops a report summarizing the panel’s recommendations concerning current deficiencies and recommendations regarding skills and competencies needed to meet the challenges.

PCOR is a form of CER.

Appendix IV: Office of the Assistant Secretary for Planning and Evaluation’s (ASPE) Projects to Build Data Capacity, as of September 30, 2017

Total obligations (dollars in millions)

Developed technical standards for how health care providers, researchers, and the public health community access and extract data from electronic health records to conduct Patient-Centered Outcomes Research (PCOR).

Identified and developed the functional and technical specifications necessary to enable electronic health record systems to retrieve, display, and fill a structured form or template and store and submit the completed form to an external repository.

Provided researchers with access to the Centers for Medicare & Medicaid Services’ Chronic Conditions Warehouse, which contains Medicare and Medicaid beneficiary, claims, and assessment data, and supported infrastructure enhancements to conduct CER.

Longitudinal follow-up of certain cancer patients to assess vital statistics, disease recurrence, disease progression, and additional treatment types. Treatment data submitted each year to the Centers for Disease Control and Prevention and provided to researchers through the National Center for Health Statistics Research Data Center.

Included clinical encounters for all patients and all conditions seen at the community health centers from 2006 to 2013 in the Community Health Applied Research Network Registry data warehouse, a research network comprising 18 community health centers. A de-identified analytic file and associated data codebook were developed to support the use of analytic files by researchers outside of the network. Established a process for investigators to access the data warehouse through the development of a data access plan.

Maintained the infrastructure for PCOR and for quality improvement in the safety net.

Developed common data elements and standards for CER .The results were the initial entries into the National Institutes of Health’s National Library of Medicine common data element repository.

Developed a conceptual framework and environmental scan; produced policy documents ranging from patient-initiated data, through research data on care processes, transitions and coordination, to researcher access to claims data; and developed the ‘HHS Strategic Roadmap for Building Data Capacity for Clinical Comparative Effectiveness Research.’ The overall CER Inventory project was to design and implement a system for the categorization and cataloguing of CER activities through a web-based tool. Due to the rapidly evolving technologies supporting web-based search engines, and the improved methods for identification of more recent CER, the development of the CER Inventory (as a web-based search engine using a retrospective algorithm) was determined to have been superseded by existing search engine tools available.

Description Designed and conducted an independent evaluation of the ASPE portfolio to systematically assess progress related to the strategic framework functionalities.

Total obligations (dollars in millions)

Conducted CER analyses on the beta release of the Multi-Payer Claims Database and evaluated beta testers’ experiences requesting and using data from the MPCD for research. Results of the beta test found that the project was successful in achieving the key objectives of building a pilot database.

Planned for development and implementation of the Centers for Medicare & Medicaid Services’ Blue Button—a service that allows patients to access their own health information in electronic form.

Linkage of data on fact, cause, and manner of death from the National Death Index to several federal population-based health data platforms in order to demonstrate the feasibility of such linkage, enable PCOR on patterns and correlates of mortality via the resulting linked data; and to facilitate collaboration between federal partners regarding strengthening the infrastructure and methods for linking healthcare data to mortality outcomes and using such linked data for PCOR.

Improve the infrastructure to support timely and complete mortality data collection through more timely delivery of state death records to the National Death Index database and by linking National Death Index database records with nationally collected hospital datasets to obtain a more complete picture of patient care.

Identify the best patient attributes to address the challenge of linking patients’ data across research, clinical, and claims data sets in order to support the PCOR data infrastructure that enables standardization and sharing of patient data across organizations.

Create a coordinated registry network for women’s health technologies that will collect patient reported outcomes and employ structured data capture from electronic health records for data collection and exchange.

Build data infrastructure for conducting PCOR using data from routine clinical settings. The sources of these data may include, but are not limited to, insurance billing claims, electronic health records, and patient registries. This project intends to harmonize several existing common data models, potentially including PCORnet and other networks.

Develop technical tools for collecting and integrating patient-reported outcome assessments into electronic health records or other health information technology products.

Create an interface that enables CMS beneficiaries to connect their MyMedicare.gov data to applications and services they trust, including research platforms related to research studies in which the beneficiary may be interested in participating.

Provide technical assistance to the Trust Fund awardees in informatics and assist ASPE in setting up additional oversight processes and procedures to monitor progress.

Description Develop a privacy and security data infrastructure blueprint, legal analysis, and ethical framework to address legal and privacy and security related policy issues that affect the use of data for various types of PCOR.

Convene clinical topic-specific working groups to discuss the data definitions currently in use and how these definitions can be harmonized to promote common definitions for outcome measures across systems. These common definitions are to be made publicly available to PCOR researchers and analysts.

Develop a natural language processing service that will be accessible and publicly available to researchers on the Public Health Community Platform – a cooperative platform for sharing interoperable technologies to address public health priority areas aimed at improving population health outcomes and health equity (e.g., tobacco use).

Leverage the Sync for Science and Blue Button application programming interface programs to enable Medicare beneficiaries to donate their medical claims data for scientific research studies.

Develop and test the capability to conduct timely and secure distributed regression analysis in distributed data networks. Additionally, explore the feasibility of creating virtual linkage capabilities to utilize data from multiple data sources and data for one specific patient with information at different institutions.

Create the infrastructure for collecting data from patients through a mobile device application, allowing patient-generated data to be linked with a single data partner that participates in the Food and Drug Administration’s Sentinel distributed network. The project will develop and pilot a mobile application to capture data from pregnant women who volunteer to participate.

Develop a policy framework for the use of patient-generated data in research and care delivery that addresses data collection tools, data donation policies, regulatory gaps, combining data with medical record data, and interoperability of data across health information systems and devices.

Create and implement a metadata standard data capture and querying system for data quality and characteristics, data source and institutional characteristics, and “fitness for use.”

Cross-Network Directory Service Create an interoperable service that allows data partners to participate in multiple data research networks, query across the networks, and share analytic capabilities and knowledge across networks. The project will be piloted across two existing networks: Food and Drug Administration’s Sentinel and PCORnet.

Generate tools and data standards that could be deployed in other CER studies by leveraging the infrastructure of an existing research study called the ADAPTABLE trial (Aspirin Dosing: A Patient-Centric Trial Assessing Benefits and Long Term Effectiveness). This trial is the first major randomized comparative effectiveness trial to be conducted by PCORnet.

Description Create a flexible, extensible, and computable mechanism for rolling data into clinically relevant equivalence groups that enable more efficient processing aggregation of laboratory data and other data from diverse health information technology systems. The primary focus of this work will be on laboratory tests.

Total obligations (dollars in millions)

Create a single point data capture approach from the electronic health record to electronic data capture systems using the Retrieve Form for Data Capture standard. Stakeholders will be provided with a tool to seamlessly integrate electronic health record and electronic data capture systems.

In addition to the projects listed, ASPE plans to obligate $2.0 million for one new project starting in fiscal year 2018.

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Karin Wallestad, Assistant Director; Michael Zose, Analyst-in-Charge; Kye Briesath; Laurie Pachter; Vikki Porter, and Jennifer Whitworth made key contributions to this report.