Comparative Effectiveness: Initial Assessment of the Patient-Centered Outcomes Research Institute
Released: 2015-03-09T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: In 2010, PPACA authorized the establishment of PCORI as a federally funded, nonprofit corporation to improve the quality and relevance of CER. PCORI, which began operation in 2010, is required to identify research priorities, establish a research project agenda, fund research consistent with its research agenda, and disseminate research results, among other things. To fund PCORI, PPACA established the Patient-Centered Outcomes Research Trust Fund, through which the institute is expected to receive an estimated $3.5 billion from fiscal years 2010 through 2019.

PPACA mandated that GAO review PCORI's activities by 2015 and 2018. This report examines (1) the extent to which PCORI established priorities and processes for funding and disseminating comparative clinical effectiveness research consistent with its legislative requirements; (2) the status of PCORI's efforts to fund comparative clinical effectiveness research; and (3) PCORI's plans, if any, to evaluate the effectiveness of its work. GAO reviewed relevant legislative requirements and PCORI documentation, including funding data, and interviewed PCORI officials. GAO also interviewed relevant stakeholders, including health policy experts and PCORI contractors. PCORI provided technical comments, which GAO incorporated as appropriate.

section_title: What GAO Found
paragraphs: The Patient-Centered Outcomes Research Institute (PCORI) has established priorities and processes for funding comparative clinical effectiveness research (CER)—which is research that evaluates and compares health outcomes and the clinical effectiveness, risks, and benefits of two or more medical treatments, services, or items such as health care interventions—and is developing dissemination plans, consistent with the legislative requirements of the Patient Protection and Affordable Care Act (PPACA). In 2012, PCORI established five broad research priorities: (1) assessment of prevention, diagnosis, and treatment options; (2) improving health care systems; (3) researching communication and dissemination strategies; (4) comparing interventions to reduce health disparities; and (5) accelerating patient-centered outcomes research and methodological research. PCORI also developed a research agenda to identify how each priority would be addressed. PCORI has established a multi-step research funding process designed to assess and select contract applications for funding. Funded contracts are monitored by PCORI staff. Per legislative requirement, PCORI is developing a peer review assessment process to review final reports submitted by contract awardees and is in the process of developing a plan for the dissemination of funded research potentially beginning in 2015, in coordination with the Agency for Healthcare Research and Quality.

PCORI has started awarding contracts for research and plans to award additional contracts through 2019. As of October 2014, PCORI has awarded 360 contracts to fund research projects, committing a total of $670.8 million to them. PCORI expects to commit about $2.6 billion to research contracts, out of $3.5 billion in total estimated spending. Approximately $106 million in commitments to date are for PCORnet, a data research network aimed at improving the capacity for and speed of conducting CER. PCORI officials stated that they expect to spend a total of $271 million on PCORnet through fiscal year 2019. PCORI officials stated that limited amounts of data will be available through PCORnet for researchers to use after September 2015 with the amount of available data increasing over time.

PCORI has established an evaluation plan and is developing efforts to measure outcomes. PCORI has developed initial plans for evaluating the institute's efforts against its three strategic goals, which are to increase information, speed implementation, and influence research. To do so, PCORI has developed primary outcome measures for assessing PCORI's progress related to these strategic goals. In its strategic plan, PCORI notes that these are meant to be long-term measures because research typically requires several years to complete and additional years for the results to be disseminated and implemented. Therefore, since 2013, PCORI has been using early and intermediate process and output measures—such as the number of people accessing or referencing PCORI information—as a way to monitor its progress toward its strategic goals. PCORI anticipates having some early results related to its primary outcome measures starting in 2017 after the first CER studies are completed and their findings released, although full evaluation of the results of these outcome measures will not be possible until around 2020.

section_title: Letter
paragraphs: In 2010, the Patient Protection and Affordable Care Act (PPACA)  authorized the establishment of the Patient-Centered Outcomes  Research Institute (PCORI) as a federally funded, nonprofit corporation to  improve the quality and relevance of comparative clinical effectiveness  research (CER). CER, research that evaluates and compares health  outcomes and the clinical effectiveness, risks, and benefits of two or more  medical treatments, services, or items such as health care interventions,  is important because more than half of all medical treatments may be  delivered without clear evidence of effectiveness, according to the  Institute of Medicine (IOM). The results of CER can be used by patients  and clinicians to make informed health care decisions about which  treatment or intervention may be most effective or beneficial for a  patient.

PCORI, which began operation in 2010, is required to identify research  priorities, establish a research project agenda, fund research consistent  with its research agenda, and disseminate research results, among other  things. PCORI seeks to make its research patient centered by  incorporating patients and other stakeholders in the research process,  and PCORI refers to the type of CER it conducts as patient-centered   To fund PCORI, PPACA established the  outcomes research (PCOR).Patient-Centered Outcomes Research Trust Fund (PCORTF), through  which the institute is expected to receive an estimated $3.5 billion from  fiscal years 2010 through 2019. Under current law, PCORI may not  make any expenditures from the PCORTF after September 30, 2019, at  which point any remaining funds are to be transferred to the general fund  of the United States Treasury.

PPACA mandated that GAO review PCORI’s activities in 2015. report examines (1) the extent to which PCORI established priorities and  processes for funding and disseminating comparative clinical  effectiveness research consistent with its legislative requirements; (2) the  status of PCORI’s efforts to fund comparative clinical effectiveness  research; and (3) PCORI’s plans, if any, to evaluate the effectiveness of  its work.

PPACA also mandated GAO to review HHS’s CER-related activities in 2015 and 2018.  GAO completed the first of these reviews in a separate report. (See GAO, Comparative  Effectiveness Research: HHS Needs to Strengthen Dissemination and Data-Capacity- Building Efforts GAO-15-280, Washington, D.C.: March 3, 2015). In addition, GAO has  completed annual reviews mandated by PPACA of PCORI’s financial statement audits  and also issued two previous reports on CER in response to Congressional requests. See  GAO, HHS Research Awards: Use of Recovery Act and Patient Protection and Affordable  Care Act Funds for Comparative Effectiveness Research, GAO-11-712R (Washington,  D.C.: June 14, 2011); Comparative Effectiveness: Agency for Healthcare Research and  Quality’s Process for Awarding Recovery Act Funds and Disseminating Results,   GAO-12-332 (Washington, D.C.: February 29, 2012); Patient-Centered Outcomes  Research Institute: Review of the Audit of Financial Statements for 2013 and 2012,   GAO-14-415R (Washington, D.C.: March 27, 2014); and Patient-Centered Outcomes  Research Institute: Review of the Audit of the Financial Statements for 2012 and 2011,  GAO-13-390R (Washington, D.C.: March 29, 2013). priorities, what plans PCORI has for updating these priorities, what  procedures PCORI employs for reviewing and selecting applications to  fund, how PCORI monitors its contractors, and how PCORI plans to  disseminate research results. We compared information gathered from  documentation and interviews to PCORI’s legislative requirements. In  addition, we determined the extent to which PCORI coordinated the  development of its research priorities and proposed or funded CER  projects with the National Institutes of Health (NIH) to ensure that its  projects were not unnecessarily duplicative of projects proposed or  funded by NIH. We also interviewed staff from both PCORI and the  Agency for Healthcare Research and Quality (AHRQ) to learn about each  organization’s efforts to coordinate dissemination activities and plans for  disseminating PCORI-funded research in the future. Finally, we  interviewed relevant stakeholders: two public policy organizations, a  health insurance trade association selected because it has provided input  to PCORI on its work, and two health policy experts selected based on  their knowledge of comparative clinical effectiveness research.

To determine the status of PCORI’s research projects, we collected and  analyzed funding data for all projects PCORI has funded as of October  2014, which was the most current data available. We relied on data  provided by PCORI; while we did not audit this reported data, we  assessed its reliability by reviewing existing information, conducting  quality control checks, and interviewing PCORI officials knowledgeable  about the data, and determined that they were sufficiently reliable for our  purposes. We also obtained written documents that outline PCORI’s  plans to fund future research, describe funded projects currently in  progress, and describe how PCORI funding is distributed across different  research areas. We interviewed PCORI officials to determine the status of  funded projects and how PCORI makes decisions to fund different areas  of research. Finally, we interviewed six PCORI contractors that are  conducting work related to a data research network being established by  PCORI.

To describe PCORI’s plans to evaluate its work, we collected relevant  documentation from PCORI on its evaluation plans, such as strategic  planning documents and documents about its evaluation framework that  contain evaluation metrics developed to date. We interviewed PCORI  officials to learn about the institute’s plans for evaluating the effectiveness  of its work.

We conducted this performance audit from April 2014 to March 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Since the enactment of the Medicare Prescription Drug, Improvement,  and Modernization Act of 2003 (MMA), AHRQ, an agency within the  Department of Health and Human Services (HHS), has been one of  several federal agencies responsible for supporting and disseminating the  results of CER. The dissemination of CER refers to developing and  distributing information derived from CER for target audiences, such as  clinicians, consumers, or policymakers, in order to inform health care  delivery or practice. This process involves translating research findings  into terminology and materials that are appropriate for the target  audience. Specifically, AHRQ has supported CER activities by awarding  grants and contracts to research centers and academic organizations to  carry out this work, which includes reviewing and synthesizing scientific  evidence through research reviews; generating new scientific evidence  and analytical tools in original research reports; compiling research  findings; and communicating those findings to a variety of audiences.  Under the American Recovery and Reinvestment Act of 2009 (Recovery  Act), AHRQ received funding of $474 million to support and disseminate  the results of CER—$300 million that was appropriated to AHRQ and  $174 million that was appropriated to the HHS Office of the Secretary and  allocated to AHRQ. The Recovery Act also required the Secretary of HHS  to enter into a contract with the IOM to produce a report that included  recommendations on research questions that should receive national  priority for study with CER funds made available under the act.

The  result of this work included a list of 100 research questions prioritized for  CER.

In 2010, PPACA authorized the establishment of PCORI as a nonprofit  corporation aimed at advancing the quality and relevance of evidence  through CER to help patients, clinicians, purchasers, and policy-makers in  making informed health care decisions. The law requires PCORI to  perform a number of duties related to CER. (See table 1.)

PCORI is governed by a 21-member Board of Governors and employs  153 staff, as of September 2014, including directors for its different  program areas as well as staff for engagement, communications, contract  management, finance, human resources, and information technology.  PCORI’s 17-member methodology committee defines methodological  standards for its research.

section_title: PCORI Has Established Priorities and Processes for Funding Comparative Clinical Effectiveness Research and Is Developing Dissemination Plans, Consistent with Its Legislative Requirements
paragraphs: PCORI has developed five broad research priorities and developed a  research agenda to identify how each priority will be addressed. The  institute has established a multi-step merit review process to score and  identify applications for funding and a process for monitoring contractors.  PCORI is also developing a peer review process for primary research and  a dissemination plan for completed research.
subsections: section_title: PCORI Has Established Research Priorities and a Research Agenda
paragraphs: In 2012, PCORI established five broad research priorities: (1) assessment  of prevention, diagnosis, and treatment options; (2) improving health care  systems; (3) communication and dissemination research; (4) addressing  disparities; and (5) accelerating patient-centered outcomes research and  methodological research. PCORI also developed a research agenda to  identify how each priority would be addressed. The research agenda  contains a set of more specific research areas within each priority. PCORI  expects its research agenda to be updated and refined over time based  on more specific analyses of gaps in research. (See table 2.)

PCORI issued its research priorities and agenda in May 2012, following a  process that began in July 2011. In the fall of 2011, PCORI formed two  workgroups within its Board of Governors—the National Priorities for  Research Workgroup and the Research Agenda Workgroup. Along with  PCORI staff and members of the Methodology Committee, these  workgroups examined the processes and products of other recent priority-  and agenda-setting efforts, including those from AHRQ and IOM. They  also engaged with and received input from stakeholder groups through a  number of public presentations and other modes of communication, such  as press releases, focus groups, and feedback through social media.

PCORI posted on its website its draft research priorities and agenda for  public comment from January 23, 2012 to March 15, 2012. In response,  PCORI received and analyzed a total of 474 formal comments, and made  changes to the draft priorities and agenda based on these comments. A  final version of PCORI’s research priorities and agenda were adopted by  the Board of Governors and made publicly available in May 2012.

PCORI established its research priorities and agenda consistent with  PPACA requirements. Specifically, PPACA directed PCORI to establish  priorities for research that take into account factors that include disease  incidence, prevalence, and burden; gaps in evidence with regard to  clinical outcomes; patient needs, outcomes, and preferences; and  practice variations and health disparities in terms of delivery and  outcomes of care, among other things. The act also directed PCORI to  develop a research agenda for addressing these priorities. The act did not  specify the content or form of the priorities or agenda. According to  documentation on the process PCORI used to develop its research  priorities and agenda, the workgroups reviewed and considered these  requirements. In the process of developing the research agenda, the  workgroups identified where, for example, certain agenda items  addressed criteria such as gaps in knowledge, variation in care, and  inclusiveness of different populations.

According to PCORI officials, PCORI intended its research priorities to be  broad in scope so that they could encompass a broad range of topics in  need of study, including different disease areas, conditions, or health care  system issues, thus making PCORI’s research agenda more flexible than  if it identified specific topics. PCORI noted in its research priorities and  agenda document that PCORI did not want to exclude any disease from  being studied. Some stakeholders we interviewed expressed concern that  PCORI’s research priorities are too broad and lack specificity. While  PCORI officials acknowledged that many of the institute’s initial funding  announcements were broad in that they did not identify specific topics,  they noted that they are in the process of increasing the proportion of  funding that goes to specific research topics. For example, PCORI will  soon begin funding awards for large pragmatic clinical studies to evaluate  patient-centered outcomes. The funding announcement for this effort  indicates a number of specific research topics of interest to PCORI.  PCORI officials noted that applications which are responsive to these  specific research topics will be given priority for funding. The specific  research topics identified in this funding announcement were developed  using, among other things, stakeholder recommendations, IOM’s CER  Priorities, and input from PCORI’s advisory panels.

To identify more specific research questions and topics for use in funding  announcements, PCORI utilizes advisory panels, as authorized by  PPACA. PPACA directed PCORI to establish advisory panels for rare  diseases and clinical trials, which PCORI established in November 2013,  and also authorized PCORI to establish other advisory panels as   needed. In addition to establishing the two advisory panels required by  PPACA, PCORI has also established five additional advisory panels:   (1) Assessment of Prevention, Diagnosis, and Treatment Options;   (2) Improving Healthcare Systems; (3) Addressing Disparities; (4) Patient  Engagement; and (5) Communication and Dissemination Research. Four  of these advisory panels align with PCORI’s research priorities. To  appoint members to its advisory panels, PCORI solicits applications via  its website. Advisory panel applicants are reviewed based on common  criteria established by PCORI and against the needs of the specific panel  for which a position is being filled. The PCORI Board of Governors makes  the final selection of advisory panel members.

PCORI’s advisory panels assist in the prioritization of research topics.  PCORI receives suggested research topics through a number of sources,  including stakeholders, social media, and workshops, as well as from  AHRQ, NIH, and professional and advocacy groups. As part of the  prioritization process, PCORI’s advisory panels evaluate and rank  suggested research topics using the following criteria: patient- centeredness, potential condition impact, assessment of current options,  likelihood of implementation in practice, and durability of information.  Topics identified by advisory panels as being of higher priority proceed for  further evaluation while the lower-priority topics enter a pool of topics that  may be reconsidered at a later date. Recommendations from PCORI’s  advisory panels are taken into consideration by PCORI’s staff and Board  of Governors and are used to refine and prioritize specific research topics  and inform the development of PCORI funding announcements. PCORI  may also use recommendations from advisory panels to commission  reviews of previous and current research on recommended high-priority  topics.

According to PCORI, its processes for identifying research include  mechanisms for avoiding unnecessary duplication and coordinating  research efforts with NIH and AHRQ. For example, when establishing its  broad research priorities, officials from both NIH and AHRQ participated  in PCORI’s workgroups to develop these priorities and the related  research agenda. As PCORI has developed more specific research  priorities for funding announcements, PCORI staff has consulted with  relevant NIH staff on specific topics in an effort to obtain expertise and an  understanding of what research is being funded on a particular topic. For  example, PCORI officials reported and NIH confirmed that when  developing a specific research topic related to cardiovascular disease,  PCORI coordinated with staff at NIH’s National Heart, Lung, and Blood  Institute to ensure that PCORI’s planned research in this area had not  already been sufficiently addressed by prior research. NIH submitted to  PCORI a list of potential topics it considered important but was not  actively funding, according to PCORI officials. PCORI officials also stated  that when a new topic is suggested to PCORI for inclusion in a funding  announcement, technical briefs are prepared to document the existing  work that has been completed on that topic. In addition, PCORI staff  stated that they conduct searches in ClinicalTrials.gov, NIH’s Research  Portfolio Online Reporting Tools Expenditure and Results (RePORTER),  and other research databases to determine if any similar research is in  progress or has already been funded.proposed research topic that is unnecessarily duplicative is eliminated.

section_title: PCORI Has Established Processes for Funding Research and Monitoring Contractors
paragraphs: PPACA directs PCORI to enter into contracts to carry out its research  priorities. PCORI has established a multi-step research funding process,  which includes merit review, that is designed to select high quality  research that has the best potential to improve patient outcomes,  according to PCORI officials. PCORI officials stated that their merit review  process is modeled on the peer review processes established by AHRQ  and NIH, which are described in law. Unlike AHRQ and NIH, however,  PCORI’s merit review process utilizes patients and stakeholders in the  review and scoring of applications. Key steps in PCORI’s research  funding process include the development and posting of funding  announcements, review and scoring of applications, and final approval by  the Board of Governors. (See figure 1.)

Submitted applications are assessed by reviewers recruited by PCORI  and selected based on expertise or knowledge in a particular subject  area. Reviewers may be patients, other stakeholders, or scientific  reviewers. Prior to reviewing applications, reviewers undergo web-based   Reviewers score applications using five standard criteria. (See  training.table 3.)

Applications are scored by reviewers through an online review, and a  subset is scored by the full panel during an in-person meeting. For the  online review, four reviewers are assigned to evaluate each application— two scientists and two stakeholders (one of them a patient). Scientific  reviewers focus on all 5 criteria, while the patient and stakeholder  reviewers focus on the potential of the study to improve health care and  outcomes, the extent to which the research is patient-centered, and the  extent to which the proposed research includes patient and stakeholder  engagement. Reviewers assign an overall score to the application and  provide written comments on each application’s specific strengths and  weaknesses. Scientific reviewers also check to determine if the  application adheres to PCORI’s methodological standards.is also involved, working with reviewers to ensure that each reviewer  understands how the applications should be assessed. Reviewers have  one month to electronically submit both their initial scores and detailed  written critiques to PCORI.

Following the online review, applications advance to in-person merit  review meetings where they are reviewed again in panels specific to the  funding announcement. To determine which applications will advance,  PCORI staff considers reviewers’ average overall scores for each  application, whether applications have scores that differ significantly  among reviewers (and could thus benefit from further discussion), and  whether applications received a good score for technical merit criteria and  are therefore a strong candidate for funding. Reviewers discuss the  applications’ merits and weaknesses and, as a panel, provide a final  overall application score. After applications are scored at the in-person  meeting, PCORI staff determine the applications that will be submitted for  review to a PCORI selection committee, composed of PCORI Board of  Governors members and up to one member from PCORI’s methodology  committee. The selection committee proposes a list of applications to  fund. According to PCORI staff, while the proposed applications generally  consist of the top-scoring applications, some lower-scoring applications  may be included to achieve program balance or fund research in critical  areas. The full Board of Governors votes to approve the recommended  applications. While the Board has the authority to make changes, PCORI  staff stated that it has not made such changes to date. Once the  recommended applications are approved, awards are announced to the  public via PCORI’s website, PCORI develops final contract requirements,  and research contracts are executed. PCORI officials stated that each  cycle of the research funding process takes 9 to 12 months to complete,  from the time a funding announcement is posted to the time  recommended applications are approved by the Board, of which the merit  review process takes about 4 to 6 months. There are currently up to four  funding cycles each year and PCORI staff noted that these cycles  overlap.

PCORI officials reported taking steps during the merit review and  application selection process to ensure PCORI’s funded research is not  duplicative of other research within the federal government or private  sector. For example, PCORI officials reported seeking input from NIH and  AHRQ during the final selection of awards through each agency’s  involvement in the selection committee and membership on PCORI’s  Board of Governors. PCORI officials also noted that in some instances  NIH and AHRQ staff assist PCORI in reviewing letters of intent submitted  in response to PCORI funding announcements. Finally, before funding an  application, PCORI program staff check databases that include  ClinicalTrials.gov and NIH’s RePORTER to identify any ongoing studies  on a particular topic that may be duplicative.

Once funded, PCORI contractors are monitored by PCORI staff.  According to PCORI staff, following an initial kickoff call with the  contractor, PCORI receives a progress report from each contractor every  6 months, and there are additional interactions via phone and through  regularly scheduled meetings. Progress reports include updates on key  project milestones, a progress statement for public use, a financial status  update, and additional documents the contractor deems relevant to the  project’s progress during the reporting period. PCORI officials stated that  if concerns arise regarding a contractor’s performance, a site visit to the  contractor could be conducted. All contractors are required to submit a  final report covering their research at the conclusion of the project.

The law also requires PCORI to develop a peer review process to assess  the integrity of primary research funded by PCORI and its adherence to  PCORI’s methodological standards. PCORI officials stated that they are  currently developing a peer review assessment process, as required by  law, to review final reports submitted by contractors at the conclusion of a  project. A draft of this peer review process was posted on PCORI’s  website for public comment in September 2014. The draft process  requires contractors to submit a draft final report to PCORI within three  months of the project’s completion for review by peer reviewers, who will  include researchers from outside of PCORI. These reviewers will consider  whether the research presented in each final report has scientific integrity  and adheres to PCORI’s methodological standards. Following receipt of  the draft final report, the peer reviewers may identify revisions, which the  contractor is required to respond to within 45 days. After revisions are  made and PCORI formally accepts the final report, its draft process states  that the institute will create and post on its website a lay abstract intended  for the general public, a medical abstract intended for researchers and  clinicians, a stand-alone table that presents key findings, and ancillary  information such as the identity of the contractor. The contractor also will  be required to ensure that the study results are submitted to  ClinicalTrials.gov and to include with that submission links to the  abstracts posted on the PCORI website.

section_title: PCORI Is Developing Dissemination Plans and Coordinating Dissemination Activities with AHRQ
paragraphs: PCORI is currently developing a plan for the dissemination of the  research it funds, as required by PPACA, and expects to begin  disseminating research results as early as 2015. Specifically, PCORI has  entered into a contract for the development of a dissemination and  implementation plan. A draft framework for this plan was provided by the  contractor to PCORI for comment in July 2014 and PCORI anticipates  that the contractor will submit the revised framework to PCORI in  February 2015. The draft framework identifies five key elements as core  components of a dissemination and implementation plan. (See table 4.)

According to PCORI, its dissemination efforts will result in research  findings being publicly available within 90 days of receiving final reports  from researchers, as required by law. PCORI has determined that this   90-day period will follow the completion of PCORI’s peer review process  for completed research. Upon accepting the final report on the research  conducted by the contractor, PCORI’s 90-day period will begin, during  which time PCORI will develop abstracts and other materials to post  publicly on its website. At the conclusion of this 90-day period, PCORI  anticipates posting abstracts, a results table that contains key findings,  and ancillary information, such as investigators and conflict of interest  information, on its website. The contractor will also be required to ensure  that the results tables are submitted to ClinicalTrials.gov, which will link to  the abstract posted on PCORI’s website. According to PCORI, these  dissemination efforts will take place at the end of the 29 to 79 month  period for funding, conducting, and disseminating research, depending on  the length of the contract. (See figure 2.)

PPACA gave both PCORI and AHRQ responsibilities for disseminating  CER results produced by PCORI. PPACA directs PCORI to make  research findings available to clinicians, patients, and the general public  within 90 days of the conduct or receipt of research findings. The act also  directs AHRQ to disseminate PCORI’s research findings as well as other  government-funded research relevant to CER. PCORI officials stated that  to coordinate AHRQ’s and PCORI’s dissemination responsibilities,  PCORI has formed a workgroup specifically to address engagement,  dissemination, and implementation activities, on which both PCORI and  AHRQ officials serve. This workgroup currently meets monthly and  discusses plans for disseminating CER results, among other things. A  PCORI official stated that PCORI and AHRQ are working together to  determine which entity will be best suited for disseminating different types  of CER results.

section_title: PCORI Has Started Awarding Contracts for Research, and Plans to Award Additional Contracts through 2019
paragraphs: As of October 2014, PCORI has awarded 360 contracts to fund research  projects across 12 funding areas. PCORI made a total of $670.8 million in  commitments to fund these contracts. Most of PCORI’s projects are  funded for periods of between 2 and 3 years, with some larger studies  funded for up to 5 years. In total, PCORI expects to make approximately  $2.6 billion in commitments for contracts starting as late as 2019, with  about $1.9 billion in commitments occurring between fiscal years 2015  and 2019.

While commitments occur when PCORI makes awards to contractors,  expenses occur when PCORI pays contractors and spends money for  PCORI’s operations. Expenses include not only money spent on research  contracts, but also on research support activities—such as merit review  and contractor monitoring—and administrative expenses. According to  PCORI, from inception through fiscal year 2014, it has incurred a total of  $235 million in expenses. fund research contracts, with the remaining amounts for research support  activities and administrative expenses. Through fiscal year 2015, PCORI  anticipates expending a total of $597 million. Overall, PCORI expects to  receive an estimated total of $3.5 billion through fiscal year 2019 from the  PCORTF to fund its work.$2.6 billion of its $3.5 billion on contracts for research and research  infrastructure, with the remaining amounts spent on research support  activities and administrative expenses.

In fiscal year 2014, PCORI reported that it was experiencing a “delay in spending,” that  is, contractors were submitting invoices to PCORI and collecting money at a slower rate  than initially expected. PCORI officials noted that they have undertaken additional efforts  to analyze and better plan for such delays, which can result because of the relatively slow  pace of spending that occurs during the beginning of research projects. PCORI has  adjusted its spending expectations based on the patterns observed to date, and PCORI  officials told us that they will monitor actual expenses as one indicator that funded work is  progressing at the appropriate pace. Officials anticipate that PCORI’s independent audit of  fiscal year 2014 will be issued in early 2015.

PCORI’s total administrative expenses for fiscal years 2012 and 2013  accounted for 20 percent and 32.5 percent of PCORI’s total operating  budget, respectively. Examples of administrative expenses include  salaries for PCORI staff, health benefits, rent for PCORI’s headquarters  office, and information technology costs. In its fiscal year 2013 financial  audit report, issued by a private, independent auditor, PCORI noted that  administrative costs for fiscal year 2013 remained high, but were not  outside of expected levels given that fiscal year 2013 was a period of  investment in systems and infrastructure that would not be sustained at  current levels in future years. According to PCORI’s financial projections,  it expects that total administrative expenses will constitute 14 percent and  8.2 percent of its total expenses in fiscal years 2014 and 2015,  respectively. (See figure 3.)

Each of the projects PCORI has committed to funding has been awarded  within one of twelve funding areas. (See figure 4.) Five of these funding  areas closely correspond to PCORI’s priority areas. Some of the other  areas—such as reducing disparities in asthma—cover specific topics  within a priority area.

According to PCORI officials, approximately $106 million in commitments  to date are for the PCORnet data research network, the aim of which is to  improve the capacity for and speed of conducting CER. PCORI officials  stated that they expect to spend a total of $271 million on PCORnet  through 2019. PCORnet is a distributed data research network, which  means that no central repository of data exists. Instead, multiple  organizations, each with their own data, agree to allow users to query  their data and combine it with data from the other organizations on a  project-by-project basis. PCORnet consists of 29 separate health data  networks called clinical data research networks (CDRN) and patient- powered research networks (PPRN), some of which existed prior to  PCORnet and some of which were created with PCORnet funding.

PCORnet is still undergoing development and testing. CDRNs and  PPRNs are currently working to map their data to the PCORnet common  data model. The common data model standardizes the definition, content,  and format of data aggregated by CDRNs and PPRNs, which is  necessary to allow researchers to use data from multiple CDRNs and  PPRNs. An initial test query was conducted using PCORnet in  September 2014. PCORnet is expected to be used to conduct an initial  clinical research trial starting in 2015. Officials stated that limited amounts  of data will be available through PCORnet for queries by researchers  after September 2015, with the amount of available data increasing over  time.

While PCORI officials, stakeholders, and PCORnet contractors noted that  PCORnet has the potential to significantly improve the ability to conduct  CER, they also noted challenges that PCORI will face with regard to the  establishment and future operation of PCORnet. For example, they  expect that the process of mapping data to the common data model will  be slow and resource intensive because of the lack of standardization  among existing data maintained by CDRNs and PPRNs, such as data  from electronic health records (EHR). PCORI officials recognize the  challenges caused by the lack of standardization. They expect that both  the common data model and a future requirement by PCORI for CDRNs  to hire additional staff with expertise related to this work will help to  address this challenge. PCORnet contractors also noted uncertainty  regarding future costs and the sustainability of the network, particularly if  the PCORTF is not reauthorized beyond fiscal year 2019. One  stakeholder we interviewed noted that some core funding would always  be required to maintain the central operation of the network. PCORI  officials acknowledge that some future core funding could be needed, but  they expect it to be lower than current funding levels. They also expect  that future research projects funded by PCORI, NIH, or others such as  the pharmaceutical and device industry will pay for the use of PCORnet’s  data. PCORI officials also noted that each CDRN and PPRN will be  required to provide a sustainability plan for continuing operation once  PCORI funds are no longer available. Another concern noted by CDRN  officials is that their data does not cover all care received by patients due  to the fragmented nature of the U.S. health care system, although such  “complete” data would be preferred by PCORI. An official from one  PPRN noted that the completeness of CDRN and PPRN data could be  improved by having PPRNs and CDRNs share data with one another,  which is not current practice. PCORI officials said that they are  collaborating with other relevant organizations, including state Medicaid  offices and private health insurers, to identify how CDRNs could link their  EHR data to claims data, which would improve data completeness. In  addition, PCORI officials stated they are requiring CDRNs to identify and  implement links with other institutions that have additional patient data.

section_title: PCORI Has Established an Evaluation Plan and Anticipates Developing Efforts to Measure Outcomes
paragraphs: PCORI’s evaluation group—a body composed of members from its Board  of Governors, methodology committee, advisory panel on patient  engagement, external experts, and PCORI staff —has developed initial  plans for evaluating PCORI’s efforts against its three strategic goals,  which are to increase information, speed implementation, and influence  research. To do so, PCORI identified primary outcome measures for each  of its strategic goals. In its strategic plan, PCORI notes that these are  meant to be long-term measures because research typically requires  several years to complete and additional years for the results to be  disseminated and implemented. Therefore, since 2013, PCORI has been  using early and intermediate process and output measures as a way to  monitor its progress toward its strategic goals. PCORI anticipates having  some early results related to its primary outcome measures starting in  2017 after the first CER studies are completed and their findings  released, although full evaluation of the results of these outcome  measures will not be possible until around 2020, after a large number of  CER studies have been completed and a few years have elapsed,  allowing time for study results to be taken up. (See table 5.)

Officials stated that to collect information for monitoring PCORI’s progress  and for evaluating PCORI’s impact on the dissemination and uptake of  CER study results, PCORI will employ a variety of data collection  methods, including surveys, focus groups, interviews, case studies, and  document reviews, to collect data on how potential consumers of CER  perceive PCORI’s work and CER in general. For example, PCORI is  conducting a number of surveys to collect both baseline and ongoing data  to gauge changes in perceptions of CER over time. (See table 6.)

PCORI officials stated that these baseline data will be compared against  future similar data collection efforts in an attempt to see whether PCORI’s  work is contributing to improved understanding and increased use of  CER. PCORI anticipates having preliminary baseline data by the end of  2014.

PCORI has identified some limitations and challenges related to their  evaluation methods. Specifically, PCORI’s evaluation plans rely on survey  development, focus groups and interviews, data extraction from PCORI  databases, and expert panels. In its plans, PCORI identifies potential  response bias and self-report bias as limitations to some of those data  collection methods. Further, PCORI officials stated that measuring  outcomes such as reducing practice variation and changing health care  delivery can be challenging, particularly within a 5- or 10-year timeframe.  PCORI staff stated that disseminating research results in an effort to  improve health care is a long-term challenge, as past research suggests  that it takes more than 17 years for research evidence to affect clinical  practice settings. PCORI staff stated that they hope the inclusion of end  users in their research process—such as patients and clinicians—will  expedite the uptake of PCORI’s research findings, but it is too soon to tell  if this will be the case.

Finally, officials stated that it will be difficult to know for certain whether  any measured changes in health care delivery and practice are  attributable to PCORI-funded research or due to other efforts. Therefore,  officials stated that they will have to rely on the measures identified in  PCORI’s strategic plan for a small subset of PCORI’s studies, to  determine the extent to which these studies may influence reductions in  practice variation or other changes in health care delivery.

PCORI is also undertaking efforts to assess the extent to which its funded  research addresses CER priority topics identified by the IOM in its 2009   PCORI issued a request  report and AHRQ’s Future Research Needs.for proposal to evaluate whether the research topics they funded address  the CER priority topics identified by IOM and AHRQ. Contractors were  selected and began conducting this work in June 2014, and it is  anticipated that the work will continue through 2015, with the possibility of  it extending into 2016. PCORI plans to use the results from this  evaluation to inform additional funding announcements in the future.

PCORI officials stated that preliminary analysis has shown that about half  of the research studies PCORI has funded to date directly related to a  CER priority topic identified in IOM’s 2009 report.

PCORI has conducted a preliminary effort to compare the extent to which  mental health research PCORI has funded aligns with mental health  topics in the IOM report, as well as some additional analyses to show the  extent to which PCORI-funded research aligns with IOM’s 100 CER  priorities, according to officials. For example, an analysis in September  2013 showed that of PCORI’s 55 projects at that time, 6 were closely  related, 23 were somewhat related, and 26 were unrelated to the IOM’s  100 CER priorities. PCORI officials noted that the IOM’s CER priorities  were developed in 2009 and that, given the amount of time that has  passed, it is likely that some of the topics listed in the IOM report are no  longer critical, while other CER topics have increased in importance.  PCORI officials stated that, as a result, the IOM’s CER priorities alone are  not a good indicator of PCORI’s progress related to CER.

section_title: Agency Comments
paragraphs: We provided a draft of this report to PCORI for review and comment.  PCORI provided technical comments, which we incorporated as  appropriate.

We are sending copies of this report to the Executive Director of PCORI  and other interested parties. In addition, the report will be available at no  charge on GAO’s website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact  me at (202) 512-7114 or at kingk@gao.gov. Contact points for our Office  of Congressional Relations and Office of Public Affairs can be found on  the last page of this report. Other major contributors to this report are  listed in appendix I.

section_title: Appendix I: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Will Simerl, Assistant Director;  Jennie Apter; LaSherri Bush; Ashley Dixon; Colbie Holderness;   Andrea E. Richardson; and Jennifer Whitworth made key contributions   to this report.


Question-summary annotations: questions: question: What has the Patient-Centered Outcomes Research Institute established for funding CER?
answer: The Patient-Centered Outcomes Research Institute (PCORI) has established priorities and processes for funding comparative clinical effectiveness research (CER)—which is research that evaluates and compares health outcomes and the clinical effectiveness, risks, and benefits of two or more medical treatments, services, or items such as health care interventions—and is developing dissemination plans, consistent with the legislative requirements of the Patient Protection and Affordable Care Act (PPACA).

question: What are the five research priorities that PCORI established in 2012?
answer: In 2012, PCORI established five broad research priorities: (1) assessment of prevention, diagnosis, and treatment options; (2) improving health care systems; (3) researching communication and dissemination strategies; (4) comparing interventions to reduce health disparities; and (5) accelerating patient-centered outcomes research and methodological research.

question: What research agenda and process has PCORI developed?
answer: PCORI also developed a research agenda to identify how each priority would be addressed. PCORI has established a multi-step research funding process designed to assess and select contract applications for funding. Funded contracts are monitored by PCORI staff.

question: What is PCORI developing in coordination with the Agency for Healthcare Research and Quality?
answer: Per legislative requirement, PCORI is developing a peer review assessment process to review final reports submitted by contract awardees and is in the process of developing a plan for the dissemination of funded research potentially beginning in 2015, in coordination with the Agency for Healthcare Research and Quality.

questions: question: How much has PCORI awarded for research?
answer: As of October 2014, PCORI has awarded 360 contracts to fund research projects, committing a total of $670.8 million to them.

question: How much does PCORI expect to commit for awards in total?
answer: PCORI expects to commit about $2.6 billion to research contracts, out of $3.5 billion in total estimated spending.

question: How much does PCORI plan to spend on PCORnet?
answer: Approximately $106 million in commitments to date are for PCORnet, a data research network aimed at improving the capacity for and speed of conducting CER. PCORI officials stated that they expect to spend a total of $271 million on PCORnet through fiscal year 2019.
child_questions: question: When will researchers be able to use data through PCORnet?
answer: PCORI officials stated that limited amounts of data will be available through PCORnet for researchers to use after September 2015 with the amount of available data increasing over time.

questions: question: What has PCORI developed to evaluate its goals?
answer: PCORI has established an evaluation plan and is developing efforts to measure outcomes. PCORI has developed initial plans for evaluating the institute's efforts against its three strategic goals, which are to increase information, speed implementation, and influence research.
child_questions: question: Why is PCORI's plan a long term measure?
answer: In its strategic plan, PCORI notes that these are meant to be long-term measures because research typically requires several years to complete and additional years for the results to be disseminated and implemented.

question: How has PCORI been measuring its strategic goals?
answer: Therefore, since 2013, PCORI has been using early and intermediate process and output measures—such as the number of people accessing or referencing PCORI information—as a way to monitor its progress toward its strategic goals.

question: When does PCORI estimate to have results from its strategic goal?
answer: PCORI anticipates having some early results related to its primary outcome measures starting in 2017 after the first CER studies are completed and their findings released, although full evaluation of the results of these outcome measures will not be possible until around 2020.

questions: question: Why did PPACA establish PCORI?
answer: In 2010, PPACA authorized the establishment of PCORI as a federally funded, nonprofit corporation to improve the quality and relevance of CER.

question: What does PCORI do?
answer: PCORI, which began operation in 2010, is required to identify research priorities, establish a research project agenda, fund research consistent with its research agenda, and disseminate research results, among other things.

question: How does PPACA fund PCORI?
answer: To fund PCORI, PPACA established the Patient-Centered Outcomes Research Trust Fund, through which the institute is expected to receive an estimated $3.5 billion from fiscal years 2010 through 2019.

questions: question: What does the GAO report examine?
answer: This report examines (1) the extent to which PCORI established priorities and processes for funding and disseminating comparative clinical effectiveness research consistent with its legislative requirements; (2) the status of PCORI's efforts to fund comparative clinical effectiveness research; and (3) PCORI's plans, if any, to evaluate the effectiveness of its work.
child_questions: question: What documents did GAO review?
answer: GAO reviewed relevant legislative requirements and PCORI documentation, including funding data, and interviewed PCORI officials. GAO also interviewed relevant stakeholders, including health policy experts and PCORI contractors.

question: What did PCORI provide to the GAO?
answer: PCORI provided technical comments, which GAO incorporated as appropriate.