Comparative Effectiveness: Agency for Healthcare Research and Quality's Process for Awarding Recovery Act Funds and Disseminating Results
Published: 2012-02-29T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The American Recovery and Reinvestment Act of 2009 (Recovery Act) provided $1.1 billion to the Department of Health and Human Services (HHS) for comparative effectiveness research (CER), which is research that compares different interventions and strategies to prevent, diagnose, treat, and monitor health conditions. Of this amount, HHSs Agency for Healthcare Research and Quality (AHRQ) received $474 million to support and disseminate the results of CER. GAO was asked to describe issues including the (1) process and criteria AHRQ used to award Recovery Act funds for CER, including steps to coordinate CER awards with other HHS entities in order to avoid unnecessary duplication of effort; and (2) plans AHRQ has for disseminating the results of CER it funded under the Recovery Act.
To address these objectives, GAO reviewed relevant documentation, including AHRQs policies and procedures for selecting the recipients of grants; internal documents that describe the award of Recovery Act grants and contracts; and Recovery Act contractors work plans. GAO also analyzed AHRQ data on the number and type of grants and contracts awarded Recovery Act CER funds. GAO interviewed AHRQ officials on the selection of Recovery Act CER grantees and contractors, including coordination with other HHS agencies that received Recovery Act CER funds, and the plans the agency has to disseminate the results of CER funded by the Recovery Act. AHRQ provided technical comments, which GAO incorporated as appropriate.
section_title: What GAO Found
paragraphs: AHRQ used its standard, competitive review processes and criteria to select the recipients of CER grants and contracts using Recovery Act funds. Specifically, to select the recipients of Recovery Act CER grants, AHRQ used its standard review process that includes peer review of grant applications, the development of funding recommendations by a team of senior officials within AHRQ, and final funding determination by the agencys director. As part of this process, AHRQ used its standard criteria to evaluate grant applications, as well as additional requirements that were specific to each funding opportunity. To select contractors who would receive Recovery Act funds, AHRQ used its standard contracting processes and criteria that are governed by the Federal Acquisition Regulation, which establishes uniform policies for acquisition of supplies and services by executive agencies, and the Public Health Service Act. These processes included an evaluation of all contract proposals using standard criteria adapted to the specific needs of each project. Between February 2009 and September 2010, AHRQ awarded $311 million of its $474 million in Recovery Act CER funds through 110 grants. AHRQ also awarded $161 million of its Recovery Act CER funding through 34 contracts. The contracts and grants AHRQ awarded supported both AHRQs agency-specific and HHSs departmentwide CER priority areas. In an effort to avoid unnecessary duplication of CER awards, AHRQ participated in HHS working groups, developed a CER spending plan, and queried HHS databases to check for duplicative awards.
According to AHRQ officials, the agency plans to disseminate the results of Recovery Act-funded CER using a range of existing mechanisms. These mechanisms include written products, training programs, social media tools, and AHRQs website. AHRQ is also developing additional strategies to disseminate CER results. AHRQ awarded four contracts using Recovery Act funds totaling approximately $42.3 million to promote innovative approaches for disseminating CER results. A variety of efforts are conducted under these contracts, including efforts to educate clinicians and develop regional dissemination offices.

section_title: Letter
paragraphs: Since the enactment of the Medicare Prescription Drug, Improvement,  and Modernization Act of 2003 (MMA), the Agency for Healthcare  Research and Quality (AHRQ), an agency within the Department of  Health and Human Services (HHS), has been one of several federal  agencies responsible for supporting and disseminating the results of  comparative effectiveness research (CER). CER is research that  compares different interventions and strategies to prevent, diagnose,  treat, and monitor health conditions. AHRQ has supported CER activities  and disseminated results by awarding grants and contracts to research  centers and academic organizations to carry out this work. These  grantees and contractors review and synthesize scientific evidence  through research reviews; generate new scientific evidence and analytical  tools in original research reports; and compile research findings that are  translated into formats for a variety of audiences. The results of CER can  be used by both patients and clinicians to make health care decisions  about which treatment or intervention may be most effective or beneficial  for a given patient.
Under the American Recovery and Reinvestment Act of 2009 (Recovery  Act), AHRQ received significant funding to support and disseminate the  results of CER. Specifically, while AHRQ’s fiscal years 2009 and 2010  budgets for CER activities were $50 million and $21 million, respectively,  the Recovery Act provided AHRQ with an additional $474 million for CER  for this period—$300 million that was appropriated to AHRQ and   $174 million that was appropriated to the HHS Office of the Secretary and  allocated to AHRQ. The Recovery Act, which was enacted on   February 17, 2009, required that AHRQ obligate these funds by  September 30, 2010. AHRQ’s $474 million in funding was part of   $1.1 billion HHS received under the Recovery Act to support CER  through its various agencies, including AHRQ and the National Institutes  of Health (NIH).
The Patient Protection and Affordable Care Act (PPACA) gave AHRQ  additional responsibilities related to CER—in particular, responsibilities  related to the Patient-Centered Outcomes Research Institute (PCORI).  PCORI is a nonprofit corporation established by PPACA to, among other  things, improve the quality and relevance of CER, and disseminate the   AHRQ is to help PCORI meet its mission in  results of this research. several ways, which include broadly disseminating the results of the  research that PCORI conducts or funds; developing a publicly available  database for this research; and promoting the incorporation of CER  findings into health information technology systems that support clinical  decision making.
In June 2011, we provided information to congressional committees on  HHS’s use of CER funds that were available under the Recovery Act and   You have also asked that we report on AHRQ’s procedures for PPACA. awarding Recovery Act CER funds and AHRQ’s plans to address the  various new responsibilities established for it by PPACA as they relate to  PCORI. In this report, we describe (1) the process and criteria AHRQ  used to award Recovery Act funds for CER, including steps to coordinate  these awards with other HHS entities in order to avoid unnecessary  duplication of effort; (2) the plans AHRQ has for disseminating the results  of CER it funded under the Recovery Act; and (3) the steps AHRQ has  taken to perform its roles and responsibilities related to PCORI under  PPACA.
subsections: section_title: Scope and Methodology
paragraphs: To describe the process and criteria AHRQ used to award its $474 million  in Recovery Act CER funds, we reviewed relevant statutes as well as  documentation on the process and criteria AHRQ uses to (1) determine  the scientific and technical merit of grant applications and contract  proposals, and (2) select grant recipients and contractors. We reviewed  spending plans, which outline AHRQ’s and HHS’s plans for spending  Recovery Act funds. We also reviewed summary statements that describe  AHRQ’s process for selecting grantees and contractors to be awarded  Recovery Act funds. In addition, we reviewed AHRQ’s Management  Operations Manual, the agency’s written guidance that provides policies  and procedures for selecting grant recipients. We interviewed AHRQ  and other HHS officials to learn about the processes and criteria they  used to select the grantees and contractors that received awards funded  with the $474 million in Recovery Act CER funds and to coordinate these  awards with other HHS agencies that also received Recovery Act CER  funds. Because NIH also received Recovery Act CER funds, we  conducted interviews with NIH officials to confirm the methods and  processes used by AHRQ to coordinate funding opportunity  announcements (FOA), contract solicitations, and awards with NIH in an  effort to prevent the unnecessary duplication of effort in awarding  Recovery Act CER funds. Finally, we obtained data from AHRQ on the  number and type of awards made between February 2009 and  September 2010 using the $474 million in Recovery Act CER funds. We  relied on Recovery Act award data provided by AHRQ and did not audit  the reported data. To determine whether AHRQ and the Office of the  Secretary’s Recovery Act CER award data were sufficiently reliable for  our analyses, we conducted a reliability assessment of the data we used  by reviewing existing information about the data, conducting quality  control checks, and interviewing agency officials knowledgeable about the  data. We determined that the data were sufficiently reliable for the  purposes of this report.
To describe AHRQ’s plans to disseminate results from CER funded with  Recovery Act funds, we reviewed the Recovery Act and other relevant  statutes to determine AHRQ’s responsibilities for disseminating CER. We  reviewed agency documents, including AHRQ contractors’ work plans  describing specific goals and activities; AHRQ’s general publications,  including reports and guides posted on AHRQ’s website and electronic  newsletters; and samples of AHRQ’s CER, including original research  reports, treatment guides, and slide presentations used for educating  clinicians. We also interviewed AHRQ officials and a contractor to  understand how the agency plans to disseminate the results of CER  funded with Recovery Act funds and to obtain information on plans AHRQ  has for assessing the effectiveness of its dissemination efforts.
To describe the steps AHRQ has taken to fulfill its roles and  responsibilities related to PCORI under PPACA, we reviewed provisions  in PPACA to identify these roles and responsibilities, which include to  broadly disseminate CER findings to various audiences; develop a  publicly available database to collect evidence and research; and  promote the timely incorporation of CER findings into health information  technology systems that support clinical decision making. While AHRQ  is required to conduct a number of activities under PPACA, we focused  our review on those activities that are related to PCORI. We also  reviewed AHRQ’s PPACA spending plan for fiscal years 2011 and 2012,  which describes the agency’s plans for using funds made available by  PPACA, as well as PCORI presentation materials and meeting reports.  We also conducted interviews with AHRQ officials to determine the steps  the agency has taken to meet its responsibilities related to PCORI under  PPACA.
We conducted this performance audit from February 2011 to February  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
section_title: Background
paragraphs: AHRQ supports CER by awarding grants and contracts to entities in order  to conduct CER and perform related activities, such as the dissemination   As 1 of 12 agencies within HHS, AHRQ’s overarching  of CER results.mission is to improve the quality, safety, efficiency, and effectiveness of  health care for all Americans. (For more information on AHRQ’s mission,  research, priorities, and budget, see app. I.)
subsections: section_title: CER and the Recovery Act
paragraphs: The Recovery Act provided a significant amount of funding for AHRQ to  conduct CER activities. (See table 1.)
HHS developed departmentwide priorities for CER. (See table 2.) Of the  total amount of $474 million in Recovery Act funds available to AHRQ for  CER, $174 million of these funds were allocated to AHRQ by the Office of  the Secretary and were used to support the HHS departmentwide  priorities for CER.
AHRQ developed seven agency-specific CER priority areas to guide its  spending of Recovery Act funds. (See table 3.) Of the total amount of  $474 million in Recovery Act funds available to AHRQ for CER,   $300 million of these funds were appropriated to the agency and,  therefore, supported the agency’s seven CER priority areas.
The enactment of PPACA in 2010 gave AHRQ new roles and  responsibilities related to disseminating CER and building capacity for  research, and appropriated funds for carrying out these activities. Several  of these responsibilities relate to work conducted by PCORI. Established  in November 2010, PCORI was authorized to help coordinate CER at a  national level by developing national priorities for CER and conducting  and funding CER activities. PPACA directs AHRQ to broadly disseminate  CER findings to physicians, other health care providers, patients, payers,  and policymakers; develop a publicly available database to collect  evidence and research; promote the timely incorporation of CER findings  into health information technology systems that support clinical decision  making; and establish a process for receiving feedback about the value of  information disseminated by AHRQ.
To fund this work, PPACA established the Patient-Centered Outcomes  Research Trust Fund (PCORTF). The act specified that percentages of  this trust fund be provided to the Secretary of HHS and AHRQ in each of   Specifically, AHRQ received $8 million  fiscal years 2011 through 2019. from this trust fund in fiscal year 2011 and will receive $24 million in fiscal  year 2012, representing 16 percent of the total amount appropriated to  this trust fund in each of these fiscal years. In subsequent fiscal years,  AHRQ will continue to receive 16 percent of the total amount appropriated  to the trust fund, which will be based on the net revenues from fees on  health insurance and self-insured plans, amounts transferred from the  Medicare trust funds, and appropriations to PCORTF from the General  Fund of the Treasury.
section_title: AHRQ’s Standard Competitive Process for Selecting Grant Recipients
paragraphs: A portion of AHRQ’s work, including its work related to CER, is conducted  through grants awarded to research centers and academic organizations  to fund research ideas developed by a grant applicant. Grant applications  are submitted in response to publicly available FOAs, which announce  AHRQ’s intention to award research grants. AHRQ has established a  standard competitive process that is governed by federal law to select   According to AHRQ officials, this multistep process  grant recipients.includes: (1) an initial review of received applications; (2) preliminary  scoring of applications; (3) review and final scoring of applications at a  peer review panel meeting; (4) the development of preliminary funding  recommendations; (5) review by a senior leadership team within AHRQ;  and (6) a final determination of funding by the agency director.
The trustee of PCORTF is to provide for the transfer from PCORTF of 20 percent of the  amounts appropriated or credited to PCORTF for each of fiscal years 2011 through 2019  to the Secretary of HHS. Of the amounts transferred, the Secretary of HHS is to distribute  80 percent to AHRQ and 20 percent to the Secretary of HHS. See 26 U.S.C. § 9511.
To evaluate the grant applications it receives in response to FOAs,  AHRQ’s peer reviewers, most of whom are authorities in their respective  fields and not government employees, use five standard core criteria to  score and rank the applications. These five standard core criteria are   (1) the significance in addressing an important problem; (2) the  investigators’ ability to carry out the research; (3) the originality or  innovation of the project; (4) the development of an adequate research  approach or framework; and (5) the scientific environment in which the  applicant plans to conduct the research. Each FOA also contains criteria  that are specific to the announcement. While these other specific criteria  are not individually scored, they are used to evaluate the applications  during the peer review panel meetings. (See fig. 1 for an overview of  AHRQ’s standard process for selecting grant recipients.)
AHRQ uses a separate competitive process to award contracts, which  fund specific activities defined by AHRQ. This process for selecting  contract proposals for award is governed by the Federal Acquisition  Regulation (FAR) and the Public Health Service Act (PHSA) and  implementing regulations.opportunities, which occurs through different types of solicitations, varies  depending on the type of contracting mechanism used. AHRQ generally  uses three types of contracting mechanisms.
The advertising of available contracting
Stand-alone contracts. This contracting vehicle involves the  issuance of new, stand-alone contracts. Proposals are submitted in  response to publicly-available solicitations referred to as requests for  proposals. A request for proposals details the specific tasks or ideas  that an agency needs a contractor to fulfill, such as delivery of a  certain service or research of a clearly defined topic.
Task orders. This contracting vehicle involves issuing task orders  under an existing, master contract, thereby giving a contractor a new  task to perform. Proposals are submitted in response to solicitations  called requests for task orders, which are issued to contractors  already awarded contracts by the agency.
General Services Administration (GSA)-schedule task orders.
This contracting vehicle involves the use of contracts that have been  awarded by GSA for governmentwide use. GSA-schedule task orders  are issued under existing master contracts awarded by GSA. These  task orders are solicited through a request for quote solicitation that is  competed among these contractors.
Upon receiving proposals in response to a solicitation, AHRQ typically  evaluates contract proposals for award using standard contracting  procedures and criteria, which are governed by the FAR and the PHSA  and implementing regulations. A technical review panel of agency officials  and external experts evaluates each proposal submitted in response to a  solicitation against standard criteria that are tailored to the specific needs  of each solicitation. These criteria include (1) demonstrated knowledge  and understanding of the contract requirements; (2) the proposed  approach to address tasks and subtasks listed in the proposal; (3) the  qualifications and experience of key management personnel, such as the  project director and project manager; (4) the potential contractor’s ability  to meet the project’s milestones; (5) the facility, equipment, and space  available to support the project goals and objectives; and (6) the past  performance of the potential contractor using information from references  or other government customers. Based on this evaluation of each  proposal’s scientific and technical merit and cost, the review panel, along   identifies  with the contracting officer’s technical representative (COTR),the entity they believe should be awarded the contract and forwards the  recommendation to the contracting officer, the federal official who has  authority to enter into a contract. The contracting officer reviews the  recommendation and makes a final award decision.
section_title: AHRQ Used Standard Competitive Processes and Criteria and Coordinated within HHS to Make Recovery Act Awards
paragraphs: AHRQ used its standard, competitive review process and criteria to select  grant recipients and award 110 Recovery Act-funded CER grants, totaling  approximately $311 million. In addition, AHRQ primarily used its standard  contracting processes and criteria to select contract proposals and enter  into 34 contracts for CER using Recovery Act funding, totaling  approximately $161 million. AHRQ also took several steps to coordinate  with other HHS agencies when soliciting and awarding CER grants and  contracts.
subsections: section_title: AHRQ Used Its Standard Competitive Review Process and Criteria to Select Grant Recipients and Award 110 CER Grants
paragraphs: Between February 2009 and September 2010, AHRQ used its standard,  competitive grant review process to select grant recipients and ultimately  award 110 CER grants using approximately $311 million in Recovery Act  CER funds. Specifically, AHRQ used its standard process for selecting  grantees, which includes an initial review and preliminary scoring of  applications; review and final scoring of applications at peer review panel  meetings; development of preliminary funding recommendations; review  of funding recommendations by a senior leadership team within AHRQ;  and a final determination of funding by the agency director.
As part of the standard process AHRQ used to select the recipients of  Recovery Act-funded CER grants, peer reviewers used the agency’s  standard core criteria to score and subsequently rank the applications for  this funding. In addition to its core criteria, AHRQ also used other criteria  in its process of selecting recipients of Recovery Act CER grants. These  other criteria were specific to each FOA and often varied depending on  the CER study requested under that announcement. These other criteria  may be used to assess, for example, a grant application in terms of the  adequacy of the protection afforded human subjects; the inclusion of   the extent to which privacy and  certain priority populations in the study;security issues have been addressed; the partnerships that the applicant  has with the proposed population; and the degree of responsiveness in  addressing the purpose and objective of the FOA. AHRQ officials told us  that the agency’s peer reviewers, program officials, and members of the  senior leadership team used both standard core criteria and other criteria  outlined in the FOAs to determine which grant applications should be  recommended to the Director of AHRQ for funding.
AHRQ issued 14 FOAs for Recovery Act CER grant opportunities. Using  its grant review process and criteria, AHRQ received 536 grant  applications and awarded 110 CER grants between February 2009 and  September 2010, totaling approximately $311 million. $311 million in grants AHRQ awarded supported HHS’s departmentwide  and AHRQ’s agency-specific CER priorities. (See fig. 2 and app. II for  more information on the award of AHRQ’s CER grants with Recovery Act  funds.)
The 14 FOAs issued and 110 CER grants awarded addressed projects in three of the  four HHS-departmentwide CER priority areas and three of AHRQ’s seven agency-specific  CER priority areas. Some priority areas were supported by more than one FOA. AHRQ  agency-specific and HHS departmentwide priority areas not supported with grants were  supported through contracts.
AHRQ made these awards by September 30, 2010, the end of the period in which the  Recovery Act funds were available for obligation. See GAO-11-712R.  AHRQ awarded an  additional $161 million through contracts and spent approximately $2 million for  administrative purposes.
AHRQ supported the remaining four CER priority areas through contracts.
AHRQ supported the one remaining HHS departmentwide CER priority area through contracts.
Percentages do not add up to 100 percent due to rounding.
For 55 of the 110 grants, the Director of AHRQ exercised her discretion to  make out-of-order funding decisions. An out-of-order funding decision  occurs when grant applications are funded out of rank order; that is, they  are not funded in accordance with the rank order of most-meritorious to  least-meritorious overall impact scores calculated during the peer review  process. Out-of-order funding decisions can be made for a number of  reasons, including the need to address important agency research  priorities, avoid duplication, or meet specific requirements in the original  FOA that the more meritorious applicants cannot meet.  Recommendations for out-of-order funding decisions are made by the  program official or senior leadership team, but the final decision to award  grants is made by the Director of AHRQ.
section_title: AHRQ Primarily Used Its Standard Competitive Contracting Process and Criteria to Award 34 CER Contracts
paragraphs: Between February 2009 and September 2010, AHRQ primarily used its  standard competitive contracting process and criteria to select contract  proposals and enter into 34 contracts using approximately $161 million in  Recovery Act CER funds. According to AHRQ officials, a review panel,  composed of external experts and AHRQ staff, evaluated all Recovery  Act CER contract proposals using the standard criteria that are tailored to  the specific needs of each contract solicitation. These criteria include  evaluating each proposal’s technical approach, management plan, and  key personnel. AHRQ officials reported that a contracting officer used the  results of the panel’s evaluation to make a final selection of contractors  that presented the best value to meet the needs of work specified in each  Recovery Act CER solicitation.
To meet the September 30, 2010, deadline established by the Recovery  Act, AHRQ made one change to its standard contracting process.  Specifically, for 1 of the 13 contract solicitations AHRQ conducted an  initial review of the contract proposals it received in order to determine  whether these contract proposals were duplicative or responsive to the  solicitation’s requirements.proposals in response to this solicitation. Agency officials explained that  because they received a large number of proposals in response to this  solicitation, they decided to conduct the initial review to identify which of  the received proposals would continue through the agency’s standard,  competitive contract review process. Four of the 23 task order contract  proposals were found to be nonresponsive or duplicative of another  previously funded study and, therefore, were not considered for further  review.
The agency received 23 task order contract  AHRQ awarded task orders under existing AHRQ master contracts, task  orders under existing GSA master contracts, and stand-alone contracts  using Recovery Act CER funds. The agency primarily awarded task  orders under existing master contracts when awarding contracts with  Recovery Act CER funds. Specifically, of the 34 contracts that AHRQ  awarded using Recovery Act CER funds, 30 were task orders under  either existing AHRQ contracts or existing GSA-schedule contracts, and  the remaining 4 were stand-alone contracts. AHRQ awarded multiple task  orders within the Evidence Synthesis and Gap Identification priority areas  under existing contracts that AHRQ entered into prior to the passage of  the Recovery Act. Officials stated that issuing task orders under existing  master contracts, which included a GSA-schedule task order, facilitated  the quick and efficient award of Recovery Act funds in instances where  the agency or GSA had existing master contracts with entities capable of  conducting work the agency wanted to support with Recovery Act CER  funds. AHRQ officials stated that this approach was faster and more  cost-effective than entering into new, stand-alone contracts. AHRQ  officials also said that they used stand-alone contracts only in instances  where there were existing contracts with entities that could perform the  planned work.
AHRQ issued 13 CER contract solicitations between February 2009 and  September 2010. Using its standard contracting process and criteria, the  agency received 80 contract proposals and entered into 34 contracts  totaling almost $161 million. departmentwide and AHRQ’s agency-specific CER priorities. (See fig. 3  and app. III for more information on the award of AHRQ’s CER contracts  with Recovery Act funds.)
The 13 contract solicitations issued and 34 CER contracts awarded addressed projects  in AHRQ and HHS-departmentwide CER priority areas. Some priority areas were  supported by more than one contract solicitation. AHRQ agency-specific and HHS  departmentwide priority areas not supported with contracts were supported through  grants.
AHRQ made these awards by September 30, 2010, the end of the period in which the  Recovery Act funds were available for obligation. See GAO-11-712R.  AHRQ awarded an  additional $311 million through grants and spent approximately $2 million for  administrative purposes.
Percentages do not add up to 100 percent due to rounding.
section_title: AHRQ Also Reported Taking Steps to Coordinate Recovery Act CER Awards with Other HHS Agencies to Avoid Unnecessary Duplication
paragraphs: AHRQ officials reported that they used five mechanisms in order to  coordinate with other HHS agencies to avoid unnecessary duplication  when creating FOAs for grants and solicitations for contracts and when  awarding Recovery Act CER funds. Specifically, AHRQ participated in a  federal interagency coordination council and an HHS working group;  contributed to an HHS spending plan to coordinate the department’s  solicitations; participated with NIH in another working group to coordinate  both solicitations and CER awards; and queried HHS databases when  awarding Recovery Act funds to identify potentially duplicative projects.
Federal Coordinating Council for CER—AHRQ participated on the  Federal Coordinating Council for CER (“the Council”), a body created  by the Recovery Act to foster coordination for CER across the federal  government in an effort to reduce duplication and encourage the  coordinated and complementary use of resources. In addition to  AHRQ, officials from the Veterans Health Administration (VHA), the  Department of Defense, and NIH also served on the Council.  According to AHRQ officials, the Council provided a mechanism for  coordinating, among other things, the establishment of CER priorities  and some Recovery Act CER grant announcements and contract  solicitations. The Council was terminated by PPACA in March 2010.
CER Coordination and Implementation Team (CER-CIT)—In  addition to the Council, AHRQ officials participated in the CER-CIT, a  departmentwide effort to coordinate investments in CER supported  with Recovery Act funds. Organized by HHS, the CER-CIT served as  a centralized forum for HHS officials to assess FOAs for grants and  solicitations for contracts. AHRQ officials stated that the CER-CIT’s  process helped ensure that the FOAs and solicitations ultimately  posted by AHRQ for grants and contracts were not duplicative of  FOAs and solicitations posted by other entities within HHS. For  example, during the CER-CIT’s review of two proposed AHRQ CER  FOAs, reviewers identified aspects of the proposed announcements  that were potentially duplicative of other proposed or existing projects.
HHS Intra-Agency Spending Plan—AHRQ contributed to an intra- agency spending plan developed by HHS that describes how all HHS  agencies anticipated using the funding they received under the  Recovery Act for CER. AHRQ contributed to this intra-agency  spending plan by developing an agency-specific spending plan that  described AHRQ’s research priorities and how the agency anticipated  using its $300 million in Recovery Act CER funds to support these  priorities. HHS incorporated AHRQ’s spending plan into the  department’s intra-agency spending plan. According to AHRQ  officials, officials from the HHS Office of the Secretary, who were  responsible for coordinating this effort, reviewed the spending plans it  received from AHRQ and other agencies, and this helped ensure that  AHRQ’s Recovery Act CER solicitations were not unnecessarily  duplicative of other CER efforts within HHS.
AHRQ-NIH Working Group—In addition to the departmentwide  working group that was primarily focused on coordination of FOAs  and contract solicitations, AHRQ and NIH formed a working group to  coordinate the award of Recovery Act funds to avoid unnecessarily  funding duplicative projects.making awards, they checked with NIH through this working group to  ensure that the two agencies were not funding duplicative projects.  For example, during one meeting members noted where issues of  duplication needed to be further discussed to ensure that studies were  complementary and not duplicative. In addition to reviewing awards,  AHRQ officials reported that this working group met regularly during  the award of the Recovery Act CER funds to share spending plans,  share solicitations, and provide updates on each agency’s respective  CER activities.
AHRQ officials stated that before
Querying of HHS Databases—AHRQ officials stated that in order to  avoid funding unnecessarily duplicative work with Recovery Act funds,  they queried HHS databases prior to awarding any Recovery Act CER  funds to ensure that other similar projects were not funded elsewhere  within HHS. AHRQ officials stated that if, in the process of querying  these databases, a duplicative award was identified, AHRQ would  contact the appropriate HHS project officer listed in the database to  discuss the award in more detail. AHRQ and NIH officials confirmed  that this process resulted in the identification of grant proposals for  training awards that were potentially duplicative of projects AHRQ had  funded. Once identified, it was decided that NIH would not fund these  potentially duplicative awards.
section_title: AHRQ Plans to Use Its Existing Mechanisms and Develop Additional Strategies to Disseminate CER Results
paragraphs: AHRQ plans to use a range of existing mechanisms, such as written  products, training, social media tools, and its website, to disseminate  results of CER funded through the Recovery Act. According to AHRQ  officials, the agency will determine which specific mechanisms will be  used to disseminate CER results by considering the unique  characteristics of the research. In addition, AHRQ awarded four contracts  using Recovery Act CER funds to develop and implement innovative  approaches for disseminating CER results, including Recovery Act- funded CER.
subsections: section_title: AHRQ Plans to Use a Range of Existing Mechanisms, Such as Written Products, Training, Social Media Tools, and Its Website, to Disseminate Recovery Act CER Results
paragraphs: AHRQ officials stated that the agency plans to use a range of existing  mechanisms to disseminate Recovery Act-funded CER results as such  results become available. As of December 6, 2011, 30 Recovery Act CER  projects were completed or in draft and some dissemination activities had  begun. The mechanisms AHRQ plans to use to disseminate Recovery  Act-funded CER include written products, training programs, social media  tools, learning networks, and AHRQ’s website. The different types of  written products that AHRQ develops for CER results and other research  include comprehensive research reviews that summarize existing  research on a CER topic; original research reports that introduce new  CER results; and plain language publications that summarize the findings  of research on the benefits and harms of different treatment options and  which are tailored to clinicians, consumers, or policymakers. AHRQ’s  training programs include web-based conferences that feature  presentations by experts accompanied by instructional slides for  clinicians. In addition, the agency employs social media tools to  disseminate notices of CER results including electronic newsletters, audio  podcasts, and Twitter. AHRQ is also drawing on an existing agency  learning network for Medicaid medical directors that was created in 2005.  This group convenes periodically to discuss ways to advance the health  of Medicaid beneficiaries, including how evidence-based research   AHRQ’s website  findings can be used to improve quality of care.provides access to CER results through search tools and links to its  written and social media formats. (See app. IV for a more detailed  description of these existing mechanisms.)
AHRQ officials explained that the agency determines which specific  mechanisms will be used to disseminate particular CER results by  considering the unique characteristics of the research such as the type of  research conducted, its potential impact, the strength of the evidence,  and the audiences that can best make use of the information. AHRQ then  develops a marketing plan that identifies key messages, target  audiences, and the mechanisms to be used to disseminate CER to those  audiences. For example, AHRQ’s marketing plan for a CER project that  examines certain treatments for type 2 diabetes targeted consumers as  well as primary care clinicians and certain specialist clinicians and other  health professionals. While this CER project was not funded with  Recovery Act funds, AHRQ officials confirmed that the process they use  to customize the dissemination for this project is the same process the  agency will follow for disseminating Recovery Act-funded CER results. To  disseminate the CER results of this project to consumers, AHRQ  developed a consumer guide and a series of audio podcasts. To reach  clinicians, AHRQ developed a clinician’s guide and a webcast program  with educational slides. These products were targeted to be distributed  through multiple channels including AHRQ’s website, as well as its  newsletters and list-servs. Notices about these products were also sent  directly to a range of general media news services; consumer health and  advocacy publications; and a wide range of key national organizations  that included those representing primary care, specialty clinicians, and  payers. (See app. V for a more detailed description of this dissemination  effort.)
section_title: AHRQ Is Developing Additional Strategies to Disseminate CER Results
paragraphs: In addition to its existing mechanisms for disseminating the results of  CER, AHRQ is in the process of developing additional dissemination  strategies. Specifically, in September 2010, AHRQ awarded four  Recovery Act-funded contracts to develop and implement innovative  approaches for disseminating CER results, including Recovery Act- funded CER. The specific purpose of each of the four contracts is  described below.
Academic Detailing. Academic detailing involves face-to-face  educational sessions by trained clinicians, including physicians,  nurses, pharmacists, and others, who visit health professionals in their  practice settings. The goal of these sessions is to share evidence- based information and facilitate use of that information to improve  patient care. AHRQ awarded a contract in the amount of $11,680,060  for the purpose of implementing academic detailing for CER from  2011 through 2013. The plan under this contract calls for academic  detailing to 1,300 primary care providers and 200 health care system  practice sites. Each provider or site will receive one face-to-face visit  every 6 months plus follow-up e-mail communications and supporting  materials. The academic detailing will focus on six CER topics over  the 3-year period. Between February 2011 and October 2011, work  completed under this contract resulted in over 1,562 visits to providers  and practice sites. These visits involved the discussion of AHRQ’s  CER results related to the treatment of type 2 diabetes.
Continuing Education Modules. AHRQ awarded a contract in the  amount of $3,981,168 for the purpose of developing and  disseminating 45 accredited online continuing education programs for  health care professionals, including physicians, physician assistants,  pharmacists, nurses, nurse practitioners, medical assistants, and  other health professionals. These programs translate CER results into  a variety of formats, for example, videos featuring case studies and  journal supplements. As of November 30, 2011, 13 approved  continuing education programs were completed, including programs  on CER results related to hip fractures, hypertension, prostate cancer,  breast cancer, heart disease, and diabetes.
Regional Dissemination and Partnership Offices. AHRQ awarded  a contract in the amount of $8,613,876 to create five regional offices  for the purpose of establishing partnerships to facilitate dissemination  and use of CER results by regional health care organizations,  businesses, unions, and consumer groups. Collaborative efforts are  expected to result in local and regional meetings, web conferences,  training programs, and distribution of CER results to partner  organizations’ memberships.
Publicity Center. AHRQ awarded a contract in the amount of  $17,999,988 to develop and implement a national strategic  communications plan for AHRQ’s CER results. The communications  plan calls for the development of national partnerships with consumer,  clinician, policymaker, and business audiences; marketing efforts,  including the use of social media, focused on disseminating results of  research; and creation of new website portals with established sites  reaching patients and clinicians. For example, under the contract,  partnerships have been established with such organizations as the  National Rural Health Association, the National Alliance for  Caregiving, the American College of Cardiology, and the American  Medical Student Association.
Along with its efforts to develop additional strategies for disseminating  CER results, AHRQ is taking steps to evaluate the effectiveness of these  strategies. Specifically, using its Recovery Act funds AHRQ awarded a  contract in the amount of $2,371,179 for the purpose of evaluating some  of AHRQ’s dissemination strategies by collecting data about  dissemination. Under the contract, information will be collected about  changes over time in the level of awareness, understanding, use, and  perceived benefits of CER . This information will be gathered from  clinicians, patients, consumers, health system decision makers,  purchasers, and policymakers. In addition, this evaluation includes plans  to collect process and outcomes data for each of the additional  dissemination strategies being developed under Recovery Act-funded  contracts including academic detailing, continuing education, regional  dissemination, and the national publicity center.
AHRQ officials noted that evaluating the impact of dissemination of its  CER results is important but also challenging. They noted that clinician  practice behavior often changes slowly and is affected by many variables,  thereby making it difficult to directly attribute changes to information  AHRQ has disseminated. In addition, once CER results are disseminated  to target audiences through, for example, AHRQ’s website or one of its  educational programs, it is often not feasible to track secondary  dissemination from those audiences to others.
section_title: AHRQ Has Begun to Monitor PCORI and Identify Resources That Could Enable It to Fulfill Its PPACA Responsibilities Related to PCORI
paragraphs: While PCORI is in the early stages of development, AHRQ has begun to  monitor PCORI’s needs to determine what resources might be needed by  AHRQ to fulfill its PPACA responsibilities related to PCORI and identify  existing resources that the agency can use to fulfill these  responsibilities. These responsibilities include broadly disseminating the  research findings published by PCORI; developing a publicly available  database to collect government-funded evidence and research from  public, private, not-for-profit, and academic sources; promoting the timely  incorporation of PCORI-generated CER findings into health information  technology systems that support clinical decision making; and  establishing a process for receiving feedback about the value of  information disseminated by AHRQ.
AHRQ officials report that they are monitoring PCORI’s needs to  determine what resources might be needed by AHRQ to fulfill its PPACA  responsibilities related to PCORI. The director of AHRQ serves on  PCORI’s Board of Governors and another high-level AHRQ official serves  on PCORI’s methodology committee, which allows AHRQ to obtain  information on the resources the institute might need and when these  resources might be needed. In addition, according to AHRQ officials, the  agency has shared information with PCORI members about AHRQ’s  existing resources at various PCORI meetings.
AHRQ officials reported that they are also in the process of identifying  existing resources, including existing capabilities and ongoing projects,  that the agency can leverage to fulfill its responsibilities related to PCORI.  For example, AHRQ officials are exploring whether contracts the agency  awarded to evaluate AHRQ’s CER dissemination efforts could be  leveraged to meet the agency’s responsibilities to obtain, on behalf of  PCORI, feedback from health care professionals on the CER information  disseminated by AHRQ. In addition, AHRQ is currently assessing whether  a research database being developed by HHS’s Office of the Assistant  Secretary for Planning and Evaluation could be used to, among other  things, store and make publicly available CER funded and generated by  PCORI.
In addition, AHRQ has developed spending plans for fiscal years 2011  and 2012 that describe how AHRQ will use the funds it receives from  PCORTF to fulfill the agency’s responsibilities related to PCORI. These  plans describe proposed FOAs and contract solicitations that would  expand opportunities for AHRQ to disseminate CER information through  a variety of channels to different target audiences, for example, public  service announcements targeting consumers and symposia and  publications targeting researchers and health care professionals. AHRQ  officials stated that the fiscal year 2011 plan has been approved by OMB,  and the fiscal year 2012 plan was under review by HHS as of December  2011. AHRQ officials stated that they have issued one FOA for a project  described in the fiscal year 2011 spending plan but, as of December  2011, have not made any awards.
section_title: Agency Comments
paragraphs: We provided a draft of this report to AHRQ for review and comment.  AHRQ provided technical comments, which we incorporated where  appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the Secretary of HHS,  interested congressional committees, and others. In addition, the report  will be available at no charge on the GAO website at http://www.gao.gov.
If you or your staffs have questions about this report, please contact me  at (202) 512-7114 or at kohnl@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made key contributions to this report are  listed in appendix VI.
section_title: Appendix I: The Agency for Healthcare Research and Quality’s (AHRQ) Mission, Research, Priorities, and Budget
paragraphs: AHRQ’s mission is to improve the quality, safety, efficiency, and  effectiveness of health care for all Americans. The purpose of AHRQ’s  research is to help people make more informed decisions and improve  the quality of health care services. AHRQ, formerly known as the Agency  for Health Care Policy and Research, is 1 of 12 agencies within the U.S.  Department of Health and Human Services (HHS). While the National  Institutes of Health focuses on biomedical research to prevent, diagnose,  and treat disease and the Centers for Disease Control and Prevention  focuses on population health and the role of community-based  interventions to improve health, AHRQ’s research focus is on long-term  and systemwide improvement of health care quality and effectiveness.
AHRQ conducts work in five broad focus areas. These areas include  comparing effectiveness of treatments; quality improvement and patient  safety; health information technology; prevention and care management;  and health care value.
Comparing the effectiveness of treatments. AHRQ’s comparative   provides patients and physicians with  effectiveness research (CER)information on which medical treatments work best for a given  condition. This includes comparisons of drugs, medical devices, tests,  surgeries, or ways to deliver health care in an effort to help patients  and their families understand which treatments work best and how  their risks compare. Initiatives under this focus area include:
The John M. Eisenberg Center for Clinical Decisions and
The Centers for Education and Research on Therapeutics
The Developing Evidence to Inform Decisions about Effectiveness  Quality improvement and patient safety. AHRQ funds and  disseminates research that identifies root causes of threats to patient  safety, provides information on the scope and impact of medical  errors, and examines effective ways to make system-level changes to  help prevent errors. Initiatives under this focus area include:
Patient safety culture assessment tools
Health information technology (health IT). AHRQ provides support  to give access to and encourage the adoption of health IT. The  agency has focused its health IT activities on the following three  goals:  Improve health care decision making Improve the quality and safety of medication management
Prevention and care management. AHRQ translates evidence- based knowledge into recommendations for clinical preventative  services. AHRQ initiatives under this focus area include:
The U.S. Preventative Services Task Force
The Patient-Centered Medical Home
The Practice-Based Research Network
Health care value. AHRQ aims to find greater value in health care by  producing the measures, data, tools, evidence, and strategies that  health care organizations, systems, insurers, purchasers, and  policymakers need to improve the value and affordability of health  care. Initiatives under this focus area include:
The Medical Expenditure Panel Survey
The Healthcare Cost and Utilization Project
The annual National Healthcare Quality Report and National
State snapshots
The Consumer Assessment of Healthcare Providers and Systems
The National Guideline Clearinghouse
The National Quality Measures Clearinghouse  In addition to the above focus areas, AHRQ also conducts crosscutting  activities related to quality, effectiveness, and efficiency. Activities include  data collection and measurement; dissemination and translation; and  program evaluation. In addition, support is provided for the investigator- initiated and targeted research grants and contracts that focus on health  services research in the areas of quality, effectiveness, and efficiency.  These activities provide the core infrastructure used by the other focus  areas.
AHRQ staff and budget. AHRQ currently employs approximately 300  staff. The agency’s fiscal year 2010 budget was $402.6 million, of which  $270.7 million went to research on health costs, quality, and outcomes.  The President’s fiscal year 2012 budget request for AHRQ was   $366.4 million, a decrease of approximately $31 million from fiscal year  2010. (See table 4 for the funding amounts under AHRQ’s focus areas.)
section_title: Appendix II: AHRQ CER Grants Awarded Using Recovery Act CER Funds, by Priority Area
paragraphs: AHRQ and the HHS Office of the Secretary jointly funded one grant. For purposes of our report, we  counted this grant under the number of grants awarded for the HHS Office of the Secretary’s  Dissemination and Translation priority area (see table 7). However, the amount of Recovery Act CER  funds the HHS Office of the Secretary and AHRQ awarded to this grant are reflected under the  amount awarded for the HHS Office of the Secretary’s Dissemination and Translation priority area  and AHRQ’s Evidence Generation priority area.
AHRQ and the HHS Office of the Secretary jointly funded one grant. For purposes of our report, we  counted this grant under the number of grants awarded for the HHS Office of the Secretary’s  Dissemination and Translation priority area. However, the amount of Recovery Act CER funds the  HHS Office of the Secretary and AHRQ awarded to this grant are reflected under the amount  awarded for the HHS Office of the Secretary’s Dissemination and Translation priority area and  AHRQ’s Evidence Generation priority area.
section_title: Appendix III: AHRQ CER Contracts Awarded Using Recovery Act CER Funds, by Priority Area
paragraphs: AHRQ combined Evidence Synthesis and Gap Identification awards under a single solicitation when  announcing the availability of these funds and within awards because, according to agency officials,  having a single solicitation for these two priority areas reduced the amount of work related to these  awards, thereby expediting the award process. As a result, AHRQ funded both of these priority areas,  but advertised projects and made awards for these priority areas under a single solicitation.
section_title: Appendix IV: AHRQ Mechanisms That Support Dissemination of CER
subsections: section_title: Types of mechanisms supporting dissemination Written products
paragraphs: Examples of AHRQ’s dissemination mechanisms
Research reviews and original research reports: These written products draw on  completed scientific studies to make comparisons of different health care  interventions or summarize original clinical research to explore practical questions  about the effectiveness of treatments.  Summary treatment guides for clinicians, consumers, policymakers: Short, plain- language guides summarize the findings of research reviews on the benefits and  harms of different treatment options.  Education modules and presentation slides: These resources are for clinicians  pursuing continuing education credits and for faculty who are instructing clinicians.
Webcasts: Researchers and clinicians participate in online programs to discuss  research findings.
Conference series: Scientific meetings on state-of-the art concepts in  communication, health literacy, and medical decision making.   Audio podcasts: The Healthcare 411 audio podcast series shares news and  information with consumers that they can use in health care decision making,  through 60-second audio news programs and longer format interviews.
Online videos: AHRQHealthTV provides videos for consumers about a range of  health topics on AHRQ’s YouTube channel.  Twitter updates: Short messages are broadcast that can be accessed by computer  or mobile phone.
RSS Feeds: Subscribers receive news and alerts about AHRQ programs through  their RSS reader.  E-mail updates: Subscribers receive e-mail updates on topics they are interested in.  The Medicaid Medical Directors Learning Network: This is one example of a network  formed by AHRQ to create an ongoing collaborative relationship to disseminate  AHRQ products, tools, and research to help members make policy and practice  decisions related to clinical treatment.  Impact Case Studies: AHRQ tracks and summarizes how AHRQ-funded research,  tools, and products are actually implemented by state governments, medical  practices, clinics, and hospitals. Makes summary information available to other  potential users.  AHRQ’s website ahrq.gov provides access to its written products, training programs,  and social media tools, as well as useful search functions and other resources.
Health Care Innovations Exchange: The Exchange offers health professionals and  researchers searchable tools to access information about evidence-based  innovations suitable for a range of health care settings and populations, as well as  opportunities to network with other professionals who have implemented these  innovations.
section_title: Appendix V: Example of Dissemination of Comparative Effectiveness Research by the Agency for Healthcare Research and Quality
paragraphs: Appendix V: Example of Dissemination of  Comparative Effectiveness Research by the  Agency for Healthcare Research and Quality Comparative Effectiveness and Safety of Premixed Insulin Analogues in Type 2  Diabetes: A Systematic Review  Type 2 diabetes is an increasingly common chronic disease that occurs in people who  have too much glucose in their blood. Blood glucose levels are high either because their  cells are resistant to insulin (a hormone that helps convert glucose into energy) or  because their pancreas does not produce enough insulin. Insulin analogues are used  approximately as commonly as human insulin by diabetics who require insulin to regulate  blood glucose levels. Created by genetically modifying human proteins, insulin analogues  were developed as an alternative to human insulin to provide tighter control of blood  sugar levels.
This study summarizes the effectiveness of insulin analogues against traditional human  insulin for type 2 diabetics. Researchers compared the effectiveness of three kinds of  synthetic insulin against their human insulin counterparts, against each other, and against  other antidiabetic medications. The report found that insulin analogues are more effective  than human insulin for treating certain diabetes-related symptoms such as high blood  sugar after meals. However, it also found that human insulin appears to be more effective  than insulin analogues in treating other aspects of diabetes, including lowering blood  sugar levels when patients go 8 hours or more without eating, typically overnight.
Consumer guide (for adults)
Clinician’s guide
Webcast and slides (for clinicians)
Audio podcast (for consumers)
Over 90 target organizations, publications, and electronic venues are identified in the  marketing plan in the following categories: clinicians, insurers, payers, pharmacy and  drugs associations, federal direct and funded medical care programs, consumer-oriented  disease organizations, and government. Provider categories targeted include retail and  health system pharmacists; family physicians and general internists; pharmacologists;  nurse practitioners; physician assistants; and endocrinologists.
General media news services including radio, television news, and major daily  newspapers; consumer and advocacy publications; African-American media; and  translation to Spanish-only and Hispanic media.
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, E. Anne Laffoon, Assistant  Director; Shaunessye Curry; Mary Giffin; Andrea E. Richardson;   Lisa Motley; Krister Friday; and Jessica C. Smith made key contributions  to this report.