NIH Biomedical Research: Agencies Involved in the Indirect Cost Rate-Setting Process Need to Improve Controls
Published: 2016-09-07T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: NIH spent over $23 billion to support extramural research in 2015, including $6.3 billion for indirect costs, which are costs not directly attributable to a specific research project or function, such as utilities expenses, for grants and cooperative agreements. NIH relies on designated cognizant agencies to design adequate internal controls over the processes for negotiating indirect cost rates with research organizations. Once set, these rates must be accepted by NIH and all federal awarding agencies.
GAO was asked to review the internal controls for overseeing the validity of indirect cost rates for NIH's research organizations. This report examines the extent to which the three primary cognizant agencies (CAS, NIH-DFAS, and ONR) that set indirect cost rates on NIH's behalf have designed internal controls to mitigate the potential for fraud, waste, and abuse in the indirect cost rate-setting process. GAO reviewed OMB guidance, the FAR, and the cognizant agencies' internal guidance on indirect cost rate negotiation; interviewed staff at the cognizant agencies; and reviewed a nongeneralizable sample of negotiation case files to obtain an understanding of the design of controls.
section_title: What GAO Found
paragraphs: Research organizations that apply for National Institutes of Health (NIH) funding participate in an indirect cost rate-setting process, which involves submitting a rate proposal; reviewing the proposal and having it audited by a third-party agency at the cognizant agency's request; and finalizing (negotiating) the rate for the organization. The Office of Management and Budget (OMB) and the Department of Health and Human Services (HHS) have designated three primary cognizant agencies to set indirect cost rates for federal financial assistance funded by NIH:
HHS's Cost Allocation Services (CAS),
NIH's Division of Financial Advisory Services (NIH-DFAS), and
the Department of Defense's (DOD) Office of Naval Research (ONR).
These cognizant agencies are responsible for ensuring that negotiated indirect cost rates comply with OMB guidance and the Federal Acquisition Regulation (FAR), as applicable.
GAO found that while the three agencies had designed controls for setting indirect cost rates, deficiencies in the design of some of these controls could result in the waste of federal resources. The deficiencies GAO identified are as follows:
None of the three agencies has updated its internal guidance to reflect current OMB guidance or changes in agency requirements, such as documentation requirements.
ONR relies on audits by the Defense Contract Audit Agency to ensure the adequacy and compliance of indirect cost proposals that it processes, but ONR has not included acceptable time frames in its internal guidance for when these audits are to be completed or what steps are to be taken when audits result in qualified opinions or when a prior audit opinion is rescinded.
All three cognizant agencies' internal guidance lacks detailed instructions to supervisors on their review responsibilities over the indirect cost rate process.
CAS and ONR have not developed internal guidance addressing differences in negotiating indirect cost rates with certain types of research organizations, such as universities and hospitals.
ONR and NIH-DFAS have not developed mechanisms to track key milestones for the indirect cost rate-setting process.
If these deficiencies are not addressed, there is an increased risk that these cognizant agencies may not properly and consistently negotiate indirect cost rates and that the rates negotiated may not comply with applicable federal regulations. Thus, there is an increased risk that the indirect cost rates used to reimburse NIH research organizations will include costs that are not allowable, allocable, and reasonable, and may result in wasted federal resources.
section_title: What GAO Recommends
paragraphs: GAO is making 12 recommendations to improve controls over their indirect cost rate process. HHS concurred with GAO's 7 recommendations to CAS and NIH-DFAS and described ongoing and planned actions to address them. DOD concurred with 4 recommendations to ONR and partially concurred with 1. GAO continues to believe that action is needed, as discussed in the report.

section_title: Letter
paragraphs: The federal government plays an important role in contributing to  American competitiveness and leadership in science through its  sponsorship of research. The Department of Health and Human Services’  (HHS) National Institutes of Health (NIH) is the largest public funder of  biomedical research in the world, sponsoring extramural research  conducted by outside organizations that has led to scientific  breakthroughs and new treatments helping people live longer and  healthier lives. In fiscal year 2015, NIH spent approximately $23.5 billion  to support extramural research, of which approximately $22 billion, or 94  percent, was funded using grants and cooperative agreements. NIH  reimburses for both direct and indirect costs incurred by the organizations  conducting extramural research. Direct costs are specifically attributed to  research projects sponsored by NIH grants, whereas indirect costs are  not directly attributable to a specific project or function, such as  administrative expenses. In fiscal year 2015, reimbursements for indirect  costs represented approximately $6.3 billion, or 28 percent, of NIH’s grant  and cooperative agreement award totals.
To be reimbursed for indirect costs, research organizations must properly  identify and claim reimbursement for these costs in accordance with  applicable federal guidance. The Office of Management and Budget  (OMB) guidance and the Federal Acquisition Regulation (FAR) establish  the principles for determining the types of indirect costs that are allowed  to be claimed and the methods for allocating such costs to federally  funded research. Because indirect costs cannot be specifically attributed  to a particular research grant, they are allocated via an indirect cost rate  that is applied to certain direct costs for each awarded grant. NIH then  uses the indirect cost rate to reimburse indirect costs to the organization.
Each research organization that applies for NIH funding participates in an  indirect cost rate-setting process, which involves the submission of a rate  proposal; reviewing the proposal and having it audited by a third-party  agency at the cognizant agency’s request; and finalizing (negotiating) the  rate for the organization. OMB and HHS have designated three primary  cognizant agencies to negotiate indirect cost rates for grants funded by  NIH: (1) HHS’s Cost Allocation Services (CAS), (2) NIH’s Division of  Financial Advisory Services (NIH-DFAS), and (3) the Department of  Defense’s (DOD) Office of Naval Research (ONR). The cognizant  agencies are responsible for ensuring that the negotiated indirect cost  rates comply with OMB guidance and the FAR, as applicable. Once an  indirect cost rate has been established for a research organization, any  federal agency will apply that rate to all grants awarded to that  organization.
You requested that we review the internal controls for overseeing the  validity of indirect cost rates for NIH grant recipients. In this report, we  discuss the extent to which the three primary cognizant agencies (CAS,  NIH-DFAS, and ONR) have designed appropriate internal controls that  mitigate the potential for fraud, waste, and abuse in the process for  setting indirect cost rates.
To determine the extent to which the three primary cognizant agencies  have designed appropriate internal controls to mitigate the potential for  fraud, waste, and abuse in the indirect cost rate-setting process, we  interviewed staff from each of the three cognizant agencies, NIH, and  OMB. We reviewed OMB’s guidance, the FAR, Standards for Internal  Control in the Federal Government, and A Framework for Managing  Fraud Risk in Federal Programs to identify controls and objectives  required to mitigate the risk for fraud, waste, and abuse in the indirect  cost rate-setting process. We then reviewed each cognizant agency’s  internal control guidance pertaining to the indirect cost rate-setting  process to determine whether the guidance was designed to mitigate the  potential for fraud, waste, and abuse. These controls are used in  negotiating with four types of organizations that received the majority of  NIH funding in fiscal year 2014: (1) institutions of higher education (here- in-after referred to as universities), (2) research institutes, (3) hospitals,  and (4) domestic for-profit organizations.
To further our understanding of the design of internal controls at the three  cognizant agencies, we selected a nongeneralizable sample of case files  for fiscal year 2014 grants at each agency. Next, we performed walk- throughs of the case files to confirm our understanding of each cognizant  agency’s design of internal controls over the indirect cost rate-setting  process. We limited the scope of our walk-throughs to three types of  research organizations: universities, research institutes, and for-profit  organizations. We did not test the operational effectiveness of the  cognizant agencies’ internal guidance and therefore cannot draw  conclusions about the effectiveness of existing internal controls over the  cognizant agencies’ indirect cost rate-setting process.
We conducted this performance audit from March 2015 to September  2016 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objective. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objective. Appendix I provides a more  detailed discussion of our objective, scope, and methodology.
subsections: section_title: Background
paragraphs: NIH funds extramural research primarily through grants. In 2015, NIH  funded approximately 50,000 research projects, totaling over $22 billion  to research organizations, through grants and cooperative agreements for  extramural research. Of the six types of organizations conducting  extramural research, NIH provided approximately $17 billion of funding to  universities in fiscal year 2015. See figure 1 for a breakout of direct and  indirect cost reimbursements received by these research organizations  that were funded through grants and cooperative agreements.
NIH generally reimburses for both direct and indirect costs incurred by the  research organizations conducting extramural grant research. Direct  costs are specifically attributed to research projects and include, for  example, a researcher’s salary, equipment cost, and travel. Indirect costs  represent an organization’s general support expenses that cannot be  specifically attributed to a specific research project or function. They  include, for example, administrative staff salaries, building utilities, and  library operations. Because indirect costs cannot be specifically attributed  to a particular research project, they are allocated via an indirect cost rate  that is applied to certain direct costs for each awarded grant. NIH then  uses the negotiated indirect cost rate to reimburse indirect cost expenses  to the organization. In fiscal year 2015, indirect costs represented  approximately $6.3 billion (28 percent) of NIH’s grants and cooperative  agreement award totals.
subsections: section_title: Cognizant Agencies Responsible for Negotiating Indirect Cost Rates
paragraphs: Each research organization that applies for NIH funding develops a  proposed indirect cost rate and subsequently negotiates with its  designated cognizant agency to set an indirect cost rate used for  reimbursement. Because many research organizations perform research  for multiple federal agencies, OMB guidance specifies that the designated  cognizant agency for a particular research organization is responsible for  negotiating and approving indirect cost rate proposals on behalf of all  other federal agencies. As noted above, OMB and HHS have designated  three primary cognizant agencies to negotiate indirect cost rates for  research organizations that receive NIH-funded grants: (1) CAS, (2) NIH- DFAS, and (3) ONR. For fiscal year 2014 NIH-funded grants, CAS  negotiated rates for approximately 460 universities, 150 research  institutes, and 90 hospitals; NIH-DFAS negotiated rates for approximately  190 for-profit organizations; and ONR negotiated rates for 25 universities  and 5 nonprofit organizations. The cognizant agencies are responsible  for ensuring that the negotiated indirect cost rates comply with OMB  guidance and the FAR, as applicable.
section_title: Calculation of the Proposed Indirect Cost Rate
paragraphs: A research organization’s proposed indirect cost rate is essentially the  ratio of its total indirect costs (after adjustments) to its total direct costs  related to all of the research organization’s grants for a particular time  period. To calculate its indirect cost rate, a research organization  divides its total claimed indirect costs (e.g., equipment and maintenance)  across the organization by its total direct costs (e.g., researcher’s salary  and travel), referred to as the distribution base, across all of the research  organization’s grants. The resulting percentage is the proposed indirect  cost rate. This proposed rate is then negotiated with the cognizant agency  until they agree upon a negotiated indirect cost rate.
To determine how much of the research organization’s indirect costs  should be allocated to each research grant, each grant’s direct cost  amount is multiplied by the negotiated indirect cost rate. In this way, if a  particular research grant contributes a higher percentage of the  organization’s direct costs, it will share in a proportionally higher  percentage of the indirect costs as well. See figure 2 for an example  calculation of an indirect cost rate and allocation at a hypothetical  research organization. This example presumes that the cognizant agency  approves the proposed rate as the negotiated rate without further  adjustment.
section_title: Indirect Cost Rate-Setting Process
paragraphs: The indirect cost rate-setting process begins when a research  organization submits an indirect cost rate proposal with supporting  documentation to its cognizant agency based on a combination of  historical and estimated costs. Examples of supporting documentation  include proposals, audited financial statements, Single Audit reports,  and certification of indirect cost rates for universities and for-profit  organizations. CAS and NIH-DFAS review proposals whereas ONR  generally sends proposals to the Defense Contract Audit Agency (DCAA)  to be audited. When reviewing a proposal, the cognizant agency is to  verify the mathematical accuracy of the rates proposed, confirm  unallowable costs have been excluded, reconcile the cost proposal to the  audited financial statements, and determine the reasonableness of the  proposed costs. After the proposal has been reviewed or audited, the  cognizant agency and the research organization negotiate and come to  an agreement on the rate. The negotiated rate is then documented in a  formal indirect cost rate agreement to be applied to all grants going  forward. Generally, the negotiated rate is set for a 1-to-4-year time frame.  This process is illustrated in figure 3.
section_title: Indirect Cost Rate Guidance and Regulations
paragraphs: The cognizant agencies are required to negotiate the indirect cost rates in  accordance with applicable federal guidance and regulations. OMB  Circular No. A-21 (Cost Principles for Educational Institutions); OMB  Circular No. A-122 (Cost Principles for Non-Profit Organizations); OMB’s  Uniform Administrative Requirements, Cost Principles, and Audit  Requirements for Federal Awards (Uniform Guidance); and FAR Part 31  (Contract Cost Principles and Procedures, applicable to for-profit  organizations) establish the principles for defining, calculating, and  negotiating indirect cost rates for applicable grants. Specifically, the  guidance and regulations describe  classification and types of allowable indirect costs; methods of allocating such costs;  reasonableness of claimed costs; exclusions and descriptions of unallowable cost elements, such as  alcohol and bad debts; and guidelines for establishing indirect cost rates.
GAO’s A Framework for Managing Fraud Risks in Federal Programs  identifies leading practices to aid program managers in managing fraud  risks. For example, A Framework for Managing Fraud Risks in Federal  Programs states that management should design and implement data  analytics as a control activity to prevent and detect fraud. Further,  Standards for Internal Control in the Federal Government provides the  overall framework for establishing and maintaining internal control across  the federal government and for identifying and addressing major  performance and management challenges and areas at greatest risk of  fraud, waste, abuse, and mismanagement. Standards for Internal  Control in the Federal Government states that internal controls comprise  the plans, methods, and procedures used to meet missions, goals, and  objectives. Management is responsible for developing detailed internal  guidance and practices to fit the agency’s operations and ensuring that  they are integral to operations.
section_title: Recent Reviews of the Cognizant Agencies’ Indirect Cost Rate-Setting Processes
paragraphs: GAO and ONR’s internal review team have each issued one report in  recent years with recommendations related to the indirect cost rate- setting processes for the cognizant agencies. GAO reviewed DOD’s  processes and procedures related to indirect costs for research and made  one recommendation. During its review, GAO found a wide variation in  indirect cost rates at universities receiving DOD funding because of  inconsistencies in the rate-setting and reimbursement processes. The  report concluded that with inconsistencies in rate-setting and  reimbursement processes and weakness in oversight methods, DOD  lacks assurance that it is reimbursing indirect costs appropriately. GAO’s  recommendation directed the Secretary of Defense to assess the current  level of audit coverage for monitoring DOD’s indirect cost reimbursement  for universities and determine what level is sufficient and whether to  expand use of closeout and other audits to oversee compliance. ONR’s  internal review office assessed the effectiveness of its Indirect Cost  Branch’s processes. While the overall rating was satisfactory, ONR’s  review office made one recommendation related to the performance  standards and procurement processes and one recommendation to  complete a staffing analysis. ONR implemented GAO’s recommendation,  but ONR’s two internal review recommendations remain unimplemented.
section_title: The Three Cognizant Agencies Have Not Adequately Designed Controls over the Indirect Cost Rate- Setting Process
paragraphs: Under OMB guidance, the cognizant agencies are responsible for  reviewing, negotiating, and approving indirect cost rate proposals on  behalf of all other federal agencies. Consequently, NIH relies on the  cognizant agencies to design adequate internal controls over their  processes for negotiating indirect cost rates with research organizations  that support the various types of NIH extramural research programs.  Since NIH reimburses the indirect costs for NIH grants based on the  negotiated rates, the controls that the cognizant agencies have designed  are essential to help protect NIH funds against fraud, waste, and abuse in  the indirect cost rate-setting process.
Based on our review of OMB guidance and the FAR, we identified  controls that are key to preventing fraud, waste, and abuse in the indirect  cost rate-setting process. These controls included steps for the negotiator  to (1) determine the allowability, allocability, and reasonableness of  proposed indirect costs and assess a research organization’s methods for  allocating such costs to federally funded research grants; (2) determine  the composition of the distribution base; and (3) maintain sufficient  documentation to support the negotiation and calculation of the indirect  cost rates.
Our review determined that while the three cognizant agencies had  designed some controls for setting indirect cost rates, deficiencies in the  design of these controls could result in the waste of federal resources.  The deficiencies we identified are as follows:  internal guidance at all three cognizant agencies was not updated to  reflect current OMB guidance or changes in agency requirements,  such as documentation requirements or formalizing agency internal  guidance;
ONR’s internal guidance lacked adequate review procedures over  DCAA’s advisory audit function when a DCAA audit was significantly  delayed or resulted in a qualified opinion or when DCAA rescinded a  previously issued opinion;  internal guidance at all three cognizant agencies lacked detailed  instructions to supervisors on their review responsibilities over the  indirect cost rate process;
CAS and ONR had not developed internal guidance addressing  differences in negotiating indirect cost rates with certain types of  research organizations; and
ONR and NIH-DFAS had not developed mechanisms to track key  milestones for the indirect cost rate-setting process.
subsections: section_title: The Three Cognizant Agencies Did Not Update Internal Guidance for Changes in Regulations or Agency Requirements
paragraphs: Standards for Internal Control in the Federal Government states that  management should develop internal guidance to help ensure that  management directives are carried out and effective and efficient control  activities are implemented to accomplish the agency’s objectives. The  standards also state that information should be recorded and  communicated to staff within a time frame that enables the staff to carry  out their responsibilities. We found that CAS and ONR had not updated  their internal guidance to reflect a majority of the federal regulations  issued as the Uniform Guidance, which became effective for grants  awarded on or after December 26, 2014. Working with the Council on  Financial Assistance Reform, OMB consolidated its grants management  circulars into the Uniform Guidance to streamline its guidance, promote  consistency among grantees, and reduce administrative burden on  nonfederal entities. New requirements include, for example, (1) the  option for research organizations that currently have a negotiated rate to  apply for a onetime extension of that rate for a period of up to 4 more  years, (2) the option for research organizations that have never  negotiated an indirect cost rate to use a de minimis indirect cost rate of 10  percent in lieu of negotiating an indirect cost rate with the cognizant  agency, and (3) additional provisions for administrative costs to be  counted as direct costs when allocable to a specific grant.
CAS officials stated that they issued a memo in July 2015 to the  negotiators providing guidance covering one of the new Uniform  Guidance requirements which allows for a one-time rate extension.   However, they have not updated their internal guidance to reflect other  new applicable requirements from the Uniform Guidance. According to  officials at CAS and ONR, they were waiting for OMB to finalize the  corresponding technical corrections and frequently asked questions and  thus had not updated their internal guidance to reflect the changes in the  Uniform Guidance as of June 2016. Further, CAS officials stated that in  addition to waiting for technical corrections and frequently asked  questions to be finalized, they were also waiting for HHS’s Office of  Grants Policy, Oversight and Evaluation to update the HHS Grants Policy  Statement to reflect the changes in the Uniform Guidance. The Grants  Policy Statement covers indirect cost policy for HHS grants and has not  been updated since 2007. However, OMB officials stated that in meetings  held in 2015 with the cognizant agencies, they requested that the  cognizant agencies update their internal guidance to reflect the  requirements of the Uniform Guidance. Figure 4 displays a timeline of key  milestones during the implementation process of the Uniform Guidance.
If they do not incorporate the new Uniform Guidance and its applicable  regulatory requirements in their internal guidance, CAS and ONR risk not  properly and consistently negotiating indirect cost rates—potentially  leading to waste of government resources. CAS and ONR as well as  research organizations stand to benefit from incorporating elements of the  Uniform Guidance, such as ONR extending negotiated rates for up to 4  years and both allowing the use of a de minimis indirect cost rate of 10  percent. These two options reduce administrative burden to both parties,  making the process more efficient and potentially reducing the waste of  government resources. Also, while OMB guidance has allowed for  charging salaries of administrative and clerical staff as a direct cost when  appropriate and when certain conditions in the guidance are met, the  Uniform Guidance has clarified the existing conditions and added new  requirements describing when administrative costs can be charged as  direct. For example, a new requirement states that costs must now be  explicitly included in the budget or have the prior written approval of the  federal awarding agency before they can be charged as direct costs.  Without updated internal guidance to reflect those provisions, CAS and  ONR may incorrectly negotiate this element as direct in cases where it  does not meet the conditions outlined in federal guidance.
Further, we found that the cognizant agencies changed certain agency  requirements, such as documentation requirements, when reviewing  indirect cost rate proposals, but did not update their internal guidance to  reflect such changes. For example:  Individual CAS field offices have established checklists, which provide  procedural steps for the negotiators to follow and acknowledge  completion of during their review of the research organization’s  indirect cost proposal. These checklists include steps such as  reconciling the proposal to the audited financial statements,  confirming that the rates are mathematically accurate, verifying the  consistency of the distribution base with prior years, and confirming  that unallowable expenses have been removed from the indirect costs  claimed. We found that although CAS considers use of these  checklists to be a control over the negotiation process, CAS has not  updated its internal guidance to require negotiators to use a checklist  or established a standardized checklist for all offices to use when  negotiating indirect cost rates.
Our walk-throughs of CAS case files determined that field offices have  developed their own checklists based on their interpretations of  negotiation requirements found in CAS’s internal guidance, and that  these interpretations were not always consistent. Although these  checklists contained similar review steps that the negotiators were to  perform, we found differences in instructions for performing trend  analyses and reviewing the indirect and direct costs claimed for  allowable and unallowable costs, which are two key controls for  minimizing the potential for fraud, waste, and abuse. Without a  standardized checklist and agency-wide internal guidance instructing  negotiators to use the standardized checklist during negotiation,  management cannot reasonably assure that negotiators and  supervisors are consistently and fairly negotiating indirect cost rates  across all CAS field offices. According to CAS officials, they are in the  process of finalizing review checklists for nonprofit organizations,  which include research institutes and universities using the simplified  proposal method and they expect the checklists to be implemented by  the end of the fiscal year 2016.
For NIH-DFAS, we were unable to determine when primary guidance  was last updated because NIH-DFAS did not have formalized internal  guidance with official dates and signatures. Its internal guidance  lacked key characteristics—such as a policy number, purpose of the  policy, effective date, and approving official—that are normally  included in formal policies and procedures. NIH-DFAS officials  acknowledged that their internal guidance did not contain the key  characteristics of internal control. These officials stated that in June  2014, NIH’s Administrative and Program Resources Office finalized  the standard operating procedures for developing formalized internal  guidance, and in October 2015, NIH-DFAS hired an outside contractor  to formalize NIH-DFAS’s internal guidance, which the officials  anticipate will be completed by December 2016.
Standards for Internal Control in the Federal Government requires  management to ensure that internal guidance and practices are integral  to their operations. We found that these deficiencies occurred because  the cognizant agencies had not designed monitoring controls to ensure  that internal guidance had been reviewed and updated to reflect changes  in federal regulations and internal procedures. According to CAS officials,  they have not designed monitoring controls to establish time frames for  periodic review and assign related roles and responsibilities because     (1) they update internal guidance as needed and (2) staff already know  their roles and responsibilities. According to ONR officials, in May 2014,  ONR issued a policy that requires staff to review and update their internal  guidance annually. However, ONR’s internal guidance was last reviewed  and updated in January 2011, so the Uniform Guidance and its applicable  regulatory requirements have not been incorporated. Without established  controls for reviewing and updating internal guidance, including setting  time frames and assigning roles and responsibilities, there is an  increased risk that management will not hold staff accountable. The  officials acknowledged that their current internal guidance needed to be  updated to reflect changes to applicable federal guidance, regulations, or  agency procedural changes, such as formalizing internal guidance. By not  ensuring that their internal guidance reflects current regulations and  procedures, the agencies increase their risk that their indirect cost rate- setting processes may not be properly and consistently carried out by the  negotiators, which in turn increases the risk that rates negotiated will not  be in accordance with federal guidance and regulations, thus increasing  the potential for fraud, waste, and abuse of federal resources.
section_title: ONR Did Not Design Adequate Review Procedures over the DCAA Advisory Audit Function
paragraphs: Unless other arrangements have been made, ONR negotiators request a  DCAA advisory audit on all indirect cost rate proposals and rely on DCAA  to issue an opinion on the adequacy and compliance of a research  organization’s indirect cost proposal prior to negotiating the indirect cost  rate. Standards for Internal Control in the Federal Government states  that controls should be designed to help ensure that management’s  directives are carried out in order to accomplish the agency’s control  objectives. While ONR’s internal guidance requires taking into account  DCAA results and comments, it does not specify review procedures to  take when DCAA (1) is unable to complete its advisory audit within a  reasonable time frame, (2) issues a qualified opinion, or (3) rescinds one  of its previously issued audit opinions.
To negotiate indirect cost rates timely, ONR’s internal guidance  requires its negotiators to provide DCAA with 45 days to complete its  advisory audit report on behalf of ONR, unless arrangements have  been made for a due date beyond 45 days (e.g., DCAA requests an  extension). In a March 2015 report to Congress, DCAA officials stated  that depending on the type of audit, the average elapsed time to  complete an audit ranges from 95 to 1,006 days. This high number  is the result of a growing backlog of DCAA audits. In light of this  backlog, ONR officials told us that their current procedure is to  routinely give DCAA additional time to complete its initial audit report  of a research organization’s cost proposals. For example, in our walk- through of ONR case files, we found that ONR initially granted as  many as 363 days—about a year—for DCAA to complete its audit,  well beyond the 45 days period in ONR’s internal guidance. In many  cases, DCAA requested and received an extension to complete its  audit of a research organization’s cost proposal. Our walk-throughs  identified instances in which ONR granted up to 139 days on top of  what was initially given to DCAA to complete its audit. ONR’s internal  guidance does not reflect this change in procedure of routinely  granting DCAA more than 45 days to prepare an advisory audit report,  and it does not include parameters defining a reasonable period for  the initial extensions beyond the 45 days. In addition, ONR’s internal  guidance does not provide procedures for negotiators to perform the  necessary audit steps if DCAA does not complete its audit within the  required time frames or a reasonable and accepted extended period.  By not including in its internal guidance reasonable and acceptable  audit completion time frames and procedures for ONR negotiators  who perform audit steps when DCAA does not complete an audit,  management cannot reasonably assure that indirect cost rate  negotiations will be accurate and completed in a timely manner.
Furthermore, we found that although ONR’s internal guidance  requires negotiators to review DCAA’s audit results, it does not  contain procedures for the negotiators to perform supplemental review  steps when DCAA’s audit results contain qualified opinions or when  DCAA rescinds one of its previously issued audit opinions. DCAA  issues a qualified opinion when it is unable to perform detailed  transaction testing on the incurred costs that are often used as the  research organization’s basis of estimate or when it does not have  enough time to complete an audit. Our walk-throughs identified  several instances of DCAA audit reports with qualified opinions (in  which DCAA was not able to perform detailed transaction testing on  incurred costs used as a basis of estimate) and one instance in which  DCAA rescinded an audit opinion (when DCAA’s Integrity and Quality  Assurance Directorate review found that DCAA’s audit was not  performed in accordance with generally accepted government  auditing standards). In those cases, ONR’s guidance does not require  the negotiator to perform any supplemental procedures to validate  costs. Without internal guidance regarding supplemental procedures  to be taken when DCAA issues a qualified opinion or rescinds one of  its previously issued audit opinions, ONR cannot reasonably assure  that its negotiators take consistent and appropriate steps, potentially  increasing the risk that ONR’s negotiated indirect cost rates could  include reimbursement for unallowable, unallocable, and  unreasonable costs.
We found that these deficiencies occurred because ONR had not  designed sufficient review procedures for those situations when DCAA   (1) is unable to complete its advisory audit within a reasonable time  frame, (2) issues a qualified opinion, or (3) rescinds one of its previously  issued audit opinions. According to ONR officials, they believe that the  procedures listed in their internal guidance will allow negotiators to  validate costs prior to negotiating an indirect cost rate. However, by not  including in its internal guidance reasonable and acceptable audit  completion time frames and supplemental procedures to be taken to  review cost proposals when DCAA cannot perform its audits in a timely  manner, issues a qualified opinion, or rescinds one of its previously  issued audit opinions, there is an increased risk that the negotiated  indirect cost rate may not comply with applicable regulations, thus  increasing the risk of fraud, waste, and abuse of federal resources.
section_title: Cognizant Agencies’ Internal Guidance Lacks Detailed Procedures on Supervisory Review over the Indirect Cost Rate- Setting Process
paragraphs: Although the cognizant agencies’ internal guidance describes broad  procedures for supervisory review, such as requiring that the supervisors  review and approve workpapers and applicable rates, these documents  do not include detailed procedures for supervisors to monitor the  negotiators’ processes. Supervisory review is a type of internal control  that provides reasonable assurance that the negotiator followed agency  procedures for negotiating indirect cost rates and reduces the risk of  inaccuracy and potential for waste of government resources. Standards  for Internal Control in the Federal Government states that management  should design internal controls to reasonably assure that ongoing  monitoring occurs in the course of normal operations, which includes  supervisory activities.
We found that none of the three cognizant agencies’ internal guidance  include adequate detailed procedures that would allow supervisors to  confirm that the negotiators have adequately performed and documented  key controls identified in their internal guidance, such as performing cost  trend analyses, executing reconciliations, analyzing adjustments for  unallowable costs, verifying the accuracy of the distribution base,  evaluating any future significant changes, and comparing the proposal to  the disclosed cost accounting practices in the approved Disclosure  Statement. These controls are meant to provide reasonable assurance  that only allowable, allocable, and reasonable indirect costs have been  proposed and that appropriate distribution bases were selected for  allocating such costs to federally funded grants. However, the cognizant  agencies’ internal guidance did not specifically require supervisors to  reasonably assure that negotiators perform those steps. In fact, as  illustrated by the examples below, our walk-throughs of case files for the  three cognizant agencies found a few instances in which supervisors  approved rates that were set by negotiators who did not perform one or  more of the controls required by the cognizant agencies’ internal  guidance.
As a tool used to assess risk, CAS’s internal guidance requires  negotiators to complete a detailed trend analysis of nonprofit and long  form research organizations’ indirect cost rates and distribution bases  for the last 3 years, including the proposal year. According to CAS’s  internal guidance, a trend analysis provides the negotiator with an  insight into the areas of the proposal needing a more detailed review.  During our walk-through of CAS case files, we found that the  negotiators did not always review the indirect cost rates and the  distribution bases for the last 3 years, including the proposal year,  when preparing a trend analysis. Additionally, CAS’s internal guidance  requires the negotiator to compare the accounting policies delineated  in the research organization’s Disclosure Statement to the indirect  cost proposal to ensure consistency in the proposal’s preparation.  However, during our walk-through of CAS case files, we found that  when the supervisors reviewed supporting documentation, they did  not confirm that the negotiators had performed this comparison.
OMB guidance states that each institution must describe the process  it uses to ensure that federal funds are not used to subsidize industry  and foreign government programs. To meet this requirement, ONR’s  submission requirements checklist states that institutions must  complete and submit several certifications and assurances, including  a Statement of Assurance attesting that federal funds are not used to  subsidize industry and foreign government programs, which cites the  OMB guidance. During our walkthrough of ONR’s case files, we found  one instance in which a university did not submit the required  Statement of Assurance. The ONR supervisor did not confirm that all  required certifications and assurances were included in the grantee’s  proposal during his review of the supporting documentation.
At NIH-DFAS, we found one instance in which the negotiator had not  reconciled the indirect cost information contained in the cost proposal  to the audited financial statements, which is a control that helps verify  that actual costs incurred were used as a basis in the proposal. The  NIH-DFAS supervisor did not verify that the negotiator had reconciled  the cost information.
Agency officials at the three cognizant agencies stated that they believe  the procedures listed in their internal guidance are sufficient for  supervisors to perform detailed review of the negotiators’ work. However,  without adequate documented supervisory review procedures, there is an  increased risk that supervisors could approve rates that were not properly  executed by the negotiators, as demonstrated in the examples above.
section_title: Two Cognizant Agencies’ Internal Guidance Does Not Address Differences in Negotiating with All Types of Research Organizations
paragraphs: Internal guidance links an agency’s objectives to its day-to-day  operations, providing a framework for employees to clearly understand  their roles and responsibilities when carrying out agency objectives.  Standards for Internal Controls in the Federal Government states that  management should develop internal guidance to help ensure that  management directives are carried out, and that effective and efficient  control activities are implemented to accomplish the agency’s objectives.  Standards also state that information should be recorded and  communicated to staff within a time frame that enables the staff to carry  out their responsibilities. However, we found that CAS and ONR lack  internal guidance for negotiating indirect cost rates with certain types of  research organizations.
CAS has not developed internal guidance that describes procedures  required for negotiating indirect cost rates for hospitals, which  represented $511 million in indirect cost reimbursements in fiscal year  2015. Currently, CAS uses a policy that the Department of Health,  Education, and Welfare (now HHS) published in 1974 (policy OASC- 3) for establishing indirect cost and patient care rates with hospitals  for grants and contracts. An HHS Office of Inspector General  memorandum from 1993 recommended that the hospital cost  principles be modernized and strengthened, as OASC-3 does not  always provide clear guidance for determining what types of costs  should be allowed and how costs should be allocated. However,  HHS has not updated OASC-3, and CAS continues to rely on this  policy for negotiating indirect cost rates with hospitals. According to  CAS officials, they have not created internal guidance for hospitals  because CAS’s national specialist—who was responsible for creating  and updating internal guidance—resigned and they have not filled the  vacancy. Without up-to-date and clear guidance for negotiating  indirect cost rates for hospitals, there is an increased risk that  negotiators will allow unallowable costs claimed by hospitals.
Additionally, CAS does not have internal guidance related to the types  of distribution bases allowed for small universities using the simplified  method (also known as the short form) for preparing facilities and  administrative cost rate proposals. According to CAS officials, they  consider the internal guidance for the long form to be sufficient and  have instructed negotiators to use the long form best practice manual  for all university proposals. However, under the simplified method,  universities are given greater flexibility in choosing between two  different types of distribution bases, whereas the long form proposal  requires universities to submit proposals using only a modified total  direct cost base. Further, OMB guidance restricts the use of the  simplified method when it produces results that appear inequitable to  the federal government. Therefore, without internal guidance for  proposals using the simplified method, there is an increased risk that  indirect costs will not be fairly distributed to grants. For example, a  distribution base that is understated will result in an inflated indirect  cost rate.
Although OMB has issued separate guidance specific to universities  and nonprofits, ONR’s only internal guidance on negotiating indirect  cost rates applies to both types of research organizations and does  not address certain key differences in OMB guidance applicable to  each type of research organization. Specifically, the distribution bases  allowed for calculating indirect cost rates differ for universities (short  and long form) and for nonprofits. For example, under the OMB  guidance, universities may use only two different types of distribution  bases, whereas nonprofit organizations are given greater flexibility as  they are instructed to select a distribution base best suited for  assigning the indirect costs to the project. However, ONR’s internal  guidance applicable to both types of research organizations does not  distinguish between the different provisions of OMB guidance. After  we identified this issue, ONR officials acknowledged that the internal  guidance needs to be updated to distinguish procedures between the  different provisions of OMB guidance and reported that they are in the  process of making the necessary revisions. ONR officials stated that  they anticipate that the draft internal guidance will be updated and  ready for final management review in June 2016. To reduce the risk  that the negotiators could use an inappropriate distribution base,  resulting in a rate calculation that is not in compliance with federal  guidance, it is important that ONR implement and issue the internal  guidance timely. However, ONR has not established a time frame for  issuance of the final internal guidance.
Without up-to-date internal guidance for all types of research  organizations, the cognizant agencies cannot clearly delineate  negotiators’ roles and responsibilities during the indirect cost rate  negotiation process. Further, by not developing adequate internal  guidance for all types of research organizations, cognizant agencies are  at an increased risk that the negotiated indirect cost rates could include  reimbursement for unallowable, unallocable, and unreasonable costs,  potentially resulting in wasted federal resources.
section_title: Two of the Cognizant Agencies Do Not Have Mechanisms to Track Milestones for Indirect Cost Rate Data
paragraphs: In order for each cognizant agency to achieve its objectives and comply  with federal guidance and regulations, it is essential for management to  have the capability to generate and review reports on indirect cost rate  data. Standards for Internal Control in the Federal Government states that  relevant information should be recorded and communicated with  management and others within the entity who need it. This information  should be in an understandable format, and provided within a time frame  that enables employees to carry out their internal control responsibilities.
We found that only one of the three cognizant agencies generates reports  for management to view data associated with all phases of the indirect  cost rate-setting process. Specifically, CAS’s system generates reports  that management and the negotiators can use to determine when  proposals are due; when signed rate agreements, proposals, or  extensions to proposals are overdue; and other rate agreement  information. In contrast, NIH-DFAS and ONR officials reported that they  do not have systems capable of generating these types of reports. ONR  officials stated that they do maintain a spreadsheet in Microsoft Excel for  universities and nonprofit organizations, which tracks information such as  whether rate agreements have been signed and whether a proposal is  awaiting an audit or management review. However, the Excel  spreadsheet does not identify when rate proposals are past due, when  the DCAA advisory audit reports are due, or the dates when the  negotiators followed up for past due proposals or DCAA advisory audit  reports.
Without the ability to produce reports enabling the cognizant agencies to  track all phases of the indirect cost rate-setting process, cognizant  agencies cannot reasonably assure compliance with federal guidance  and regulations and accomplishment of agency objectives. For example,  in accordance with federal guidance and regulation, a research  organization must submit its indirect cost rate proposal to the cognizant  agency within 6 months after the close of the fiscal year. However, we  found one instance in which an ONR research organization had not  submitted its final indirect cost rate proposal, which was due in December  2014. As of January 2016 (13 months overdue), the organization had still  not provided the final indirect cost rate proposal to ONR. In June 2016,  upon further inquiry by us, ONR officials stated that the research  organization did not submit a rate proposal because it had gone out of  business.
When indirect cost rate proposals are not submitted in a timely manner,  the cognizant agencies are also unable to close out grants and can incur  additional costs to the government, leading to potential waste. Effective  management reporting tools are therefore critical for enabling  management to make better decisions and meet agency objectives and  goals for effective and efficient use of resources.
section_title: Conclusions
paragraphs: Grants are an important form of federal financial assistance that NIH uses  to carry out its mission as the nation’s leading sponsor of biomedical  research. NIH reimbursed $6.3 billion in indirect costs in fiscal year 2015,  representing over a quarter of the total amount that NIH awarded through  grants and cooperative agreements. Because the designated cognizant  agencies negotiate the indirect cost rates on behalf of all federal  agencies, including NIH, it is critical that the cognizant agencies  adequately design internal controls to reasonably assure that their indirect  cost rate negotiation process run efficiently and effectively and reduce the  risk that federal resources may be subject to fraud, waste, and abuse.
Although the three cognizant agencies had established controls for the  indirect cost rate-setting process, we identified deficiencies in the design  of some of these controls. Until the three cognizant agencies update their  internal guidance to reflect current regulations and agency procedures  and include instructions to supervisors about their review responsibilities,  there is an increased risk that taxpayer funds will not be adequately  protected. In addition, ONR lacks needed guidance to address delayed  audits, audits that have qualified opinions, and audits where previously  issued opinions are rescinded. Also, both CAS and ONR lack internal  guidance to address differences in negotiating for all research  organization types. Finally, ONR and NIH-DFAS are missing mechanisms  to track key milestones for indirect cost rate-setting information. If these  deficiencies are not addressed, there is an increased risk that the  cognizant agencies may not properly and consistently negotiate indirect  cost rates and that the rates negotiated may not comply with applicable  federal regulations. Thus, there is an increased risk that the indirect cost  rates used to reimburse NIH research organizations will include costs that  are not allowable, allocable, and reasonable and may result in wasted  federal resources.
section_title: Recommendations for Executive Action
paragraphs: To improve the design of internal controls over the indirect cost rate- setting process, we recommend that the Director of CAS take the  following four actions: establish internal controls to periodically review and update internal  guidance when changes are made to applicable regulations to  reasonably assure the guidance reflects current requirements; develop a standardized checklist and document procedures in its  internal guidance instructing negotiators to use the checklist during  negotiation; develop detailed internal guidance for the completion and  documentation of supervisory review of the indirect cost rate  negotiation process to provide reasonable assurance that key control  activities have been performed by the negotiators; and develop internal guidance for negotiating indirect cost rates with all  types of research organizations, including hospitals, as well as  universities using the simplified method.
As NIH-DFAS begins formalizing its internal guidance, we recommend  that the Director of NIH-DFAS take the following three actions: update internal guidance to include key characteristics, such as policy  number, purpose of the policy, effective date, and approving official,  that are normally included in formal policy and procedures; develop detailed procedures for the completion and documentation of  supervisory review of the indirect cost rate negotiation process to  provide reasonable assurance that key control activities have been  performed by the negotiator; and establish a mechanism for tracking key milestones in the indirect cost  rate-setting process, such as when indirect cost rate proposals are  due.
To improve the design of internal controls over the indirect cost rate- setting process, we recommend that the Director of ONR take the  following five actions:  implement the May 2014 policy requiring an annual review of  guidance so that internal guidance is updated when changes are  made to applicable regulations and procedures to reasonably assure  that the guidance reflects current requirements;  include in its internal guidance acceptable DCAA audit completion  time frames and identify supplemental procedures to be performed by  negotiators if DCAA cannot perform its audits timely or if DCAA issues  a qualified opinion or rescinds one of its previously issued audit  opinions, to reasonably assure that the indirect cost rate proposal has  been adequately reviewed and the negotiated rate complies with  applicable regulations; develop detailed procedures for the completion and documentation of  supervisory review of the indirect cost rate negotiation process to  provide reasonable assurance that required certifications and  assurances are obtained and follow-up with the research organization  is documented;  finalize and issue internal guidance for negotiating indirect cost rates  with universities and nonprofit organizations, including establishing a  time frame for issuance of the internal guidance, to help ensure that  the procedures are implemented in a timely manner; and update ONR’s existing process for tracking key milestones in the  indirect cost rate-setting process to include information such as when  indirect cost rate proposals are overdue and when DCAA’s audit  reports are due.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to HHS, DOD, and OMB for their review  and comment. In written comments, reprinted in appendixes II and III,  HHS concurred with our recommendations and provided information on  actions planned or under way to address them and DOD concurred with  four of our five recommendations and partially concurred with one. Our  report did not include recommendations to OMB, and the OMB Liaison to  GAO responded in an e-mail that OMB did not have any formal  comments on the report. HHS, DOD, and OMB each provided technical  comments, which we incorporated as appropriate.
In response to the recommendations directed toward CAS, HHS said that  by December 31, 2016, CAS will establish a written procedure requiring  periodic reviews and updates of internal guidance whenever changes are  made to applicable regulations, update and complete standardized  checklists for each type of indirect cost review and instruct the staff to use  the checklists, establish and implement standardized procedures for  supervisory review of workpapers and rate agreements, and update  internal guidance for negotiating indirect cost rates with universities using  the simplified method. HHS also stated that CAS would develop internal  guidance for negotiating with hospitals as soon as possible.
In response to the recommendations directed toward NIH-DFAS, HHS  said that by December 31, 2016, NIH-DFAS will update internal guidance  to include key characteristics that are normally included in formal policies  and procedures and develop detailed procedures for completing and  documenting supervisory review of indirect cost rate negotiations. HHS  also stated that NIH-DFAS will establish a mechanism for tracking key  milestones in the indirect cost rate-setting process. HHS said that NIH- DFAS is currently looking into the feasibility of incorporating key  milestones into two major initiatives, and if it is unable to do so, NIH- DFAS will develop an alternative tracking system by March 31, 2017.
HHS’s CAS and NIH-DFAS actions, if implemented effectively, would  address our recommendations.
In response to the recommendations directed toward ONR, DOD  concurred with four of our five recommendations. For the four  recommendations that it concurred with, DOD stated that ONR will  comply with its requirement for an annual review of its internal guidance  and update its internal guidance to provide more realistic DCAA audit  report dates, including general procedures for negotiators to perform in  the case of untimely audits and qualified or rescinded opinions.  Additionally, DOD said that ONR will update its internal guidance for  negotiating indirect cost rates with universities and nonprofit organizations  by December 31, 2016, and will update its existing processes for tracking  key milestones to include information such as due dates for rate  proposals and DCAA audit reports. If implemented effectively, these  actions would address the four recommendations.
In response to the recommendation to develop supervisory review  procedures, DOD partially concurred. First, DOD disagreed that a  certification declaring that federal funds were not used to subsidize  industry and foreign government programs is required. DOD cited 2  C.F.R. pt. 200, appendix. III (formerly located in OMB Circular A-21, 2  C.F.R. pt. 220, app. A prior to December 2014), which states that each  institution must describe the process it uses to ensure that federal funds  are not used to subsidize industry and foreign government programs, and  stated that a certification is only one way to accomplish this. While we  agree there may be different ways to meet this requirement, ONR’s  submission requirements checklist states that institutions must complete  and submit several certifications and assurances, including a Statement  of Assurance attesting that federal funds are not used to subsidize  industry and foreign government programs.The Statement of Assurance  specifically cites the C.F.R. provision. Consequently, ONR has chosen to  use the Statement of Assurance to meet the C.F.R. provision, and we  found no other documentation that would otherwise meet the  requirement. We revised the body of the report to avoid suggesting that  OMB requires a certification and to specifically cite the Statement of  Assurance, and similarly revised the recommendation for clarification, but  our overall finding stands.Therefore, we continue to believe that ONR  needs to develop supervisory review procedures to reasonably assure  that these Statements of Assurance are obtained.
Further, DOD did not agree that ONR lacks procedures to ensure  supervisors confirm that negotiators adequately performed and  documented key controls. DOD noted that both the primary and  secondary supervisors are required to review and approve the Business  Clearance Memorandum, which records steps performed by the  negotiator. While we agree that the Business Clearance Memorandum  documents steps performed by the negotiator, these steps are  documented at a high level and do not include detailed procedures for  supervisors to follow to reasonably assure that the negotiator has  performed and documented all key control activities, such as obtaining all  required certifications and assurances. DOD agreed in its response that  ONR’s Business Clearance Memorandum can be improved and stated  that ONR will update it to require the negotiator to cross-reference the  review steps to the proposal to facilitate the supervisor’s review process.  However, it is not clear whether the planned Business Clearance  Memorandum revisions will include providing detailed procedures for  supervisory review as we recommended.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 21 days from the  report date. At that time, we will send copies to the Secretaries of Health  and Human Services and Defense and the Director of the Office of  Management and Budget. In addition, the report will be available at no  charge on the GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-2623 or davisbh@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who make key contributions to this report  are listed in appendix IV.
section_title: Appendix l: Objective, Scope, and Methodology
paragraphs: The objective of this review was to determine the extent to which the  cognizant agencies have designed internal controls to mitigate fraud,  waste, and abuse in the indirect cost rate-setting process for National  Institutes of Health (NIH) grants. Generally, the agency that provides the  most funding to a particular research organization has rate-setting  cognizance for that organization. However, the Office of Management and  Budget (OMB) and the Department of Health and Human Services (HHS)  have designated three primary cognizant agencies to negotiate indirect  cost rates for grants funded by NIH: (1) HHS’s Cost Allocation Services  (CAS), (2) NIH’s Division of Financial Advisory Services (NIH-DFAS), and  (3) the Department of Defense’s (DOD) Office of Naval Research (ONR).  Specifically, OMB guidance (OMB Circular No. A-21, Cost Principles for  Educational Institutions, whose provisions have been incorporated in  OMB’s Uniform Administrative Requirements, Cost Principles, and Audit  Requirements for Federal Awards (Uniform Guidance)) specifically  assigns responsibility for negotiating rates for universities to either HHS’s  CAS or DOD’s ONR. OMB guidance (OMB Circular No. A-122, Cost  Principles for Non-Profit Organizations, whose provisions have been  incorporated in the Uniform Guidance) assigns responsibility for  negotiating rates for research institutes to the cognizant agency with the  largest dollar value of grants with an organization, unless different  arrangements are agreed to by the agencies concerned. HHS Acquisition  Regulations Part 342, Contract Administration, assigns the responsibility  for negotiating indirect cost rates for hospitals to HHS’s CAS.  Furthermore, the Secretary of HHS authorized NIH-DFAS to have HHS- wide cognizance of the indirect cost rate negotiation function with for- profit grantees, which are not covered by the OMB circulars on grant cost  principles. Therefore, we focused our review on the three primary  cognizant agencies: (1) CAS, (2) NIH-DFAS, and (3) ONR. rates for the four types of research organizations reviewed and  corresponding federal guidance and regulations. for these key characteristics and determined, through the inclusion of the  characteristics, whether the policies would be considered formal.
We obtained NIH grant funding data for extramural research for fiscal  year 2014 and performed procedures to determine whether the data were  reliable enough for our purposes. Specifically, we interviewed  knowledgeable agency officials about the quality control procedures the  agency had in place when collecting and creating the data and tested the  data for unusual items. Based on the results of these procedures, we  determined that the data were reliable enough for our purposes. To  further our understanding of the design of internal controls at the three  cognizant agencies, we selected a nongeneralizable sample of  negotiation case files and performed walk-throughs for each type of  research organization selected for our review. To select the  nongeneralizable sample, we took the total population of grants funded  for fiscal year 2014 and stratified the population by funding dollars (i.e.,  high, medium, and low) for each type of research organization and each  cognizant agency. Finally, we selected six case files from each of the  populations (see fig. 6). obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objective. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objective.
section_title: Appendix II: Comments from the Department of Health and Human Services
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Kim McGatlin (Assistant  Director), Rathi Bose, Francine DelVecchio, Wilfred Holloway, Alec  Hulbert, Diana Lee, Sophie Geyer, Jason Kelly, Kevin McAloon, and  Kailey Seibert made key contributions to this report.