prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 1.

Report 1: The RAD program was authorized by Congress and signed into law by the President in November 2011 under the Consolidated and Further Continuing Appropriations Act, 2012 with amendments in 2014, 2015, 2016, and 2017  The RAD program consists of two components  The first component of the RAD program—and the focus of our review— provides PHAs the opportunity to convert units subsidized under HUD’s public housing program and owned by the PHAs to properties with long- term (typically, 15–20 years) project-based voucher (PBV) or project- based rental assistance (PBRA) contracts  These are two forms of Section 8 rental assistance that tie the assistance to the unit to provide subsidized housing to low-income residents  In a RAD conversion, PHA- owned public housing properties can be owned by the PHA, transferred to new public or nonprofit owners, or transferred to private, for-profit owners when necessary to access LIHTC financing, if the PHA preserves its interest in the property in a HUD-approved manner  The second component of RAD converts privately owned properties with expiring subsidies under old rental assistance programs to PBV or PBRA in order to preserve affordability and encourage property rehabilitation  The goals of the RAD program include preserving the affordability of federally assisted rental properties and improving their physical and financial condition  Specifically, postconversion owners (PHAs, nonprofits, or for-profit entities) can leverage the subsidy payments under the newly converted contracts to raise capital through private debt and equity investments, or conventional private debt, to make improvements  The RAD program provides added flexibility for PHAs to access private and public funding sources to supplement public housing funding  These financing sources may include debt financing through public or private lenders; mortgage financing insured by FHA; PHA operating reserves; replacement housing factor funds; seller or take-back financing; deferred developer fees; equity investment generated by the availability of 4 percent and 9 percent LIHTC; or other private or philanthropic sources  PHAs also may pursue various options for their conversions, which often depend on property needs and available financing, including property rehabilitation or new construction  Additionally, PHAs may undertake conversion involving no property rehabilitation or new construction to meet certain financial goals or for future rehabilitation or new construction, as long as the PHA can demonstrate to HUD that the property does not need immediate rehabilitation and can be physically and financially maintained for the term of the Section 8 Housing Assistance Payment contract (HAP contract)  The RAD authorizing legislation and RAD Notice also specify requirements for ownership and control of converted properties  That is, converted properties must have public or nonprofit ownership or control, with limited exceptions  The RAD authorizing legislation, RAD Notice, HAP contracts, and RAD Use Agreement also establish procedures to help ensure that public housing remains a public asset should challenges arise, such as default, bankruptcy, or foreclosure  Oversight of RAD conversion and properties is primarily divided among three HUD offices  The Office of Recapitalization is responsible for administering the conversion process but generally does not oversee converted properties  Before conversion, the Office of Public and Indian Housing oversees the properties  After conversion, oversight remains with Public and Indian Housing for properties that convert to PBV contracts and transfers to the Office of Multifamily Housing Programs for PBRA  The RAD program has been implemented and expanded in phases  Since its authorization, the RAD unit cap gradually increased from 60,000 units in 2011 to 225,000 units in May 2017  The RAD program is currently fully subscribed with all 225,000 units allocated  As of September 30, 2017, 689 conversions were closed that involved a total of 74,709 units (see fig  1 for a breakdown by fiscal year)  Additionally, 706 conversions involving 79,078 units were in the process of structuring conversion plans  The remaining conversions under the cap were allocated to specific properties and in the process of having commitments issued or reserved under multi-phase or portfolio awards, according to HUD officials  RAD conversions begin with the submission of an application by PHAs after which they are notified of selection  The PHA is then required to submit a financing plan within 180 days or a later deadline based on the nature of the financing proposed  A RAD conversion is considered closed when the HAP contract is signed and financial documents are executed  The properties are considered converted to Section 8 assisted housing on the effective date of the HAP, which is generally the first day of the following month  Once the RAD conversion is closed, the PHA or ownership entity can move forward with its submitted proposals or RAD-related rehabilitation or new construction and is responsible for complying with RAD requirements and associated contracts  In some cases, rehabilitation can take place in advance of conversion closing if public housing funds are being used  Most RAD conversions involved some type of construction  Our analysis of HUD data showed that as of September 30, 2017 417 of 689 closed conversions (61 percent) involved planned rehabilitation to the property, 86 (12 percent) new construction, and 186 (27 percent) no construction; and 361 of 706 active RAD conversions (51 percent) involved planned rehabilitation, 89 (13 percent) new construction, and 256 (36 percent) no construction  HUD officials stated that they approve conversions that involve no immediate planned rehabilitation or new construction as long as the property has no immediate needs to be addressed  Such conversions allow PHAs to better position themselves to access additional capital to address future rehabilitation or construction plans  Our review of 31 conversion files also showed that the scope of proposed physical changes varied among RAD conversions  For properties that included scope of work narratives, physical changes included renovations to mitigate hazardous materials, aesthetic renovations, code and accessibility compliance, and construction of new buildings, among other changes  Financing for RAD conversions involved multiple public and private sources, but many conversions used LIHTC  Our analysis of HUD data showed that as of September 30, 2017, 173 of 689 closed RAD conversions (25 percent) utilized 4 percent LIHTC, 99 (14 percent) utilized 9-percent LIHTC, and 416 (60 percent) did not use LIHTC  By dollar amount, major financing sources were 4 percent LIHTC at $2 4 billion; new first mortgages at $1 8 billion; and 9 percent LIHTC at $1 1 billion  Construction costs constituted the highest-dollar use of financing for RAD conversions, but not all conversions incurred construction costs, as discussed earlier  On average, construction costs per closed conversion were $6 4 million (ranging from no construction costs to $236 million) and nearly $60,000 per-unit converted to RAD  Construction costs represented the highest-dollar use of financing for closed RAD conversions at $4 4 billion followed by building and land acquisition costs, and developer fees  For more information on financing sources and uses, see appendix II  PHA officials and developers we interviewed cited various factors that influence financing sources needed for RAD conversions  For example, property needs assessments help establish the level of rehabilitation or new construction that would address the capital needs of the property  In turn, needs assessments can derive from physical assessment results and incorporate federal, state, or local compliance requirements  For instance, rehabilitation or construction would need to address the accessibility requirements of the Americans with Disabilities Act and local building codes, among other requirements  PHA officials and developers we interviewed also said they had to consider competition or access to financing for RAD conversions  For example, PHAs noted that tax credit applications and other financing had to be competitive  Some PHAs we interviewed also noted that while the 9 percent LIHTC provides more equity to finance low-income units (finances 70 percent of the costs of the units), there is more competition for the 9 percent LIHTC, while the 4 percent LIHTC can be automatically awarded for certain deals involving tax exempt bonds and federally subsidized projects  Thus, while some PHAs and developers might prefer to obtain 9 percent LIHTC, they often apply for 4 percent LIHTC to increase the chances of obtaining some tax credit equity  For example, one particular PHA that had used both 4 percent and 9 percent LIHTCs noted that in one transaction it had to compete against 74 applicants for 25 available awards of 9 percent credits  The RAD authorizing statute requires HUD to assess and publish findings regarding the amount of private capital leveraged as a result RAD conversions  A leverage ratio relates the dollars other sources provide to the dollars a program provides to an institution or a project  HUD uses various quantitative, qualitative, and processing and efficiency metrics to measure conversion outcomes  To meet the RAD statutory requirement, HUD published an overall RAD leverage ratio that has fluctuated between 19:1 and 9:1 since 2014  HUD’s most recent leverage ratio in fiscal year 2017 was 19:1, nearly double what the agency reported the prior year  We asked HUD officials why the leverage ratio nearly doubled between 2016 and 2017 and received conflicting information during the course of our audit  Initially, officials noted that the ratio was intended to replicate the methodology used by PD&R in its September 2016 report  Subsequently, the officials clarified that they did not follow PD&R’s methodology for categorizing financial source data  Specifically, officials did not review or make manual adjustments to the financial data PHAs entered in open source fields to ensure sources actually represented public, private, or other funding categories when calculating the leverage ratio  Finally, they noted that they disagreed with the methodology used in the PD&R September 2016 report and stated that there are various ways to calculate leverage  For the purposes of announcing the most recent leverage ratio in 2017, HUD officials decided that a leverage ratio comparing federally appropriated public housing resources would reflect the amount of financing leveraged had RAD not existed  We found, and officials from HUD acknowledged, three limitations to the RAD leverage calculation  First, HUD generally had data on funding sources and amounts a RAD conversion proposed to use (at the time of its application to HUD and at the time of closing of construction financing) rather than data after construction is completed on funding sources and amounts  HUD officials stated that they were reviewing final closing packages to confirm that the data reflect the latest reported information on sources and uses of funds for each conversion at closing  However, sources and uses of funds and amounts at the time the RAD conversion is closed may differ from amounts upon completion of construction  In October 2017, HUD implemented procedures to verify completion of planned construction activities and costs, which we discuss later in this report  Second, HUD’s leverage ratio, published in 2017, did not manually adjust funding source data to accurately account for all sources in calculating the leverage ratio for RAD  Specifically, HUD did not isolate funding sources that were federally appropriated, contributed by the PHA, or contributed by state or local municipalities to calculate leverage  For example, among approximately $2 billion from other financial sources, HUD included Moving to Work (MTW) funding (which may include public housing capital funds, public housing operating funds, and voucher funds) and tax credit equity as leveraged sources  However, these are not necessarily private sources, which we explain later in this report  As a result, HUD’s current calculation does not reflect the amount of private- sector leveraging  HUD calculated and published a RAD leverage ratio in May of 2017 using the following formula: Total leverage ratio = (total dollars from all sources – public housing dollars) To calculate the RAD leverage ratio, HUD uses some but not all financial source data it collects (see app  II for a list of data fields collected by HUD)  For example, HUD mistakenly excluded data that capture private funds, reducing the amount of total sources in the numerator  HUD calculates “public housing dollars” by adding data that capture replacement factor funds, public housing operating reserve funds, and prior-year public housing capital funds  HUD considers tax credit equity, new first mortgages, and “other funding” data to be non-public housing dollars (see app  II for a list of fields in HUD’s calculation)  PHAs enter a description and amount for other funding sources in “other funding” data fields (see app  II)  For example, a PHA may enter a federal financial source in one of the open-entry “other funding” data fields, requiring a manual adjustment to properly account for the financial source  According to HUD, additional fields were included in mid-2016 to better differentiate certain sources such as from the HOME Investment Partnerships Program (HOME) and seller take-back financing  Prior to this point, these financial sources were placed into “other” fields, and the standard resource desk report had not been updated until mid-2017 to include all of these fields  Third, HUD does not categorize and report its leveraging by private and public sources  According to HUD officials, informative leverage methodologies could calculate the ratio based on the leveraging of public housing program funds, the leveraging of all federally appropriated funds, or the leveraging of PHA funds (i e , sources in the transaction that have come from the PHA itself even if not federally appropriated through the public housing program), among other methodologies  The RAD authorizing statute requires HUD to assess and publish findings on the amount of private-sector leveraging  In addition, Standards for Internal Control in the Federal Government require agencies to communicate quality information with external parties, such as other government entities, to make informed decisions and evaluate the entity’s performance in achieving key objectives  HUD also does not use final (postcompletion) funding data in another metric of RAD leveraging  Specifically, in June 2017 HUD publicly reported that RAD “has leveraged more than $4 billion in capital investment in order to make critical repairs and improvements ” HUD calculates this figure by summing the construction costs—a subcomponent of total costs—with data from the time a conversion closes and not upon completion of construction  HUD officials we spoke with clarified that this metric solely reports construction investments and does not reflect any conclusion regarding private leverage of public funds  But, HUD publically characterized this measure in different ways, including as the amount of “public-private investment in distressed public housing,” the amount of “construction achieved under RAD,” and the amount of “new private and public funds leveraged by RAD ” HUD’s 2016 interim report calculated and published multiple leverage ratios, but chose to highlight a RAD leverage ratio that is consistent with ratios used for other HUD programs  However, the ratio does not specifically follow the prescribed ratio language in the authorizing statute because the report states that the ratio represents the amount of private and public external sources invested for every dollar invested by PHAs but the statutory language only discusses private-sector leveraging  Officials further noted that the statute does not require a particular methodology and HUD relies on PD&R—and its independent contractor— to determine the appropriate methodology for purposes of compliance with the statute  Lastly, the statute does not preclude the use of other leverage metrics for other purposes, such as using the ratio to measure the amount of nonpublic housing funds leveraged in RAD transactions that would not be available to the property absent RAD  As a result, HUD’s leverage metrics announced in May 2017 do not accurately reflect the amount of private-sector leveraging achieved through RAD, do include public funding as private sources, and inconsistently measured sources that were federally appropriated or contributed by PHAs, potentially under- or over-reporting the program’s performance  Additionally, in October 2017, HUD began implementing procedures to collect data after construction is completed and is not yet able to calculate a leverage metric using final (postcompletion) financial sources rather than the financial sources collected at closing  The lack of a consistent metric for private leveraging could also lead to inconsistent reporting of the leverage ratio, as has occurred in prior years  We recalculated RAD leverage ratios in a number of different ways, including to correct errors we identified during our review  For example, HUD’s 2016 interim report noted that data on closed transactions do not provide detailed description of “other sources,” requiring a crosswalk between applications and closed transactions to develop estimates for the allocation of “other sources” across financial source categories  Abbreviated descriptions are provided in the form of notes that are not always clear and consistent; therefore public housing sources may include federally appropriated sources, as well state, city, or county sources  Through our estimates, we found that the overall leverage ratio could range from 7 44:1 for a ratio recalculating HUD’s leverage ratio to 1 23:1 for a ratio estimating private-sector leveraging  Recalculation with HUD methodology and financial source recategorization  As discussed previously, HUD’s methodology does not account for all financial data collected by HUD and includes “other” funding sources erroneously considered as leveraged funds  Thus, we manually adjusted RAD funding source data and found that nearly $1 2 billion were erroneously considered leveraged funds because they are not private funds  For example, HUD included MTW funds; public housing operating reserves; public housing capital funds; replacement housing factor funds; other federal funds; other state, local, or county funds; and take-back financing funds as leveraged financial sources  For more information, see appendix II  We obtained documentation from HUD to replicate their methodology and recategorized financial sources that corrected errors in the data, and found that the RAD leverage ratio was less than half of HUD’s most recently publicly reported leverage ratio (19:1), approximately 7 44:1 (see app  II)  Recalculation to exclude LIHTC and other federal sources  We previously reported that LIHTCs are considered a federal source because tax credit equity represents foregone federal tax revenue and, therefore, are a direct cost to the government  Accordingly, we recalculated the RAD leverage ratio by excluding all federal funding sources and obtained a ratio of approximately 1 43:1 (see app  II)  Recalculation of private-sector leveraging  Lastly, the RAD authorizing statute requires HUD to assess and publish findings on the amount of private-sector leveraging, but HUD’s current calculation does not present the amount of private-sector leveraging and does not include all available data (for example, the “Other Private” funds collected by HUD)  We estimated the amount of private-sector leveraging by grouping public housing sources, other public sources, and private sources, resulting in a leverage ratio of approximately 1 23:1 (see app  II)  In October 2017, HUD implemented procedures to certify completion after developers finish RAD-approved rehabilitation or construction  Previously, HUD had a limited ability to monitor and evaluate final (postcompletion) physical and financial changes in RAD projects with existing data  According to HUD officials, HUD did not implement completion certification procedures before October 2017 because it had been addressing what it considered to be the highest risks first (such as clarifying requirements for RAD participants, resident safeguards, and other procedural and administrative requirements)  HUD’s October 2017 completion certification procedures include instructions for owners to report final construction costs and documentation on completion of repairs or construction within 45 days of the completion date recorded in the RAD Conversion Commitment  More specifically, HUD requires owners to list a final construction cost amount—a subcomponent of total costs—in the RAD resource desk, describe variances from the approved construction cost amount in a comment box, and describe how increases in costs were addressed  Additionally, a third-party must certify that the repairs in the scope of work were completed by providing an attestation to HUD  However, HUD’s procedures do not require documentation from the owners to support the final total cost figures, which include not only construction costs but also building and land acquisition costs, and developer fees, among others as noted earlier in this report  These procedures also do not require a certification from owners on all financing sources and costs recorded in the RAD Conversion Commitment  Standards for Internal Control in the Federal Government require that management implement control activities through documented policies and procedures to provide reasonable assurance that the objectives of the agency will be achieved, and also communicate quality information with external parties to make informed decisions and evaluate the entity’s performance in achieving key objectives  While HUD now has certification completion procedures in place, this process provides the agency limited financial information from owners  As a result, HUD is unable to report metrics that reflect final (postcompletion) RAD financial outcomes after construction is completed  Furthermore, HUD is limited in its ability to effectively oversee conversion budget and cost variances, and expenditures that require HUD approval  Lastly, the RAD authorizing statute requires that the Secretary of HUD demonstrate the feasibility of the RAD conversion model to recapitalize and operate public housing properties under various situations and by leveraging other sources of funding to recapitalize properties  Without metrics that reflect the final (postcompletion) financial outcomes of RAD after construction is completed, HUD and congressional decisionmakers are unable to make informed decisions concerning the RAD program  HUD has not systematically tracked or analyzed household data on residents in RAD-converted units that are available from its public housing or Section 8 databases or from PHAs or other postconversion owners—the main sources of resident data for the RAD program  In addition, HUD has not yet developed monitoring procedures for all the resident safeguards in the RAD program  Finally, residents told us of some concerns about information they received on RAD conversions, communications opportunities, and the relocation process  HUD officials told us that the agency does not systematically track or analyze household-level data on residents in RAD-converted units across existing program databases (HUD maintains household data for the public housing and Section 8 rental assistance programs in two databases)  In particular, HUD does not track changes in household characteristics before and after conversion, such as changes in rent, as well as relocations or displacement of individual households  However, according to HUD officials, their databases are not designed to track the impact of RAD conversion on residents and they are unable to electronically link household information submitted before RAD conversion to information submitted after conversion  Once a property is converted, the property and corresponding household information are removed from the public housing database  Owners of converted properties are to use software to manually enter household information into the databases for the Section 8 program when submitting tenant certifications and information for assistance payments  This procedure is the standard for administration of all project-based Section 8 properties  HUD officials stated that they have explored the possibility of transferring household data from one system to another at the time of a property’s conversion  While HUD has not systematically analyzed household information from its public housing and Section 8 databases, we were able to perform a limited analysis  We requested and received data from HUD on the households affected by RAD  Using the data provided that were current as of June 2017, we were able to identify about 26,000 households that lived in units that were converted to a PBV subsidy, but we were unable to identify the total number of households converted to a PBRA subsidy  Based on our analysis of 26,000 PBV households, we found about 2,700 (about 11 percent of) households were headed by an about 6,800 (about 26 percent of) households were headed by an individual with a disability; about 2,700 households (about 10 percent of) households were headed by an elderly person who also had a disability; over half (about 14,000 or 54 percent) of the households were headed by an individual identified as black; close to 11,000 households (about 41 percent) were identified as white; and about 1,000 households (about 4 percent) were identified as Asian  Close to 3,100 households (about 12 percent) were headed by an individual identified as Hispanic; about half (about 49 percent) of the PBV households were single- person households; the median annual income of PBV households both before and after RAD conversion was about $10,000; and about 5,300 (about 20 percent) of households were paying a flat rent rather than income-based rent before RAD conversion  However, the data on PBV households were not comprehensive  For example, while about 10,000 residents (about 57 percent) experienced a rent increase following RAD conversion under PBV, we could not determine if the rent increase was the result of an increase in resident income  We also could not determine changes in location among the PBV households following RAD conversion  Rather than relying on the public housing and Section 8 databases for tracking household information during conversion, HUD officials indicated that the agency will rely on locally maintained resident logs, which contain household information collected by property owners, as the starting point when HUD determines a compliance review is warranted  The logs will be the primary way the agency collects household information for compliance reviews under the RAD program, according to HUD officials  In November 2016, HUD issued a notice, which requires the PHA or other postconversion owner to maintain a log about every household at a converting project, including information on race and ethnicity, household size, and disability  The notice also requires owners to track residence status throughout the relocation process, including whether the resident has returned, moved elsewhere, was permanently relocated or evicted; relocation dates; and details on any temporary housing and moving assistance provided  Owners are required to make the information available to HUD upon request for audits and other purposes  According to HUD officials, the agency expects the information in the resident logs to be more robust than what they would collect through the public housing and Section 8 databases, which do not track residents while they are relocated  HUD officials stated that the agency plans to review selected resident logs as part of an ongoing limited compliance review of about 90 RAD conversion projects  HUD officials told us they are developing procedures for performing compliance reviews—such as developing a mechanism to review a sample of logs on a periodic basis—but they have not yet done so because they have been focusing on developing procedures for activities that present a high risk to the program as described in the following section  HUD has not established a time frame for developing these procedures  However, HUD officials indicated that they plan to select resident logs for review based on risk of noncompliance and do not plan to analyze program-wide information currently collected in the public housing and Section 8 databases for program monitoring  HUD officials also noted that that PD&R is planning to track a sample of residents through its evaluation of the program, which we previously mentioned  While HUD has decided to rely on resident logs because of the difficulty of tracking household information across its program databases, using resident logs to assess the effects of the RAD program on residents has limitations  While the resident logs would contain detailed household information, they were not required prior to November 2016 and may not contain information on households converted before that date (RAD conversions started in 2013)  HUD’s public housing and Section 8 databases contain information on such households  Second, as previously mentioned, HUD plans to review resident logs only when there is a risk of noncompliance, but they collect household information in their databases on a rolling basis  Standards for Internal Control in the Federal Government require agencies to use quality information to achieve their objectives, and obtain and evaluate relevant and reliable data in a timely manner for use in effective monitoring  Without a comprehensive review of household information—one based on information in HUD data systems as well as resident logs—HUD cannot reasonably assess the effects of ongoing and completed RAD conversions on residents and compliance with resident safeguards, as discussed in the next section  HUD has not yet developed monitoring procedures for certain resident safeguards under the RAD program  RAD requirements include those intended to ensure that residents whose units are converted through RAD are informed about the conversion process; can continue to live in a converted property following RAD conversion; are afforded certain protections carried over from the public housing are afforded a phase-in of any rent increases under Section 8 program requirements  Currently, based on HUD notice requirements, PHAs must document compliance with three safeguards (PHA plan amendments, resident notification, and procedural rights) in their RAD application and other conversion paperwork  For example, PHAs must submit comprehensive written responses to resident comments received in connection with the required resident meetings with their RAD application  For one safeguard, PHAs are not required to report to HUD but must retain documentation of compliance to be made available to HUD as part of the monitoring for the program  For others, the HUD notice does not specify reporting and monitoring requirements  Based on our review of files for selected conversions, which we previously discussed, we found PHAs generally submitted documentation of their efforts to inform residents about RAD conversion, such as providing evidence to HUD of meetings with residents and written responses to resident questions as required  However, the specific documents for these requirements were not available from HUD in all cases  HUD’s review of amendments to PHA plans was documented in all but one of the conversions we reviewed  Documentation requirements for resident relocations have changed since RAD was introduced, which made the documentation more difficult to assess  HUD developed and started implementing procedures in October 2017 that require owners to certify and provide data supporting compliance with the resident right-to-return requirements  For example, owners must certify the number of residents who exercised their right to return to a converted property compared with the number of residents who did not return  HUD is also developing standard operating procedures to review each conversion for compliance with RAD relocation provisions  Specifically, the procedures would describe the review steps required at different stages of the conversion process, a process for identifying risks, and how to address instances of noncompliance with RAD requirements  Additionally, HUD noted that they have 2 compliance reviews under way including 1 involving a set of HUD requirements that affect relocations of more than 1 year and the limited compliance review of 90 projects that we previously described  HUD officials noted that they are developing additional guidance in other areas  First, HUD officials indicated that as part of an overall update of RAD standard operating procedures, they are developing additional protocols on resident notification and how residents’ comments are addressed through conversion planning  Second, the agency had not been consistently collecting required documentation on “house rules,” which describe the conditions and procedures for evicting residents and terminating assistance at RAD PBRA properties, so it has developed and implemented additional legal review procedures as part of the implementation of RAD resident eviction and grievance procedural rights requirements  According to HUD officials, they have been focusing primarily on right-to-return and relocation requirements because they represent areas of highest risk  HUD has not developed separate monitoring procedures for other resident safeguards—the phase-in of tenant rent increases, resident representation through tenant organizations, and choice mobility requirements  However, HUD officials told us that they plan to assess how administrative data can be used to monitor choice mobility as part of the planning for a separate PD&R evaluation of this safeguard  HUD officials also indicated that there are procedures for residents to report complaints to HUD if resident representation and organization requirements are not met  Standards for Internal Control in the Federal Government require agencies to implement control activities through documented policies and procedures to provide reasonable assurance that agency objectives will be achieved  These standards also require agencies to design procedures to achieve goals and objectives, and identify, analyze, and respond to risks related to achieving the defined objectives  Table 1 includes a description and information on implementation of resident safeguards that most directly affect residents’ experience with the conversion process and ability to live at the property following conversion  Appendix III describes these and other RAD resident safeguards  HUD officials indicated that the safeguards for the phase-in of tenant rent increases, resident representation, procedural rights, and choice mobility presented a lower risk than the right-to-return requirements, so they were a lower priority, and in some cases were addressed through general monitoring of the Section 8 program  For choice mobility options, HUD indicated that its data systems are not designed to track whether residents are able to exercise these options, such as tracking whether residents left a property to exercise choice mobility or for other reasons  All but two of the resident safeguards do not take effect until after a property has been converted and is part of the Section 8 program  For example, residents are only eligible to use vouchers through choice mobility after they have lived in the converted property for 1 or 2 years depending on the assistance contract involved (PBV or PBRA)  Moreover, certain RAD safeguards are not typically available for Section 8 residents  For example, RAD establishes resident representation provisions and procedural rights that are more in line with public housing rather than Section 8 requirements  While HUD has indicated that the Section 8 program has experience administering different types of assistance contracts, RAD nonetheless creates separate requirements for certain provisions from the public housing and Section 8 programs  As previously mentioned, RAD conversions have been completed at an increasing pace in the last 5 years  However, because HUD has not yet developed separate monitoring procedures for certain requirements—the phase-in of tenant rent increases, resident representation through tenant organizations, and choice mobility requirements, many of which take effect after a conversion—and without using all available household data, the agency will not be able to reasonably ensure that these safeguards were implemented  Residents who participated in our focus groups expressed some concerns about information they received on RAD conversions, communications opportunities, and the relocation process  Residents indicated that they were notified about RAD conversion in a variety of ways  Residents in 5 of 14 focus groups found the information presented to them on RAD to be helpful  Residents in 7 of 14 focus groups indicated that the information they received was not helpful  Across these focus groups, a range of concerns was expressed, including that the information provided was not always clear or reflective of the final changes resulting from RAD conversion, and that the PHA and management were not always forthcoming with information about the RAD changes  Residents in some focus groups also indicated that they were not involved in the RAD conversion  Residents in 5 of 14 groups indicated that they were not given the opportunity to provide input into the RAD changes, while residents in 6 of 14 groups indicated that their concerns were not addressed and their suggestions were not incorporated  Residents also described problems with relocations  Some of the concerns expressed by resident focus groups on relocation related to the location of the temporary units (3 of 14 focus groups), the timing of relocation or amount of notice given (7 of 14 focus groups), and moving issues (such as items damaged during moves)  Residents were asked to describe ways in which RAD conversion improved or harmed their living conditions  Residents in several focus groups indicated that RAD improved their living conditions, including both the condition (7 of 14 focus groups) and appearance of their units or the property in which they lived (6 of 14 focus groups)  Some of the changes residents liked included the installation of new appliances, mold and pest removal, and safety and energy efficiency improvements  However, residents in several of the focus groups identified problems with their living conditions following RAD conversion  The problems residents identified included security concerns (10 of 14 focus groups); renovations that were of poor quality (6 of 14 focus groups); and other problems with the units (10 of 14 focus groups), such as pest problems; decreased amenities (8 of 14 focus groups), such as the removal of common areas or in-unit washing machines; and issues with property management (11 of 14 focus groups)  For example, in several instances, residents stated that new managers or owners in place following RAD conversion were not responsive to their needs or concerns  During our site visits, residents described other experiences with RAD conversion  Residents in all of the groups identified being notified about RAD  Residents in 9 of 14 focus groups indicated that their rent was the same following RAD conversion  Residents in a few focus groups indicated that their rent had increased because of changes in their income or conversion from a flat rent  However, residents in a few focus groups experienced challenges in how their income was certified for the purpose of calculating rents, such as problems with requests for information (2 of 14 focus groups) and other issues with the process (4 of 14 focus groups)  For example, residents reported having to provide the same paperwork multiple times  No instances in which residents were permanently involuntarily displaced were reported  One resident organization expressed concerns about fewer eviction protections and resident representation after RAD conversion  We spoke with 18 PHAs, some of which cited benefits as well as several challenges of RAD participation and some noted HUD responsiveness to their circumstances and concerns  According to many of the PHAs we spoke with, benefits of participating in the RAD program included reducing administrative requirements in Section 8 programs and opening avenues for additional sources of funding  In particular, many of the PHAs noted that RAD allowed them to access tax credit equity and other funding to complete the bulk of their repairs and renovations at once  Over half of the PHAs we spoke with also found HUD to be flexible and responsive to individual PHA circumstances  The majority of PHAs we spoke with indicated that remaining in the public housing program was not tenable because funding for the public housing program was not enough to meet their long-term capital needs  PHAs we contacted also noted several challenges of participating in RAD: financing constraints, timing challenges, and evolving requirements  Financing constraints  Some PHAs noted that program rent requirements can limit PHA participation in RAD  Each year, HUD calculates a contract rent—the total rent for a unit, including operating subsidy and resident contribution  PHAs must use the contract rent to calculate Section 8 subsidies for properties converting under RAD  According to HUD and several PHAs, contract rents for RAD-converted Section 8 units are lower than rents in traditional Section 8 assisted units, and are almost always lower than market-rate rents  Several PHAs and HUD officials have described the difficulty of converting units from the public housing program with this rent limitation  For example, when the cost of needed rehabilitation or construction is high, low allowable contract rents might not be sufficient to access appropriate capital for the conversion  In certain localities, PHAs have found solutions to augment rents and have used RAD flexibilities to allow them to convert and plan for operating expenses  For example, the PHA in Tacoma, Washington, used the Moving to Work program flexibilities to increase contract rents and housing officials in San Francisco used an allowable procedure to transfer RAD assistance from converted buildings to properties throughout its portfolio (each is a blend of traditional project-based vouchers with higher contract rents and RAD assistance)  In Montgomery County, Maryland, the PHA similarly included RAD assistance in some mixed-finance properties that contain other high-rent subsidies and market-rate rents  Timing challenges  Some PHAs said they faced major challenges in coordinating RAD timelines with HUD, lenders, or other parties or with the requirements of the LIHTC process  HUD officials acknowledged that PHAs with more complex transactions, including those involved in the LIHTC process, struggle to implement their conversion plans within RAD time frames  HUD officials noted that because there is a statutory cap on the number of units that can be converted under RAD, they have established time frames to stay under the cap and ensure that PHAs that are planning to convert are ready to participate in the program  Additionally, according to HUD, it has made technical assistance available to all PHAs that receive a Commitment to enter into a Housing Assistance Payment contract during the RAD process to help ensure their readiness for RAD closing and to meet remaining conversion deadlines  On the other hand, some PHAs expressed concern to us about delays in the conversion process that put them at risk for missing state LIHTC deadlines  HUD officials described putting conversions on a fast-track on a case-by-case basis to meet LIHTC deadlines  For example, in one case a PHA relocated residents before closing and without HUD approval  HUD required the PHA to fund an escrow account until it was able to determine any payments that might need to be made to residents and any other necessary corrective action  This was done so that HUD could look into the issue while mitigating additional harm to the residents and continuing to move the PHA toward closing and aligned with tax credit application deadlines  The timing of conversion can also create gaps in the payment of Section 8 funds to PHAs  Section 8 funding should begin in January of the year following conversion  PHAs rely on annual public housing subsidies for the conversion year—public housing program funds are paid to PHAs annually and are not recaptured by HUD following RAD conversion  However, according to some PHAs we interviewed, Section 8 funding did not begin on time  For example, in Baltimore, Maryland, subsidy flow after conversion had not begun as of June of the following year  HUD officials told us inadequate guidance from HUD and confusion from PHAs regarding the necessary steps to request payment in a timely manner have been the major cause of the problems  HUD has tried to remedy delays and updated its notice to provide clearer guidance on the timing of subsidy flow around the time of conversion to Section 8  Moreover, HUD officials indicated that there has been confusion among PHAs on how to request funds, so HUD is currently revising and updating the guidance on steps PHAs must take to request payment under the PBRA program  HUD officials also indicated that it has begun monitoring whether new participants are taking the steps needed well before their first request for funding  Some PHAs we contacted also mentioned difficulty in coordinating with HUD on fulfilling internal RAD requirements and reviews  According to some, the different offices involved in RAD conversions within HUD were not well aligned and had different interpretations of the rules  For example, some RAD conversions require a civil rights review by HUD’s Office of Fair Housing and Equal Opportunity Office, including those transactions that require new construction or resident relocations  Some PHAs indicated that such reviews occurred too late in the conversion process even after other HUD offices had approved the conversion  HUD officials acknowledge that different HUD offices have different objectives in the RAD process  HUD officials indicated that the agency is trying to coordinate more effectively among these offices and streamline the conversion process as much as possible  Evolving requirements  While the majority of PHAs with which we spoke said that HUD provided clear, sufficient, and timely information, some PHAs noted that it also was challenging to adapt to evolving requirements  Some PHAs noted that as HUD identified problems in the early years of the program, it would change the guidance in response  For example, HUD officials explained that it had clarified fair housing review requirements in response to PHA concerns that the fair housing review occurred too late in the process and could affect successful conversion of projects  The most recent RAD notice (effective January 2017) is the third version since 2013 and revisions have involved substantial changes  For example, this notice provided PHAs with greater flexibilities on the funding sources they can use to raise initial contract rents and the ways they can demonstrate ownership and control of a converted property  In addition, HUD introduced a notice in November 2016 to strengthen resident protections  Some PHAs told us they found the pace or timing of the evolving requirements difficult to manage and also noted confusion about conversion instructions and guidance due to changing requirements  For example, one PHA indicated that the agency had problems reporting information into a new RAD data field in HUD’s Voucher Management System because there was no guidance at the time on how to complete this field  However, HUD has since included additional instructions in the user’s manual that became effective in April 2017  The Committee has included language to establish procedures that will ensure that public housing remains a public asset in the that event that the project experiences problems, such as default or foreclosure  In each RAD conversion, HUD and the property owner execute a use agreement, which specifies affordability and use restrictions for the property  The use agreement generally exists concurrently with the HAP contract, which is executed to govern the provision of either the PBRA or PBV subsidy for the unit  The use agreement must be recorded in a superior position to new or existing financing or other encumbrances on the converted property  Under a Section 8 HAP contract, residents pay 30 percent of adjusted household income  In the absence of the HAP contract, the use agreement is set up to control the amount paid: If the HAP contract is removed due to breach, noncompliance, or insufficiency of appropriations, under the use agreement new households in all units previously covered under the HAP contract must have incomes at or below 80 percent of the area median income for households of the size occupying an appropriately sized unit for their family size at the time of admission, and rents may not exceed 30 percent of 80 percent of area median income for the remainder of the term of the use agreement  For new residents at or below 80 percent of the area median income, under the use agreement the resident rent contribution without a HAP contract generally would be higher than that paid under a HAP contract, which is based on household income instead of the universally determined area median income  Although the use agreement maintains some level of affordability, the owner receives no subsidy under PBRA or PBV without a HAP contract and resident rent contribution is not tied to individual household income but rather based on a universal area income calculation (see fig  3)  According to HUD officials, other program requirements support the goal of long-term preservation: HAP contracts are executed for 20 years for PBRA or 15–20 years for PBV properties and compliance with all affordability requirements in the HAP and statute and regulation governing the PBRA and PBV programs must be maintained while the contract is in force  According to the authorizing statute, PHAs (for PBV contracts) and HUD (for PBRA contracts) shall offer and project owners shall accept a renewal contract at the expiration of the initial HAP contract and at each subsequent renewal  Each renewal contract will be subject to a RAD use agreement, governing the use of the property consistent with HUD requirements  According to the RAD notice, the project owner also is to establish and maintain a replacement reserve to aid in funding extraordinary maintenance and repair and replacement of capital items  The reserve account must be built up to and maintained at a level determined by HUD to be sufficient to meet projected requirements  According to HUD officials, during the conversion, HUD staff review each capital needs assessment to try to determine whether a property’s capital needs can be addressed over the forthcoming 20-year period  We reviewed 31 completed conversion files, the set of documentation required by HUD to enable a PHA to convert units from public housing to a Section 8 subsidy, and associated RAD contracts  In each file, key contractual protections appeared consistent with program requirements  Specifically, in all cases executed use agreements (which included requirements to limit residency eligibility to households making less than 80 percent of area median income) were included and not altered from the HUD template  In most files we reviewed, we found foreclosure riders were included and that they stated that use agreements would survive foreclosure, meaning that any new owners would take ownership subject to the agreements  Executed HAP contracts, requiring that residents’ contributions be set at 30 percent of adjusted household income, also were present in all files we reviewed  According to HUD officials, PHAs, and two housing groups we spoke with, provisions in the RAD use agreement to keep units affordable appear to be strong, with use and affordability protections designed to survive foreclosure, but the strength of provisions cannot yet be fully determined because the provisions have not yet been tested in foreclosure proceedings or in courts  According to HUD officials, as of October 2017 no RAD properties had entered foreclosure  The RAD authorizing statute requires that ownership be transferred to a capable public entity or, if not one, a capable entity as determined by HUD, or if necessary to fulfill LIHTC requirements for the property, to a HUD-approved for-profit entity (provided the PHA retained sufficient interest in the property)  HUD also subjects any subsequent transfer of the property to HUD review and requires the successor ownership to meet these same requirements  As stated in the use agreement, a lien holder must give HUD notice prior to declaring a default and provide HUD concurrent notice with any written filing of foreclosure (providing that the foreclosure sale must not be sooner than 60 days after the notice), but the use agreement does not prohibit a lien holder from foreclosing on the lien or accepting a deed in lieu of foreclosure  The RAD use agreement, which is recorded superior to other liens and places use and affordability restrictions on the property, survives foreclosure  With or without a HAP contract in place, the lender or new owner must maintain the units for low- income households according to the terms of the use agreement  Therefore, according to HUD officials, the lender or new owner has an incentive to identify an appropriate owner and secure HUD approval to avoid a default under the HAP contract, which provides a Section 8 subsidy to the owner  That is, if no HAP contract were in place, the owner would collect only the tenant rent contribution (30 percent of 80 percent of area median income), rather than the tenant rent contribution plus the subsidy  HUD has discretion to enforce or waive certain use and affordability protections  According to the authorizing statute, in the case of foreclosure, bankruptcy, or termination and transfer of assistance for material violation or substantial default, the priority for ownership or control must be provided to a capable public entity, or, if no such entity can be found, to a capable entity as determined by the Secretary of HUD  Additionally, the statute allows the transfer of property to for-profit entities to facilitate the use of LIHTC financing, with requirements to maintain the PHA’s interest, which was discussed above  As of September 30, 2017, about 40 percent of RAD conversions involved LIHTC financing  According to the RAD notice, in the event of a default of a property’s use agreement or HAP contract, HUD may terminate the HAP contract and transfer assistance to another location to retain affordable units  HUD will determine the appropriate location and owner entity for the transferred assistance consistent with statutory goals and requirements for RAD  The RAD use agreement will remain in effect even in the case of abatement or termination of the HAP contract for the term the contract would have run, unless HUD agreed differently in writing  In this case, the RAD notice limits HUD discretion to terminate the use agreement to only cases involving a transfer of assistance to another property  HUD has not yet developed procedures to monitor RAD projects for risks to long-term affordability of units, including default or foreclosure  HUD officials described an ongoing effort to develop oversight procedures it would need to reasonably ensure compliance with RAD agreements and avoid risks to long-term affordability once conversions closed and units moved to Section 8 but, as previously discussed, the agency has not yet completed this effort or fully implemented a monitoring system  HUD officials told us they also planned to develop protocols to more closely monitor properties at risk of foreclosure, including developing indicators, procedures, roles, and responsibilities within HUD, but they have not finalized the design of procedures or fully implemented them  To develop protocols, HUD created an asset management working group in September 2016  The officials also stressed that no one can take possession of or foreclose on a property without HUD involvement and approval  For example, HUD officials said they expect few foreclosures among RAD-converted properties because lenders tend to communicate with the agency early so that it can become involved to prevent foreclosure  HUD officials pointed to a robust structure to oversee program properties in the PBRA program, but stated PBV property oversight continues to be developed by the Office of Public and Indian Housing  According to Standards for Internal Control in the Federal Government, agencies should design procedures to achieve goals and objectives, such as the preservation of unit affordability, and respond to risks, in this case the risk of default or foreclosure or noncompliance with program requirements  Additionally, management should identify, analyze, and respond to risks related to achieving its goals and objectives  According to HUD officials, the agency had not yet fully developed and implemented oversight procedures for postconversion monitoring because since 2012, the agency has focused on RAD start-up and review and oversight procedures for the conversion process  HUD officials also said that many projects would receive ongoing monitoring from other parties, which also could serve as a safeguard for unit affordability and help ensure the appropriate financial and physical condition of the property after RAD conversion  For example, just under half of all RAD properties use LIHTC financing as part of financing packages, which can also include local and state bonds  According to HUD officials, oversight by tax credit allocating agencies, investors, and lenders, while not alone sufficient, helps secure affordable units in a property for the long-term  However, tax credit allocating agencies, investors, and lenders are not signatories to the HAP contract or use agreement and have no formal role in reasonably ensuring that properties meet requirements exclusive to RAD  Although other entities may exercise some oversight of properties, by not developing and implementing procedures for ongoing oversight, HUD in its role as program administrator will not be able to reasonably ensure that properties adhere to requirements or meet basic program goals  Furthermore, without such monitoring HUD would be limited in its ability to identify and assist with properties at risk of foreclosure  RAD was created to demonstrate the feasibility of converting public housing units to other rental assistance programs to help preserve affordable rental units and address the significant backlog of capital needs in the public housing program  However, demonstrating the feasibility of RAD conversion is contingent on collecting and assessing quality information about the conversion projects  HUD has an opportunity to improve the demonstration’s metrics  For instance, implementing robust postclosing oversight and collecting information on financial outcomes upon completion of construction would not only improve HUD’s oversight capabilities but also allow it to report quality information  Moreover, limitations in HUD’s methodology for calculating leverage ratios for RAD may obscure the effect of funding sources used to help fund RAD conversions, potentially under- or over-reporting the program’s capital leveraging  By collecting comprehensive information on final (postcompletion) financing sources and costs and developing quality metrics, HUD would be better positioned to more accurately report the results of the demonstration program  Additionally, a focus on the conversion process itself (and less on its results), and limitations in HUD’s data have contributed to limited monitoring by HUD in other areas  Specifically, by not developing and implementing monitoring procedures to assess the effect of RAD on residents HUD cannot ensure compliance with resident safeguards  Further, HUD collects and maintains household data for the public housing and Section 8 programs, yet it does not systematically use this information to ensure that resident safeguards are in place  Finally, HUD could benefit from additional procedures to assess RAD properties for risks to long-term preservation to be able to respond to property default or foreclosure  We are making the following five recommendations to HUD: HUD’s Assistant Secretary for Housing should include provisions in its postclosing monitoring procedures to collect comprehensive high quality data on financial outcomes upon completion of construction, which could include requiring third-party certification of and collecting supporting documentation for all financing sources and costs  (Recommendation 1) HUD’s Assistant Secretary for Housing should improve the accuracy of RAD leverage metrics—such as better selecting inputs to the leverage ratio calculation and clearly identifying what the leverage ratio measures—and calculate a private-sector leverage ratio  (Recommendation 2) HUD’s Assistant Secretary for Housing should prioritize the development and implementation of monitoring procedures to ensure that resident safeguards are implemented  (Recommendation 3) HUD’s Assistant Secretary for Housing should determine how it can use available program-wide data from public housing and Section 8 databases, in addition to resident logs, for analysis of the use and enforcement of RAD resident protections  (Recommendation 4) HUD’s Assistant Secretary for Housing should prioritize the development and implementation of procedures to assess risks to the preservation of unit affordability  (Recommendation 5) We provided a draft of this report to HUD for comment  HUD provided written comments on the draft report, which are summarized below and reproduced in appendix IV  HUD also provided technical comments, which we incorporated as appropriate  In its comment letter, HUD stated that it agreed with our findings that HUD can improve metrics used to assess program impact and build on existing oversight structures  HUD described actions it intends to take to implement our recommendations to the extent possible and consistent with resource limitations  More specifically, HUD agreed with our first recommendation to ensure it collects comprehensive quality data on financial outcomes in its postclosing monitoring procedures (which could include supporting documentation for all financing sources and costs)  HUD agreed it should routinely collect an updated list of funding sources and uses and related documentation when projects had cost overruns or other significant changes  HUD intends to review and revise, as appropriate, required postcompletion certifications  HUD added that in most cases, funding sources and uses do not materially change between closing and construction completion  HUD stated that securing the postclosing information in such cases might be of minimal benefit relative to the additional reporting burden  However, it is not clear how HUD would determine if projects had significant changes in costs or uses because HUD lacks postcompletion information that would show the magnitude of changes  In relation to reporting burden, HUD already has implemented procedures to collect limited financial information following the completion of construction in October 2017  We believe any additional reporting would not be disproportionate to the benefits of improving HUD's oversight capabilities through project completion and enhancing its reporting to more accurately reflect the results of the demonstration program  For our second recommendation to improve the accuracy of RAD leverage metrics and calculate a private-sector leverage ratio, HUD agreed that RAD leverage metrics can be improved  HUD will ensure that the private-sector leverage ratio required by statute is clearly identified and included in its RAD evaluation  HUD also intends to identify a small number of relevant leverage ratios with distinct methodologies and will routinely publish these ratios with clear identification and explanations  In relation to our finding of misidentified funding sources, HUD plans to re- examine its chart of accounts and review prior transaction records to address errors and properly classify transaction sources  In response to our third recommendation to prioritize the development and implementation of monitoring procedures for resident safeguards, HUD agreed that it is important to better document and expedite development and implementation of monitoring procedures  HUD also agreed that additional monitoring was needed to ensure the right of residents to request and move with a tenant-based voucher after a period of residency (choice-mobility)  HUD noted that its Office of Policy Development and Research is seeking funding for additional research on RAD with a focus on the use and effect of choice-mobility options, which would inform HUD's monitoring efforts  Finally, while HUD said that we did not find the safeguards to be weak or inadequate, we did not perform an audit designed to assess the safeguards and therefore cannot opine on their adequacy  On the basis of our findings, we found that HUD’s implementation of these safeguards could be strengthened  Regarding our fourth recommendation that HUD determine how it can use available program-wide data and resident logs for analysis of RAD resident protections, HUD agreed to examine how it could use its existing data systems to further enhance its monitoring efforts  HUD added that the systems have limitations, so that the agency also uses other mechanisms to track and monitor implementation of resident protections  For our fifth recommendation to prioritize the development and implementation of procedures to assess risks to the preservation of unit affordability, HUD agreed that it is important to assess and mitigate risks to unit affordability  HUD stated that it employs robust underwriting standards prior to permitting conversion, and relies on existing procedures to conduct ongoing oversight of Project-Based Rental Assistance (PBRA) properties, which we discussed in the draft  However, as we noted, HUD has not yet developed procedures to more closely monitor RAD properties at risk of foreclosure, though it plans to establish indicators of foreclosure risk and oversight roles and responsibilities within HUD  HUD said that since the summer of 2017, it has been evaluating what additional oversight procedures might be needed for RAD Project-Based Voucher properties  HUD also described plans to augment its existing oversight procedures to preserve affordable units in the event of foreclosure by developing protocols in the following areas: transfer of property ownership to a capable entity, transfer of the rental assistance to another site, and protection of residents in the event a Housing Assistance Payment contract was terminated  As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date  At that time, we will send copies to the Secretary of Housing and Urban Development and other interested parties  In addition, the report will be available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact me at (202) 512-8678 or garciadiazd@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs are listed on the last page of this report  GAO staff who made major contributions to this report are listed in appendix IV  This report examines aspects of the Department of Housing and Urban Development’s (HUD) Rental Assistance Demonstration (RAD) program  More specifically, this report addresses (1) HUD’s assessment of the physical and financial outcomes of RAD conversion to date; (2) how RAD conversions affected residents and what safeguards were in place to protect them, including while temporarily relocated and during conversion; (3) what challenges, if any, public housing agencies (PHA) faced in implementing RAD; and (4) the extent to which RAD provisions are designed to help preserve the long-term affordability of units  To address all four objectives, we analyzed agency documentation and interviewed officials from HUD  The documentation we reviewed included policies and procedures for RAD; manuals describing HUD data systems; draft policies and procedures for implementing postclosing oversight; and reports on RAD performance  We interviewed HUD headquarters officials from the Office of Recapitalization within the Office of Housing, which oversees the administration of RAD, and the Office of Policy Development and Research (PD&R)  We also interviewed PHA officials and developers involved in RAD transactions, as well as selected experts and other stakeholders to obtain their perspectives on RAD  Additionally, we conducted a literature search to identify publications related to RAD  We visited a nonprobability sample of eight PHAs in Maricopa County, Arizona; Alameda County, California; Montgomery County, Maryland; and in the cities of San Francisco, California; Baltimore, Maryland; New Bern, North Carolina; El Paso, Texas; and Tacoma, Washington, to observe housing units before, during, or after renovation when possible as well as common areas that had planned or undergone renovation  We selected sites to include varying PHA sizes, RAD subsidy types, planned rehabilitation and resident relocation, numbers and sizes of RAD transactions, closing dates, constructions costs, and geographic locations across the United States  At each site, we conducted semistructured interviews with PHA officials and, when available, developers (5 sites)  We also conducted one or two focus-group interviews with groups of 6– 15 residents who lived at the converted properties to obtain their perspectives and experiences  In each location, we asked the PHAs to invite residents to participate in the focus groups based on their availability  We also met with the Resident Advisory Board in each location that had one  For 7 of 8 site visits, we selected two RAD properties to conduct resident focus groups (in Alameda County, California we held one focus group)  We conducted a content analysis based on resident focus group interviews to describe resident experiences and the RAD program’s effects on residents  Utilizing the selection criteria noted above, we conducted semistructured telephonic interviews with an additional nonprobability sample of 10 PHAs in Fresno, California; Fort Collins, Colorado; Dekalb County, Georgia; Chicago, Illinois; Ypsilanti, Michigan; Cuyahoga County, Ohio; Philadelphia; Pennsylvania; Spartanburg, South Carolina; McKinney, Texas; and Yakima, Washington  Because we selected a non-probability sample of PHAs to visit and interview, the information we obtained cannot be generalized more broadly to all PHAs  However, it provides context on RAD particularly on implementation challenges and perspectives on physical and financial impacts, long-term affordability, and resident protections  We also selected the following 11 individuals and organizations as experts and stakeholders: 1  Council of Large Public Housing Authorities 2  National Association of Housing and Redevelopment Officials 3  Center on Budget and Policy Priorities 4  Public Housing Authorities Directors Association 5  National Housing Law Project 6  Community Legal Services of Philadelphia 7  Maryland Legal Aid 8  Disability Rights Maryland 9  Jaime Alison Lee, Associate Professor of Law and Director, Community Development Clinic, University of Baltimore School of Law 10  Yumiko Aratani, Assistant Professor, Columbia University Mailman 11  University of California, Berkeley, Terner Center for Housing We interviewed experts and stakeholders on resident impacts and implementation challenges associated with RAD  The entities may not represent all views on these topics, but their views provide insights on RAD  To select these individuals and groups, we met with three major PHA associations and two resident advocacy groups, and asked for referrals for organizations or individuals with expertise in RAD  We also selected a nonprobability, random sample of 31 RAD conversion files to review  Utilizing HUD RAD Resource Desk data, we randomly selected 31 RAD files for properties that had closed conversion as of June 30, 2017 and that planned to incur construction costs  We used the files to help us determine physical changes to RAD conversions and the impacts of RAD on residents through, for example, relocation  We excluded RAD conversions with no construction costs from the random sample because they would not have physical changes and no resident relocation would occur before or during our review  To address our first objective on the physical and financial outcomes of RAD conversion to date and how HUD measured these outcomes, we first obtained and analyzed HUD data on RAD conversions since RAD’s authorization (from fiscal years 2013 through 2017)  We assessed the reliability of these data by reviewing system documentation, interviewing knowledgeable officials about system controls, and conducting electronic testing  We determined that the data were sufficiently reliable for the purposes of describing rehabilitation and new construction in RAD projects and evaluating RAD leveraging metrics  We included in our analysis all RAD conversions that were active or closed  We used these data to determine the number of closed RAD conversions, associated financial sources and uses, subsidy types, and type of construction (rehabilitation, new construction, and no rehabilitation or new construction)  In addition, during our interviews with PHAs and developers, we obtained their perspectives on potential contributing factors to financial decisions and type of construction pursued through RAD conversion  As noted earlier, we also reviewed 31 randomly selected files of converted properties with construction costs to describe property physical changes in RAD conversions  Furthermore, we reviewed HUD documents, such as HUD and PD&R evaluations, publications, and policies and procedures to gain additional context for how HUD measures RAD outcomes  We also interviewed HUD officials, including PD&R and Office of Recapitalization officials, on RAD data and metrics, as well as other performance monitoring activities  We further analyzed data from the HUD RAD Resource Desk to determine how these data support HUD’s metrics and performance monitoring activities  As previously mentioned, we determined that these HUD data were sufficiently reliable for the purposes of this report  Specifically, we assessed and calculated RAD leverage ratio and construction activity  We assessed HUD’s performance monitoring activities and reporting against the RAD authorizing statute, Standards for Internal Control in the Federal Government  To recalculate estimates of the RAD leverage metric, we obtained documentation from the Office of Recapitalization to review the methodology used to calculate their most recent leverage ratio  We aligned the methodology they provided with RAD Resource Desk Transaction Log data that was downloaded on August 7, 2017  We replicated HUD’s methodology and matched the data utilized with the descriptors from the Transaction Log  To isolate financial sources and manually adjust the “other source” data, we compiled matched descriptors and funding amounts and categorized each observation based on the funding source description, as a federal source, state/county/city source, or PHA source, among others  For additional information and results, see appendix II  To determine how RAD affected residents in converted units, we analyzed HUD public housing and Section 8 household data before and after conversion (demographic characteristics of residents and changes in rent, income, and location)  Specifically, we examined data from 2013— when the first transactions closed—through June 30, 2017  HUD compiled and provided custom extracts of data on households in RAD- converted properties from the Inventory Management System/Public and Indian Housing Information Center (IMS/PIC) (public housing and Section 8 PBV) and Tenant Rental Assistance Certification System (Section 8 PBRA)  We assessed the reliability of the data extracts provided by HUD by (1) performing electronic testing of required data elements, (2) reviewing existing information about the data and the system that produced them, and (3) interviewing agency officials knowledgeable about the data  We determined the data on PBV households were sufficiently reliable for the purposes of our reporting objectives, but that the data on PBRA households was not sufficiently reliable for purposes of describing some characteristics of RAD households  For example, in trying to determine participation in the RAD program by year, we received several thousand PBRA entries that preceded the establishment of the RAD program  Moreover, as we previously mentioned, the postconversion household data for PBRA conversions is in a separate data system, so some variables, such as those related to race, ethnicity, rent, and income, differ from the other household data for that program  Because of these limitations, the data for PBRA households were not reliable for purposes of comparing RAD household characteristics before and after conversion as we had intended  To describe safeguards for residents and help ascertain how HUD implemented protections, we reviewed legal protections and requirements in HUD notices, reviewed selected conversion files, and interviewed HUD officials about monitoring and compliance processes  Finally, as previously described, we held focus groups with residents to better understand any effects on their living conditions and quality of life  To determine challenges PHAs faced in implementing RAD, we reviewed HUD guidance and related documents for PHAs in the program  We also interviewed eight PHAs during our site visits and spoke with another 10 PHAs by telephone about the benefits and challenges of participating in the RAD program  To examine provisions designed to help preserve long-term affordability of units, we reviewed the RAD authorizing statute and amendments and HUD notices and interviewed HUD staff to verify our understanding of agency affordability protections  For a sample of 31 randomly selected properties, we examined templates for contractual agreements for RAD closings and analyzed closing documents and contracts to determine if agreements matched program requirements  We interviewed HUD staff and staff of 18 PHAs to obtain viewpoints on the potential strengths or weaknesses of preservation in the case of default or foreclosure  We conducted this performance audit from February 2016 to February 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  The Department of Housing and Urban Development’s (HUD) Office of Recapitalization collects financial sources and use data from Rental Assistance Demonstration (RAD) participants  Table 2 lists the financial source fields collected by HUD  Table 3 lists the financial cost fields collected by HUD  Table 4 provides additional financial source detail pertaining to HUD’s leverage ratio calculation  Table 5 and Table 6 show the total financial source amounts collected by HUD  Specifically, Table 5 shows total financial source amounts prior to recategorization, while Table 6 shows total financial source amounts after manual adjustments  Manual adjustments included isolating funding source observations in “other funding” fields 1-6 and incorporating them into existing fields, as appropriate  Table 7 replicates HUD’s methodology for calculating the RAD leverage metrics after manual adjustments in HUD data  See Table 4, above, to compare changes in each category  Table 8 recalculates the leverage ratio by deducting federal sources as leveraged sources  Table 9 recalculates the leverage ratio by deducting public sources as leveraged sources (compare to Table 8 above)  The Rental Assistance Demonstration (RAD) program has numerous requirements intended to ensure residents whose units are converted through RAD receive certain protections  The following is a description of these safeguards and their reporting and monitoring requirements  The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people  GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions  GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability  The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website (https://www gao gov)  Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence  To have GAO e-mail you a list of newly posted products, go to https://www gao gov and select “E-mail Updates ” The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white  Pricing and ordering information is posted on GAO’s website, https://www gao gov/ordering htm  Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537  Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order  Call for additional information  Connect with GAO on Facebook, Flickr, Twitter, and YouTube  Subscribe to our RSS Feeds or E-mail Updates  Listen to our Podcasts  Visit GAO on the web at https://www gao gov  James-Christian Blockwood, Managing Director, spel@gao gov, (202) 512-4707 U S  Government Accountability Office, 441 G Street NW, Room 7814, Washington, DC 20548 Please Print on Recycled Paper  In addition to the individual named above, Paul Schmidt (Assistant Director), Julie Trinder-Clements (Analyst in Charge), Meghana Acharya, Enyinnaya David Aja, Alyssia Borsella, Juan J  Garcia, Ron La Due Lake, Amanda Miller, Marc Molino, Barbara Roesmann, Jessica Sandler, MaryLynn Sergent, Rachel Stoiko, and William Woods made major contributions to this report 


Report 2: The Department of Defense (DOD) obligates more than $300 billion annually to pay for goods and services (including research and development)  Most of these acquisitions are governed by numerous statutes and regulations found in Title 10 of the United States Code, the Federal Acquisition Regulation (FAR), and the Defense Federal Acquisition Regulation Supplement (DFARS)  DOD can also enter into certain transactions without triggering most of the standard acquisition statutes and regulations by using other transaction (OT) authorities  In recent years, Congress has expanded these authorities and DOD is increasingly using OTs for research, prototyping, and production  This report examines (1) how OTs work, (2) why they were established, (3) potential benefits and risks of using OTs, and (4) whether there are data available against which to measure their effectiveness  Appendix A provides a legislative history of DOD's other transaction authorities  On October 4, 1957, the Soviet Union triggered a space race with the United States when it successfully launched Sputnik I into orbit, becoming the first nation to send a man-made satellite into space  Congress, concerned that the United States was falling behind in space, held a series of hearings on an "emergency" effort to respond to the Soviet launch of Sputnik  At the same time, a bill was introduced in the Senate to create an agency with the means to quickly and efficiently develop a national space program  These efforts led to passage of the National Aeronautics and Space Act of 1958 (Space Act, P L  85-568) in July 1958, which established the National Aeronautics and Space Administration (NASA)  In an effort to give the new agency "the necessary freedom to carry on research, development, and exploration     to insure the full development of these peaceful and defense uses without unnecessary delay," the Space Act granted NASA broad authority to "enter into and perform such contracts, leases, cooperative agreements, or other transactions as may be necessary" to accomplish its mission of research and exploration (emphasis added)  Congress extended different variations of OT authorities to other select agencies, granting the authority to DOD in the FY1990 & 1991 National Defense Authorization Act (NDAA, P L  101-189 )  For an analysis of which non-DOD agencies can use OTs and how the authorities differ by agency, see Appendix B   Other transactions are legally binding contracts that are generally exempt from federal procurement laws and regulations such as the Competition in Contracting Act and the Federal Acquisition Regulation  In contrast, traditional procurement contracts must adhere to the procurement rules set forth in statute and regulation  Generally, DOD can use other transaction authorities for three purposes: 1  conduct research, 2  develop prototypes, or 3  contract for follow-on production of a successful prototype project  DOD's other transaction authorities are found in two sections of law: 10 U   S   C   2371 grants DOD the authority to use other transactions to carry out basic, applied, and advanced research projects  DOD regulations treat these projects as financial assistance instruments, not as contracts  10 U   S   C   2371b permits the use of other transactions to conduct prototype projects and follow-on production  OTs can only be used for prototypes if one of the following applies: at least one nontraditional defense contractor significantly participating in the project; all significant participants are small businesses or nontraditional defense contractors; at least one-third of the total cost of the prototype project is provided by nongovernment participants; or the senior procurement acquisition official provides in writing an explanation of the exceptional circumstances justifying an OT  Follow-on production can only be conducted when the underlying prototype OT was competitively awarded, and the prototype project was successfully completed  10 U   S   C   2373 , while generally not considered an o ther t ransaction authority , allows DOD to buy certain items and designs for experimental or test purposes without having to adhere to the procurement laws set forth in Chapter 137 of Title 10  OT authorities grant government officials the flexibility to include, amend, or exclude contract clauses and requirements that are mandatory in traditional procurements (e g , termination clauses, payments, audit requirements, intellectual property, and contract disputes)  OTs can be structured in numerous ways, including a direct relationship between a single government agency and a single provider; joint ventures; partnerships; multiple agencies joining together to fund an agreement encompassing multiple providers; or through a consortium  One common application of OTs is to forge an agreement with a consortium  A consortium is an organized group— it can consist of nonprofits, academic institutions, or contractors— focusing on a specific technology area  Generally, a lead entity coordinates and directs a consortium's activities  Consortia have consisted of a handful to as many as 1,000 members  Consortia can act to facilitate multiparty agreements whereby each member is akin to its own co-prime contractor with the government  In such a case, the government articulates the need or problem it is trying to solve, and the various members of the consortium can submit white papers for consideration  In this scenario, OTs can serve as an efficient way for all members to send unsolicited technology suggestions and solutions to solve a defined challenge  Consortia can also be used to develop an ecosystem of entities working together on a project, whereby members of a consortium pool resources and collaborate with DOD  A number of analysts argue that using consortia in this way gives the federal government a unique ability to leverage and pool the technological expertise and innovation of multiple entities in a particular sector, thereby strengthening and advancing a sector of the industrial base that may have defense applications  Seen like this, OTs can be considered a mechanism to promote defense technology and the defense industrial base, with the potential added benefit of advancing the domestic commercial technology base  Some analysts, however, have argued that many of today's consortia do not operate as collaborative organizations, but function more like managed multiple award task order contracts  These analysts argue that DOD should seek to foster more collaboration in consortia  Some analysts have argued that consortia reduce competition, since only members of the consortium under contract can participate in the project or submit white papers for consideration and funding  Others counter that even when an OT is signed with a single consortium, competition could be increased, since all members of the consortium are notified of the opportunity (expanding the pool of potential competitors aware of the opportunity), and the OT will foster internal competition among consortium members  OTs are not procurement contracts and thus are exempt from numerous procurement statutes and regulations, including the statutes in Chapter 137 of Title 10 ( Procurement Generally )  They are, however, bound by standard contract and other select laws and regulations  Examples of laws that do not apply include the Truth in Negotiations Act, Competition in Contracting Act, Cost Accounting Standards, Contract Disputes Act, and select intellectual property statutes such as the Bayh-Dole Act  A number of these laws are aimed at oversight and protecting the interests of taxpayers  While the Competition in Contracting Act does not apply, 10 U S C  2371b requires that research and prototype projects be competed "to the maximum extent practicable " DOD policy mirrors the statutory language, stating "[C]ompetition is a good thing  It helps keep prices low, quality high, and gives the government leverage in negotiations " To the extent that OTs induce nontraditional contractors to work with DOD, OTs can be viewed as promoting competition among (and within) entities that would not normally compete for DOD contracts  However, the lack of explicitly defined competition requirements could result in less competition for certain OTs  OTs are not free from all legislative and regulatory requirements  Generally, statutes and regulations that are not procurement-specific apply, including the Trade Secrets Act, (18 U S C  1905); the Economic Espionage Act (18 U S C  1831-39); elements of the Freedom of Information Act (5 U S C  552); and fiscal and property laws, such as the Anti-deficiency Act (31 U S C  1341); and the Tucker Act (28 U S C  1491)  As can be seen by these citations, many of these statutes are not located in the procurement titles of Title 10 or Title 41  The Government Accountability Office (GAO) has held that OTs are not procurement contracts, and therefore it will not review protests of such an award or solicitation  However, GAO will review "a timely protest that an agency is improperly using its other transaction authority " OT contracts can be protested to the Court of Federal Claims, although there is debate as to the extent of the court's jurisdiction  The limited protest jurisdiction of GAO is appealing to many government procurement officials  One senior Air Force official reportedly stated that OTs are "just so much faster and so much more attuned to getting something that we want today and not have to spend a couple of years going through a protest, going through this huge process to get something we wanted two years ago " It generally does not take that long to go through a GAO protest  By statute, GAO must issue an opinion on a protest within 100 days of the protest being filed, and 70% of cases are resolved in less than 60 days  While exempt from GAO bid protests, OTs are not exempt from GAO audits  Generally, OTs that include government payments exceeding $5 million are required to include a clause granting GAO the right to examine the records of any related party  However, this requirement has a number of limitations  Used properly, OTs can provide significant benefits to DOD  Along with the potential benefits come certain risks  Some of the potential benefits to OTs highlighted by analysts and officials include providing a mechanism to pool R&D resources with industry to facilitate development of, and obtain "the latest state-of-the-art, dual use technologies"; attracting nontraditional contractors with promising technological capabilities to work with DOD; lowering costs by eliminating requirements associated with the Federal Acquisition Regulations (i e , costs associated with required reporting and administrative activities) or sharing costs with industry; and "speeding up" the acquisition process  A number of experts argue that OTs provide a unique mechanism for DOD to invest in, and influence the direction of, technology development even when the end result is not directly tied to a military capability  As one observer noted, the real benefit to DOD may be that [t]he R&D has been accomplished and is available to the technical and scientific communities  As a result, a subsequent phase of research can begin or a particular approach can be demonstrated to be of no value  Some argue that OTs have particular import today  Drawing parallels to the space race, these analysts argue that DOD is engaged in a defense technology race  According to the 809 Panel: DoD is now in a period during which the time a particular technology is a dominant force on the battlefield is getting increasingly shorter, disruptive technologies are emerging at a faster pace, and these technologies are more widely dispersed…In a world with rapidly changing technology, time is a valuable resource that must not be taken for granted  It is difficult to predict what capabilities DoD will need 5 to 10 years from now—biotechnology, nanotechnology, artificial intelligence, robotics, or a new technology area not even known today  It also is unclear on what plane the military will conduct warfare—traditional battlefields, space, cyberspace, or some other domain  The current acquisition system lacks the agility needed to adapt to new paradigms  These analysts argue that OTs and similar rapid acquisition authorities are critical for DOD to compete in such a fast-paced global environment where technology and innovation are no longer driven by DOD, but by industry and foreign competitors  In 1960, the United States accounted for 69% of global R&D, with U S  defense-related R&D alone accounting for more than one-third of global R&D  The federal government funded approximately twice as much R&D as U S  business  However, from 1960 to 2016, the U S  share of global R&D fell to 28%, and the federal government's share of total U S  R&D fell from 65% to 24%, while business's share more than doubled from 33% to 67%  As a result of these global, national, and federal trends, federal defense R&D's share of total global R&D fell to 3 7% in 2016  Given the shift in the global R&D landscape, and the diminishing influence of DOD as a market mover, analysts suggest that the current procurement system is overburdened by regulations and bureaucratic processes that slow the system, increase costs, and dissuade companies from doing business with DOD  In contrast, OTs are viewed as faster, attracting companies that would otherwise forgo working with DOD and promoting broader investment in critical defense capabilities  As one analyst wrote: OTAs are currently the only way to remove the barriers necessary to get these nontraditional sources of innovation to do business with the military  Properly constructed, OTAs help speed up the process, respect a company's IP through negotiation rather than regulatory fiat, and result in contracting under commercial terms and conditions  Congress also appears to have shifted its view on appropriate use of other transactions  In the FY1999 conference report, Congress stated that [OT] authority should only be used in the exceptional cases where it can be clearly demonstrated that a normal contract or grant will not allow sufficient access to affordable technologies  By comparison, in the FY2018 NDAA, Congress expanded OT authorities and stated the following: In the execution of science and technology and prototyping programs, the Secretary of Defense shall establish a preference, to be applied in circumstances determined appropriate by the Secretary, for using transactions other than contracts, cooperative agreements, and grants  Along with the potential benefits come potential risks, including that of diminished oversight and exemption from laws and regulations designed to protect government and taxpayer interests  Some analysts, while acknowledging the important role of OTs, raise concerns over transparency and how these agreements are being employed  As one industry official stated, OTs are a contracting method, not a substitute for good acquisition practices  Discussing a particular OT for cloud services that was protested and ultimately cancelled by DOD, one observer argued The cloud contract provides a teachable moment for procurement reform-minded officials in the Pentagon and Capitol Hill  The problem was not with the OTA mechanism, which remains an essential element of reforming Pentagon procurement  Rather, the problem was with a lack of transparency with how the mechanism was employed  Scott Amey, general counsel of the Project on Government Oversight, cautioned We have to seriously consider how we are using [OTs]; whether we are using them as intended, whether we are getting the goods and services that we really want and need, whether we are getting them at the best cost and process, and we are using this procurement vehicle as a way to just circumvent the rules and have contractors not have the administration and oversight they need to hold them accountable  I'm just afraid this is going to result in a lot of waste, fraud, and abuse in the future  Congress has expressed repeated concerns that OTs could be used to circumvent congressional intent  In the FY1999 NDAA, the committees emphasized that the authority should only be used in a limited manner  The conference report stated the following: The conferees are especially concerned that such authority not be used to circumvent the appropriate management controls in the standard acquisition and budgeting process  Congress echoed a similar concern in the FY2019 NDAA  According to the House report: The committee also urges the Department to reiterate through established guidelines that OTA is not a means for circumventing appropriate use of the Federal Acquisition Regulations, and that full and open competition should be used to the maximum extent possible to maintain a sense of integrity, fairness, and credibility in the Federal Procurement process  Other transactions are also exempt from many of the socioeconomic policies put in place by Congress to promote public policies, including some Buy America requirements  Some analysts have raised concerns that OTs are a way to circumvent many of the public policies enshrined in the acquisition process  A number of analysts and officials have raised concerns that if DOD uses OTs in ways not intended by Congress—or is perceived to abuse the authority—Congress could clamp down on the authority  Under Secretary of the Army Ryan McCarthy reportedly stated that the military department is "trying to be very judicious about this authority so we don't lose it " Some analysts argue that Congress is already clamping down on the use of OTs  These analysts point to language in the FY2019 NDAA and FY2019 appropriations legislation ( P L  115-245 ) requiring additional reporting and notification (see Appendix A )  Such notification and reporting requirements, however, are not new; when Congress expanded OT authorities in the past, reporting and notification requirements were commonly included  The reporting requirements may also be a result of congressional frustration with a lack of transparency and data on how DOD uses OTs  DOD lacks authoritative data that can be used to assess OT effectiveness and better understand broader trends associated with these agreements  The most frequently cited source for such data is the Federal Procurement Data System-Next Generation (FPDS-NG), which is the primary source for tracking data on contract obligations, including other transactions for prototypes and follow-on production  FPDS-NG is configured to track data on cost-sharing, other transaction award type, and prevalence of nontraditional contractors  Obligations connected to OTs for research are tracked by the Defense Assistance Awards Data System, which is primarily used to track grants and cooperative agreements  This bifurcation of how OT data are tracked makes it more difficult to get a consolidated view of OT data  According to DOD, all OT data will be reported through FPDS-NG starting in late 2019  The procurement data in FPDS-NG are not fully reliable  There are quality issues relating to accuracy, completeness, and timeliness of data  CRS reviewed FPDS-NG data for prototype OT agreements signed or modified between FY2015 and FY2017 and found similar data inconsistencies  DOD officials acknowledge that they do not have sufficiently reliable data upon which to conduct analysis on the use of OTs and are taking steps to try to improve the data  The analyses below reflect CRS's effort to analyze DOD's use of other transaction authority based on the best available data  According to FPDS-NG, in FY2017, DOD obligated $2 1 billion—and received $360 million in cost-share contributions—on prototype other transaction agreements, representing less than 1% of DOD's total FY2017 contract obligations (approximately $320 billion)  Despite the small percentage of obligations, OTs are growing quickly and are expected to continue to grow at a rapid pace  From FY2013 to FY2017, the number of new prototype agreements increased from 12 to 94, an increase of over 650% (see Table 1 )  DOD's Defense Innovation Unit (DIU, formerly known as the Defense Innovation Unit Experimental, or DIUx) was involved with approximately half of the prototype agreements executed in 2017  DOD officials have stated their intent to further increase the department's use of OTs  Officials say that this increase is due to Congress expanding the statutory authority  The Army executed more than 66% of the prototype OT agreements between FY2013 and FY2017, often on behalf of other military departments and components (see Table 2 )  Army Contracting Command-New Jersey at Picatinny Arsenal executes many of these agreements  DIU currently uses Picatinny Arsenal to execute all of its other transaction agreements  A number of private sector companies do not pursue federal government contracts because they are unwilling to forfeit intellectual property rights or adhere to some of the procurement regulations  One of the goals of OTs is to expand the defense marketplace by creating a mechanism for access to technologies and services of companies that would not otherwise work with DOD, particularly startups and companies developing innovative technology  As one industry representative stated: Because they are "outside" the FAR, OT agreements do not require such cumbersome oversight and audit requirements such as those imposed by the Truth in Negotiations Act, cost and pricing data or an expensive Cost Accounting System (CAS) qualified financial system  CAS compliant financial systems can cost a company millions of dollars to implement and maintain — and are therefore a significant, if not potentially fatal, barrier to government market entry for startups and small, innovative companies  These requirements tend to reinforce the "legacy advantage" of large traditional contractors, who can afford to hire and staff these requirements with large staffs of accountants and lawyers  A number of nontraditional companies told CRS that they are more likely to work with DOD because of the department's other transaction authorities  Despite these claims, some observers question whether OTs are effectively bringing nontraditional contractors into the defense marketplace  A DOD Inspector General report examining other transactions from FY1994 to FY2001 found that OTs did not attract significant numbers of nontraditional defense contractors to do business with DOD  The report found that of the 209 prototype agreements examined, traditional defense contractors received 95% of the $5 7 billion in funds awarded  A recent analysis of FPDS-NG data by Federal News Network had similar findings  According to the report, from FY2015 to FY2017, while nontraditional defense contractors were awarded most of the new OTs (66% vs  33% for traditional defense contractors), the dollar value of the OTs favored traditional contractors ($20 8 billion vs  $7 4 billion for nontraditional contractors)  Some observers have questioned the accuracy of the data published by Federal News Network  According to Charlie McBride, president of Consortium Management Group (which manages two consortia working with DOD through OTs), 88% of the total dollar value of awards to CMG Group has gone to nontraditional prime contractors, and nontraditional entities have participated in the remaining 12%  This debate highlights the lack of authoritative data on OTs  The currently available data may not accurately reflect the extent to which nontraditional contractors are engaged in OT agreements  FPDS-NG does not collect data regarding subcontractors or consortia composition, making it difficult to determine the nature and extent to which nontraditional defense contractors and entities may be working under OT agreements with DOD directly or as subcontractors  When Congress extended OT authority to DOD, it authorized inserting a clause requiring a person or entity to make payments to DOD as a condition of receiving support under the agreement  Such funds were to be merged into an account dedicated to support DARPA advanced research projects  The intent of this provision was to permit DARPA to "recoup the fruits of such arrangements, when there is a 'dual use' potential for commercial application" and reinvest the funds to develop other technologies  In addition to the recoupment authority, DOD can share costs with other parties under an OT  Using this approach, the amount of each party's share is negotiated and incorporated into the agreement  Congress believed that OTs and cooperative agreements were ideal vehicles for promoting DOD-industry collaboration in developing dual-use technologies  For example, the FY1992 & 1993 NDAA ( P L  102-190 ) authorized DOD to enter into cooperative and other transaction agreements to develop critical dual-use technologies as set forth in the Defense Critical Technologies Plan  According to the Senate report:     the United States tends to underinvest in dual-use technologies  National security requirements alone often do not justify major DOD support, and market prospects alone often appear to be too long-term or high risk to justify US industry carrying the entire development burden     The committee encourages use of cooperative agreements and other transactions in lieu of grants or contracts     The provision would require that at least 50 percent of funding over the life of a partnership derive from non-federal sources but would allow for a smaller industry share at the start  In the 1990s, some DARPA OTs required participants to share costs because the types of work completed generally involved R&D that was mutually beneficial to government needs and industry commercial goals  In certain instances, present-day OTs have cost-sharing requirements that foster collaborative research  Some analysts believe that DOD is not always realizing all the benefits that OTs have to offer, such as sufficiently leveraging private capital or forming true consortia of multiple parties pooling resources  Some of these analysts believe that DOD does not sufficiently use consortia to leverage private investment through the pursuit of collaborative, mutually beneficial, dual-use technologies  Some observers argue that as the legislation on OTs has evolved, the cost-sharing provision for prototype projects has come to create an unfair playing field biased against traditional defense contractors  For prototype projects, traditional contractors generally are required to assume one-third of costs whereas nontraditional defense contractors and small businesses generally do not have to cost share  From a fairness perspective, these observers argue that traditional contractors should not have a mandatory cost share  These observers also point out that some nontraditional defense contractors are companies with billions of dollars of revenue that should not be granted a competitive cost advantage  Putting traditional defense contractors at a competitive disadvantage could deny DOD access to those companies with the most experience working on defense products, potentially depriving the military of access to leading defense-related research and technology  Other observers argue that the cost share as currently structured is appropriate: traditional defense contractors hold a significant competitive edge in their understanding of, and have the systems in place to manage, traditional contracts  In contrast, nontraditional and small businesses, which generally cannot compete with the traditional defense contractors, need the exemption from the cost-share requirement to be able to work with DOD  These observers also argue that traditional contractors are awarded the majority of dollars obligated to OTs, proving how difficult it is for nontraditional suppliers to break into the defense marketplace  Additionally, traditional contractors could avoid the cost-sharing requirement by teaming with a nontraditional contractor  DOD has not effectively tracked data on cost sharing  A 2017 report to Congress indicated that in FY2016 DOD obligated $1 billion for prototype agreements and received $68 million in cost-share contributions  A CRS review found numerous concerns with the data underlying the report, and with DOD's analytical conclusions  See Appendix C for further discussion  A number of analysts and industry officials have raised concerns that DOD could use OTs to avoid competitions, as OTs are exempt from the Competition in Contracting Act  According to statute, follow-on production using other transaction authority can only be awarded if the underlying R&D agreement was competed  In addition, 10 U S C  2371b states that "to the maximum extent practicable" OTs must be competed  Determining whether OTs are being used to circumvent competition requires a two-step analysis: 1  Are OTs competed less often that traditional contracts? 2  Is there a benefit to these OTs being competed? Available FPDS-NG data suggest that DOD is broadly complying with 10 U S C  2371b's competition mandate  Between FY2013 and FY2017, approximately 89% of all new OT prototype agreements were competed in some fashion  Many observers and analysts believe that OT agreements can be executed substantially faster, sometimes in a matter of weeks, compared to the months or years it typically takes to execute traditional contracts  Based in part on this belief, some officials and analysts are touting OTs as a new model for conducting acquisitions and the answer to many of the problems in defense acquisition  These analysts argue that in a world of increasingly fast technology development, DOD acquisitions must go faster or risk being left behind  Many acquisition professionals argue that OT contracts are not inherently faster than traditional contracting; instead, they are executed faster because they are not encumbered by the reviews, protests, and bureaucratic layers that have been overlaid on traditional contracting  According to these officials, OTs take just as long as traditional contracts if the same execution and oversight processes are applied  And because all terms are negotiable, complex negotiations could make OTs take longer to execute than traditional contracts that have required, nonnegotiable conditions  The Other Transactions Guide states The OT award process will not always be faster than the traditional procurement processes and sometimes can be as long or longer  The speed of award is tied to many factors, many of which are internal to the organization  DOD has not tracked data on the relative time it takes to execute OTs vs  traditional contracts, making it impossible to objectively assess these claims  Analysts and officials generally agree that the workforce plays a critical role in determining the success or failure of an acquisition  Because there are fewer predefined requirements, OTs can be more difficult to negotiate than traditional contracts, putting DOD at greater risk of not getting what it wants at a reasonable price  The complexity and difficulty of negotiations is particularly high when there are intellectual property/patent rights issues, as is the case with most OTs  Given these challenges, OTs often require more experienced and capable government representatives to ensure implementation of agreements that are in the government's best interest  Some analysts question the extent to which the workforce is sufficiently trained and equipped to negotiate OTs  In response to this concern, in the FY2018 NDAA, Congress required workforce education and training for OTs, and required DOD to establish a cadre of intellectual property experts to advise, assist, and provide resources to program offices that are developing intellectual property strategies for contracts and agreements (see Appendix A )  DOD officials acknowledge that more training and education is required  Given the complexity of OTs and the limited extent to which they are used, some analysts and industry officials suggested that there may be a benefit to establishing a centralized office within DOD responsible for executing or overseeing all other transaction agreements  Such a structure could help ensure that those members of the acquisition workforce engaged in other transactions are sufficiently experienced, trained, and qualified  A number of analysts have argued that DOD should take steps to improve its use of OTs  Many of these analysts have suggested that data are not consistently and accurately tracked, regulations and guidance on when and how to use OTs are vague or insufficient, and the workforce is not sufficiently prepared to effectively use OTs  A number of officials have acknowledged these shortcomings and DOD is reportedly taking steps to address them  For example, in December 2018, DOD issued an updated Other Transactions Guide , a comprehensive guide containing best practices, case studies, and a clarification of myths related to other transaction authorities  In addition, Defense Acquisition University developed new course materials addressing OTs and is working to expand its offerings of relevant training and classes  However, numerous acquisition officials question whether it is possible, or even desirable, to try to quickly implement training aimed at preparing the thousands of DOD acquisition officials to execute OTs  Some of these officials have suggested it might be appropriate to only allow a limited and vetted number of acquisition professionals to be OT agreements officers  Some argue that OTs are just one "tool in the tool box," appropriate for only specific types of contracts, and should not be used to avoid the statutory and regulatory framework or to try to accelerate the process just for the sake of speed  Others have suggested that OTs should be used to cut through bureaucracy, speed up the acquisition process, avoid regulations and bid protests, and perhaps eventually supplant the regular FAR-based contracting process  Given the benefits and risks associated with OTs, questions for Congress include the following: 1  To what extent and in what circumstances do the potential benefits of OTs in terms of cost, schedule, and added capabilities outweigh concerns over potential fraud, waste, abuse, diminished oversight, and other public policy objectives? 2  Should OT authorities be extended further, curtailed, or maintained? The FY2019 NDAA required DOD to submit a report annually through 2021, summarizing DOD's use of OTs, including organizations involved; number of transactions; amounts of payments; and purpose, description, and status of projects  The NDAA also required the Defense Innovation Unit to submit a report to Congress, to include the number of traditional and nontraditional defense contractors with DOD contracts or other transactions resulting directly from the unit's initiatives  The conference report for the FY2019 defense appropriations bill included language expressing the conferees' "[concern] with the lack of transparency surrounding the employment of OTA, particularly for follow-on production " The conferees directed DOD to provide quarterly reports to the House and Senate appropriations committees listing each active OT, and to include additional information for each agreement  The conferees also directed GAO to review DOD's use of OTs to determine whether the "employment of this authority conforms to applicable statutes and guidelines, to include the identification of any potential conflicts " GAO was also required to report on the extent to which OTs have been used since FY2016  The multitude of reporting requirements, and the questionable reliability of the available data, raise a number of questions that Congress may wish to explore, such as the following: 1  To what extent, if any, should the current reporting requirements be consolidated to create a more streamlined and consistent flow of information to Congress? 2  What specific data does Congress need in these reports to effectively conduct oversight? For example, what percentage of research OTs result in prototype projects and follow-on production? 3  To what extent are the data sufficiently reliable, and will such data be easily retrievable in the future, to allow Congress to conduct effective, timely, and ongoing oversight? If the data are not sufficiently reliable or accessible in the future, what other data collection and tracking methods could Congress mandate to ensure ongoing access to reliable data? 4  How are OTs being used? Where and when in the acquisition lifecycle is the authority being used? To what extent is the requirements process being circumvented when DOD awards an OT follow-on production contract for a major system? One analyst suggested that "no efficiency is lost if only the most able personnel are authorized to procure and administer" OT agreements and further argued that expanding the use of OTs would increase the training costs by expanding the number of people who can "weave complex agreements in a relatively unstructured environment " Congress may consider whether DOD should establish an acquisition innovation lab or center of excellence responsible for overseeing, executing, and approving all OTs across the department  Such a lab or center could be staffed and supported by a cadre of professionals with experience across the acquisition lifecycle who have a willingness and ability to embrace new ideas and rethink existing practices  Alternatively, such labs or centers could be established in the military departments  Having centers in each organization could allow for consideration of the different missions and business approaches of the departments and help educate the workforce on a more systematic basis  Proponents argue that such an office would help ensure that only experienced and capable officials, with the appropriate training, use OT authorities  Such an office could also help protect against layering internal DOD policies and bureaucracies onto OTs by placing OTs outside of the traditional bureaucratic acquisition process  Proponents could further argue that such an office could better propagate best practices and ensure that OTs are used appropriately and are consistent with guidance and legislation  Having a single office responsible for executing or approving all OTs could also help ensure more timely and accurate information, giving Congress more visibility into DOD's use of other transaction authorities  To the extent that a single, high-level official is responsible for managing and overseeing all OTs, Congress might wish to consider repealing or modifying existing statutory approval requirements  If such an office was able to provide timely and accurate information, Congress might also consider some of the current reporting requirements unnecessary, and may choose to repeal some of the reporting requirements  Opponents of such a proposal argue that centralizing OTs would have the opposite effect, increasing bureaucracy by adding yet another office within DOD  Opponents also argue that such an office could make it more time-consuming to get a project underway and may discourage program offices from attempting or suggesting OTs  Some also argue that a single office may not have the resources to execute and approve agreements in a timely manner, and would inhibit spreading expertise on how to execute OT agreements more broadly across the acquisition workforce  Even proponents who might in theory support establishing such an office could raise significant concerns regarding how such an office would function in practice  A number of alternative options could be pursued to address concerns raised by opponents of establishing a centralized office  Some of these alternative options include the following: Granting such an office primary, but not exclusive, authority to execute OTs  For example, agreement officers specifically authorized to do so could execute OTs, with the centralized office conducting a peer review  Under this construct, the office could also be charged with providing information and expertise/consulting services on the use of OTs to program offices contemplating using the authorities  Establishing centers within each military department, with a designated office in OSD serving a coordinating function (with nondelegable approval authority residing in the military department office designated for OTs)  Creating the office as a pilot program for three years, to help DOD manage OTs until such time as the workforce becomes more experienced and proficient in using these agreements  Appendix A  Legislative History Other transaction authority first appeared in the National Aeronautics and Space Act of 1958  Since then, Congress has extended OT authorities to 11 federal agencies and a number of other federal offices (see Appendix B for information on other federal entities with similar authorities)  This appendix traces the legislative history of OT authorities and select related statutes applicable to DOD  To read the full text of the three statutory provisions related to OTs (10 U S C  2371, 2371b, and 2373), see Appendix D   National Aeronautics and Space Act of 1958 (P L  85-568) Creation of Other Transaction Authority On July 29, 1958 President Dwight D  Eisenhower signed into law the National Aeronautics and Space Act (P L  85-568), which established the National Aeronautics and Space Administration (NASA)  The purpose of the act included the expansion of human knowledge, preservation of the role of the United States as a leader in space science and technology, and pursuing the most effective utilization of the scientific and engineering resources of the United States  Section 203(b)(5) of the Space Act provided NASA the authority (emphasis added) to enter into and perform such contracts, leases, cooperative agreements, or other transactions as may be necessary in the conduct of its work and on such terms as it may deem appropriate, with any instrumentality of the United States     or with any person, firm, association, corporation, or educational institution  To the maximum extent practicable and consistent with the accomplishments of the purpose of this Act, such contracts, leases, agreements, and other transactions shall be allocated by the Administrator in a manner which will enable small-business concerns to participate equitably and proportionately in the conduct of the work of the Administration  Intellectual Property Rights The Space Act specifically addressed NASA's "property rights in inventions " Section 305 stated that any invention made in the performance of any work under any contract is the exclusive property of the United States "unless the Administrator waives all or any part of the rights " This was true even when the person who created the invention "was not employed or assigned to perform research, development, or exploratory work, but the invention is nevertheless related to the contract" and was made during working hours, or with a contribution of the government  The act granted the Administrator wide latitude to "waive all or any part of the rights of the United States under this section" if doing so was deemed to be in the best interests of the United States  When such rights were waived, NASA retained an irrevocable, nonexclusive, nontransferable royalty-free license by or on behalf of the United States  National Defense Authorization Act for FY1990 & FY1991 ( P L  101-189 ) The FY1990 & FY1991 NDAA granted DARPA temporary authority to enter into "cooperative agreements and other transactions" for the purpose of conducting advanced research projects  The statute clarified that OTs should only be used when "the use of standard contracts or grants is not feasible or appropriate " Congress restricted funding for OTs and cooperative agreements to $25 million of appropriated funds for FY1990 and FY1991, and set the authority to expire on September 30, 1991  Cost Sharing The FY1990 & FY1991 NDAA permitted OTs (or cooperative agreements) to include a clause requiring a person or entity to make payments to DOD as a condition of receiving support under the agreement  Such funds were to be merged into an account dedicated to supporting DARPA advanced research projects using cooperative agreements and other transactions  The act required, to the extent practicable, that funds provided by the government not exceed the total amount provided by the other parties to the project  According to the Senate report, one of the intents of the cost-sharing provision was to permit DARPA to "recoup the fruits of such arrangements, when there is a 'dual use' potential for commercial application" and to reinvest the funds to develop other technologies  Reporting Requirements The FY1990 & FY1991 NDAA required DOD to submit an annual report on the use of OTs and cooperative agreements, to include a description of each agreement and the technologies involved, the potential military and commercial utility of the technology, the reasons a contract or grant was not feasible to support the research, and the amount of payments, if any, received by the federal government under the agreement  National Defense Authorization Act for FY1991 ( P L  101-510 ) Section 244 increased the funds authorized for cooperative agreements and OTs from $25 million to $50 million  However, no such funding was appropriated  Reporting and Notification Requirements The conference report required DOD to submit to Congress a report listing the cooperative agreements and consortia intended to be used in FY1991-1992  The conference report also required DOD to provide the armed services and appropriations committees 30 days' notice prior to DOD signing a cooperative agreement or agreement with a consortia under OT authority  The Senate report focused on consortia as a method to pool resources, share research among numerous participants, and promote critical dual-use technology  Department of Defense Appropriations Act, 1992 ( P L  102-172 ) Limitations on the Use of OTs Section 8113A of P L  102-172 placed temporary limitations on the use of agreements undertaken pursuant to 10 U S C  2371: Section 8113A limited the use of OTs and cooperative agreements exclusively to DARPA (to the exclusion of the rest of DOD) for FY1992  Section 8113A limited DARPA to obligating or expending no more than $37 5 million in FY1992 for cooperative agreements or OTs undertaken pursuant to 10 U S C  2371  Section 8113A further established that no more than $75 million could be obligated or expended by DARPA in FY1992 for DOD dual-use critical technology partnerships  National Defense Authorization Act for FY1992 & FY1993 ( P L  102-190 ) Expanded Authority Section 826 extended other transaction authority to the military departments, and established separate fund accounts in each department for cost sharing  Section 826 also repealed the sunset for cooperative agreements and OTs, making the authorities permanent  Section 821 authorized DOD to enter into cooperative and other transaction agreements to develop critical dual-use technologies as set forth in the Defense Critical Technologies Plan  According to the Senate report   

ground_truth: HUD administers the Public Housing program, which provides federally assisted rental units to low-income households through PHAs. In 2010, HUD estimated its aging public housing stock had $25.6 billion in unmet capital needs. To help address these needs, the RAD program was authorized in fiscal year 2012. RAD allows PHAs to move (convert) properties in the public housing program to Section 8 rental assistance programs, and retain property ownership or transfer it to other entities. The conversion enables PHAs to access additional funding, including investor equity, generally not available for public housing properties. GAO was asked to review public housing conversions under RAD and any impact on residents. This report addresses, among other objectives, HUD's (1) assessment of conversion outcomes; (2) oversight of resident safeguards; and (3) provisions to help preserve the long-term affordability of units. GAO analyzed data on RAD conversions through fiscal year 2017; reviewed a sample of randomly selected, nongeneralizable RAD property files; and interviewed HUD officials, PHAs, developers, academics, and affected residents. The Department of Housing and Urban Development (HUD) put procedures in place to evaluate and monitor the impact of conversion of public housing properties under the Rental Assistance Demonstration (RAD) program. RAD's authorizing legislation requires HUD to assess and publish findings about the amount of private-sector leveraging. HUD uses a variety of metrics to measure conversion outcomes. But, the metric HUD uses to measure private-sector leveraging—the share of private versus public funding for construction or rehabilitation of assisted housing—has limitations. For example, HUD's leveraging ratio counts some public resources as leveraged private-sector investment and does not use final (post-completion) data. As a result, HUD's ability to accurately assess private-sector leveraging is limited. HUD does not systematically use its data systems to track effects of RAD conversions on resident households (such as changes in rent and income, or relocation) or monitor use of all resident safeguards. Rather, since 2016, HUD has required public housing agencies (PHA) or other post-conversion owners to maintain resident logs and collect such information. But the resident logs do not contain historical program information. HUD has not developed a process for systematically reviewing information from its data systems and resident logs on an ongoing basis. HUD has been developing procedures to monitor compliance with some resident safeguards—such as the right to return to a converted property—and begun a limited review of compliance with these safeguards. However, HUD has not yet developed a process for monitoring other safeguards—such as access to other housing voucher options. Federal internal control standards require agencies to use quality information to achieve objectives, and obtain and evaluate relevant and reliable data in a timely manner for use in effective monitoring. Without a comprehensive review of household information and procedures for fully monitoring all resident safeguards, HUD cannot fully assess the effects of RAD on residents. RAD authorizing legislation and the program's use agreements (contracts with property owners) contain provisions intended to help ensure the long-term availability of affordable units, but the provisions have not been tested in situations such as foreclosure. For example, use agreements between HUD and property owners specify affordability and use restrictions that according to the contract would survive a default or foreclosure. HUD officials stated that HUD intends to develop procedures to identify and respond to risks to long-term affordability, including default or foreclosure in RAD properties. However, HUD has not yet done so. According to federal internal control standards, agencies should identify, analyze, and respond to risks related to achieving goals and objectives. Procedures that address oversight of affordability requirements would better position HUD to help ensure RAD conversions comply with program requirements, detect potential foreclosure and other risks, and take corrective actions. GAO makes five recommendations to HUD intended to improve leveraging metrics, monitoring of the use and enforcement of resident safeguards, and compliance with RAD requirements. HUD agreed with our recommendations to improve metrics and build on existing oversight.

prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 2.



Report 2:  the United States tends to underinvest in dual-use technologies  National security requirements alone often do not justify major DOD support, and market prospects alone often appear to be too long-term or high risk to justify US industry carrying the entire development burden     The committee encourages use of cooperative agreements and other transactions in lieu of grants or contracts     The provision would require that at least 50 percent of funding over the life of a partnership derive from non-federal sources but would allow for a smaller industry share at the start  The conference report stated that the partnerships should focus on programs that fit into the security needs within DARPA  The conference report also stated that OTs are appropriate for those cases where the "regulations applicable to the allocation of patent and data rights under the procurement statutes may not be appropriate to partnership arrangements in certain cases " National Defense Authorization Act for FY1993 ( P L  102-484 ) Cost Sharing Section 4221 established 10 U S C  2511, which required DOD to establish cooperative arrangements with industry, educational institutions, federal labs, and other entities, to pursue research, development, and application of dual-use technologies  The section authorized DOD to use grants, contracts, cooperative agreements, or OTs to create these partnerships, and that the Federal government should not contribute more than 50% of the costs related to projects under this authority  National Defense Authorization Act for FY1994 ( P L  103-160 ) Expanded Authority Section 827 established 10 U S C  2358, which gave the Secretary of Defense and the Secretaries of the military departments the authority to conduct basic, advanced, and applied research through the use of contracts, cooperative agreements, grants, and OTs  Previously, OTs were only authorized for advanced research  Prototype Authorities Section 845 granted DARPA the authority to use OTs for prototype projects directly related to weapons or weapon systems proposed to be acquired by DOD  Section 845 required that "to the maximum extent practicable," prototypes be competitively awarded  This authority was set to terminate after three years  Section 845 remained as a note to 10 U S C  2371 until separately codified as 10 U S C  2371b in the FY2016 NDAA ( P L  114-92 )  Federal Acquisition Streamlining Act of 1994 ( P L  103-355 ) Section 1301 redesignated the language in 10 U S C  2358 (granting the authority to use OTs) to 10 U S C  2371  Reporting Requirements Section 1301 also required DOD to submit an annual report to the armed services committees, to include a general description of the other transactions, including the technologies involved in the research, the potential military and, if any, commercial utility of such technologies, the reasons for not using a contract or grant to provide support for such research, and the amount of payments, if any, received during the fiscal year pursuant to a clause in the other transactions and to what accounts such payments were credited  National Defense Authorization Act for FY1997 ( P L  104-201 ) Section 203 required that a senior DOD official be designated in OSD, and that the officials' sole responsibility be developing policy related to, and ensuring implementation of, DOD's dual-use technology program  This section authorized DOD to use OTs (as well as contracts, cooperative agreements, and grants) for dual-use projects only if the project "is entered into through the use of competitive procedures " Section 743 granted DOD the authority to use OTs to conduct research on Gulf War Syndrome, to determine its relationship to possible exposures of members of the Armed Forces to chemical warfare agents and hazardous materials, and the use of inoculations and new drugs  Expanded Prototype Authorities Section 804 amended Section 845 of the FY1994 NDAA by extending to the military departments and officials designated by the Secretary of Defense, the authority to use OTs for certain prototype projects  This authority, originally granted solely to DARPA and set to expire after three years, was given a new termination date of September 30, 1999  Reporting Requirements Section 267 modified elements of the annual report to the armed services committees  National Defense Authorization Act for FY1998 ( P L  105-85 ) Section 832 amended 10 U S C  2371 by clarifying that certain information submitted to DOD (i e  a proposal, business plan, technical information) be protected from disclosure pursuant to 5 U S C  552 for a period of five years  Strom Thurmond National Defense Authorization Act for FY1999 ( P L  105-261 ) Section 241 extended the sunset day for the authority to use OTs for prototypes from September 30, 1999, to September 30, 2001  Section 817 amended Section 2371 of Title 10, United States Code, clarifying that information submitted by outside parties in cooperative agreements for basic, applied, and advanced research is protected from disclosure under Section 552 of Title 5, United States Code  Department of Defense Appropriations Act, 1999 ( P L  105-262 ) While the enacted FY1999 defense appropriations bill ( P L  105-262 ) did not include legislative language addressing OTs, H Rept  105-591 , which accompanied the House-reported version of H R  4103 , included language expressing the House Appropriations Committee's "serious reservations" regarding the Air Force's then-proposed use of an OT agreement—instead of a contract— to develop the Evolved Expendable Launch Vehicle (EELV) program  The committee noted that "under [OTs] traditional safeguards which protect the government's interest in large acquisition programs are largely absent," and required the Under Secretary of Defense for Acquisition, Technology, and Logistics (now the Under Secretary of Defense for Acquisition and Sustainment) and the DOD Inspector General to certify to the congressional defense committees that the use of an OT was appropriate for the EELV program, and that "adequate safeguards exist[ed] to protect the government's interest and monitor program performance " National Defense Authorization Act for FY2000 ( P L  106-65 ) Section 801 required that for prototypes using OT authorities, DOD ensure that GAO, under its audit authority, have access to records relating to other transaction prototype agreements exceeding $5 million  Section 801 allowed for a waiver to GAO access and exempted entities that over the last year have not entered into an agreement with DOD that provided for audit access by a government entity  According to the Senate report: Senior DOD officials have sought legislation to extend other transaction authority to production contracts  Under current authority, there is some debate about whether GAO has audit access to other transactions  As the size, costs, and complexity of programs being funded using other transactions increases, the committee wants to ensure that the GAO has audit access in relation to the higher levels of spending and added risks  Reporting Requirements The Senate report also addressed reporting requirements and congressional intent to review the use of OTs  The report stated the following: The committee is assessing the utility of other transaction prototype authority  The statement of managers accompanying the Strom Thurmond National Defense Authorization Act of 1999 directed the Secretary of Defense to report on the use of this authority to the congressional defense committees, no later than March 1, 1999  In addition, both the Department of Defense Inspector General and the General Accounting Office are reviewing the use of other transaction prototype authority and will report to Congress in the coming year  The committee is interested in the extent that new commercial firms are entering the DOD marketplace through the use of other transaction authority, as well as the degree of cost sharing between the government and non-federal government parties  The committee is also interested in any lessons learned from the broad exemptions to federal law provided by other transaction authority  For example, other transactions are exempt from the Competition in Contracting Act, Truth in Negotiations Act, Contract Disputes Act, Antikickback Act of 1986, Procurement Integrity Act, Service Contract Act, Buy American Act, and chapter 137 of title 10, United States Code  Questions have been raised about whether the government's interest is adequately protected in the absence of the applicability of these statutes  Conversely, advocates of the view that the government should take advantage of the flexibility of other transactions have championed proposals to extend other transaction authority to production  The committee directs the Secretary of Defense to provide a new report that updates information in the March 1999 report on the use of other transaction prototype authority to the congressional defense committees by February 1, 2000  National Defense Authorization Act for FY2001 ( P L  106-398 ) Limitations on the Use of OTs Section 803 limited the use of OTs for prototype projects to only those circumstances when at least one nontraditional defense contractor significantly participates in the project, one-third of the total cost of the project is paid out of funds provided by parties to the transaction other than the federal government, or the senior procurement executive determines in writing that exceptional circumstances justify use of an OT  Nontraditional defense contractor was defined as an entity that for a period of one year has not entered into or performed "any contract that is subject to full coverage under the cost accounting standards" or "any other contract in excess of $500,000 to carry out prototype projects or to perform basic, applied, or advanced research projects for a Federal agency, that is subject to the Federal Acquisition Regulation '' Section 803 also extended the authority to use OTs for prototypes from September 30, 2001, to September 30, 2004  According to the Senate report, the intent of using OTs for prototypes is to attract companies that typically do not do business with the Department of Defense and encourage cost sharing and experimentation in potentially more efficient ways of doing business with traditional defense contractors  Other transaction authority is an important acquisition tool that can facilitate the incorporation of commercial technology into military weapon systems  In an environment where, in many areas, commercial technology is now more advanced than defense technology, it is imperative that the Department continue to have the flexibility to use innovative contractual instruments that provide access to this technology  There are, however, improvements that can be made in managing and overseeing these contractual arrangements  Section 804 clarified the extent of GAO's access to records in instances where the party in question has only done business with the government in the preceding year through an OT or cooperative agreement  National Defense Authorization Act for FY2002 ( P L  107-107 ) Expanded Authority—Follow-on Production Section 822 of the FY2002 NDAA granted DOD the authority to award a follow-on production contract for prototype projects when at least one-third of the total cost of the prototype project is to be paid out of funds provided by non-federal government sources  Under this authority, such a follow-on contract could be awarded without competition if the prototype project was successfully completed, the number of units in the production contract does not exceed the number of units specified in the underlying prototype agreement, and the price for each unit does not exceed the price specified in the underlying transaction  Department of Defense and Emergency Supplemental Appropriations for Recovery from and Response to Terrorist Attacks on the United States Act, 2002 ( P L  107-117 ) Establishment of Army Venture Capital Initiative (AVCI) Section 8150 designated $25 million of the FY2002 funds made available for Army Research, Development, Test, and Evaluation (RDT&E) to be made available to the Secretary of the Army for the purpose of funding a venture capital investment corporation established pursuant to 10 U S C  2371  A 2014 RAND report stated that OT authorities were used only to form the AVCI, and were not used to acquire products or services: "While OT authorities were used to form [AVCI] itself, any volume of the Army's purchase of products and services from [AVCI] companies is conducted under the FAR " National Defense Authorization Act for FY2004 ( P L  108-136 ) Expanded Authority Section 847 of the FY2004 NDAA extended the authority to use an OT for developing prototypes to improve weapons or weapon systems currently in use by the Armed Forces  Previously, such authority was restricted to prototypes "directly relevant to weapons or weapon systems proposed to be acquired or developed" by DOD  Section 847 also established a pilot program for transitioning prototypes to follow-on contracts for production for nontraditional defense contractors  Under the pilot program, such a follow-on contract could be treated as a commercial item or an item developed with both federal and private sector funds (for purposes of negotiating intellectual property rights)  The pilot program was restricted to contracts with nontraditional defense contractors, where the value of the contract does not exceed $50 million (approximately $70 million in FY2018 dollars), and that are firm-fixed price or fixed price with economic adjustment  The pilot program was set to sunset September 30, 2008  Section 1441 authorized any agency that engages in basic, applied, or advanced research and development projects that facilitated defense against or recovery from terrorism or nuclear, biological, chemical, or radiological attack to exercise the same general authority given to DOD as found in 10 U S C  2371 (including for prototype projects)  Reporting Requirements Section 1031 sunset the annual reporting requirement for OT after the report covering FY2006 was submitted  National Defense Authorization Act for FY2006 ( P L  109-163 ) Restricted Authority and Notification Requirements Section 212 of the FY2006 NDAA directed the Army to procure the Future Combat System using contract procedures set forth in part 15 of the Federal Acquisition Regulation, in lieu of an OT  Section 823 extended ethics requirements to OT prototype authority and required the congressional defense committees be notified in writing at least 30 days before such authority is exercised  Certification Requirements Section 823 also amended Section 845 of the FY1994 NDAA, requiring a written determination by a senior procurement executive for other transaction prototype projects estimated between $20 million and $100 million, and for a written determination by the Under Secretary of Defense for Acquisition, Technology, and Logistics for prototype projects that exceed $100 million  According to the Senate report: Section 845 was intended to be used for limited prototype projects, particularly those in which the Department seeks to engage nontraditional defense contractors that may be averse to the requirements imposed by a standard Department procurement contract  For this reason, the statement of managers accompanying the Strom Thurmond National Defense Authorization Act for Fiscal Year 1999 (Public Law 105–261) states: The conferees continue to believe that the section 845 authority should only be used in the exceptional cases where it can be clearly demonstrated that a normal contract or grant will not allow sufficient access to affordable technologies  The conferees are especially concerned that such authority not be used to circumvent the appropriate management controls in the standard acquisition and budgeting process  …  The committee does not believe that the $20 9 billion agreement entered between the Army and the Lead Systems Integrator for the FCS program is consistent with the language and intent of section 845 authority  Section 845 authority is intended to be used for limited prototype projects, particularly those in which the Department of Defense seeks to engage nontraditional defense contractors that may be averse to the requirements imposed by a standard Department contract  Department of Defense Appropriations Act, 2007 ( H R  5631 ) Reporting Requirement The FY2007 defense appropriations bill ( P L  109-289 ) did not include language addressing OTs  The conference report ( H Rept  109-676 ) included language expressing the conferees' "[concern] with the continued use of OTA contracts by the Missile Defense Agency," as such contracts "lack the customary safeguards found under FAR-based contracts for organizational conflict of interest, truth in negotiations and submission of cost and pricing data " The conferees "strongly encourage[d]" the Missile Defense Agency to convert "large development and procurement contracts using OTA to FAR-based contracts," and directed the Missile Defense Agency to submit a report to the congressional defense committees on the use of OTs, to include the number, value, and justification for the use of such agreements  National Defense Authorization Act for FY2008 ( P L  110-181 ) Section 823 extended the authority for prototype projects for five more years, from September 30, 2008, to September 30, 2013  National Defense Authorization Act for FY2009 ( P L  110-417 ) Section 822 required DOD to issue guidance on rights in technical data under non-FAR agreements, including OTs  Section 822 also required that appropriate provisions relating to rights in technical data be included in non-FAR agreements, consistent with policy guidance  This requirements is in statute at 10 U S C  2320 note  Section 824 expanded the scope of the pilot program for transition to follow-on contracts for certain prototype projects to include research projects carried out under 10 U S C  2371  Authority to use the pilot program, set to expire September 30, 2008, was extended to September 30, 2010  Section 874 required OT data be included in the Federal Procurement Data System  National Defense Authorization Act for FY2011 ( P L  111-383 ) Section 866 changed the definition of nontraditional defense contractor, conforming the definition to that found in 10 U S C  2302(9)  National Defense Authorization Act for FY2013 ( P L  112-239 ) Section 863 extended the authority for using OTs for prototype projects from September 30, 2013, to September 30, 2018  Carl Levin and Howard P  "Buck" McKeon National Defense Authorization Act for FY2015 ( P L  113-291 ) Expanded Authority Section 812 expanded the authority to use OT for prototypes, to include those "directly related to enhancing the mission effectiveness of military personnel and the supporting platforms, systems, components, or materials proposed to be acquired or developed by the Department of Defense, or to improvement of platforms, systems, components, or materials in use by the Armed Forces " Prior to the FY2015 NDAA, OTs could only be used for prototypes relating to weapons or weapon systems proposed to be developed, or for the improvement of weapons or weapon systems currently in use  Reporting Requirements Section 1071 repealed the reporting requirement language found in 10 U S C  2371, relating to OTs for research projects  National Defense Authorization Act for FY2016 ( P L  114-92 ) Until the FY2016 NDAA, the prototyping and follow-on production authority established in Section 845 of the FY1994 NDAA (as amended) was found in 10 U S C  2371 note  Section 815 of the FY2016 NDAA simultaneously repealed Section 845 of the FY1994 NDAA and put the repealed language into the newly created 10 U S C  2371b  The FY2016 NDAA also modified 2371b by making the authority permanent  Expanded Authority and Small Business The authorities in Section 2371b were expanded to allow their use when "all significant participants in the transaction other than the Federal government are small businesses or nontraditional contractors" and when the agency determines that using an OT would expand the defense supply base in a manner that could not be accomplished through a contract  Section 815 eased the restriction on follow-on production contracts or transactions  Section 815 amended the definition of a nontraditional defense contractor found in 10 U S C  2302 to be an entity that is not currently performing, and for one year prior to an OT has not performed on any contract or subcontract that is subject to full coverage under the cost accounting standards pursuant to Section 1502 of Title 41, U S C  Section 815 also required DOD to update its guidance to reflect changes in the statute  The conference report stated that Congress believed OTs are an attractive option for firms and organizations that do not usually participate in government contracting due to typical overhead burdens and the "one size fits all" rules governing defense acquisition  The report also stated that OTs could support DOD's effort to access new sources of technological innovation, specifically with Silicon Valley startup firms and small commercial firms  National Defense Authorization Act for FY2018 ( P L  115-91 ) Expanded Authority Section 216 of the FY2018 NDAA authorized nonprofit research institutions to enter into OTs with DOD for prototype projects  Section 862 amended 10 U S C  2358, granting the Secretary of Defense and the military departments the authority to pursue basic research, applied research, advanced research, and development projects under the OT authorities granted in Sections 2371 and 2371b of Title 10  Workforce Section 802 required DOD to establish a cadre of intellectual property experts to advise, assist, and provide resources to program offices who are developing intellectual property strategies for contracts and agreements  Section 863 required training and education for personnel involved in OTs and other innovative contracting methods  Certification Requirements and Small Business Section 864 adjusted the language of the statute to state that the dollar threshold relates to the specific transaction for a prototype project and not for the value of the entire project  Section 864 defined a transaction for follow-on production to include "all individual prototype sub-projects awarded under the transaction to a consortium of United States industry and academic institutions " Section 864 also increased the dollar thresholds for required approvals and defined the term small business to include small businesses under Section 9 of the Small Business Act to ensure that companies participating in the Small Business Innovation Research and Small Business Technology Transfer programs were considered small businesses for the purposes of the cost-sharing requirements  Miscellaneous Section 867 required the Secretary of Defense to establish a preference for OTs in the "execution of science and technology and prototyping programs " Section 1711 required DOD to carry out a pilot program to "assess the feasibility and advisability of increasing the manufacturing capability of the defense industrial base " Pursuant to the pilot, Section 1711 authorized DOD to use OTs to support production capabilities in small and medium-sized manufacturers  John S  McCain National Defense Authorization Act for FY2019 ( P L  115-232 ) Section 211 of the FY2019 NDAA clarified that follow-on production of a prototype or subproject within a consortium may occur if the individual prototype or subproject is complete; all projects associated with the consortium do not need to be completed before follow-on production of a specific prototype  Expanded Authority The FY2019 NDAA authorized the use of OTs to develop enhanced personal protective equipment (Section 226) and to carry out research under the Explosive Ordnance Disposal Defense Program (Section 311)  Reporting Requirements Section 244 required the Defense Innovation Unit to submit a report to Congress, to include the number of traditional and nontraditional defense contractors with DOD contracts or other transactions resulting directly from the unit's initiatives  Section 873 required DOD to submit an annual report through 2021, summarizing DOD's use of OTs, including organizations involved; number of transactions; amounts of payments; and purpose, description, and status of projects  Department of Defense Appropriations Act, 2019 ( P L  115-245 ) Reporting Requirements While the enacted FY2019 defense appropriations bill did not include legislative language addressing OTs, the conference report included language expressing the conferees' "[concern] with the lack of transparency surrounding the employment of OTA, particularly for follow-on production " The conferees directed DOD to provide quarterly reports to the House and Senate appropriations committees listing each active OT, and to include the following information on each agreement: funding military service or DOD component; major command (if applicable); contracting activity; appropriation title; budget line item; minimum and maximum award value; vendor; obligations and expenditures to date; product service code; period of performance; and indication if the OT agreement included an option for follow-on production (with a description of the scope of anticipated follow-on production)  The conferees also directed GAO to review DOD's use of OTs to determine whether the "employment of this authority conforms to applicable statutes and guidelines, to include the identification of any potential conflicts " GAO was also required to report on the extent to which OTs have been used since FY2016  Notification Requirements The House report to accompany H R  6157 acknowledged OTs as an "important tool to provide flexibility and agility for cutting-edge research and development projects and prototypes " However, the report stated its concern "with the lack of transparency on the use of OTA authority for follow-on production procurements," and directed that no funds could be obligated or expended for a follow-on production contract or a transaction carried out under 10 U S C  2371b, until 30 days after the Secretary of Defense provides the congressional defense committees with a notification of the proposed contract or transaction, to include a justification of why an OT is being used for production  Appendix B  Non-DOD Federal Agencies with Agency-Wide OT or Related Authorities A number of agencies have varying other transaction or similar authorities, as reflected in Table B-1   The table below is not a comprehensive or definitive listing of every federal government entity with OT or related authorities  In some instances, offices, agencies, commissions, and other federal government entities have OT or related authorities that are only associated with certain programs or projects, such as the National Institutes of Health (which has OT authority for such specific activities such as the National Heart, Blood Vessel, Lung, and Blood Diseases and Blood Resources Program [42 U S C  §285b-3] and the Cures Acceleration Network [42 U S C  §287a])  Appendix C  Reliability of Data on Other Transactions All data have imperfections and limitations  FPDS-NG data can be used to identify broad trends and produce rough estimates, or to gather information about specific contracts  Some observers say that despite their shortcomings, FPDS-NG data are substantially more comprehensive than what is available in most other countries in the world  Understanding the limitations of government procurement data—including knowing when, how, and to what extent to rely on data—can help policymakers incorporate FPDS-NG data more effectively into their decisionmaking process  FPDS-NG OT Data Quality and Accuracy Issues Decisionmakers should be cautious when using data from FPDS-NG to develop policy or otherwise draw conclusions, especially with respect to OTs  In some cases, the data themselves may not be reliable  In other instances, a query for particular data may return differing results, depending on the parameters and timing of the analysis  In particular, all DOD data entered into FPDS-NG are subject to a 90-day delay, and updates to "data, including new actions, modifications, and corrections are made on a regular basis," which could result in changes to "data     for current and/or prior fiscal years " Inconsistencies in FPDS-NG Data Within FPDS-NG, two primary collections of obligation data exist: one associated with standard government procurement contracts or modifications to such contracts, and the other associated with prototype OT agreements or modifications to such agreements  FPDS-NG's collection of prototype OT data allows for the input of additional data elements—such as nongovernment dollars associated with cost-share prototype OT agreements—not included in FPDS-NG's collection of standard government procurement contract data  More consequentially, FPDS-NG's prototype OT data include two similar data elements that allow users to identify the fiscal year a prototype OT agreement was signed or modified  One, labeled in the database as "Fiscal Year," appears to allow users entering data into the system to manually assign a fiscal year to a transaction  FPDS-NG users entering data into the system appear to have interpreted this data element in various, conflicting ways  For example, a Department of the Air Force OT agreement was signed in February 2016 for the development of rocket propulsion system prototypes under the Evolved Expendable Launch Vehicle (EELV) program  FPDS-NG records an obligation of $115 million in FY2020 for this agreement  Fiscal law bars DOD from obligating money now for future fiscal years that have not yet occurred  The second, labeled "Contract Fiscal Year," appears to be based on the date the prototype OT agreement was signed or modified  See Table C-1 for a comparison of the "Fiscal Year" and "Contract Fiscal Year" elements for selected new prototype OT agreements signed between FY2013 and FY2017  If a user selects the Fiscal Year data element when attempting to review high-level data on recent trends in the use of prototype OT agreements within DOD—such as the total amount obligated for prototype OT agreements on an annual basis—that user will obtain a substantially different result than if he or she selects the Contract Fiscal Year data element for a similar analysis  See Table C-2 for a comparison of action obligations and nongovernment contributions using the Fiscal Year and Contract Fiscal Year data elements for new prototype OT agreements signed between FY2013 and FY2017  DOD OT Data Analysis Methodological Issues in Congressional Reports A March 2017 report to Congress entitled "An Assessment of Cost-Sharing in Other Transaction Agreements for Prototype Projects," completed by the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics indicated that In FY2016, DOD obligated (Note: Two outliers in 2016 excluded) $1 billion in section 2371b awards and received $68 million in cost-share contributions after excluding two significant trend outliers  DOD cited FPDS-NG as the source for its analysis, and defined the "significant trend outliers" excluded as "two $40 million OTAs with total cost-share of $270 million," likely referring to two DARPA prototype OT agreements conducted on a cost-share basis initiated in FY2016  However, a CRS analysis of the same FPDS-NG data identified numerous inconsistencies in DOD's methodological approach  Recreation of DOD Methodology Specifically, DOD appears to have conducted its analysis using the "Fiscal Year" data element referenced in this report's discussion of FPDS-NG OT data quality and accuracy issues, which would attribute some prototype OT activities to the wrong fiscal year for the purposes of comparative trend analysis  DOD also compared two disparate transaction types: the reported "$1 billion in Section 2371b awards" includes all action obligations associated with ongoing prototype OT activities in FY2016, including transactions associated with prototype OT indefinite delivery contracts, while the "$68 million in cost-share contributions" includes only cost-share contributions associated with new prototype OT agreements  If DOD used the "Contract Fiscal Year" data element, excluded the identified "trend outliers," and focused on all action obligations and cost-share contributions associated with ongoing prototype OT activities, it would have found instead that the department obligated $1 4 billion for prototype OT agreements in FY2016, with an additional $313 5 million in cost-share contributions from the private sector  On the other hand, if DOD used the "Contract Fiscal Year" data element, excluded the identified "trend outliers," and focused on only action obligations and cost-share contributions associated with new prototype OT agreements, it would have found instead that the department obligated $400 million for prototype OT agreements in FY2016, with an additional $272 1 million in cost-share contributions from the private sector  Appendix D  Other Transaction Authority Statutes 10 U S C  §2371  Research projects: transactions other than contracts and grants (a) ADDITIONAL FORMS OF TRANSACTIONS AUTHORIZED — The Secretary of Defense and the Secretary of each military department may enter into transactions (other than contracts, cooperative agreements, and grants) under the authority of this subsection in carrying out basic, applied, and advanced research projects  The authority under this subsection is in addition to the authority provided in Section 2358 of this title to use contracts, cooperative agreements, and grants in carrying out such projects  (b) EXERCISE OF AUTHORITY BY SECRETARY OF DEFENSE — In any exercise of the authority in subsection (a), the Secretary of Defense shall act through the Defense Advanced Research Projects Agency or any other element of the Department of Defense that the Secretary may designate  (c) ADVANCE PAYMENTS — The authority provided under subsection (a) may be exercised without regard to Section 3324 of Title 31  (d) RECOVERY OF FUNDS — (1) A cooperative agreement for performance of basic, applied, or advanced research authorized by Section 2358 of this title and a transaction authorized by subsection (a) may include a clause that requires a person or other entity to make payments to the Department of Defense or any other department or agency of the Federal Government as a condition for receiving support under the agreement or other transaction  (2) The amount of any payment received by the Federal Government pursuant to a requirement imposed under paragraph (1) may be credited, to the extent authorized by the Secretary of Defense, to the appropriate account established under subsection (f)  Amounts so credited shall be merged with other funds in the account and shall be available for the same purposes and the same period for which other funds in such account are available  (e) CONDITIONS — (1) The Secretary of Defense shall ensure that- (A) to the maximum extent practicable, no cooperative agreement containing a clause under subsection (d) and no transaction entered into under subsection (a) provides for research that duplicates research being conducted under existing programs carried out by the Department of Defense; and (B) to the extent that the Secretary determines practicable, the funds provided by the Government under a cooperative agreement containing a clause under subsection (d) or a transaction authorized by subsection (a) do not exceed the total amount provided by other parties to the cooperative agreement or other transaction  (2) A cooperative agreement containing a clause under subsection (d) or a transaction authorized by subsection (a) may be used for a research project when the use of a standard contract, grant, or cooperative agreement for such project is not feasible or appropriate  (f) SUPPORT ACCOUNTS — There is hereby established on the books of the Treasury separate accounts for each of the military departments and the Defense Advanced Research Projects Agency for support of research projects and development projects provided for in cooperative agreements containing a clause under subsection (d) and research projects provided for in transactions entered into under subsection (a)  Funds in those accounts shall be available for the payment of such support  (g) EDUCATION AND TRAINING —The Secretary of Defense shall— (1) ensure that management, technical, and contracting personnel of the Department of Defense involved in the award or administration of transactions under this section or other innovative forms of contracting are afforded opportunities for adequate education and training; and (2) establish minimum levels and requirements for continuous and experiential learning for such personnel, including levels and requirements for acquisition certification programs  (h) REGULATIONS — The Secretary of Defense shall prescribe regulations to carry out this section  (i) Protection of Certain Information From Disclosure -(1) Disclosure of information described in paragraph (2) is not required, and may not be compelled, under Section 552 of Title 5 for five years after the date on which the information is received by the Department of Defense  (2)(A) Paragraph (1) applies to information described in subparagraph (B) that is in the records of the Department of Defense if the information was submitted to the Department in a competitive or noncompetitive process having the potential for resulting in an award, to the party submitting the information, of a cooperative agreement for performance of basic, applied, or advanced research authorized by Section 2358 of this title or another transaction authorized by subsection (a)  (B) The information referred to in subparagraph (A) is the following: (i) A proposal, proposal abstract, and supporting documents  (ii) A business plan submitted on a confidential basis  (iii) Technical information submitted on a confidential basis  10 U S C  §2371b  Authority of the Department of Defense to carry out certain prototype projects (a) AUTHORITY — (1) Subject to paragraph (2), the Director of the Defense Advanced Research Projects Agency, the Secretary of a military department, or any other official designated by the Secretary of Defense may, under the authority of Section 2371 of this title, carry out prototype projects that are directly relevant to enhancing the mission effectiveness of military personnel and the supporting platforms, systems, components, or materials proposed to be acquired or developed by the Department of Defense, or to improvement of platforms, systems, components, or materials in use by the armed forces  (2) The authority of this section- (A) may be exercised for a transaction (for a prototype project) that is expected to cost the Department of Defense in excess of $100,000,000 but not in excess of $500,000,000 (including all options) only upon a written determination by the senior procurement executive for the agency as designated for the purpose of Section 1702(c) of Title 41, or, for the Defense Advanced Research Projects Agency or the Missile Defense Agency, the director of the agency that- (i) the requirements of subsection (d) will be met; an d (ii) the use of the authority of this section is essential to promoting the success of the prototype project; and (B) may be exercised for a transaction (for a prototype project) that is expected to cost the Department of Defense in excess of $500,000,000 (including all options) only if- (i) the Under Secretary of Defense for Acquisition, Technology, and Logistics determines in writing that- (I) the requirements of subsection (d) will be met; an d (II) the use of the authority of this section is essential to meet critical national security objectives; and (ii) the congressional defense committees are notified in writing at least 30 days before such authority is exercised  (3) The authority of a senior procurement executive or director of the Defense Advanced Research Projects Agency or Missile Defense Agency under paragraph (2)(A), and the authority of the Under Secretary of Defense for Acquisition, Technology, and Logistics under paragraph (2)(B), may not be delegated  (b) EXERCISE OF AUTHORITY — (1) Subsections (e)(1)(B) and (e)(2) of such Section 2371 shall not apply to projects carried out under subsection (a)  (2) To the maximum extent practicable, competitive procedures shall be used when entering into agreements to carry out projects under subsection (a)  (c) COMPTROLLER GENERAL ACCESS TO INFORMATION — (1) Each agreement entered into by an official referred to in subsection (a) to carry out a project under that subsection that provides for payments in a total amount in excess of $5,000,000 shall include a clause that provides for the Comptroller General, in the discretion of the Comptroller General, to examine the records of any party to the agreement or any entity that participates in the performance of the agreement  (2) The requirement in paragraph (1) shall not apply with respect to a party or entity, or a subordinate element of a party or entity, that has not entered into any other agreement that provides for audit access by a Government entity in the year prior to the date of the agreement  (3) (A) The right provided to the Comptroller General in a clause of an agreement under paragraph (1) is limited as provided in subparagraph (B) in the case of a party to the agreement, an entity that participates in the performance of the agreement, or a subordinate element of that party or entity if the only agreements or other transactions that the party, entity, or subordinate element entered into with Government entities in the year prior to the date of that agreement are cooperative agreements or transactions that were entered into under this section or Section 2371 of this title  (B) The only records of a party, other entity, or subordinate element referred to in subparagraph (A) that the Comptroller General may examine in the exercise of the right referred to in that subparagraph are records of the same type as the records that the Government has had the right to examine under the audit access clauses of the previous agreements or transactions referred to in such subparagraph that were entered into by that particular party, entity, or subordinate element  (4) The head of the contracting activity that is carrying out the agreement may waive the applicability of the requirement in paragraph (1) to the agreement if the head of the contracting activity determines that it would not be in the public interest to apply the requirement to the agreement  The waiver shall be effective with respect to the agreement only if the head of the contracting activity transmits a notification of the waiver to Congress and the Comptroller General before entering into the agreement  The notification shall include the rationale for the determination  (5) The Comptroller General may not examine records pursuant to a clause included in an agreement under paragraph (1) more than three years after the final payment is made by the United States under the agreement  (d) APPROPRIATE USE OF AUTHORITY — (1) The Secretary of Defense shall ensure that no official of an agency enters into a transaction (other than a contract, grant, or cooperative agreement) for a prototype project under the authority of this section unless one of the following conditions is met: (A) There is at least one nontraditional defense contractor or nonprofit research institution participating to a significant extent in the prototype project  (B) All significant participants in the transaction other than the Federal Government are small businesses (including small businesses participating in a program described under Section 9 of the Small Business Act (15 U S C  638)) or nontraditional defense contractors  (C) At least one third of the total cost of the prototype project is to be paid out of funds provided by sources other than other than the Federal Government  (D) The senior procurement executive for the agency determines in writing that exceptional circumstances justify the use of a transaction that provides for innovative business arrangements or structures that would not be feasible or appropriate under a contract, or would provide an opportunity to expand the defense supply base in a manner that would not be practical or feasible under a contract  (2) (A) Except as provided in subparagraph (B), the amounts counted for the purposes of this subsection as being provided, or to be provided, by a party to a transaction with respect to a prototype project that is entered into under this section other than the Federal Government do not include costs that were incurred before the date on which the transaction becomes effective  (B) Costs that were incurred for a prototype project by a party after the beginning of negotiations resulting in a transaction (other than a contract, grant, or cooperative agreement) with respect to the project before the date on which the transaction becomes effective may be counted for purposes of this subsection as being provided, or to be provided, by the party to the transaction if and to the extent that the official responsible for entering into the transaction determines in writing that- (i) the party incurred the costs in anticipation of entering into the transaction; and (ii) it was appropriate for the party to incur the costs before the transaction became effective in order to ensure the successful implementation of the transaction  (e) Definitions —In this section: (1) The term "nontraditional defense contractor" has the meaning given the term under Section 2302(9) of this title  (2) The term "small business" means a small business concern as defined under Section 3 of the Small Business Act (15 U S C  632)  (f) FOLLOW-ON PRODUCTION CONTRACTS OR TRANSACTIONS — (1) A transaction entered into under this section for a prototype project may provide for the award of a follow-on production contract or transaction to the participants in the transaction  A transaction includes all individual prototype subprojects awarded under the transaction to a consortium of United States industry and academic institutions  (2) A follow-on production contract or transaction provided for in a transaction under paragraph (1) may be awarded to the participants in the transaction without the use of competitive procedures, notwithstanding the requirements of Section 2304 of this title, if - (A) competitive procedures were used for the selection of parties for participation in the transaction; and (B) the participants in the transaction successfully completed the prototype project provided for in the transaction  (3) Contracts and transactions entered into pursuant to this subsection may be awarded using the authority in subsection (a), under the authority of Chapter 137 of this title, or under such procedures, terms, and conditions as the Secretary of Defense may establish by regulation  (g) AUTHORITY TO PROVIDE PROTOTYPES AND FOLLOW-ON PRODUCTION ITEMS AS GOVERNMENT-FURNISHED EQUIPMENT — An agreement entered into pursuant to the authority of subsection (a) or a follow-on contract or transaction entered into pursuant to the authority of subsection (f) may provide for prototypes or follow-on production items to be provided to another contractor as Government-furnished equipment  (h) APPLICABILITY OF PROCUREMENT ETHICS REQUIREMENTS — An agreement entered into under the authority of this section shall be treated as a Federal agency procurement for the purposes of Chapter 21 of Title 41  10 U S C  §2373  Procurement for experimental purposes (a) AUTHORITY — The Secretary of Defense and the Secretaries of the military departments may each buy ordnance, signal, chemical activity, transportation, energy, medical, space-flight, and aeronautical supplies, including parts and accessories, and designs thereof, that the Secretary of Defense or the Secretary concerned considers necessary for experimental or test purposes in the development of the best supplies that are needed for the national defense  (b) PROCEDURES — Purchases under this section may be made inside or outside the United States and by contract or otherwise  Chapter 137 of this title applies only when such purchases are made in quantities greater than necessary for experimentation, technical evaluation, assessment of operational utility, or safety or to provide a residual operational capability 
Report 2:  The program has begun phase three, including approving 19 percent of Boeing’s phase three reports  Program Requirements  While the program manager told us that all of the requirements contribute to the safety of the commercial systems, safety officials are required to approve a subset of these requirements  Examples of requirements approved by safety officials include the ability to leave the spacecraft in an emergency or to abort a launch  When a contractor is ready for NASA to verify that it has met a requirement, the contractor submits data for NASA to review through a verification closure notice  We define “safety-specific notices” as those requiring safety officials’ approval  As shown in table 2, as of March 2018, the program had approved 2 percent of Boeing’s safety-specific notices and 0 percent of SpaceX’s safety-specific notices  Testing  The program also requires testing to verify and validate the crew transportation system  Agency officials emphasized the importance of testing to safety, stating that testing reduces uncertainty about a system’s performance and can uncover unknown problems  As noted above, both contractors will be conducting an uncrewed and a crewed flight test prior to being certified  While a certain level of risk needs to be accepted to conduct human spaceflight, these flight tests help to mitigate this risk by validating the integrated performance of the hardware and software  Agency and program officials stated that the contractors’ flight tests are critical evidence to support certification of a safe and reliable system  As evidenced by the data above, the program still has a significant amount of work ahead with respect to approving certification packages and closing hazard reports and verification closure notices  We have previously found that the program’s workload was an emerging schedule risk, and the contractors have continued to express concern about program officials’ ability to process and approve certification paperwork in a timely manner  Workload has also been a concern for two of NASA’s independent review organizations  For example, the Aerospace Safety Advisory Panel noted in its January 2018 annual report that the sheer volume of work that remains for the program in terms of closing hazard reports and verification closure notices is significant  In addition, the program’s safety and mission assurance office identified the upcoming bow wave of work in a shrinking time period as a top risk to achieving certification  One mechanism the program put in place to assess the overall safety of each spacecraft—loss of crew—has been a focus of the Aerospace Safety Advisory Panel, Members of Congress, our prior work, and the program itself  Loss of crew captures the probability of death or permanent disability to one or more crew members  It has received a lot of attention, in part, because it has been a top risk for the program since 2015  Specifically, the program has been concerned that neither contractor would be able to meet the contract requirement of a 1 in 270 probability of incurring loss of crew  We identified two key concerns with how NASA is using the loss of crew metric: (1) inconsistent approaches to assess the loss of crew metric and (2) no identified plan to share lessons learned about using the loss of crew metric as a safety threshold  A loss of crew value is generated through a probabilistic safety analysis, which models scenarios that could result in the loss of crew using various inputs  According to the program’s analysis, the probability of on-orbit debris damaging the vehicle has the greatest effect on a loss of crew value  This probability is informed by an orbital debris (debris) model, which was updated in 2014, after the loss of crew requirement was established  The updated debris model makes it harder to meet a loss of crew value, in part, because the modeling environment where the contractors’ systems will operate has changed  For example, the updated model includes a larger span of orbit, greater range of debris sizes, and the addition of material density classifications, which were not included in the former model  Further, the probabilistic safety analysis may include operational mitigations, such as on-orbit inspections that would include using cameras on the ISS to visually survey the spacecraft for damage, which, according to officials, makes it easier to meet a loss of crew value  NASA describes the probabilistic safety analysis as a powerful tool that should be used as part of the overall risk management process to ensure the risk associated with development and operation of a system is understood, evaluated, managed, and mitigated  However, we found differences in the approaches that officials plan to use to assess loss of crew as well as in the loss of crew value being measured that could limit the usefulness of this tool  Agency Certification  The agency certification review for each contractor will include an assessment of whether its crew transportation system meets a loss of crew threshold of 1 in 150 for missions to the ISS, which is based on a May 2011 safety memo from the Office of Safety and Mission Assurance  A loss of crew value with a higher denominator, such as 1 in 270, is harder to meet than with a lower denominator, such as 1 in 150  According to the Chief of the Office of Safety and Mission Assurance, he will assess the 1 in 150 threshold using a probabilistic safety analysis that includes the updated debris model and operational mitigations, such as the on- orbit inspections cited above  Program Office  According to program officials, they will assess whether either contractor meets a 1 in 270 loss of crew value based on a probabilistic safety analysis using the former debris model (not the updated model) and not including operational mitigations  Contracting Officer  According to the contracting officer, each contractor’s loss of crew requirement is 1 in 270 without including operational mitigations  The contracting officer stated that SpaceX’s contract requirement uses the updated debris model in the probabilistic safety analysis, whereas Boeing’s contract requirement uses the former debris model in the probabilistic safety analysis  Program’s Chief Safety and Mission Assurance Officer  According to the program’s chief safety and mission assurance officer, he will conduct a probabilistic safety analysis using the updated debris model and will not include operational mitigations to assess whether each contractor meets a 1 in 200 loss of crew value  This loss of crew value stems from a program update that occurred after the initial contracts were signed  These different approaches are summarized in table 3 below  Agency policy requires human spaceflight programs to set a safety threshold, which NASA did for the Commercial Crew Program when it identified the 1 in 150 loss of crew threshold in the May 2011 safety memo  Subsequently, the program set more rigorous loss of crew values in contract and program documents  NASA also updated the debris model, which we previously noted makes it more difficult to meet a loss of crew value  As a result, NASA does not have a consistent approach for how to incorporate key inputs to the probabilistic safety analysis, including changes to the debris model  Instead, the risk tolerance that NASA is accepting with loss of crew varies based upon which entity is presenting the results of its probabilistic safety analysis  For example, it is possible that the program’s assessment will determine that neither contractor will meet the 1 in 270 contract requirement, but that the agency’s assessment will determine that the contractors meet the 1 in 150 agency certification value because that analysis will include operational mitigations  Or the program’s assessment could determine that Boeing meets the 1 in 270 contractual loss of crew requirement, but the agency’s assessment may determine that Boeing does not meet the 1 in 150 agency certification value because that analysis will use the updated debris model  Federal internal controls state that agency management should define objectives clearly to enable the identification of risks and define risk tolerances  Specifically, management should define risk tolerances in specific and measurable terms, so they are clearly stated and can be measured  In this case, because there will be multiple analyses conducted using different inputs, NASA risks not clearly capturing or documenting, in a coherent manner, its overall risk tolerance with respect to loss of crew before a final decision must be made on whether to certify either crew transportation system  Moreover, capturing the challenges and lessons learned from using the loss of crew metric is critical, particularly because agency officials told us that this is the first time this metric has been used as a safety threshold  Also, there are different viewpoints about the utility of the metric as a safety threshold across the agency  The program manager repeatedly told us that loss of crew is best used as a design tool  For example, program officials told us that both contractors incorporated additional orbital debris shielding into their designs to mitigate the orbital debris risk and improve their loss of crew values  In addition, the Aerospace Safety Advisory Panel reported in 2018 that loss of crew should not be viewed as an absolute measure of actual risk during operations  However, the May 2011 agency safety memo states that a breach of the loss of crew threshold would initiate a termination review of the Commercial Crew Program, which is a more strict application of the loss of crew metric  Both program and safety officials told us that, after the agency certification is complete and lessons learned are available to be compiled, sharing those lessons learned across NASA would be a good idea given the complexities associated with assessing the loss of crew metric  As of April 2018, however, agency officials said they did not have a plan for loss of crew knowledge-sharing  We have previously found that lessons learned provide a powerful method of sharing good ideas for improving work processes, facility or equipment design and operation, quality, safety, and cost-effectiveness  Further, according to NASA’s Knowledge Policy on Program and Projects, which is managed through the Office of the Chief Engineer, a principle of each center and mission directorate’s knowledge strategy is that knowledge is the cornerstone of NASA’s ability to achieve mission success  The policy acknowledges that NASA faces continuous challenges in using what it knows effectively  These challenges include, but are not limited to, enabling the identification and flow of knowledge across organizational boundaries; preserving knowledge at risk of being lost; and providing means for individuals, teams, and the organization to learn from experiences  If NASA does not capture lessons learned from the Commercial Crew Program on using the loss of crew requirement to set a program’s safety threshold and whether it met the agency’s intended goal, future programs will not be able to benefit from the knowledge gained from this multibillion dollar investment  NASA’s governance model prescribes a management structure that employs checks and balances among key organizations to ensure that decisions have the benefit of different points of view and are not made in isolation  As part of this structure, NASA established the technical authority process as a system of checks and balances to provide independent oversight of programs and projects in support of safety and mission success through the selection of specific individuals with delegated levels of authority  The technical authority process has been used in other parts of the government for acquisitions, including the Department of Defense and Department of Homeland Security  The Commercial Crew Program is organizationally connected to three technical authorities within NASA: the Office of the Chief Engineer technical authority, the Office of Chief Health and Medical technical authority, and the Office of Safety and Mission Assurance (safety) technical authority  The safety technical authority is responsible for ensuring from an independent standpoint that the program’s products and processes satisfy NASA’s safety, reliability, and mission assurance policies  The NASA safety technical authority has delegated authority through the Kennedy Space Center Director to the Chief Safety and Mission Assurance Officer for the Commercial Crew Program  We have previously reviewed how NASA has organized its technical authorities for its Exploration Systems Development organization—an organization that oversees the development of the Space Launch System, Orion crew capsule, and associated ground systems that have the goal of extending human presence beyond low-Earth orbit  In October 2017, we found that the Exploration Systems Development organization had established an organizational structure in which the technical authorities for engineering and safety and mission assurance were dual hatted simultaneously in programmatic positions  We found that having the same individual simultaneously fill both a technical authority role and a program role created an environment of competing interests, where the technical authority’s ability to impartially and objectively assess the programs while at the same time acting on behalf of the Exploration Systems Development organization in programmatic capacities may be subject to impairments  We found that this was in contrast to a recommendation from the Columbia Accident Investigation Board report—the result of an in-depth assessment of the technical and organizational causes of the 2003 Space Shuttle Columbia accident—for NASA to establish a technical authority to serve independently of the Space Shuttle program, so that employees would not feel hampered to bring forward safety concerns or disagreements with programmatic decisions  The board’s findings that led to this recommendation included a broken safety culture in which it was difficult for minority and dissenting opinions to percolate up through the hierarchy; dual center and programmatic roles vested in one person that had confused lines of authority, responsibility, and accountability and made the oversight process susceptible to conflicts of interest; and oversight personnel in positions within the program, increasing the risk that these staffs’ perspectives would be hindered by too much familiarity with the programs they were overseeing  In October 2017, we recommended that the division no longer dual hat two individuals who had both programmatic and technical authority responsibilities  As of April 2018, NASA had taken steps to separate the engineering technical authority position from the programmatic position, and NASA’s Chief of Safety and Mission Assurance said he planned to separate the safety position but had not yet completed that action  The Commercial Crew Program employs a similar structure to the Exploration Systems Development organization in that the safety technical authority is dual hatted simultaneously in a programmatic position as the Commercial Crew Program’s Safety and Mission Assurance Manager  According to the program’s safety technical authority, in his programmatic role for the program, he helps set priorities for safety issues, including how staff will be utilized to meet those priorities  In the technical authority role, he provides independent oversight in support of safety and mission success  In his dual-hatted role, this official will be responsible for endorsing the program’s certification recommendations in two different capacities: as the technical authority and as a program authority  As a result, this structure relies on the same individual to completely separate two roles—one to manage the Commercial Crew Program’s safety issues within programmatic cost and schedule constraints, and the other to assess the same issues in an independent oversight role  While the Commercial Crew Program may have an additional level of separation between the safety technical authority and the program’s involvement in the design of commercial systems due to its shared assurance model with the commercial providers, the Commercial Crew Program still maintains a structure where one individual simultaneously serves in both technical authority and programmatic roles  Figure 5 describes some of the conflicting roles and responsibilities of this official in his two different positions  During our review, officials cited several factors in support of a dual- hatted approach: The safety technical authority retains independence because his technical authority reporting path and performance reviews are not under the purview of the Commercial Crew Program chain of command  Due to the Commercial Crew Program’s shared assurance model with commercial providers, the program is operating in a quality assurance role that provides an additional level of separation between the safety technical authority and the program’s involvement in the design of commercial systems  For safety decisions involving cost and schedule where the individual who is dual hatted with both technical authority and programmatic responsibilities may feel conflicted, he stated that he would discuss these matters with his management to validate the logic behind his decision  There are cost and knowledge efficiencies gained from one individual serving in both programmatic and safety technical authority capacities  NASA’s Chief of Safety and Mission Assurance stated that he has great confidence in the individual currently serving in the dual hatted role for the Commercial Crew Program, but acknowledged there is inherent conflict even with the program’s shared assurance model  In December 2017, he stated that, based on our previous work and current discussions, he intends to decouple the programmatic and technical authority responsibilities for the Commercial Crew Program but had not done so as of April 2018  Federal internal control standards state that an agency should design control activities to achieve objectives and respond to risks, which includes segregation of key duties and responsibilities to reduce the risk of error, misuse, or fraud  By overlapping technical authority and programmatic responsibilities, NASA will continue to run the risk of creating an environment of competing interests for the Commercial Crew Program’s safety technical authority  NASA’s Commercial Crew Program is a multibillion dollar effort to facilitate the commercial development of a crew transportation system that can end the United States’ reliance on Russia to maintain an uninterrupted presence on the ISS  Boeing and SpaceX continue to make progress developing a capability to fly to the ISS, but both have continued to experience delays  Program analysis indicates risks of further delays in each contractor’s current schedule, but NASA has not provided that information to Congress in its routine briefings  Without this information, Congress does not know the full extent of potential delays to inform decision making  Additional delays could also disrupt U S  access to the ISS  While NASA is working on potential solutions, there is no contingency plan in place to address this potential gap  Without a viable contingency plan, NASA puts at risk achievement of the U S  goal and objective for the ISS  NASA must balance safety with acceptable risk for human spaceflight  As part of the certification process for each contractor’s spacecraft, NASA has developed one key safety metric, loss of crew  However, the complicated nature of this metric is further muddled by the inconsistent approaches being used across NASA about what inputs to be considered  As a result, there is no clear articulation of what level of risk NASA will accept with respect to this program  In addition, NASA does not have plans to capture lessons learned from how the Commercial Crew Program has used this metric to assess safety and is missing an opportunity to capture this knowledge for future human spaceflight programs  Finally, a space program’s management and oversight approach is an integral part of ensuring that human spaceflight is as safe and successful as possible  Independence of the program management and oversight functions is key to achieving the balance between safety and success  The Commercial Crew Program’s approach, however, burdens the safety technical authority with both programmatic and independent technical authority responsibilities  As a result, NASA has limited assurance that independence can be maintained as part of its institutional process to ensure safety and success  We are making the following five recommendations to NASA: The NASA Associate Administrator for Human Exploration and Operations should direct the Commercial Crew Program to include the results of its schedule risk analysis in its mandatory quarterly reports to Congress  (Recommendation 1) The NASA Administrator should develop and maintain a contingency plan for ensuring a presence on the ISS until a Commercial Crew Program contractor is certified  (Recommendation 2) The NASA Administrator should direct the Chief of Safety and Mission Assurance, the NASA Associate Administrator for Human Exploration and Operations, the Commercial Crew Program Manager, and the Commercial Crew Program Contracting Officer to collectively determine and document before the agency certification review how the agency will determine its risk tolerance level with respect to loss of crew  (Recommendation 3) After completing the agency certification review, NASA’s Chief Engineer and Chief of Safety and Mission Assurance, with support from the NASA Associate Administrator for Human Exploration and Operations and the Commercial Crew Program Manager, should document lessons learned related to loss of crew as a safety threshold for future crewed spaceflight missions, given the complexity of the metric  (Recommendation 4) The NASA Chief of Safety and Mission Assurance should restructure the technical authority within the Commercial Crew Program to ensure that the technical authority for the Office of Safety and Mission Assurance is no longer dual hatted with programmatic and independent technical authority responsibilities  (Recommendation 5) We provided a draft of this report to NASA for review and comment  NASA provided written comments that are reprinted in appendix II  In its response, NASA concurred with three of our recommendations, did not concur with one, and partially concurred with another  NASA concurred with our recommendation to develop and maintain a contingency plan to ensure a U S  presence on the ISS and expects to take action to close this recommendation by the end of December 2018  NASA concurred with our recommendation to document lessons learned related to the loss of crew requirement and expects to take action to close this recommendation by the end of May 2019  NASA concurred with our recommendation to restructure the safety technical authority so that it is no longer dual hatted with programmatic and independent technical authority responsibilities  NASA expects to take action to close this recommendation by the end of August 2018  NASA did not concur with our recommendation that the Commercial Crew Program should include the results of its schedule risk analysis in its quarterly reports to Congress  NASA stated that it uses the contractors’ schedules as a baseline to provide qualitative statements in the NASA summary that accompanies each contractor’s quarterly reports to Congress  NASA believes that this approach is appropriate and is in accordance with the explanatory statement accompanying the Consolidated and Further Continuing Appropriations Act, 2015  NASA also stated that it will be working to ensure that the contractors’ schedules and the program’s internal assessments sync up as the program gets closer to launch  As a result, NASA explained that there will not be a requirement for a detailed NASA assessment, because the contractors’ schedule will either match NASA’s analysis or NASA will discuss its position as it has done in previous reports to Congress  We continue to believe the recommendation is valid because the program’s schedule risk analysis would provide Congress with valuable insight into potential delays, which are likely  Both contractors have repeatedly stated that their schedules are aggressive and that the dates are ambitious  As a result, we found that the contractors frequently delay dates for key events  For example, Boeing has delayed its certification milestone by 17 months and SpaceX by 22 months since the original schedules were established  The program’s recent schedule risk analysis indicates that more delays to certification are likely, but that information is not presented to Congress in NASA’s quarterly reports  Without this information, Congress does not know the full extent of potential delays to inform decision making  NASA partially concurred with our recommendation that the Chief of Safety and Mission Assurance, the NASA Associate Administrator for Human Exploration and Operations, the Commercial Crew Program Manager, and the Commercial Crew Program Contracting Officer should collectively determine and document how the agency will determine its risk tolerance level with respect to loss of crew before the agency certification review  In its response, NASA stated that it documented the agency’s risk tolerance level with respect to loss of crew for the program in its May 2011 safety memo  Further, NASA stated that it documented the requirement to limit risks to the loss of crew in a certification requirements document  NASA stated that ultimately the Commercial Crew Program is accountable for ensuring that the contractors’ systems meet the loss of crew value in this certification requirements document, which is a loss of crew value of 1 in 270  If a contractor’s system cannot meet that loss of crew value, or any other requirement, the program will request a waiver as part of the human rating certification process to ensure transparency  NASA acknowledged in its response that the existence of multiple documents defining residual risk requirements and an agency threshold for loss of crew can be confusing  NASA’s response, however, does not address our finding that it does not have a consistent approach for how to incorporate key inputs, including which debris model should be used or whether to include operational mitigations  NASA stated that it had taken action to address this recommendation; however, NASA did not outline any steps it took to resolve the concern that the risk tolerance for the loss of crew requirement depends on which entity is presenting the results of its analysis  We continue to believe that, before the agency certification review, the key parties must collectively determine how the agency will determine its risk tolerance with respect to loss of crew  We believe this approach will reduce confusion and increase transparency  We are sending copies of this report to NASA Administrator and interested congressional committees  In addition, the report is available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact me at (202)512-4841 or chaplainc@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made major contributions to this report are listed in appendix III  The objectives of our review were to assess (1) the extent to which the contractors have made progress towards meeting the National Aeronautics and Space Administration’s (NASA) certification requirements and NASA’s plans to help ensure continued access to the International Space Station (ISS); and (2) how NASA’s certification process addresses safety of the contractors’ crew transportation systems  To assess the contractors’ progress towards certification, we obtained and reviewed program and contractor documents, including monthly and quarterly updates from April 2017 through May 2018  We interviewed program and contractor officials to discuss the contractors’ recent progress, including upcoming events and any expected delays, and to understand technical risks, potential consequences, and planned mitigation activities  To identify total delays to date, we compared original contract schedules to Boeing and SpaceX’s calendar year 2018 first quarter proposed schedules, which are the most recent  Based on our review of program and contractor documents, we defined the contractors’ key events as: the uncrewed and crewed flight tests, the design certification reviews for each of those flights, and the certification milestone  We selected the two flight tests for each contractor as key events because they are intended to test key system capabilities, including the ability to launch, dock with the ISS, and return safely to Earth  We selected the design certification reviews because they verify the contractors’ crew transportation systems’ capability to safely approach, dock, mate, and depart from the ISS, among other requirements  We selected the certification milestone because it determines whether the crew transportation system meets the Commercial Crew Program’s requirements  To determine the extent to which contractors have delayed these key events over time, we analyzed the contractors’ schedule data from the 13 quarterly progress reports to date, from first quarter 2015 through first quarter calendar year 2018  We also obtained the results of the program’s April 2018 schedule risk analysis  We presented the schedule analysis range from the end of the month of the earliest possible completion date to the end of the month of the latest possible completion date  We reviewed the program’s Congressional requirements to report on cost, schedule, and technical status  Finally, to assess the potential effects of any certification delays on NASA’s access to ISS, we reviewed NASA’s contracts with Boeing and the Russian Federal Space Agency for transportation on the Soyuz vehicle  We interviewed officials from the ISS program and NASA’s Human Exploration and Operations Mission Directorate to determine if the agency had developed contingency plans to mitigate the effects of any certification delays on its access to the ISS  To assess how NASA’s certification process addresses safety of the contractors’ crew transportation systems, we reviewed agency safety policies, program plans, and contract documents to identify what safety assessments were required of the program, which safety factors would be considered in certification reviews, and when certification approval would be granted  We also interviewed program officials and the contractors about their safety policies and procedures as well as about the certification process  We identified the loss of crew requirements for the program and the contractors, and interviewed program and agency officials to determine how loss of crew would be assessed and considered throughout the certification process  To gain a broader understanding of the relative importance of loss of crew, we reviewed NASA safety policies, prior GAO reports, and annual reports from the Aerospace Safety Advisory Panel  To determine how the Orbital Debris Engineering Model (ORDEM) was updated and to assess differences between the former and updated models, we reviewed NASA documentation about the ORDEM 2000 and ORDEM 3 0 models and obtained information about the models from NASA’s Orbital Debris Program Office  To assess the program’s progress in closing requirements, including those requirements specifically related to safety, we reviewed program data for each contractor on contract requirements and closure status  We limited this analysis to requirements included in the CCT-REQ-1130 ISS Crew Transportation and Services Requirements Document because these requirements are verified by the Commercial Crew Program, whereas requirements contained within SSP 50808 ISS to Commercial Orbital Transportation Services Interface Requirements Document are managed by the ISS Transportation and Integration Office  We classified requirements as “safety-specific” when the program’s Office of Safety and Mission Assurance was listed as a verification closure notice signatory (i e , approver)  We then analyzed the program’s data to determine how many verification closure notices had been approved for all requirements and for the subset of safety-specific requirements  We reviewed program and agency documentation, such as organizational charts, program plans, and safety policies as well as interviewed program and agency officials, to determine the role of the safety and mission assurance technical authority in the program  We also reviewed the 2003 Columbia Accident Investigation Board’s Report’s findings and recommendations related to culture and organizational management of human spaceflight programs  We reviewed annual briefings and reports and met with representatives from two organizations that provide NASA with independent assessments of the program, the program’s standing review board and the Aerospace Safety Advisory Panel, to gain their perspectives on the contractor’s progress and how NASA addresses safety in its certification process  We also met with representatives from the National Transportation Safety Board and three experts with background on safety in human spaceflight in order to increase our contextual understanding of the role of safety in human spaceflight missions  We conducted this performance audit from April 2017 to July 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  Cristina T  Chaplain, (202) 512-4841 or chaplainc@gao gov  In addition to the contact named above, Molly Traci, Assistant Director; Kazue Chinen; Lorraine Ettaro; Lisa Fisher; Laura Greifner; Kurt Gurka; Miranda Riemer; Juli Steinhouse; Roxanna T  Sun; Hai Tran; Kristin Van Wychen; and Alyssa Weir made significant contributions to this report 

ground_truth: The Department of Defense (DOD) obligates more than $300 billion annually to buy goods and services, and to support research and development. Most of these acquisitions are governed by procurement statutes and regulations found in Title 10 (and parts of other select titles) of the United States Code, the Federal Acquisition Regulation (FAR), and the Defense Federal Acquisition Regulation Supplement. Under certain circumstances, DOD can enter into an other transaction (OT) agreement instead of a traditional contract. OT agreements are generally exempt from federal procurement laws and regulations. These exemptions grant government officials the flexibility to include, amend, or exclude contract clauses and requirements that are mandatory in traditional procurements (e.g., termination clauses, cost accounting standards, payments, audit requirements, intellectual property, and contract disputes). OT authorities also grant more flexibility to structure agreements in numerous ways, including joint ventures; partnerships; consortia; or multiple agencies joining together to fund an agreement encompassing multiple providers. Other transaction agreements are legally binding contracts; they are referred to as agreements to distinguish them from the traditional procurement contracts governed by the FAR and procurements laws. Other transaction authorities are set forth in two sections of law: 10 U.S.C. 2371—granting authority to use OTs for basic, applied, and advanced research projects. 10 U.S.C. 2371b—granting authority to use OTs for prototype projects and follow-on production. Under this authority, a prototype project can only be conducted if at least one nontraditional defense contractor significantly participates in the project; all significant participants are small businesses or nontraditional defense contractors; at least one-third of the total cost of the prototype project is provided by nongovernment participants; or the senior procurement acquisition official provides a written justification for using an OT. Follow-on production can only be conducted when the underlying prototype OT was competitively awarded, and the prototype project was successfully completed. OTs have the potential to provide significant benefits to DOD, including attracting nontraditional contractors with promising technological capabilities to work with DOD, establishing a mechanism to pool resources with other entities to facilitate development of, and obtain, state-of-the-art dual-use technologies, and offering a unique mechanism for DOD to invest in, and influence the direction of, technology development. A number of analysts warn that along with the potential benefits come significant risks, including potentially diminished oversight and exemption from laws and regulations designed to protect government and taxpayer interests. In FY2017, DOD obligated $2.1 billion on prototype OT agreements, representing less than 1% of contract obligations for the year. However, the use of OTs is expected to grow at a rapid pace, due in part to recent statutory changes expanding other transaction authorities. A number of analysts and officials have raised concerns that if DOD uses OTs in ways not intended by Congress—or is perceived to abuse the authority—Congress could clamp down on the authority. Generally, DOD lacks authoritative data that can be used to measure and evaluate the use of other transaction authorities.