prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 1.



Report 2: The RAD program was authorized by Congress and signed into law by the President in November 2011 under the Consolidated and Further Continuing Appropriations Act, 2012 with amendments in 2014, 2015, 2016, and 2017  The RAD program consists of two components  The first component of the RAD program—and the focus of our review— provides PHAs the opportunity to convert units subsidized under HUD’s public housing program and owned by the PHAs to properties with long- term (typically, 15–20 years) project-based voucher (PBV) or project- based rental assistance (PBRA) contracts  These are two forms of Section 8 rental assistance that tie the assistance to the unit to provide subsidized housing to low-income residents  In a RAD conversion, PHA- owned public housing properties can be owned by the PHA, transferred to new public or nonprofit owners, or transferred to private, for-profit owners when necessary to access LIHTC financing, if the PHA preserves its interest in the property in a HUD-approved manner  The second component of RAD converts privately owned properties with expiring subsidies under old rental assistance programs to PBV or PBRA in order to preserve affordability and encourage property rehabilitation  The goals of the RAD program include preserving the affordability of federally assisted rental properties and improving their physical and financial condition  Specifically, postconversion owners (PHAs, nonprofits, or for-profit entities) can leverage the subsidy payments under the newly converted contracts to raise capital through private debt and equity investments, or conventional private debt, to make improvements  The RAD program provides added flexibility for PHAs to access private and public funding sources to supplement public housing funding  These financing sources may include debt financing through public or private lenders; mortgage financing insured by FHA; PHA operating reserves; replacement housing factor funds; seller or take-back financing; deferred developer fees; equity investment generated by the availability of 4 percent and 9 percent LIHTC; or other private or philanthropic sources  PHAs also may pursue various options for their conversions, which often depend on property needs and available financing, including property rehabilitation or new construction  Additionally, PHAs may undertake conversion involving no property rehabilitation or new construction to meet certain financial goals or for future rehabilitation or new construction, as long as the PHA can demonstrate to HUD that the property does not need immediate rehabilitation and can be physically and financially maintained for the term of the Section 8 Housing Assistance Payment contract (HAP contract)  The RAD authorizing legislation and RAD Notice also specify requirements for ownership and control of converted properties  That is, converted properties must have public or nonprofit ownership or control, with limited exceptions  The RAD authorizing legislation, RAD Notice, HAP contracts, and RAD Use Agreement also establish procedures to help ensure that public housing remains a public asset should challenges arise, such as default, bankruptcy, or foreclosure  Oversight of RAD conversion and properties is primarily divided among three HUD offices  The Office of Recapitalization is responsible for administering the conversion process but generally does not oversee converted properties  Before conversion, the Office of Public and Indian Housing oversees the properties  After conversion, oversight remains with Public and Indian Housing for properties that convert to PBV contracts and transfers to the Office of Multifamily Housing Programs for PBRA  The RAD program has been implemented and expanded in phases  Since its authorization, the RAD unit cap gradually increased from 60,000 units in 2011 to 225,000 units in May 2017  The RAD program is currently fully subscribed with all 225,000 units allocated  As of September 30, 2017, 689 conversions were closed that involved a total of 74,709 units (see fig  1 for a breakdown by fiscal year)  Additionally, 706 conversions involving 79,078 units were in the process of structuring conversion plans  The remaining conversions under the cap were allocated to specific properties and in the process of having commitments issued or reserved under multi-phase or portfolio awards, according to HUD officials  RAD conversions begin with the submission of an application by PHAs after which they are notified of selection  The PHA is then required to submit a financing plan within 180 days or a later deadline based on the nature of the financing proposed  A RAD conversion is considered closed when the HAP contract is signed and financial documents are executed  The properties are considered converted to Section 8 assisted housing on the effective date of the HAP, which is generally the first day of the following month  Once the RAD conversion is closed, the PHA or ownership entity can move forward with its submitted proposals or RAD-related rehabilitation or new construction and is responsible for complying with RAD requirements and associated contracts  In some cases, rehabilitation can take place in advance of conversion closing if public housing funds are being used  Most RAD conversions involved some type of construction  Our analysis of HUD data showed that as of September 30, 2017 417 of 689 closed conversions (61 percent) involved planned rehabilitation to the property, 86 (12 percent) new construction, and 186 (27 percent) no construction; and 361 of 706 active RAD conversions (51 percent) involved planned rehabilitation, 89 (13 percent) new construction, and 256 (36 percent) no construction  HUD officials stated that they approve conversions that involve no immediate planned rehabilitation or new construction as long as the property has no immediate needs to be addressed  Such conversions allow PHAs to better position themselves to access additional capital to address future rehabilitation or construction plans  Our review of 31 conversion files also showed that the scope of proposed physical changes varied among RAD conversions  For properties that included scope of work narratives, physical changes included renovations to mitigate hazardous materials, aesthetic renovations, code and accessibility compliance, and construction of new buildings, among other changes  Financing for RAD conversions involved multiple public and private sources, but many conversions used LIHTC  Our analysis of HUD data showed that as of September 30, 2017, 173 of 689 closed RAD conversions (25 percent) utilized 4 percent LIHTC, 99 (14 percent) utilized 9-percent LIHTC, and 416 (60 percent) did not use LIHTC  By dollar amount, major financing sources were 4 percent LIHTC at $2 4 billion; new first mortgages at $1 8 billion; and 9 percent LIHTC at $1 1 billion  Construction costs constituted the highest-dollar use of financing for RAD conversions, but not all conversions incurred construction costs, as discussed earlier  On average, construction costs per closed conversion were $6 4 million (ranging from no construction costs to $236 million) and nearly $60,000 per-unit converted to RAD  Construction costs represented the highest-dollar use of financing for closed RAD conversions at $4 4 billion followed by building and land acquisition costs, and developer fees  For more information on financing sources and uses, see appendix II  PHA officials and developers we interviewed cited various factors that influence financing sources needed for RAD conversions  For example, property needs assessments help establish the level of rehabilitation or new construction that would address the capital needs of the property  In turn, needs assessments can derive from physical assessment results and incorporate federal, state, or local compliance requirements  For instance, rehabilitation or construction would need to address the accessibility requirements of the Americans with Disabilities Act and local building codes, among other requirements  PHA officials and developers we interviewed also said they had to consider competition or access to financing for RAD conversions  For example, PHAs noted that tax credit applications and other financing had to be competitive  Some PHAs we interviewed also noted that while the 9 percent LIHTC provides more equity to finance low-income units (finances 70 percent of the costs of the units), there is more competition for the 9 percent LIHTC, while the 4 percent LIHTC can be automatically awarded for certain deals involving tax exempt bonds and federally subsidized projects  Thus, while some PHAs and developers might prefer to obtain 9 percent LIHTC, they often apply for 4 percent LIHTC to increase the chances of obtaining some tax credit equity  For example, one particular PHA that had used both 4 percent and 9 percent LIHTCs noted that in one transaction it had to compete against 74 applicants for 25 available awards of 9 percent credits  The RAD authorizing statute requires HUD to assess and publish findings regarding the amount of private capital leveraged as a result RAD conversions  A leverage ratio relates the dollars other sources provide to the dollars a program provides to an institution or a project  HUD uses various quantitative, qualitative, and processing and efficiency metrics to measure conversion outcomes  To meet the RAD statutory requirement, HUD published an overall RAD leverage ratio that has fluctuated between 19:1 and 9:1 since 2014  HUD’s most recent leverage ratio in fiscal year 2017 was 19:1, nearly double what the agency reported the prior year  We asked HUD officials why the leverage ratio nearly doubled between 2016 and 2017 and received conflicting information during the course of our audit  Initially, officials noted that the ratio was intended to replicate the methodology used by PD&R in its September 2016 report  Subsequently, the officials clarified that they did not follow PD&R’s methodology for categorizing financial source data  Specifically, officials did not review or make manual adjustments to the financial data PHAs entered in open source fields to ensure sources actually represented public, private, or other funding categories when calculating the leverage ratio  Finally, they noted that they disagreed with the methodology used in the PD&R September 2016 report and stated that there are various ways to calculate leverage  For the purposes of announcing the most recent leverage ratio in 2017, HUD officials decided that a leverage ratio comparing federally appropriated public housing resources would reflect the amount of financing leveraged had RAD not existed  We found, and officials from HUD acknowledged, three limitations to the RAD leverage calculation  First, HUD generally had data on funding sources and amounts a RAD conversion proposed to use (at the time of its application to HUD and at the time of closing of construction financing) rather than data after construction is completed on funding sources and amounts  HUD officials stated that they were reviewing final closing packages to confirm that the data reflect the latest reported information on sources and uses of funds for each conversion at closing  However, sources and uses of funds and amounts at the time the RAD conversion is closed may differ from amounts upon completion of construction  In October 2017, HUD implemented procedures to verify completion of planned construction activities and costs, which we discuss later in this report  Second, HUD’s leverage ratio, published in 2017, did not manually adjust funding source data to accurately account for all sources in calculating the leverage ratio for RAD  Specifically, HUD did not isolate funding sources that were federally appropriated, contributed by the PHA, or contributed by state or local municipalities to calculate leverage  For example, among approximately $2 billion from other financial sources, HUD included Moving to Work (MTW) funding (which may include public housing capital funds, public housing operating funds, and voucher funds) and tax credit equity as leveraged sources  However, these are not necessarily private sources, which we explain later in this report  As a result, HUD’s current calculation does not reflect the amount of private- sector leveraging  HUD calculated and published a RAD leverage ratio in May of 2017 using the following formula: Total leverage ratio = (total dollars from all sources – public housing dollars) To calculate the RAD leverage ratio, HUD uses some but not all financial source data it collects (see app  II for a list of data fields collected by HUD)  For example, HUD mistakenly excluded data that capture private funds, reducing the amount of total sources in the numerator  HUD calculates “public housing dollars” by adding data that capture replacement factor funds, public housing operating reserve funds, and prior-year public housing capital funds  HUD considers tax credit equity, new first mortgages, and “other funding” data to be non-public housing dollars (see app  II for a list of fields in HUD’s calculation)  PHAs enter a description and amount for other funding sources in “other funding” data fields (see app  II)  For example, a PHA may enter a federal financial source in one of the open-entry “other funding” data fields, requiring a manual adjustment to properly account for the financial source  According to HUD, additional fields were included in mid-2016 to better differentiate certain sources such as from the HOME Investment Partnerships Program (HOME) and seller take-back financing  Prior to this point, these financial sources were placed into “other” fields, and the standard resource desk report had not been updated until mid-2017 to include all of these fields  Third, HUD does not categorize and report its leveraging by private and public sources  According to HUD officials, informative leverage methodologies could calculate the ratio based on the leveraging of public housing program funds, the leveraging of all federally appropriated funds, or the leveraging of PHA funds (i e , sources in the transaction that have come from the PHA itself even if not federally appropriated through the public housing program), among other methodologies  The RAD authorizing statute requires HUD to assess and publish findings on the amount of private-sector leveraging  In addition, Standards for Internal Control in the Federal Government require agencies to communicate quality information with external parties, such as other government entities, to make informed decisions and evaluate the entity’s performance in achieving key objectives  HUD also does not use final (postcompletion) funding data in another metric of RAD leveraging  Specifically, in June 2017 HUD publicly reported that RAD “has leveraged more than $4 billion in capital investment in order to make critical repairs and improvements ” HUD calculates this figure by summing the construction costs—a subcomponent of total costs—with data from the time a conversion closes and not upon completion of construction  HUD officials we spoke with clarified that this metric solely reports construction investments and does not reflect any conclusion regarding private leverage of public funds  But, HUD publically characterized this measure in different ways, including as the amount of “public-private investment in distressed public housing,” the amount of “construction achieved under RAD,” and the amount of “new private and public funds leveraged by RAD ” HUD’s 2016 interim report calculated and published multiple leverage ratios, but chose to highlight a RAD leverage ratio that is consistent with ratios used for other HUD programs  However, the ratio does not specifically follow the prescribed ratio language in the authorizing statute because the report states that the ratio represents the amount of private and public external sources invested for every dollar invested by PHAs but the statutory language only discusses private-sector leveraging  Officials further noted that the statute does not require a particular methodology and HUD relies on PD&R—and its independent contractor— to determine the appropriate methodology for purposes of compliance with the statute  Lastly, the statute does not preclude the use of other leverage metrics for other purposes, such as using the ratio to measure the amount of nonpublic housing funds leveraged in RAD transactions that would not be available to the property absent RAD  As a result, HUD’s leverage metrics announced in May 2017 do not accurately reflect the amount of private-sector leveraging achieved through RAD, do include public funding as private sources, and inconsistently measured sources that were federally appropriated or contributed by PHAs, potentially under- or over-reporting the program’s performance  Additionally, in October 2017, HUD began implementing procedures to collect data after construction is completed and is not yet able to calculate a leverage metric using final (postcompletion) financial sources rather than the financial sources collected at closing  The lack of a consistent metric for private leveraging could also lead to inconsistent reporting of the leverage ratio, as has occurred in prior years  We recalculated RAD leverage ratios in a number of different ways, including to correct errors we identified during our review  For example, HUD’s 2016 interim report noted that data on closed transactions do not provide detailed description of “other sources,” requiring a crosswalk between applications and closed transactions to develop estimates for the allocation of “other sources” across financial source categories  Abbreviated descriptions are provided in the form of notes that are not always clear and consistent; therefore public housing sources may include federally appropriated sources, as well state, city, or county sources  Through our estimates, we found that the overall leverage ratio could range from 7 44:1 for a ratio recalculating HUD’s leverage ratio to 1 23:1 for a ratio estimating private-sector leveraging  Recalculation with HUD methodology and financial source recategorization  As discussed previously, HUD’s methodology does not account for all financial data collected by HUD and includes “other” funding sources erroneously considered as leveraged funds  Thus, we manually adjusted RAD funding source data and found that nearly $1 2 billion were erroneously considered leveraged funds because they are not private funds  For example, HUD included MTW funds; public housing operating reserves; public housing capital funds; replacement housing factor funds; other federal funds; other state, local, or county funds; and take-back financing funds as leveraged financial sources  For more information, see appendix II  We obtained documentation from HUD to replicate their methodology and recategorized financial sources that corrected errors in the data, and found that the RAD leverage ratio was less than half of HUD’s most recently publicly reported leverage ratio (19:1), approximately 7 44:1 (see app  II)  Recalculation to exclude LIHTC and other federal sources  We previously reported that LIHTCs are considered a federal source because tax credit equity represents foregone federal tax revenue and, therefore, are a direct cost to the government  Accordingly, we recalculated the RAD leverage ratio by excluding all federal funding sources and obtained a ratio of approximately 1 43:1 (see app  II)  Recalculation of private-sector leveraging  Lastly, the RAD authorizing statute requires HUD to assess and publish findings on the amount of private-sector leveraging, but HUD’s current calculation does not present the amount of private-sector leveraging and does not include all available data (for example, the “Other Private” funds collected by HUD)  We estimated the amount of private-sector leveraging by grouping public housing sources, other public sources, and private sources, resulting in a leverage ratio of approximately 1 23:1 (see app  II)  In October 2017, HUD implemented procedures to certify completion after developers finish RAD-approved rehabilitation or construction  Previously, HUD had a limited ability to monitor and evaluate final (postcompletion) physical and financial changes in RAD projects with existing data  According to HUD officials, HUD did not implement completion certification procedures before October 2017 because it had been addressing what it considered to be the highest risks first (such as clarifying requirements for RAD participants, resident safeguards, and other procedural and administrative requirements)  HUD’s October 2017 completion certification procedures include instructions for owners to report final construction costs and documentation on completion of repairs or construction within 45 days of the completion date recorded in the RAD Conversion Commitment  More specifically, HUD requires owners to list a final construction cost amount—a subcomponent of total costs—in the RAD resource desk, describe variances from the approved construction cost amount in a comment box, and describe how increases in costs were addressed  Additionally, a third-party must certify that the repairs in the scope of work were completed by providing an attestation to HUD  However, HUD’s procedures do not require documentation from the owners to support the final total cost figures, which include not only construction costs but also building and land acquisition costs, and developer fees, among others as noted earlier in this report  These procedures also do not require a certification from owners on all financing sources and costs recorded in the RAD Conversion Commitment  Standards for Internal Control in the Federal Government require that management implement control activities through documented policies and procedures to provide reasonable assurance that the objectives of the agency will be achieved, and also communicate quality information with external parties to make informed decisions and evaluate the entity’s performance in achieving key objectives  While HUD now has certification completion procedures in place, this process provides the agency limited financial information from owners  As a result, HUD is unable to report metrics that reflect final (postcompletion) RAD financial outcomes after construction is completed  Furthermore, HUD is limited in its ability to effectively oversee conversion budget and cost variances, and expenditures that require HUD approval  Lastly, the RAD authorizing statute requires that the Secretary of HUD demonstrate the feasibility of the RAD conversion model to recapitalize and operate public housing properties under various situations and by leveraging other sources of funding to recapitalize properties  Without metrics that reflect the final (postcompletion) financial outcomes of RAD after construction is completed, HUD and congressional decisionmakers are unable to make informed decisions concerning the RAD program  HUD has not systematically tracked or analyzed household data on residents in RAD-converted units that are available from its public housing or Section 8 databases or from PHAs or other postconversion owners—the main sources of resident data for the RAD program  In addition, HUD has not yet developed monitoring procedures for all the resident safeguards in the RAD program  Finally, residents told us of some concerns about information they received on RAD conversions, communications opportunities, and the relocation process  HUD officials told us that the agency does not systematically track or analyze household-level data on residents in RAD-converted units across existing program databases (HUD maintains household data for the public housing and Section 8 rental assistance programs in two databases)  In particular, HUD does not track changes in household characteristics before and after conversion, such as changes in rent, as well as relocations or displacement of individual households  However, according to HUD officials, their databases are not designed to track the impact of RAD conversion on residents and they are unable to electronically link household information submitted before RAD conversion to information submitted after conversion  Once a property is converted, the property and corresponding household information are removed from the public housing database  Owners of converted properties are to use software to manually enter household information into the databases for the Section 8 program when submitting tenant certifications and information for assistance payments  This procedure is the standard for administration of all project-based Section 8 properties  HUD officials stated that they have explored the possibility of transferring household data from one system to another at the time of a property’s conversion  While HUD has not systematically analyzed household information from its public housing and Section 8 databases, we were able to perform a limited analysis  We requested and received data from HUD on the households affected by RAD  Using the data provided that were current as of June 2017, we were able to identify about 26,000 households that lived in units that were converted to a PBV subsidy, but we were unable to identify the total number of households converted to a PBRA subsidy  Based on our analysis of 26,000 PBV households, we found about 2,700 (about 11 percent of) households were headed by an about 6,800 (about 26 percent of) households were headed by an individual with a disability; about 2,700 households (about 10 percent of) households were headed by an elderly person who also had a disability; over half (about 14,000 or 54 percent) of the households were headed by an individual identified as black; close to 11,000 households (about 41 percent) were identified as white; and about 1,000 households (about 4 percent) were identified as Asian  Close to 3,100 households (about 12 percent) were headed by an individual identified as Hispanic; about half (about 49 percent) of the PBV households were single- person households; the median annual income of PBV households both before and after RAD conversion was about $10,000; and about 5,300 (about 20 percent) of households were paying a flat rent rather than income-based rent before RAD conversion  However, the data on PBV households were not comprehensive  For example, while about 10,000 residents (about 57 percent) experienced a rent increase following RAD conversion under PBV, we could not determine if the rent increase was the result of an increase in resident income  We also could not determine changes in location among the PBV households following RAD conversion  Rather than relying on the public housing and Section 8 databases for tracking household information during conversion, HUD officials indicated that the agency will rely on locally maintained resident logs, which contain household information collected by property owners, as the starting point when HUD determines a compliance review is warranted  The logs will be the primary way the agency collects household information for compliance reviews under the RAD program, according to HUD officials  In November 2016, HUD issued a notice, which requires the PHA or other postconversion owner to maintain a log about every household at a converting project, including information on race and ethnicity, household size, and disability  The notice also requires owners to track residence status throughout the relocation process, including whether the resident has returned, moved elsewhere, was permanently relocated or evicted; relocation dates; and details on any temporary housing and moving assistance provided  Owners are required to make the information available to HUD upon request for audits and other purposes  According to HUD officials, the agency expects the information in the resident logs to be more robust than what they would collect through the public housing and Section 8 databases, which do not track residents while they are relocated  HUD officials stated that the agency plans to review selected resident logs as part of an ongoing limited compliance review of about 90 RAD conversion projects  HUD officials told us they are developing procedures for performing compliance reviews—such as developing a mechanism to review a sample of logs on a periodic basis—but they have not yet done so because they have been focusing on developing procedures for activities that present a high risk to the program as described in the following section  HUD has not established a time frame for developing these procedures  However, HUD officials indicated that they plan to select resident logs for review based on risk of noncompliance and do not plan to analyze program-wide information currently collected in the public housing and Section 8 databases for program monitoring  HUD officials also noted that that PD&R is planning to track a sample of residents through its evaluation of the program, which we previously mentioned  While HUD has decided to rely on resident logs because of the difficulty of tracking household information across its program databases, using resident logs to assess the effects of the RAD program on residents has limitations  While the resident logs would contain detailed household information, they were not required prior to November 2016 and may not contain information on households converted before that date (RAD conversions started in 2013)  HUD’s public housing and Section 8 databases contain information on such households  Second, as previously mentioned, HUD plans to review resident logs only when there is a risk of noncompliance, but they collect household information in their databases on a rolling basis  Standards for Internal Control in the Federal Government require agencies to use quality information to achieve their objectives, and obtain and evaluate relevant and reliable data in a timely manner for use in effective monitoring  Without a comprehensive review of household information—one based on information in HUD data systems as well as resident logs—HUD cannot reasonably assess the effects of ongoing and completed RAD conversions on residents and compliance with resident safeguards, as discussed in the next section  HUD has not yet developed monitoring procedures for certain resident safeguards under the RAD program  RAD requirements include those intended to ensure that residents whose units are converted through RAD are informed about the conversion process; can continue to live in a converted property following RAD conversion; are afforded certain protections carried over from the public housing are afforded a phase-in of any rent increases under Section 8 program requirements  Currently, based on HUD notice requirements, PHAs must document compliance with three safeguards (PHA plan amendments, resident notification, and procedural rights) in their RAD application and other conversion paperwork  For example, PHAs must submit comprehensive written responses to resident comments received in connection with the required resident meetings with their RAD application  For one safeguard, PHAs are not required to report to HUD but must retain documentation of compliance to be made available to HUD as part of the monitoring for the program  For others, the HUD notice does not specify reporting and monitoring requirements  Based on our review of files for selected conversions, which we previously discussed, we found PHAs generally submitted documentation of their efforts to inform residents about RAD conversion, such as providing evidence to HUD of meetings with residents and written responses to resident questions as required  However, the specific documents for these requirements were not available from HUD in all cases  HUD’s review of amendments to PHA plans was documented in all but one of the conversions we reviewed  Documentation requirements for resident relocations have changed since RAD was introduced, which made the documentation more difficult to assess  HUD developed and started implementing procedures in October 2017 that require owners to certify and provide data supporting compliance with the resident right-to-return requirements  For example, owners must certify the number of residents who exercised their right to return to a converted property compared with the number of residents who did not return  HUD is also developing standard operating procedures to review each conversion for compliance with RAD relocation provisions  Specifically, the procedures would describe the review steps required at different stages of the conversion process, a process for identifying risks, and how to address instances of noncompliance with RAD requirements  Additionally, HUD noted that they have 2 compliance reviews under way including 1 involving a set of HUD requirements that affect relocations of more than 1 year and the limited compliance review of 90 projects that we previously described  HUD officials noted that they are developing additional guidance in other areas  First, HUD officials indicated that as part of an overall update of RAD standard operating procedures, they are developing additional protocols on resident notification and how residents’ comments are addressed through conversion planning  Second, the agency had not been consistently collecting required documentation on “house rules,” which describe the conditions and procedures for evicting residents and terminating assistance at RAD PBRA properties, so it has developed and implemented additional legal review procedures as part of the implementation of RAD resident eviction and grievance procedural rights requirements  According to HUD officials, they have been focusing primarily on right-to-return and relocation requirements because they represent areas of highest risk  HUD has not developed separate monitoring procedures for other resident safeguards—the phase-in of tenant rent increases, resident representation through tenant organizations, and choice mobility requirements  However, HUD officials told us that they plan to assess how administrative data can be used to monitor choice mobility as part of the planning for a separate PD&R evaluation of this safeguard  HUD officials also indicated that there are procedures for residents to report complaints to HUD if resident representation and organization requirements are not met  Standards for Internal Control in the Federal Government require agencies to implement control activities through documented policies and procedures to provide reasonable assurance that agency objectives will be achieved  These standards also require agencies to design procedures to achieve goals and objectives, and identify, analyze, and respond to risks related to achieving the defined objectives  Table 1 includes a description and information on implementation of resident safeguards that most directly affect residents’ experience with the conversion process and ability to live at the property following conversion  Appendix III describes these and other RAD resident safeguards  HUD officials indicated that the safeguards for the phase-in of tenant rent increases, resident representation, procedural rights, and choice mobility presented a lower risk than the right-to-return requirements, so they were a lower priority, and in some cases were addressed through general monitoring of the Section 8 program  For choice mobility options, HUD indicated that its data systems are not designed to track whether residents are able to exercise these options, such as tracking whether residents left a property to exercise choice mobility or for other reasons  All but two of the resident safeguards do not take effect until after a property has been converted and is part of the Section 8 program  For example, residents are only eligible to use vouchers through choice mobility after they have lived in the converted property for 1 or 2 years depending on the assistance contract involved (PBV or PBRA)  Moreover, certain RAD safeguards are not typically available for Section 8 residents  For example, RAD establishes resident representation provisions and procedural rights that are more in line with public housing rather than Section 8 requirements  While HUD has indicated that the Section 8 program has experience administering different types of assistance contracts, RAD nonetheless creates separate requirements for certain provisions from the public housing and Section 8 programs  As previously mentioned, RAD conversions have been completed at an increasing pace in the last 5 years  However, because HUD has not yet developed separate monitoring procedures for certain requirements—the phase-in of tenant rent increases, resident representation through tenant organizations, and choice mobility requirements, many of which take effect after a conversion—and without using all available household data, the agency will not be able to reasonably ensure that these safeguards were implemented  Residents who participated in our focus groups expressed some concerns about information they received on RAD conversions, communications opportunities, and the relocation process  Residents indicated that they were notified about RAD conversion in a variety of ways  Residents in 5 of 14 focus groups found the information presented to them on RAD to be helpful  Residents in 7 of 14 focus groups indicated that the information they received was not helpful  Across these focus groups, a range of concerns was expressed, including that the information provided was not always clear or reflective of the final changes resulting from RAD conversion, and that the PHA and management were not always forthcoming with information about the RAD changes  Residents in some focus groups also indicated that they were not involved in the RAD conversion  Residents in 5 of 14 groups indicated that they were not given the opportunity to provide input into the RAD changes, while residents in 6 of 14 groups indicated that their concerns were not addressed and their suggestions were not incorporated  Residents also described problems with relocations  Some of the concerns expressed by resident focus groups on relocation related to the location of the temporary units (3 of 14 focus groups), the timing of relocation or amount of notice given (7 of 14 focus groups), and moving issues (such as items damaged during moves)  Residents were asked to describe ways in which RAD conversion improved or harmed their living conditions  Residents in several focus groups indicated that RAD improved their living conditions, including both the condition (7 of 14 focus groups) and appearance of their units or the property in which they lived (6 of 14 focus groups)  Some of the changes residents liked included the installation of new appliances, mold and pest removal, and safety and energy efficiency improvements  However, residents in several of the focus groups identified problems with their living conditions following RAD conversion  The problems residents identified included security concerns (10 of 14 focus groups); renovations that were of poor quality (6 of 14 focus groups); and other problems with the units (10 of 14 focus groups), such as pest problems; decreased amenities (8 of 14 focus groups), such as the removal of common areas or in-unit washing machines; and issues with property management (11 of 14 focus groups)  For example, in several instances, residents stated that new managers or owners in place following RAD conversion were not responsive to their needs or concerns  During our site visits, residents described other experiences with RAD conversion  Residents in all of the groups identified being notified about RAD  Residents in 9 of 14 focus groups indicated that their rent was the same following RAD conversion  Residents in a few focus groups indicated that their rent had increased because of changes in their income or conversion from a flat rent  However, residents in a few focus groups experienced challenges in how their income was certified for the purpose of calculating rents, such as problems with requests for information (2 of 14 focus groups) and other issues with the process (4 of 14 focus groups)  For example, residents reported having to provide the same paperwork multiple times  No instances in which residents were permanently involuntarily displaced were reported  One resident organization expressed concerns about fewer eviction protections and resident representation after RAD conversion  We spoke with 18 PHAs, some of which cited benefits as well as several challenges of RAD participation and some noted HUD responsiveness to their circumstances and concerns  According to many of the PHAs we spoke with, benefits of participating in the RAD program included reducing administrative requirements in Section 8 programs and opening avenues for additional sources of funding  In particular, many of the PHAs noted that RAD allowed them to access tax credit equity and other funding to complete the bulk of their repairs and renovations at once  Over half of the PHAs we spoke with also found HUD to be flexible and responsive to individual PHA circumstances  The majority of PHAs we spoke with indicated that remaining in the public housing program was not tenable because funding for the public housing program was not enough to meet their long-term capital needs  PHAs we contacted also noted several challenges of participating in RAD: financing constraints, timing challenges, and evolving requirements  Financing constraints  Some PHAs noted that program rent requirements can limit PHA participation in RAD  Each year, HUD calculates a contract rent—the total rent for a unit, including operating subsidy and resident contribution  PHAs must use the contract rent to calculate Section 8 subsidies for properties converting under RAD  According to HUD and several PHAs, contract rents for RAD-converted Section 8 units are lower than rents in traditional Section 8 assisted units, and are almost always lower than market-rate rents  Several PHAs and HUD officials have described the difficulty of converting units from the public housing program with this rent limitation  For example, when the cost of needed rehabilitation or construction is high, low allowable contract rents might not be sufficient to access appropriate capital for the conversion  In certain localities, PHAs have found solutions to augment rents and have used RAD flexibilities to allow them to convert and plan for operating expenses  For example, the PHA in Tacoma, Washington, used the Moving to Work program flexibilities to increase contract rents and housing officials in San Francisco used an allowable procedure to transfer RAD assistance from converted buildings to properties throughout its portfolio (each is a blend of traditional project-based vouchers with higher contract rents and RAD assistance)  In Montgomery County, Maryland, the PHA similarly included RAD assistance in some mixed-finance properties that contain other high-rent subsidies and market-rate rents
Report 2: , freestanding medical schools)  Specific educational programs are often accredited by a specialized accrediting agency, and the institution at which the program is offered is accredited by a regional or national accrediting organization  Generally, an institution must be accredited by an ED-recognized accrediting agency that has the authority to cover all of the institution's programs  Alternatively, a public or private nonprofit IHE may be preaccredited by an agency recognized by ED to grant such preaccreditation, and a public postsecondary vocational institution may be accredited by a state agency that ED determines is a reliable authority  Proprietary institutions must be accredited by an ED-recognized accrediting agency  The accreditation process begins with an institution or program requesting accreditation  Institutional accreditation is cyclical, with a cycle ranging from every few years up to 10 years  Initial accreditation does not guarantee subsequent renewal of the accredited status  Typically, an institution seeking accreditation will first perform a self-assessment to determine whether its operations and performance meet the basic standards required by the relevant accrediting agency  Next, an outside group of higher education peers (e g , faculty and administrators) and members of the public conduct an on-site visit at the institution during which the team determines whether the accrediting organization's standards are being met  Based on the results of the self-assessment and site visit, the accrediting organization determines whether accreditation will be awarded, renewed, denied, or provisionally awarded to an institution  Educational programs within institutions can be accredited by programmatic accrediting agencies; however, a program is not required to be accredited by a programmatic accrediting agency for Title IV purposes  Rather, it only needs to be covered by the IHE's primary accrediting agency  Frequently, programmatic accrediting agencies review a specific program within an IHE that is accredited by a regional or national accrediting agency  An institution that has had its accreditation revoked or terminated for cause cannot be recertified as an IHE eligible to participate in Title IV programs for 24 months following the loss of accreditation, unless the accrediting agency rescinds the loss  The same rules apply if an institution voluntarily withdraws its accreditation  The Secretary can, however, continue the eligibility of a religious institution whose loss of accreditation, whether voluntary or not, is related to its religious mission and not to the HEA accreditation standards  If an institution's accrediting agency loses its recognition from ED, it has up to 18 months to obtain accreditation from another ED-recognized agency  Although the federal government does not set specific standards for institutional or programmatic accreditation, generally, it does require that institutions be accredited or preaccredited by a recognized accrediting organization to be eligible for Title IV participation  ED's primary role in accreditation is to recognize an accrediting agency as a "reliable authority regarding the quality of education or training offered" at IHEs through the processes and conditions set forth in the HEA and federal regulations  For ED recognition, Section 496 of the HEA specifically requires that an accrediting agency be a state, regional, or national agency that demonstrates the ability to operate as an accrediting agency within the relevant state or region or nationally  Additionally, agencies must meet one of the following criteria: IHE membership with the agency must be voluntary, and one of the primary purposes of the agency must be accreditation of the IHEs  The agency must be a state agency approved by the Secretary as an accrediting agency on or before October 1, 1991  The agency must either conduct accreditation through a voluntary membership of individuals in a profession, or it must have as its primary purpose the accreditation of programs within institutions that have already been accredited by another ED-recognized agency  Agencies that meet the first or third criterion listed above must also be administratively and financially separate and independent of any related trade association or membership organization  For an agency that meets the third criterion and that was ED-recognized on or before October 1, 1991, the Secretary may waive the requirement that the agency be administratively and financially independent of any related organization, but only if the agency can show that the existing relationship with the related organization has not compromised its independence in the accreditation process  All types of accrediting agencies must show that they consistently apply and enforce standards that ensure that the education programs, training, or courses of study offered by an IHE are of sufficient quality to meet the stated objectives for which the programs, training, or courses are offered  The standards used by the accrediting agencies must assess student achievement in relation to the institution's mission; this may include course completion, job placement rates, and passage rates of state licensing exams  Agencies must also consider curricula, faculty, facilities, fiscal and administrative capacity, student support services, and admissions practices  Accrediting agencies must also meet requirements that focus on the review of an institution's operating procedures, including reviewing an institution's policies and procedures for determining credit hours, the application of those policies and procedures to programs and coursework, and reviewing any newly established branch campuses  They must also perform regular on-site visits that focus on the quality of education and program effectiveness  The final component of the program integrity triad is eligibility and certification by ED  Here, ED is responsible for verifying an institution's legal authority to operate within a state and its accreditation status  ED also evaluates an institution's financial responsibility and administrative capability to administer Title IV student aid programs  An institution can be certified to participate in Title IV for up to six years before applying for recertification  ED determines an IHE's financial responsibility based on its ability to provide the services described in its official publications, to administer the Title IV programs in which it participates, and to meet all of its financial obligations  A public IHE is deemed financially responsible if its debts and liabilities are backed by the full faith and credit of the state or another government entity  A proprietary or private nonprofit IHE is financially responsible if it meets specific financial ratios (e g , equity ratio) established by ED, has sufficient cash reserves to make any required refunds (including the return of Title IV funds), is meeting all of its financial obligations, and is current on its debt payments  Even if an institution meets the above requirements, ED does not consider it financially responsible if the IHE does not meet third-party financial audit requirements or if the IHE violated past performance requirements, such as failing to satisfactorily resolve any compliance issues identified in program reviews or audits  Alternatively, if an institution does not meet the above standards of financial responsibility, ED may still consider it financially responsible or give it provisional certification, under which it may operate for a time, if it qualifies under an alternative standard  These alternative standards include submitting an irrevocable letter of credit to ED that is equal to at least 50% of the Federal Student Aid (FSA) program funds that the IHE received during its most recently completed fiscal year, meeting specific monitoring requirements, or participating in the Title IV programs under provisional certification  Along with demonstrating financial responsibility, an institution must demonstrate its ability to properly administer the Title IV programs in which it participates and to provide the education it describes in public documents (e g , marketing brochures)  Administrative capability focuses on the processes, procedures, and personnel used in administering Title IV funds and indicators of student success  Administrative capability standards address numerous aspects of Title IV administration  For example, to administer Title IV programs an institution must use ED's electronic processes and develop a system to identify and resolve discrepancies in Title IV information received by various institutional offices  The IHE must also refer cases of Title IV student fraud or criminal misconduct to ED's Office of Inspector General for resolution, and it must provide all enrolled and prospective students financial aid counseling  Finally, the IHE must have an adequate internal system of checks and balances that includes dividing the functions of authorizing payments and disbursing funds between two separate offices  Institutions are required to have a capable staff member to administer Title IV programs and coordinate those programs with other aid received by students  This person must also have an adequate number of qualified staff to assist with aid administration  Before receiving Title IV funds, an IHE must certify that neither it nor its employees have been debarred or suspended by a federal agency; similar limitations apply to lenders, loan servicers, and third-party servicers  Relating to indicators of student success, an institution must have satisfactory academic progress (SAP) standards for students receiving Title IV funds  In general, IHEs must develop SAP standards that establish a minimum grade point average (or its equivalent) for students and a maximum time frame in which students must complete their educational programs  A student who fails to meet the SAP requirements becomes ineligible to receive Title IV funds  Also related to student success indicators, an institution that seeks to participate in Title IV programs for the first time may not have an undergraduate withdrawal rate for regular students that is greater than 33% during its most recently completed award year  An institution may be deemed administratively incapable if it has a high cohort default rate (CDR)  In general, the CDR is the number of an IHE's federal loan recipients who enter repayment in a given fiscal year (the cohort fiscal year) and who default within a certain period of time after entering repayment (cohort default period; CDP), divided by the total number of borrowers who entered repayment in the cohort fiscal year  Since 2014, ED has used a three-year CDP in calculating an institution's CDR  An IHE will be found administratively incapable if one of the following conditions is met: 1  an institution's CDR is greater than 40% in one year for loans made under the FFEL and Direct Loans programs; 2  an institution's CDR is 30% or greater for each of the three most recent fiscal years for loans made under the FFEL and Direct Loans programs; or 3  an institution's CDR is 15% or greater in any single year for loans made under the Federal Perkins Loan Program  When an IHE is determined to be administratively incapable due to a high CDR, it may become ineligible to participate in the Direct Loan, Pell Grant, and/or Perkins Loan programs (but not other Title IV programs)  ED may grant provisional certification for up to three years to an institution that would be deemed administratively capable except for its high cohort default rates  If an institution is seeking initial certification, ED can grant it up to one year of provisional certification  ED can also grant an institution provisional certification for up to three years if ED is determining the IHE's administrative capacity and financial responsibility for the first time, if the IHE has experienced a partial or total change in ownership, or if ED determines that the administrative or financial condition of the IHE may hinder its ability to meet its financial responsibilities  Additionally, if an accrediting agency loses its ED recognition, any institution that was accredited by that agency may continue to participate in Title IV programs for up to 18 months after ED's withdrawal of recognition  To ensure that an institution is conforming to eligibility requirements, ED can conduct program reviews  During a program review, ED evaluates an institution's compliance with Title IV requirements and identifies actions the IHE must take to correct any problem(s)  Review priority is given to those institutions with high cohort default rates; IHEs with significant fluctuations in Pell Grant awards or Direct Loan volume that are not accounted for by changes in programs offered; IHEs that are reported to have deficiencies or financial aid problems by their state or accrediting agency; IHEs with high annual dropout rates; and IHEs determined by ED to pose a significant risk of failing to comply with the administrative capability or financial responsibility requirements  If, during a review, ED determines that an institution is not administratively capable or financially responsible or is violating Title IV program rules, ED may grant it provisional certification, take corrective actions, or impose sanctions  ED has the authority to impose a variety of sanctions and corrective actions on an institution that violates Title IV program rules, a Program Participation Agreement (discussed later in this report) or any other agreement made under the laws or regulations, or if it substantially misrepresents the nature of its educational programs, financial charges, or graduates' employability  Sanctions include fines, limitations, suspensions, emergency actions, and terminations  ED can also sanction third-party servicers performing tasks related to the institution's Title IV programs  ED may impose several types of sanctions on institutions for statutory and regulatory violations, including fines, limitations, and suspensions  ED can fine an institution up to $55,907 for each statutory or regulatory violation it commits, depending on the size of the IHE and the seriousness of the violation  Under a limitation, ED imposes specific conditions or restrictions on an institution related to its administration of Title IV funds  A limitation lasts for at least 12 months, and if an institution fails to abide by the limitation, ED may initiate a termination proceeding  Finally, under a suspension, an institution is not allowed to participate in Title IV programs for up to 60 days  Each of these sanctions may require an institution to take corrective actions as well, which may include repaying illegally used funds or making payments to eligible students from the IHE's own funds  ED can take emergency action to withhold Title IV funds from an institution if it receives reliable information that an IHE is violating applicable laws or regulations, agreements, or limitations  ED must determine that the institution is misusing federal funds, that immediate action is necessary to stop misuses, and that the potential losses outweigh the importance of using established procedures for limitation, suspension, or termination  An emergency action suspends an institution's participation in Title IV programs and prohibits it from disbursing such funds  Typically, the emergency action may not last more than 30 days  The final action ED can take is the termination of an institution's participation in Title IV programs  Generally, an institution that has had its participation terminated cannot reapply to be reinstated for at least 18 months  To request reinstatement, an institution must submit a fully completed application to ED and demonstrate that it has corrected the violation(s) for which its participation was terminated  ED may then approve, approve subject to limitations, or deny the institution's request  Several other requirements affect institutional eligibility for Title IV programs  Some of these requirements include institution Program Participation Agreements, which include provisions related to incentive compensation and campus crime reporting requirements; return of Title IV funds; and distance education  The failure to meet the requirements for any of these may result in the loss of Title IV eligibility or other sanctions  HEA Section 487 specifies that each institution wanting to participate in Title IV student aid programs is required to have a current Program Participation Agreement (PPA)  A PPA is a document in which the institution agrees to comply with the laws, regulations, and policies applicable to the Title IV programs; it applies to an IHE's branch campuses and locations that meet Title IV requirements, as well as its main campus  It also lists all of the Title IV programs in which the IHE is eligible to participate, the date on which the PPA expires, and the date on which the IHE must reapply for participation  By signing a PPA, an institution agrees that it will act as a fiduciary responsible for properly administering Title IV funds, will not charge students a processing fee to determine a student's eligibility for such funds, and will establish and maintain administrative and fiscal procedures to ensure the proper administration of Title IV programs  The PPA reiterates many provisions required for institutional eligibility and ED certification discussed earlier in this report and contains several additional notable requirements that may affect an IHE's Title IV eligibility, which are described below  Along with the general participation requirements with which an institution must comply, a PPA may also contain institution-specific requirements  As part of their PPAs, domestic and foreign proprietary IHEs must agree to derive at least 10% of their revenue from non-Title IV funds (i e , no more than 90% of their revenue can come from Title IV funds)  This is known as the 90/10 rule  Examples of non-Title IV funds include private education loans and some military and veterans' benefits, such as benefits provided under the Post-9/11 GI Bill program  If an IHE violates the 90/10 rule in one year, it does not immediately lose its Title IV eligibility  Rather, it is placed on a provisional eligibility status for two years  If the IHE violates the 90/10 rule for two consecutive years, it loses its eligibility for at least two years  In a PPA, an IHE must agree it will not provide any commission or incentive compensation to individuals based directly or indirectly on their success in enrolling students or the enrolled students' obtaining financial aid; however, some exceptions apply to this general rule  For instance, IHEs can provide incentive compensation to individuals for the recruitment of foreign students who are ineligible to receive Title IV funds or they can provide incentive compensation through a profit-sharing plan  The ban on incentive compensation only applies to the activities of securing enrollment (recruitment) and securing financial aid  Other activities are not banned, and ED draws a distinction between activities that involve directly working with individual students and policy-level determinations that affect recruitment and financial aid awards  For instance, an individual who is responsible for contacting potential student applicants or assisting students in filling out an enrollment application cannot receive incentive compensation, but an individual who conducts marketing activities, such as the broad dissemination of informational brochures or the collection of contact information, can receive incentive compensation  HEA Section 485(f), referred to as the Clery Act, requires domestic Title IV participating IHEs (1) to report to ED campus crime statistics and (2) establish and disseminate campus safety and security policies  Both the campus crime statistics and campus safety and security policies must be compiled and disseminated to current and prospective students and employees in an IHE's annual security report (ASR)  Campus crime statistics required to be reported to ED and included in an ASR include data on the occurrence on campus of a range of offenses specified in statute, including murder, burglary, robbery, domestic violence, rape, and other forms of sexual violence  In addition to campus crime statistics, ASRs must include statements of campus safety and security policies regarding, for example, procedures and facilities for students and others to report criminal actions or other emergencies occurring on campus and an IHE's response to such reports; security and access to campus facilities; campus law enforcement, including the law enforcement authority of campus security personnel, and the working relationship between campus security personnel and state and local law enforcement; programs designed to inform students and employees about the prevention of crimes; and the possession, use, and sale of alcoholic beverages and illegal drugs; enforcement of state underage drinking laws; enforcement of federal and state drug laws; and any drug or alcohol abuse education programs required under the HEA  An ASR must also include statements of policies specifically relating to incidence of domestic and sexual violence  For example, an ASR must include statements of policy regarding programs to prevent such incidents; procedures a victim should follow if such an incident as occurred; procedures an IHE will follow once such an incident has been reported and procedures for institutional disciplinary actions in cases of alleged incidents (including a statement of the standard of evidence that will be used in any school proceeding arising from the incident report); and possible sanctions and protective measures that an IHE may impose following a final determination in an institutional proceeding regarding such incidences  The Clery Act prohibits the Secretary of Education from requiring IHEs to adopt particular policies, procedures, or practices; and prohibits retaliation against anyone exercising his or her rights or responsibilities under the act  HEA Section 484B specifies that when a Title IV aid recipient withdraws from an IHE before the end of the payment or enrollment period for which funds were disbursed, Title IV funds must be returned to ED according to a statutorily prescribed schedule  In general, when a student withdraws from an IHE, an IHE first determines the portion of Title IV aid considered to be "earned" by the student while enrolled and the portion considered to be "unearned " Unearned aid must be returned to ED  Up to the 60% point of a payment or enrollment period, unearned funds must be returned on a pro rata schedule  After the 60% point of a payment or enrollment period, the total amount of funds awarded is considered to have been earned by the student and no funds are required to be returned  Whether an IHE and/or the student is required to return the funds to ED depends on a variety of circumstances, including whether Title IV funds have been applied directly to a student's institutional charges  Unearned funds must be returned to their respective programs in a specified order, with loans being returned first, followed by Pell Grants, and then other Title IV aid  In some instances, a student may have earned more aid than has been disbursed, and the difference is disbursed to the student after the student withdraws  Generally, distance education and correspondence education refers to educational instruction with a separation in time, place, or both between the student and instructor  It is a way in which institutions can increase student access to postsecondary education by offering alternatives to traditional on-campus instruction  Recently, due to the greater availability of new technologies, there has been substantial growth in the amount and types of courses institutions offer  Section 103(7)(A) and (B) of the HEA and the accompanying regulations define distance education as instruction that uses "(1) the internet; (2) one-way and two-way transmissions through open broadcast, closed circuit, cable, microwave, broadband lines, fiber optics, satellite, or wireless communications devices; [or]     (3) audio conferencing" to deliver instruction to students separated from the instructor  A course taught through a video cassette, DVD, or CD-ROM is considered a distance education course if one of the above-mentioned technologies is used to support student-instructor interaction  Regardless of the technology used, "regular and substantive interaction between the students and the instructor" must be ensured  Correspondence courses are expressly excluded from the definition of distance education  A correspondence course is one for which an institution provides instructional materials and exams for students who do not physically attend classes at the IHE, but does not include those courses that are delivered with "regular and substantive interaction between the students and the instructor" via one of the above-described technologies  In 1992, partially in response to cases of some correspondence institutions' fraudulent and abusive practices used to attract unqualified students to enroll in programs of poor or questionable quality, Congress incorporated provisions referred to as the "50% rules" into the HEA  The rules affected both the eligibility of institutions offering correspondence courses and their students' eligibility for Title IV aid  In general, under the rules, an institution is ineligible for Title IV aid if more than 50% of its courses are offered by correspondence, or if 50% or more of its students are enrolled in correspondence courses  As discussed earlier in this report, rules promulgated in 2016 would have required an IHE offering postsecondary distance or correspondence education in a state in which it is not physically located to meet any state authorization requirements within that state  Under the regulations, an IHE could meet this requirement if it participates in a state authorization reciprocity agreement  These regulations were scheduled to become effective July 1, 2018  However, on July 3, 2018 (and effective June 29, 2018), the Secretary of Education issued a final rule delaying the implementation of these requirements until July 1, 2020  The distinction between distance education and traditional instruction is also important for the purposes of Title IV program eligibility  Distance education programs provided by domestic IHEs are eligible for Title IV participation if they have been accredited by an accrediting agency recognized by ED to evaluate distance education programs  A program offered by a foreign IHE, in whole or in part, through distance education (including telecommunications) or correspondence is ineligible for Title IV participation 

ground_truth: In 2014, NASA awarded two firm-fixed-price contracts to Boeing and SpaceX, worth a combined total of up to $6.8 billion, to develop crew transportation systems and conduct initial missions to the ISS. In February 2017, GAO found that both contractors had made progress, but their schedules were under mounting pressure. The contractors were originally required to provide NASA all the evidence it needed to certify that their systems met its requirements by 2017. A House report accompanying H.R. 5393 included a provision for GAO to review the progress of NASA's human exploration programs. This report examines the Commercial Crew Program, including (1) the extent to which the contractors have made progress towards certification and (2) how NASA's certification process addresses safety of the contractors' crew transportation systems. GAO analyzed contracts, schedules, and other documentation and spoke with officials from NASA, the Commercial Crew Program, Boeing, SpaceX, and two of NASA's independent review bodies that provide oversight. Both of the Commercial Crew Program's contractors, Boeing and Space Exploration Technologies Corporation (SpaceX), are making progress finalizing designs and building hardware for their crew transportation systems, but both contractors continue to delay their certification milestone (see figure). Certification is the process that the National Aeronautics and Space Administration (NASA) will use to ensure that each contractor's system meets its requirements for human spaceflight for the Commercial Crew Program. Further delays are likely as the Commercial Crew Program's schedule risk analysis shows that the certification milestone is likely to slip. The analysis identifies a range for each contractor, with an earliest and latest possible completion date, as well as an average. The average certification date was December 2019 for Boeing and January 2020 for SpaceX, according to the program's April 2018 analysis. Since the Space Shuttle was retired in 2011, the United States has been relying on Russia to carry astronauts to and from the International Space Station (ISS). Additional delays could result in a gap in U.S. access to the space station as NASA has contracted for seats on the Russian Soyuz spacecraft only through November 2019. NASA is considering potential options, but it does not have a contingency plan for ensuring uninterrupted U.S. access. NASA's certification process addresses the safety of the contractors' crew transportation systems through several mechanisms, but there are factors that complicate the process. One of these factors is the loss of crew metric that was put in place to capture the probability of death or permanent disability to an astronaut. NASA has not identified a consistent approach for how to assess loss of crew. As a result, officials across NASA have multiple ways of assessing the metric that may yield different results. Consequently, the risk tolerance level that NASA is accepting with loss of crew varies based upon which entity is presenting the results of its assessment. Federal internal controls state that management should define risk tolerances so they are clear and measurable. Without a consistent approach for assessing the metric, the agency as a whole may not clearly capture or document its risk tolerance with respect to loss of crew. GAO is making five recommendations, including that NASA develop a contingency plan for ensuring a U.S. presence on the ISS and clarify how it will determine its risk tolerance for loss of crew. NASA concurred with three recommendations; partially concurred on the recommendation related to loss of crew; and non-concurred with a recommendation to report its schedule analysis to Congress. GAO believes these recommendations remain valid, as discussed in the report.

prompt_front: You are given two reports by the government agency.



Question: Write a one-page summary of the report 2.

Report 1: NASA’s Commercial Crew Program is a multi-phased effort that began in 2010  Across the phases, NASA has engaged several companies, using both agreements and contract vehicles to develop and demonstrate crew transportation capabilities  As the program has passed through these phases, NASA has generally narrowed down the number of participants  The early phases of the program were under Space Act agreements, which is what NASA calls the agreements entered into pursuant to its other transaction authority  These types of agreements are generally not subject to the Federal Acquisition Regulation (FAR) and allow the government and its contractors greater flexibility in many areas  Under these Space Act agreements, NASA relied on the commercial companies to propose specifics related to their crew transportation systems, including their design, the capabilities they would provide, and the level of private investment  In these phases, NASA provided technical support and determined whether the contractors met certain technical milestones  In most cases, NASA also provided funding  For the final two phases of the program, NASA awarded FAR-based contracts  By using FAR-based contracts, NASA gained the ability to procure missions to the ISS, while continuing to provide technical expertise and funding to the contractors  NASA levied two sets of requirements on the contractors: the ISS program requirements, which must be met by all spacecraft visiting the ISS whether they carry cargo or crew; and the Commercial Crew Program requirements, which have a focus on system capabilities and safety rather than design  The program also established a verification closure notice process, in which the contractors submit data to NASA to verify they have met all the requirements to be certified  This certification must occur before contractors are allowed to fly initial crewed missions to the ISS  In September 2014, NASA awarded firm-fixed-price contracts to Boeing and SpaceX, valued at up to $4 2 billion and $2 6 billion, respectively, for the Commercial Crew Transportation Capability phase  Under a firm- fixed-price contract, the contractor must perform a specified amount of work for the price negotiated by the contractor and government  This is in contrast to a cost-reimbursement contract, in which the government generally agrees to pay the contractor’s allowable costs regardless of whether work is completed  During this phase, the contractors will complete development of crew transportation systems  Boeing’s spacecraft—CST-100 Starliner—is composed of a crew module and a service module  The crew module will carry the crew and cargo  It also includes communication systems, docking mechanisms, and return systems for Earth landing  The service module provides propulsion on-orbit and in abort scenarios as well as radiators for thermal control  SpaceX’s spacecraft—Dragon 2—is composed of a capsule, which we refer to as the crew module, and a trunk, which we refer to as the support module  The crew module is composed of a pressure section and a service section  This module will carry the crew and cargo  It also includes avionics, docking mechanisms, and return systems for a water landing  The support module includes solar arrays for on-orbit power and guidance fins for escape abort scenarios  Figure 1 shows the spacecraft and launch vehicles for Boeing and SpaceX’s crew transportation systems  The Commercial Crew Transportation Capability phase contracts include three types of services: Contract Line Item 001 encompasses the firm-fixed-price design, development, test, and evaluation work needed to support NASA’s certification of the contractor’s spacecraft, launch vehicle, and ground support systems  Contract Line Item 002 covers any service missions that NASA orders to transport astronauts to and from the ISS  Under this indefinite-delivery, indefinite-quantity line item, NASA has ordered six post-certification missions from each contractor  Each service mission is its own firm-fixed-price task order  NASA must certify the contractors’ systems before they can fly these missions  Contract Line Item 003 is an indefinite-delivery, indefinite-quantity line item for any special studies, tests, and analyses that NASA may request  These tasks do not include any work necessary to accomplish the requirements under contract line item 001 and 002  As of April 2018, NASA had funded studies worth approximately $30 million to Boeing, including approximately $27 million for additional testing of the parachute system  NASA had funded studies worth approximately $44 million to SpaceX, including approximately $34 million for additional testing of the parachute system  For each contractor, the maximum value of this contract line item is $150 million  NASA has made changes to the contracts that have increased their value  While the contracts are fixed-price, their values can increase if NASA adds work or otherwise changes requirements, among other means  As of April 2018, NASA requirement changes had increased the value of contract line item 001 for Boeing by approximately $191 million and for SpaceX by approximately $91 million  NASA divided the certification work under contract line item 001 into two acceptance events: the design certification review and the certification milestone  An acceptance event occurs when NASA approves a contractor’s designs and acknowledges that the contractor’s work is complete and meets the requirements of the contract  The first acceptance event—the design certification review—verifies the contractor’s crew transportation system’s capability to safely approach, dock, mate, and depart from the ISS, among other requirements  After the contractor has successfully completed all of its flight tests, as well as various other activities, the second acceptance event—the certification milestone—determines whether the crew transportation system meets the Commercial Crew Program’s requirements  Following this contract milestone is an agency certification review, which authorizes the use of a contractor’s system to transport NASA crew to and from the ISS  Figure 2 shows a notional path leading up to the agency certification review  The Commercial Crew Program’s certification plan outlines how the program will incrementally review required deliverables leading up to, and supporting, the agency certification review  For each review, the plan describes the information that the contractor and the program will present  At the agency certification review, which is chaired by the Associate Administrator of the Human Exploration and Operations Mission Directorate, the agency will review the program’s formal recommendation to certify the contractor’s crew transportation system  Program officials said that their goal is to develop and review certification evidence incrementally in order to reduce the risk that issues will be identified during the agency certification review  In our February 2017 report, we evaluated the progress made by the two contractors on the Commercial Crew Program and found the following: Both of the Commercial Crew Program’s contractors had made progress developing their crew transportation systems, but both also had aggressive development schedules that were increasingly under pressure  We reported that both Boeing and SpaceX had determined that they would not be able to meet their original 2017 certification dates, and both expected certification to be delayed until 2018  We found that the schedule pressures were amplified by NASA’s need to provide a viable crew transportation option because its contract with Russia’s space agency was to provide crew transportation to the ISS for six astronauts through 2018 with rescue and return through late spring 2019  Purchasing additional seats from Russia involves a contracting process that typically takes 3 years  Without a viable contingency option for ensuring uninterrupted access to the ISS in the event of further Commercial Crew delays, we concluded that NASA was at risk of not being able to maximize the return on its multibillion dollar investment in the space station  The Commercial Crew Program was using mechanisms laid out in its contracts to gain a high level of visibility into the contractors’ crew transportation systems, but maintaining that level of visibility through certification could add schedule pressures  We noted that, for example, due to NASA’s acquisition strategy for this program, its personnel were less involved in the testing, launching, and operation of the crew transportation system  While the program developed productive working relationships with both contractors, obtaining the level of visibility that the program required had also taken more time than the program or contractors had anticipated  Ultimately, we noted that the program had the responsibility for ensuring the safety of U S  astronauts, and its contracts gave it deference to determine the level of visibility required to do so  We concluded that the program office could face difficult choices moving forward about how to maintain the level of visibility it feels it needs without adding to the program’s schedule pressures  In order to ensure that the United States had continued access to the ISS if the Commercial Crew Program’s contractors experienced additional schedule delays, we recommended in our February 2017 report that the NASA Administrator develop a contingency plan for maintaining a presence on the ISS beyond 2018, including options to purchase additional Russian Soyuz seats, and report to Congress on the results  NASA concurred with this recommendation, and in February 2017, NASA executed a contract modification that purchased two seats and included an option to purchase three additional crewmember seats from Boeing on the Russian Soyuz vehicle  These seats represent a contingency plan for U S  access to the ISS through 2019  In April 2017, NASA informed the Congress of this action  Boeing and SpaceX continue to make progress developing their crew transportation systems, but both contractors have further delayed the certification milestone to early 2019  These changes have occurred as the contractors continue to work to aggressive schedules, and they have had to delay key events regularly  Further delays are likely as the Commercial Crew Program’s schedule risk analysis shows that the certification milestone is likely to further slip  In addition, as of mid-June 2018, NASA officials told us that these dates may change soon but that both contractors have not yet provided official updates to their schedules to NASA  NASA has not fully shared information with Congress regarding the risks of future schedule delays for the contractors and, as a result, Congress lacks insight into when the contractors will be certified  Also, there may be a gap in access to the ISS if the Commercial Crew Program experiences additional delays  While NASA has begun to discuss potential options, it currently does not have a contingency plan for how to ensure an uninterrupted presence on the ISS beyond 2019  Boeing and SpaceX have continued to make progress finalizing their designs and building hardware as they work toward their certification milestones  The contractors are manufacturing test articles to demonstrate system performance and flight spacecraft to support the uncrewed and crewed flight tests, which are expected to demonstrate the ability to meet contract requirements  As table 1 shows, these test articles and spacecraft vary in levels of completion  Some are built and undergoing testing while others are starting the manufacturing phase  Should any issues arise during integration and test or the flight tests, the contractors may have to complete rework on the spacecraft already under construction  While both contractors are making progress, the Commercial Crew Program is tracking risks that each contractor has to address through testing and other means as they work towards the certification milestone  As we have previously reported, these types of risks are inherent in NASA’s major acquisitions, which are highly complex, specialized, and often pushing the state of the art in space technology, but they could also delay the contractors’ progress if issues arise during testing  The Commercial Crew Program’s top programmatic risks identified for Boeing include challenges related to its abort system performance, parachutes, and launch vehicle  Abort System: Boeing is addressing a risk that its abort system, which it needs for human spaceflight certification, may not meet the program’s requirement to have sufficient control of the vehicle through an abort  In some abort scenarios, Boeing has found that the spacecraft may tumble, which could pose a threat to the crew’s safety  To validate the effectiveness of its abort system, Boeing has conducted extensive wind tunnel testing and plans to complete a pad abort test in July 2018  Parachute System: Boeing is also addressing a risk that during descent, a portion of the spacecraft’s forward heat shield may re- contact the spacecraft after it is jettisoned and damage the parachute system  Boeing’s analysis indicates the risk exists only if one of two parachutes that pull the forward heat shield away from the spacecraft does not deploy as expected, and that potential re-contact is non- detrimental  However, NASA’s independent analysis indicates that this may occur even if both parachutes deploy as expected  If the program determines this risk is unacceptable, Boeing would need to redesign the parachute system, which the program estimates could result in at least a 6-month delay  Launch Vehicle Data: One of the program’s top programmatic and safety concerns is that it may not have enough information from Boeing’s launch vehicle provider, United Launch Alliance, to assess whether the Atlas V launch vehicle prevents or controls cracking that could lead to catastrophic failures  NASA estimates that unfinished work in this area could take Boeing and the United Launch Alliance until the fourth quarter of 2018 to complete  Additionally, the first stage of the Atlas V is powered by the Russian built RD-180 engine, and, according to program and Boeing officials, access to its data is highly restricted by agreements between the U S  and Russian governments  Since our last report, the Commercial Crew Program has lowered the risk that certification of the launch vehicle might not occur by negotiating steps to access necessary data, but work is still ongoing  The Commercial Crew Program’s top programmatic risks identified for SpaceX are in part related to ongoing design and development efforts related to its launch vehicle design, the Falcon 9 Block 5  Composite Overwrap Pressure Vessel: This Block 5 design includes SpaceX’s redesign of the composite overwrap pressure vessel, which is intended to contain a gas under high pressure  SpaceX officials stated the newly designed vessel aims to eliminate risks identified in the older design, which was involved in an anomaly that caused a mishap in September 2016  SpaceX plans to qualify the updated design for flight prior to the uncrewed flight test design certification review  Engine Turbine Cracking: The Block 5 design also includes design changes to address cracks in the turbine of its engine identified during development testing  NASA program officials told us that they had informed SpaceX that the cracks were an unacceptable risk for human spaceflight  SpaceX officials told us that they have made design changes to this Block 5 upgrade that did not result in any cracking during initial testing  However, this risk will not be closed until SpaceX successfully completes qualification testing in accordance with NASA’s standards without any cracks  As of March 2018, SpaceX had not yet completed this testing  Propellant Loading Procedures: Both the program and a NASA advisory group have raised SpaceX’s plan to fuel the launch vehicle after the astronauts are on board the spacecraft to be a potential safety risk  In the May 2018 meeting minutes, however, the Aerospace Safety Advisory Panel stated that with appropriate controls in place, this approach could be a viable option for the program to consider  SpaceX’s perspective is that this operation may be a lower risk to the crew because it reduces the crew exposure time while the launch vehicle is being loaded with propellant  To better understand the propellant loading procedures, the program and SpaceX agreed to demonstrate the loading process five times from the launch site in the final crew configuration prior to the crewed flight test  The five events include the uncrewed flight test and the in-flight abort test  Therefore, delays to those events would lead to delays to the agreed upon demonstrations, which could in turn delay the crewed flight test and certification milestone  Both contractors have notified NASA that their certification milestones have slipped to January 2019 for Boeing and February 2019 for SpaceX, but the Commercial Crew Program’s schedule risk analysis indicates more delays are likely  This analysis identifies a range for each contractor, with an earliest and latest possible completion date, as well as an average  In April 2018, the program’s schedule risk analysis found there was zero percent chance that either contractor would achieve its current proposed certification milestone  The analysis’s average certification date was December 2019 for Boeing and January 2020 for SpaceX  Figure 3 shows the original Boeing and SpaceX contract schedules and the current proposed schedule for five key events in each contract, as well as NASA’s schedule risk analysis for the certification milestone  Each month, the program updates its schedule risk analysis based on the contractors’ internal schedules as well as program officials’ perspectives and insight into specific technical risks  The Commercial Crew Program manager told us that differences between the contractors’ proposed schedules and the program’s schedule risk analysis include: The contractors are aggressive and use their schedule dates to motivate their teams, while NASA adds additional schedule margin for testing  Both contractors assume an efficiency factor in getting to the crewed flight test that NASA does not factor into its analysis  The program manager also told us that the program meets with each contractor monthly to discuss schedules and everyone agrees to the relationships between events in the schedule even if they disagree on the length of time required to complete events  The program manager added, however, that she relies on her prior experience to estimate schedule time frames as opposed to relying on the contractors’ schedules, which are often optimistic  Our analysis also shows that the contractors often delay their schedules  Both contractors have repeatedly stated that their schedules are aggressive and have set ambitious—rather than realistic—dates, only to frequently delay them  Since the current contracts were awarded in 2014, the Commercial Crew Program has held 13 quarterly reviews for each contractor  For the five key events identified above, Boeing has reported a delay at 7 of those quarterly reviews and SpaceX has reported a delay at 9 of them  In mid-June 2018, NASA officials told us that the dates for these key events may change soon  The information presented in Figure 3 above is based on first quarter calendar year 2018 data  NASA officials stated both contractors have not yet officially communicated new schedule dates to NASA as of the second quarter calendar year 2018  We found that both contractors have updated schedules that indicate delays are forthcoming for at least one key event, but NASA officials told us they lack confidence in those dates until they are officially communicated to NASA by the contractors  As a result, NASA is managing a multibillion dollar program without confidence in its schedule information as it approaches several big events, including uncrewed and crewed flight tests  The risk of future delays in the contractors’ schedules is critical information that NASA has not fully shared with Congress  Moreover, NASA has not yet developed a contingency plan to address the potential gaps that these delays could have on U S  access to the ISS after 2019  Specifically, in the Explanatory Statement accompanying the fiscal year 2018 Consolidated and Further Continuing Appropriations Act, the House Appropriations Committee stated its expectation that NASA report quarterly to the Senate and House Committees on Appropriations on the status of the Commercial Crew Program contracts  Previously, members of Congress had asked for this information in order to ensure that Congress had adequate insight into this program  While NASA includes both contractors’ proposed schedules in its quarterly report to Congress, NASA does not include the results of its own schedule risk analysis  Given the frequency with which the contractors delay key events in their schedules, the program’s schedule risk assessment provides valuable insight into potential delays that NASA currently is not providing to Congress  In addition, as previously mentioned, NASA executed a contract modification that purchased two seats and included an option to purchase three additional crew member seats through Boeing for an undisclosed value and reported this action to Congress in April 2017  Ultimately, the option was exercised, and NASA purchased a total of five seats on four different Soyuz flights  Boeing obtained these seats through a separate settlement with the Russian firm RSC Energia, which manufactures the Soyuz  These seats were intended to serve as a contingency plan based on schedule information available at that time  However, subsequent delays, as well as the risk of future delays as discussed above, indicate that this contingency plan will likely no longer be sufficient  The earliest and latest possible completion dates for certification in NASA’s April 2018 schedule risk analysis indicate it is possible that neither contractor would be ready before August 2020, leaving a potential gap in access of at least 9 months  We calculated the potential gap based on the contractor certification milestone dates, but there could be some additional time required between that review and the first post-certification service mission to the ISS  As seen in figure 4, if the contractors can maintain their current proposed schedules for their respective certification milestones, a gap in access to the ISS is not expected  However, there would be a gap in access to the ISS if neither contractor has its certification milestone before November 2019, which is when NASA expects the final Russian Soyuz seat for a U S  astronaut to return  Senior NASA officials told us that sustaining a U S  presence on the ISS is essential to maintain and operate integral systems, without which the ISS cannot function  Given the importance of maintaining a U S  presence on the ISS, NASA officials have stated they are working on options to address the potential gap in access  However, officials told us that planning for contingencies is difficult given the extensive international negotiations required for some options  Obtaining additional Soyuz seats seems unlikely, as the process for manufacturing the spacecraft and contracting for those seats typically takes 3 years—meaning additional seats would not be available before 2021  As a result, according to NASA’s Associate Administrator for Human Exploration and Operations, the options NASA is considering include: Refine the remaining Soyuz launch schedule to allow for a return in January 2020, as opposed to November 2019  This would provide 2 additional months of access to the ISS before the commercial crew flights need to start  Use the crewed flight tests as operational flights to transport U S  astronauts to and from the ISS  In March 2018, NASA modified Boeing’s contract to allow NASA to add a third crew member and extend the length of the flight test, if NASA chooses to do so  This would have limited usefulness, however, in filling a potential gap in access to the ISS if the schedule for Boeing’s crewed flight test slips past the return date for the last Soyuz flight and SpaceX also continues to experience delays  NASA’s Associate Administrator for Human Exploration and Operations stated that he is “brainstorming” other options to ensure access to the ISS but does not have a formal plan  While options are not unlimited and decisions have to be made within the context of the current geopolitical environment, Congress stated in the NASA Authorization Act of 2005 that it is U S  policy to possess the capability for human access to space on a continuous basis  In 2010, Congress further stated that one of the key objectives of the United States’ human spaceflight policy is to sustain the capability for long-duration presence in low-Earth orbit through full utilization of the ISS  If NASA does not develop options for ensuring access to the ISS in the event of further Commercial Crew delays, it will not be able to ensure that the U S  policy goal and objective for the ISS will be met  The Commercial Crew Program relies on several contractual mechanisms to assess safety throughout the certification process, and those mechanisms are in varying stages of completion  The program itself, its contractors, and two of NASA’s independent review organizations have raised concerns about the program’s ability to assess and evaluate all of the deliverables in a timely manner  In addition, one of the key safety requirements levied by the program is loss of crew, which captures the probability of death or disability to a crew member  NASA does not have a consistent approach for how to incorporate key inputs to assess this metric, which means the agency as a whole may not clearly capture or document its risk tolerance with respect to loss of crew  Further, the program’s chief safety and mission assurance officer is dual hatted to serve simultaneously in a programmatic position as well as the program’s safety technical authority  This approach creates an environment of competing interests because it relies on the same individual to manage technical and safety aspects on behalf of the program while also serving as the independent oversight of those same areas  The contractors are required to provide several key deliverables to the Commercial Crew Program, which inform the agency certification review and help NASA determine the level of risk it is accepting with respect to safety of each spacecraft  As described below, these deliverables are in varying stages of completion and the program itself, its contractors, and two of NASA’s independent review organizations have raised concerns about the program’s ability to assess and evaluate all of the deliverables in a timely manner  Certification Data Package  Among other things, the certification data package includes a list of seven system safety assessments  For example, the certification data package includes a fault tolerance assessment, which describes the system’s ability to sustain a certain number of undesired events, such as software or operational anomalies  A human error analysis—one of the seven assessments in the data package—evaluates human errors to minimize their negative effects on the system  Boeing held its uncrewed flight test design certification review in December 2017 and submitted its certification data package for NASA approval  Boeing plans three more updates to this data package prior to the final certification milestone  SpaceX has begun to submit data and plans to submit its final certification data package as part of its crewed flight test design certification review, which is scheduled for September 2018  According to the program’s certification review plan, program officials will review and approve the contractors’ certification data packages, which will be used to inform the agency certification review  Phased Safety Review Process  A three-phased safety review process informs the program’s quality assurance activities, and it is intended to ensure that the contractors have identified all safety-critical hazards and implemented associated controls prior to the first crewed flight test  In phase one, the contractors identified risks in their designs and developed reports on potential hazards, the controls they put in place to mitigate them, and explanations for how the controls will mitigate the hazards  In phase two, which is nearing completion, the program reviews and approves the contractors’ hazard reports and develops strategies to verify and validate that the controls are effective  For example, if a control requires that an item be waterproofed, verification and validation strategies could include inspections and tests to confirm that the item is waterproof  As of April 2018, the program had yet to complete this phase, having approved 97 percent of Boeing’s phase two reports and 72 percent of SpaceX’s phase two reports  In phase three, the contractors will conduct the verification activities and submit the hazard reports to the program for approval  The program has begun phase three, including approving 19 percent of Boeing’s phase three reports  Program Requirements  While the program manager told us that all of the requirements contribute to the safety of the commercial systems, safety officials are required to approve a subset of these requirements  Examples of requirements approved by safety officials include the ability to leave the spacecraft in an emergency or to abort a launch  When a contractor is ready for NASA to verify that it has met a requirement, the contractor submits data for NASA to review through a verification closure notice  We define “safety-specific notices” as those requiring safety officials’ approval  As shown in table 2, as of March 2018, the program had approved 2 percent of Boeing’s safety-specific notices and 0 percent of SpaceX’s safety-specific notices  Testing  The program also requires testing to verify and validate the crew transportation system  Agency officials emphasized the importance of testing to safety, stating that testing reduces uncertainty about a system’s performance and can uncover unknown problems  As noted above, both contractors will be conducting an uncrewed and a crewed flight test prior to being certified  While a certain level of risk needs to be accepted to conduct human spaceflight, these flight tests help to mitigate this risk by validating the integrated performance of the hardware and software  Agency and program officials stated that the contractors’ flight tests are critical evidence to support certification of a safe and reliable system  As evidenced by the data above, the program still has a significant amount of work ahead with respect to approving certification packages and closing hazard reports and verification closure notices  We have previously found that the program’s workload was an emerging schedule risk, and the contractors have continued to express concern about program officials’ ability to process and approve certification paperwork in a timely manner  Workload has also been a concern for two of NASA’s independent review organizations  For example, the Aerospace Safety Advisory Panel noted in its January 2018 annual report that the sheer volume of work that remains for the program in terms of closing hazard reports and verification closure notices is significant  In addition, the program’s safety and mission assurance office identified the upcoming bow wave of work in a shrinking time period as a top risk to achieving certification  One mechanism the program put in place to assess the overall safety of each spacecraft—loss of crew—has been a focus of the Aerospace Safety Advisory Panel, Members of Congress, our prior work, and the program itself  Loss of crew captures the probability of death or permanent disability to one or more crew members  It has received a lot of attention, in part, because it has been a top risk for the program since 2015  Specifically, the program has been concerned that neither contractor would be able to meet the contract requirement of a 1 in 270 probability of incurring loss of crew  We identified two key concerns with how NASA is using the loss of crew metric: (1) inconsistent approaches to assess the loss of crew metric and (2) no identified plan to share lessons learned about using the loss of crew metric as a safety threshold  A loss of crew value is generated through a probabilistic safety analysis, which models scenarios that could result in the loss of crew using various inputs  According to the program’s analysis, the probability of on-orbit debris damaging the vehicle has the greatest effect on a loss of crew value  This probability is informed by an orbital debris (debris) model, which was updated in 2014, after the loss of crew requirement was established  The updated debris model makes it harder to meet a loss of crew value, in part, because the modeling environment where the contractors’ systems will operate has changed  For example, the updated model includes a larger span of orbit, greater range of debris sizes, and the addition of material density classifications, which were not included in the former model  Further, the probabilistic safety analysis may include operational mitigations, such as on-orbit inspections that would include using cameras on the ISS to visually survey the spacecraft for damage, which, according to officials, makes it easier to meet a loss of crew value  NASA describes the probabilistic safety analysis as a powerful tool that should be used as part of the overall risk management process to ensure the risk associated with development and operation of a system is understood, evaluated, managed, and mitigated  However, we found differences in the approaches that officials plan to use to assess loss of crew as well as in the loss of crew value being measured that could limit the usefulness of this tool  Agency Certification  The agency certification review for each contractor will include an assessment of whether its crew transportation system meets a loss of crew threshold of 1 in 150 for missions to the ISS, which is based on a May 2011 safety memo from the Office of Safety and Mission Assurance  A loss of crew value with a higher denominator, such as 1 in 270, is harder to meet than with a lower denominator, such as 1 in 150  According to the Chief of the Office of Safety and Mission Assurance, he will assess the 1 in 150 threshold using a probabilistic safety analysis that includes the updated debris model and operational mitigations, such as the on- orbit inspections cited above  Program Office  According to program officials, they will assess whether either contractor meets a 1 in 270 loss of crew value based on a probabilistic safety analysis using the former debris model (not the updated model) and not including operational mitigations  Contracting Officer  According to the contracting officer, each contractor’s loss of crew requirement is 1 in 270 without including operational mitigations  The contracting officer stated that SpaceX’s contract requirement uses the updated debris model in the probabilistic safety analysis, whereas Boeing’s contract requirement uses the former debris model in the probabilistic safety analysis  Program’s Chief Safety and Mission Assurance Officer  According to the program’s chief safety and mission assurance officer, he will conduct a probabilistic safety analysis using the updated debris model and will not include operational mitigations to assess whether each contractor meets a 1 in 200 loss of crew value  This loss of crew value stems from a program update that occurred after the initial contracts were signed  These different approaches are summarized in table 3 below  Agency policy requires human spaceflight programs to set a safety threshold, which NASA did for the Commercial Crew Program when it identified the 1 in 150 loss of crew threshold in the May 2011 safety memo  Subsequently, the program set more rigorous loss of crew values in contract and program documents  NASA also updated the debris model, which we previously noted makes it more difficult to meet a loss of crew value  As a result, NASA does not have a consistent approach for how to incorporate key inputs to the probabilistic safety analysis, including changes to the debris model  Instead, the risk tolerance that NASA is accepting with loss of crew varies based upon which entity is presenting the results of its probabilistic safety analysis  For example, it is possible that the program’s assessment will determine that neither contractor will meet the 1 in 270 contract requirement, but that the agency’s assessment will determine that the contractors meet the 1 in 150 agency certification value because that analysis will include operational mitigations  Or the program’s assessment could determine that Boeing meets the 1 in 270 contractual loss of crew requirement, but the agency’s assessment may determine that Boeing does not meet the 1 in 150 agency certification value because that analysis will use the updated debris model  Federal internal controls state that agency management should define objectives clearly to enable the identification of risks and define risk tolerances  Specifically, management should define risk tolerances in specific and measurable terms, so they are clearly stated and can be measured  In this case, because there will be multiple analyses conducted using different inputs, NASA risks not clearly capturing or documenting, in a coherent manner, its overall risk tolerance with respect to loss of crew before a final decision must be made on whether to certify either crew transportation system  Moreover, capturing the challenges and lessons learned from using the loss of crew metric is critical, particularly because agency officials told us that this is the first time this metric has been used as a safety threshold  Also, there are different viewpoints about the utility of the metric as a safety threshold across the agency  The program manager repeatedly told us that loss of crew is best used as a design tool  For example, program officials told us that both contractors incorporated additional orbital debris shielding into their designs to mitigate the orbital debris risk and improve their loss of crew values  In addition, the Aerospace Safety Advisory Panel reported in 2018 that loss of crew should not be viewed as an absolute measure of actual risk during operations  However, the May 2011 agency safety memo states that a breach of the loss of crew threshold would initiate a termination review of the Commercial Crew Program, which is a more strict application of the loss of crew metric  Both program and safety officials told us that, after the agency certification is complete and lessons learned are available to be compiled, sharing those lessons learned across NASA would be a good idea given the complexities associated with assessing the loss of crew metric  As of April 2018, however, agency officials said they did not have a plan for loss of crew knowledge-sharing  We have previously found that lessons learned provide a powerful method of sharing good ideas for improving work processes, facility or equipment design and operation, quality, safety, and cost-effectiveness  Further, according to NASA’s Knowledge Policy on Program and Projects, which is managed through the Office of the Chief Engineer, a principle of each center and mission directorate’s knowledge strategy is that knowledge is the cornerstone of NASA’s ability to achieve mission success  The policy acknowledges that NASA faces continuous challenges in using what it knows effectively  These challenges include, but are not limited to, enabling the identification and flow of knowledge across organizational boundaries; preserving knowledge at risk of being lost; and providing means for individuals, teams, and the organization to learn from experiences  If NASA does not capture lessons learned from the Commercial Crew Program on using the loss of crew requirement to set a program’s safety threshold and whether it met the agency’s intended goal, future programs will not be able to benefit from the knowledge gained from this multibillion dollar investment  NASA’s governance model prescribes a management structure that employs checks and balances among key organizations to ensure that decisions have the benefit of different points of view and are not made in isolation  As part of this structure, NASA established the technical authority process as a system of checks and balances to provide independent oversight of programs and projects in support of safety and mission success through the selection of specific individuals with delegated levels of authority  The technical authority process has been used in other parts of the government for acquisitions, including the Department of Defense and Department of Homeland Security  The Commercial Crew Program is organizationally connected to three technical authorities within NASA: the Office of the Chief Engineer technical authority, the Office of Chief Health and Medical technical authority, and the Office of Safety and Mission Assurance (safety) technical authority  The safety technical authority is responsible for ensuring from an independent standpoint that the program’s products and processes satisfy NASA’s safety, reliability, and mission assurance policies  The NASA safety technical authority has delegated authority through the Kennedy Space Center Director to the Chief Safety and Mission Assurance Officer for the Commercial Crew Program  We have previously reviewed how NASA has organized its technical authorities for its Exploration Systems Development organization—an organization that oversees the development of the Space Launch System, Orion crew capsule, and associated ground systems that have the goal of extending human presence beyond low-Earth orbit  In October 2017, we found that the Exploration Systems Development organization had established an organizational structure in which the technical authorities for engineering and safety and mission assurance were dual hatted simultaneously in programmatic positions  We found that having the same individual simultaneously fill both a technical authority role and a program role created an environment of competing interests, where the technical authority’s ability to impartially and objectively assess the programs while at the same time acting on behalf of the Exploration Systems Development organization in programmatic capacities may be subject to impairments  We found that this was in contrast to a recommendation from the Columbia Accident Investigation Board report—the result of an in-depth assessment of the technical and organizational causes of the 2003 Space Shuttle Columbia accident—for NASA to establish a technical authority to serve independently of the Space Shuttle program, so that employees would not feel hampered to bring forward safety concerns or disagreements with programmatic decisions  The board’s findings that led to this recommendation included a broken safety culture in which it was difficult for minority and dissenting opinions to percolate up through the hierarchy; dual center and programmatic roles vested in one person that had confused lines of authority, responsibility, and accountability and made the oversight process susceptible to conflicts of interest; and oversight personnel in positions within the program, increasing the risk that these staffs’ perspectives would be hindered by too much familiarity with the programs they were overseeing  In October 2017, we recommended that the division no longer dual hat two individuals who had both programmatic and technical authority responsibilities  As of April 2018, NASA had taken steps to separate the engineering technical authority position from the programmatic position, and NASA’s Chief of Safety and Mission Assurance said he planned to separate the safety position but had not yet completed that action  The Commercial Crew Program employs a similar structure to the Exploration Systems Development organization in that the safety technical authority is dual hatted simultaneously in a programmatic position as the Commercial Crew Program’s Safety and Mission Assurance Manager  According to the program’s safety technical authority, in his programmatic role for the program, he helps set priorities for safety issues, including how staff will be utilized to meet those priorities  In the technical authority role, he provides independent oversight in support of safety and mission success  In his dual-hatted role, this official will be responsible for endorsing the program’s certification recommendations in two different capacities: as the technical authority and as a program authority  As a result, this structure relies on the same individual to completely separate two roles—one to manage the Commercial Crew Program’s safety issues within programmatic cost and schedule constraints, and the other to assess the same issues in an independent oversight role  While the Commercial Crew Program may have an additional level of separation between the safety technical authority and the program’s involvement in the design of commercial systems due to its shared assurance model with the commercial providers, the Commercial Crew Program still maintains a structure where one individual simultaneously serves in both technical authority and programmatic roles  Figure 5 describes some of the conflicting roles and responsibilities of this official in his two different positions  During our review, officials cited several factors in support of a dual- hatted approach: The safety technical authority retains independence because his technical authority reporting path and performance reviews are not under the purview of the Commercial Crew Program chain of command  Due to the Commercial Crew Program’s shared assurance model with commercial providers, the program is operating in a quality assurance role that provides an additional level of separation between the safety technical authority and the program’s involvement in the design of commercial systems  For safety decisions involving cost and schedule where the individual who is dual hatted with both technical authority and programmatic responsibilities may feel conflicted, he stated that he would discuss these matters with his management to validate the logic behind his decision  There are cost and knowledge efficiencies gained from one individual serving in both programmatic and safety technical authority capacities  NASA’s Chief of Safety and Mission Assurance stated that he has great confidence in the individual currently serving in the dual hatted role for the Commercial Crew Program, but acknowledged there is inherent conflict even with the program’s shared assurance model  In December 2017, he stated that, based on our previous work and current discussions, he intends to decouple the programmatic and technical authority responsibilities for the Commercial Crew Program but had not done so as of April 2018  Federal internal control standards state that an agency should design control activities to achieve objectives and respond to risks, which includes segregation of key duties and responsibilities to reduce the risk of error, misuse, or fraud  By overlapping technical authority and programmatic responsibilities, NASA will continue to run the risk of creating an environment of competing interests for the Commercial Crew Program’s safety technical authority  NASA’s Commercial Crew Program is a multibillion dollar effort to facilitate the commercial development of a crew transportation system that can end the United States’ reliance on Russia to maintain an uninterrupted presence on the ISS  Boeing and SpaceX continue to make progress developing a capability to fly to the ISS, but both have continued to experience delays  Program analysis indicates risks of further delays in each contractor’s current schedule, but NASA has not provided that information to Congress in its routine briefings  Without this information, Congress does not know the full extent of potential delays to inform decision making  Additional delays could also disrupt U S  access to the ISS  While NASA is working on potential solutions, there is no contingency plan in place to address this potential gap  Without a viable contingency plan, NASA puts at risk achievement of the U S  goal and objective for the ISS  NASA must balance safety with acceptable risk for human spaceflight  As part of the certification process for each contractor’s spacecraft, NASA has developed one key safety metric, loss of crew  However, the complicated nature of this metric is further muddled by the inconsistent approaches being used across NASA about what inputs to be considered  As a result, there is no clear articulation of what level of risk NASA will accept with respect to this program  In addition, NASA does not have plans to capture lessons learned from how the Commercial Crew Program has used this metric to assess safety and is missing an opportunity to capture this knowledge for future human spaceflight programs  Finally, a space program’s management and oversight approach is an integral part of ensuring that human spaceflight is as safe and successful as possible  Independence of the program management and oversight functions is key to achieving the balance between safety and success  The Commercial Crew Program’s approach, however, burdens the safety technical authority with both programmatic and independent technical authority responsibilities  As a result, NASA has limited assurance that independence can be maintained as part of its institutional process to ensure safety and success  We are making the following five recommendations to NASA: The NASA Associate Administrator for Human Exploration and Operations should direct the Commercial Crew Program to include the results of its schedule risk analysis in its mandatory quarterly reports to Congress  (Recommendation 1) The NASA Administrator should develop and maintain a contingency plan for ensuring a presence on the ISS until a Commercial Crew Program contractor is certified  (Recommendation 2) The NASA Administrator should direct the Chief of Safety and Mission Assurance, the NASA Associate Administrator for Human Exploration and Operations, the Commercial Crew Program Manager, and the Commercial Crew Program Contracting Officer to collectively determine and document before the agency certification review how the agency will determine its risk tolerance level with respect to loss of crew  (Recommendation 3) After completing the agency certification review, NASA’s Chief Engineer and Chief of Safety and Mission Assurance, with support from the NASA Associate Administrator for Human Exploration and Operations and the Commercial Crew Program Manager, should document lessons learned related to loss of crew as a safety threshold for future crewed spaceflight missions, given the complexity of the metric  (Recommendation 4) The NASA Chief of Safety and Mission Assurance should restructure the technical authority within the Commercial Crew Program to ensure that the technical authority for the Office of Safety and Mission Assurance is no longer dual hatted with programmatic and independent technical authority responsibilities  (Recommendation 5) We provided a draft of this report to NASA for review and comment  NASA provided written comments that are reprinted in appendix II  In its response, NASA concurred with three of our recommendations, did not concur with one, and partially concurred with another  NASA concurred with our recommendation to develop and maintain a contingency plan to ensure a U S  presence on the ISS and expects to take action to close this recommendation by the end of December 2018  NASA concurred with our recommendation to document lessons learned related to the loss of crew requirement and expects to take action to close this recommendation by the end of May 2019  NASA concurred with our recommendation to restructure the safety technical authority so that it is no longer dual hatted with programmatic and independent technical authority responsibilities  NASA expects to take action to close this recommendation by the end of August 2018  NASA did not concur with our recommendation that the Commercial Crew Program should include the results of its schedule risk analysis in its quarterly reports to Congress  NASA stated that it uses the contractors’ schedules as a baseline to provide qualitative statements in the NASA summary that accompanies each contractor’s quarterly reports to Congress  NASA believes that this approach is appropriate and is in accordance with the explanatory statement accompanying the Consolidated and Further Continuing Appropriations Act, 2015  NASA also stated that it will be working to ensure that the contractors’ schedules and the program’s internal assessments sync up as the program gets closer to launch  As a result, NASA explained that there will not be a requirement for a detailed NASA assessment, because the contractors’ schedule will either match NASA’s analysis or NASA will discuss its position as it has done in previous reports to Congress  We continue to believe the recommendation is valid because the program’s schedule risk analysis would provide Congress with valuable insight into potential delays, which are likely  Both contractors have repeatedly stated that their schedules are aggressive and that the dates are ambitious  As a result, we found that the contractors frequently delay dates for key events  For example, Boeing has delayed its certification milestone by 17 months and SpaceX by 22 months since the original schedules were established  The program’s recent schedule risk analysis indicates that more delays to certification are likely, but that information is not presented to Congress in NASA’s quarterly reports  Without this information, Congress does not know the full extent of potential delays to inform decision making  NASA partially concurred with our recommendation that the Chief of Safety and Mission Assurance, the NASA Associate Administrator for Human Exploration and Operations, the Commercial Crew Program Manager, and the Commercial Crew Program Contracting Officer should collectively determine and document how the agency will determine its risk tolerance level with respect to loss of crew before the agency certification review  In its response, NASA stated that it documented the agency’s risk tolerance level with respect to loss of crew for the program in its May 2011 safety memo  Further, NASA stated that it documented the requirement to limit risks to the loss of crew in a certification requirements document  NASA stated that ultimately the Commercial Crew Program is accountable for ensuring that the contractors’ systems meet the loss of crew value in this certification requirements document, which is a loss of crew value of 1 in 270  If a contractor’s system cannot meet that loss of crew value, or any other requirement, the program will request a waiver as part of the human rating certification process to ensure transparency  NASA acknowledged in its response that the existence of multiple documents defining residual risk requirements and an agency threshold for loss of crew can be confusing  NASA’s response, however, does not address our finding that it does not have a consistent approach for how to incorporate key inputs, including which debris model should be used or whether to include operational mitigations  NASA stated that it had taken action to address this recommendation; however, NASA did not outline any steps it took to resolve the concern that the risk tolerance for the loss of crew requirement depends on which entity is presenting the results of its analysis  We continue to believe that, before the agency certification review, the key parties must collectively determine how the agency will determine its risk tolerance with respect to loss of crew  We believe this approach will reduce confusion and increase transparency  We are sending copies of this report to NASA Administrator and interested congressional committees  In addition, the report is available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact me at (202)512-4841 or chaplainc@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report  GAO staff who made major contributions to this report are listed in appendix III  The objectives of our review were to assess (1) the extent to which the contractors have made progress towards meeting the National Aeronautics and Space Administration’s (NASA) certification requirements and NASA’s plans to help ensure continued access to the International Space Station (ISS); and (2) how NASA’s certification process addresses safety of the contractors’ crew transportation systems  To assess the contractors’ progress towards certification, we obtained and reviewed program and contractor documents, including monthly and quarterly updates from April 2017 through May 2018  We interviewed program and contractor officials to discuss the contractors’ recent progress, including upcoming events and any expected delays, and to understand technical risks, potential consequences, and planned mitigation activities  To identify total delays to date, we compared original contract schedules to Boeing and SpaceX’s calendar year 2018 first quarter proposed schedules, which are the most recent  Based on our review of program and contractor documents, we defined the contractors’ key events as: the uncrewed and crewed flight tests, the design certification reviews for each of those flights, and the certification milestone  We selected the two flight tests for each contractor as key events because they are intended to test key system capabilities, including the ability to launch, dock with the ISS, and return safely to Earth  We selected the design certification reviews because they verify the contractors’ crew transportation systems’ capability to safely approach, dock, mate, and depart from the ISS, among other requirements  We selected the certification milestone because it determines whether the crew transportation system meets the Commercial Crew Program’s requirements  To determine the extent to which contractors have delayed these key events over time, we analyzed the contractors’ schedule data from the 13 quarterly progress reports to date, from first quarter 2015 through first quarter calendar year 2018  We also obtained the results of the program’s April 2018 schedule risk analysis  We presented the schedule analysis range from the end of the month of the earliest possible completion date to the end of the month of the latest possible completion date  We reviewed the program’s Congressional requirements to report on cost, schedule, and technical status  Finally, to assess the potential effects of any certification delays on NASA’s access to ISS, we reviewed NASA’s contracts with Boeing and the Russian Federal Space Agency for transportation on the Soyuz vehicle  We interviewed officials from the ISS program and NASA’s Human Exploration and Operations Mission Directorate to determine if the agency had developed contingency plans to mitigate the effects of any certification delays on its access to the ISS  To assess how NASA’s certification process addresses safety of the contractors’ crew transportation systems, we reviewed agency safety policies, program plans, and contract documents to identify what safety assessments were required of the program, which safety factors would be considered in certification reviews, and when certification approval would be granted  We also interviewed program officials and the contractors about their safety policies and procedures as well as about the certification process  We identified the loss of crew requirements for the program and the contractors, and interviewed program and agency officials to determine how loss of crew would be assessed and considered throughout the certification process  To gain a broader understanding of the relative importance of loss of crew, we reviewed NASA safety policies, prior GAO reports, and annual reports from the Aerospace Safety Advisory Panel  To determine how the Orbital Debris Engineering Model (ORDEM) was updated and to assess differences between the former and updated models, we reviewed NASA documentation about the ORDEM 2000 and ORDEM 3 0 models and obtained information about the models from NASA’s Orbital Debris Program Office  To assess the program’s progress in closing requirements, including those requirements specifically related to safety, we reviewed program data for each contractor on contract requirements and closure status  We limited this analysis to requirements included in the CCT-REQ-1130 ISS Crew Transportation and Services Requirements Document because these requirements are verified by the Commercial Crew Program, whereas requirements contained within SSP 50808 ISS to Commercial Orbital Transportation Services Interface Requirements Document are managed by the ISS Transportation and Integration Office  We classified requirements as “safety-specific” when the program’s Office of Safety and Mission Assurance was listed as a verification closure notice signatory (i e , approver)  We then analyzed the program’s data to determine how many verification closure notices had been approved for all requirements and for the subset of safety-specific requirements  We reviewed program and agency documentation, such as organizational charts, program plans, and safety policies as well as interviewed program and agency officials, to determine the role of the safety and mission assurance technical authority in the program  We also reviewed the 2003 Columbia Accident Investigation Board’s Report’s findings and recommendations related to culture and organizational management of human spaceflight programs  We reviewed annual briefings and reports and met with representatives from two organizations that provide NASA with independent assessments of the program, the program’s standing review board and the Aerospace Safety Advisory Panel, to gain their perspectives on the contractor’s progress and how NASA addresses safety in its certification process  We also met with representatives from the National Transportation Safety Board and three experts with background on safety in human spaceflight in order to increase our contextual understanding of the role of safety in human spaceflight missions  We conducted this performance audit from April 2017 to July 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  Cristina T  Chaplain, (202) 512-4841 or chaplainc@gao gov  In addition to the contact named above, Molly Traci, Assistant Director; Kazue Chinen; Lorraine Ettaro; Lisa Fisher; Laura Greifner; Kurt Gurka; Miranda Riemer; Juli Steinhouse; Roxanna T  Sun; Hai Tran; Kristin Van Wychen; and Alyssa Weir made significant contributions to this report 


Report 2:  Timing challenges  Some PHAs said they faced major challenges in coordinating RAD timelines with HUD, lenders, or other parties or with the requirements of the LIHTC process  HUD officials acknowledged that PHAs with more complex transactions, including those involved in the LIHTC process, struggle to implement their conversion plans within RAD time frames  HUD officials noted that because there is a statutory cap on the number of units that can be converted under RAD, they have established time frames to stay under the cap and ensure that PHAs that are planning to convert are ready to participate in the program  Additionally, according to HUD, it has made technical assistance available to all PHAs that receive a Commitment to enter into a Housing Assistance Payment contract during the RAD process to help ensure their readiness for RAD closing and to meet remaining conversion deadlines  On the other hand, some PHAs expressed concern to us about delays in the conversion process that put them at risk for missing state LIHTC deadlines  HUD officials described putting conversions on a fast-track on a case-by-case basis to meet LIHTC deadlines  For example, in one case a PHA relocated residents before closing and without HUD approval  HUD required the PHA to fund an escrow account until it was able to determine any payments that might need to be made to residents and any other necessary corrective action  This was done so that HUD could look into the issue while mitigating additional harm to the residents and continuing to move the PHA toward closing and aligned with tax credit application deadlines  The timing of conversion can also create gaps in the payment of Section 8 funds to PHAs  Section 8 funding should begin in January of the year following conversion  PHAs rely on annual public housing subsidies for the conversion year—public housing program funds are paid to PHAs annually and are not recaptured by HUD following RAD conversion  However, according to some PHAs we interviewed, Section 8 funding did not begin on time  For example, in Baltimore, Maryland, subsidy flow after conversion had not begun as of June of the following year  HUD officials told us inadequate guidance from HUD and confusion from PHAs regarding the necessary steps to request payment in a timely manner have been the major cause of the problems  HUD has tried to remedy delays and updated its notice to provide clearer guidance on the timing of subsidy flow around the time of conversion to Section 8  Moreover, HUD officials indicated that there has been confusion among PHAs on how to request funds, so HUD is currently revising and updating the guidance on steps PHAs must take to request payment under the PBRA program  HUD officials also indicated that it has begun monitoring whether new participants are taking the steps needed well before their first request for funding  Some PHAs we contacted also mentioned difficulty in coordinating with HUD on fulfilling internal RAD requirements and reviews  According to some, the different offices involved in RAD conversions within HUD were not well aligned and had different interpretations of the rules  For example, some RAD conversions require a civil rights review by HUD’s Office of Fair Housing and Equal Opportunity Office, including those transactions that require new construction or resident relocations  Some PHAs indicated that such reviews occurred too late in the conversion process even after other HUD offices had approved the conversion  HUD officials acknowledge that different HUD offices have different objectives in the RAD process  HUD officials indicated that the agency is trying to coordinate more effectively among these offices and streamline the conversion process as much as possible  Evolving requirements  While the majority of PHAs with which we spoke said that HUD provided clear, sufficient, and timely information, some PHAs noted that it also was challenging to adapt to evolving requirements  Some PHAs noted that as HUD identified problems in the early years of the program, it would change the guidance in response  For example, HUD officials explained that it had clarified fair housing review requirements in response to PHA concerns that the fair housing review occurred too late in the process and could affect successful conversion of projects  The most recent RAD notice (effective January 2017) is the third version since 2013 and revisions have involved substantial changes  For example, this notice provided PHAs with greater flexibilities on the funding sources they can use to raise initial contract rents and the ways they can demonstrate ownership and control of a converted property  In addition, HUD introduced a notice in November 2016 to strengthen resident protections  Some PHAs told us they found the pace or timing of the evolving requirements difficult to manage and also noted confusion about conversion instructions and guidance due to changing requirements  For example, one PHA indicated that the agency had problems reporting information into a new RAD data field in HUD’s Voucher Management System because there was no guidance at the time on how to complete this field  However, HUD has since included additional instructions in the user’s manual that became effective in April 2017  The Committee has included language to establish procedures that will ensure that public housing remains a public asset in the that event that the project experiences problems, such as default or foreclosure  In each RAD conversion, HUD and the property owner execute a use agreement, which specifies affordability and use restrictions for the property  The use agreement generally exists concurrently with the HAP contract, which is executed to govern the provision of either the PBRA or PBV subsidy for the unit  The use agreement must be recorded in a superior position to new or existing financing or other encumbrances on the converted property  Under a Section 8 HAP contract, residents pay 30 percent of adjusted household income  In the absence of the HAP contract, the use agreement is set up to control the amount paid: If the HAP contract is removed due to breach, noncompliance, or insufficiency of appropriations, under the use agreement new households in all units previously covered under the HAP contract must have incomes at or below 80 percent of the area median income for households of the size occupying an appropriately sized unit for their family size at the time of admission, and rents may not exceed 30 percent of 80 percent of area median income for the remainder of the term of the use agreement  For new residents at or below 80 percent of the area median income, under the use agreement the resident rent contribution without a HAP contract generally would be higher than that paid under a HAP contract, which is based on household income instead of the universally determined area median income  Although the use agreement maintains some level of affordability, the owner receives no subsidy under PBRA or PBV without a HAP contract and resident rent contribution is not tied to individual household income but rather based on a universal area income calculation (see fig  3)  According to HUD officials, other program requirements support the goal of long-term preservation: HAP contracts are executed for 20 years for PBRA or 15–20 years for PBV properties and compliance with all affordability requirements in the HAP and statute and regulation governing the PBRA and PBV programs must be maintained while the contract is in force  According to the authorizing statute, PHAs (for PBV contracts) and HUD (for PBRA contracts) shall offer and project owners shall accept a renewal contract at the expiration of the initial HAP contract and at each subsequent renewal  Each renewal contract will be subject to a RAD use agreement, governing the use of the property consistent with HUD requirements  According to the RAD notice, the project owner also is to establish and maintain a replacement reserve to aid in funding extraordinary maintenance and repair and replacement of capital items  The reserve account must be built up to and maintained at a level determined by HUD to be sufficient to meet projected requirements  According to HUD officials, during the conversion, HUD staff review each capital needs assessment to try to determine whether a property’s capital needs can be addressed over the forthcoming 20-year period  We reviewed 31 completed conversion files, the set of documentation required by HUD to enable a PHA to convert units from public housing to a Section 8 subsidy, and associated RAD contracts  In each file, key contractual protections appeared consistent with program requirements  Specifically, in all cases executed use agreements (which included requirements to limit residency eligibility to households making less than 80 percent of area median income) were included and not altered from the HUD template  In most files we reviewed, we found foreclosure riders were included and that they stated that use agreements would survive foreclosure, meaning that any new owners would take ownership subject to the agreements  Executed HAP contracts, requiring that residents’ contributions be set at 30 percent of adjusted household income, also were present in all files we reviewed  According to HUD officials, PHAs, and two housing groups we spoke with, provisions in the RAD use agreement to keep units affordable appear to be strong, with use and affordability protections designed to survive foreclosure, but the strength of provisions cannot yet be fully determined because the provisions have not yet been tested in foreclosure proceedings or in courts  According to HUD officials, as of October 2017 no RAD properties had entered foreclosure  The RAD authorizing statute requires that ownership be transferred to a capable public entity or, if not one, a capable entity as determined by HUD, or if necessary to fulfill LIHTC requirements for the property, to a HUD-approved for-profit entity (provided the PHA retained sufficient interest in the property)  HUD also subjects any subsequent transfer of the property to HUD review and requires the successor ownership to meet these same requirements  As stated in the use agreement, a lien holder must give HUD notice prior to declaring a default and provide HUD concurrent notice with any written filing of foreclosure (providing that the foreclosure sale must not be sooner than 60 days after the notice), but the use agreement does not prohibit a lien holder from foreclosing on the lien or accepting a deed in lieu of foreclosure  The RAD use agreement, which is recorded superior to other liens and places use and affordability restrictions on the property, survives foreclosure  With or without a HAP contract in place, the lender or new owner must maintain the units for low- income households according to the terms of the use agreement  Therefore, according to HUD officials, the lender or new owner has an incentive to identify an appropriate owner and secure HUD approval to avoid a default under the HAP contract, which provides a Section 8 subsidy to the owner  That is, if no HAP contract were in place, the owner would collect only the tenant rent contribution (30 percent of 80 percent of area median income), rather than the tenant rent contribution plus the subsidy  HUD has discretion to enforce or waive certain use and affordability protections  According to the authorizing statute, in the case of foreclosure, bankruptcy, or termination and transfer of assistance for material violation or substantial default, the priority for ownership or control must be provided to a capable public entity, or, if no such entity can be found, to a capable entity as determined by the Secretary of HUD  Additionally, the statute allows the transfer of property to for-profit entities to facilitate the use of LIHTC financing, with requirements to maintain the PHA’s interest, which was discussed above  As of September 30, 2017, about 40 percent of RAD conversions involved LIHTC financing  According to the RAD notice, in the event of a default of a property’s use agreement or HAP contract, HUD may terminate the HAP contract and transfer assistance to another location to retain affordable units  HUD will determine the appropriate location and owner entity for the transferred assistance consistent with statutory goals and requirements for RAD  The RAD use agreement will remain in effect even in the case of abatement or termination of the HAP contract for the term the contract would have run, unless HUD agreed differently in writing  In this case, the RAD notice limits HUD discretion to terminate the use agreement to only cases involving a transfer of assistance to another property  HUD has not yet developed procedures to monitor RAD projects for risks to long-term affordability of units, including default or foreclosure  HUD officials described an ongoing effort to develop oversight procedures it would need to reasonably ensure compliance with RAD agreements and avoid risks to long-term affordability once conversions closed and units moved to Section 8 but, as previously discussed, the agency has not yet completed this effort or fully implemented a monitoring system  HUD officials told us they also planned to develop protocols to more closely monitor properties at risk of foreclosure, including developing indicators, procedures, roles, and responsibilities within HUD, but they have not finalized the design of procedures or fully implemented them  To develop protocols, HUD created an asset management working group in September 2016  The officials also stressed that no one can take possession of or foreclose on a property without HUD involvement and approval  For example, HUD officials said they expect few foreclosures among RAD-converted properties because lenders tend to communicate with the agency early so that it can become involved to prevent foreclosure  HUD officials pointed to a robust structure to oversee program properties in the PBRA program, but stated PBV property oversight continues to be developed by the Office of Public and Indian Housing  According to Standards for Internal Control in the Federal Government, agencies should design procedures to achieve goals and objectives, such as the preservation of unit affordability, and respond to risks, in this case the risk of default or foreclosure or noncompliance with program requirements  Additionally, management should identify, analyze, and respond to risks related to achieving its goals and objectives  According to HUD officials, the agency had not yet fully developed and implemented oversight procedures for postconversion monitoring because since 2012, the agency has focused on RAD start-up and review and oversight procedures for the conversion process  HUD officials also said that many projects would receive ongoing monitoring from other parties, which also could serve as a safeguard for unit affordability and help ensure the appropriate financial and physical condition of the property after RAD conversion  For example, just under half of all RAD properties use LIHTC financing as part of financing packages, which can also include local and state bonds  According to HUD officials, oversight by tax credit allocating agencies, investors, and lenders, while not alone sufficient, helps secure affordable units in a property for the long-term  However, tax credit allocating agencies, investors, and lenders are not signatories to the HAP contract or use agreement and have no formal role in reasonably ensuring that properties meet requirements exclusive to RAD  Although other entities may exercise some oversight of properties, by not developing and implementing procedures for ongoing oversight, HUD in its role as program administrator will not be able to reasonably ensure that properties adhere to requirements or meet basic program goals  Furthermore, without such monitoring HUD would be limited in its ability to identify and assist with properties at risk of foreclosure  RAD was created to demonstrate the feasibility of converting public housing units to other rental assistance programs to help preserve affordable rental units and address the significant backlog of capital needs in the public housing program  However, demonstrating the feasibility of RAD conversion is contingent on collecting and assessing quality information about the conversion projects  HUD has an opportunity to improve the demonstration’s metrics  For instance, implementing robust postclosing oversight and collecting information on financial outcomes upon completion of construction would not only improve HUD’s oversight capabilities but also allow it to report quality information  Moreover, limitations in HUD’s methodology for calculating leverage ratios for RAD may obscure the effect of funding sources used to help fund RAD conversions, potentially under- or over-reporting the program’s capital leveraging  By collecting comprehensive information on final (postcompletion) financing sources and costs and developing quality metrics, HUD would be better positioned to more accurately report the results of the demonstration program  Additionally, a focus on the conversion process itself (and less on its results), and limitations in HUD’s data have contributed to limited monitoring by HUD in other areas  Specifically, by not developing and implementing monitoring procedures to assess the effect of RAD on residents HUD cannot ensure compliance with resident safeguards  Further, HUD collects and maintains household data for the public housing and Section 8 programs, yet it does not systematically use this information to ensure that resident safeguards are in place  Finally, HUD could benefit from additional procedures to assess RAD properties for risks to long-term preservation to be able to respond to property default or foreclosure  We are making the following five recommendations to HUD: HUD’s Assistant Secretary for Housing should include provisions in its postclosing monitoring procedures to collect comprehensive high quality data on financial outcomes upon completion of construction, which could include requiring third-party certification of and collecting supporting documentation for all financing sources and costs  (Recommendation 1) HUD’s Assistant Secretary for Housing should improve the accuracy of RAD leverage metrics—such as better selecting inputs to the leverage ratio calculation and clearly identifying what the leverage ratio measures—and calculate a private-sector leverage ratio  (Recommendation 2) HUD’s Assistant Secretary for Housing should prioritize the development and implementation of monitoring procedures to ensure that resident safeguards are implemented  (Recommendation 3) HUD’s Assistant Secretary for Housing should determine how it can use available program-wide data from public housing and Section 8 databases, in addition to resident logs, for analysis of the use and enforcement of RAD resident protections  (Recommendation 4) HUD’s Assistant Secretary for Housing should prioritize the development and implementation of procedures to assess risks to the preservation of unit affordability  (Recommendation 5) We provided a draft of this report to HUD for comment  HUD provided written comments on the draft report, which are summarized below and reproduced in appendix IV  HUD also provided technical comments, which we incorporated as appropriate  In its comment letter, HUD stated that it agreed with our findings that HUD can improve metrics used to assess program impact and build on existing oversight structures  HUD described actions it intends to take to implement our recommendations to the extent possible and consistent with resource limitations  More specifically, HUD agreed with our first recommendation to ensure it collects comprehensive quality data on financial outcomes in its postclosing monitoring procedures (which could include supporting documentation for all financing sources and costs)  HUD agreed it should routinely collect an updated list of funding sources and uses and related documentation when projects had cost overruns or other significant changes  HUD intends to review and revise, as appropriate, required postcompletion certifications  HUD added that in most cases, funding sources and uses do not materially change between closing and construction completion  HUD stated that securing the postclosing information in such cases might be of minimal benefit relative to the additional reporting burden  However, it is not clear how HUD would determine if projects had significant changes in costs or uses because HUD lacks postcompletion information that would show the magnitude of changes  In relation to reporting burden, HUD already has implemented procedures to collect limited financial information following the completion of construction in October 2017  We believe any additional reporting would not be disproportionate to the benefits of improving HUD's oversight capabilities through project completion and enhancing its reporting to more accurately reflect the results of the demonstration program  For our second recommendation to improve the accuracy of RAD leverage metrics and calculate a private-sector leverage ratio, HUD agreed that RAD leverage metrics can be improved  HUD will ensure that the private-sector leverage ratio required by statute is clearly identified and included in its RAD evaluation  HUD also intends to identify a small number of relevant leverage ratios with distinct methodologies and will routinely publish these ratios with clear identification and explanations  In relation to our finding of misidentified funding sources, HUD plans to re- examine its chart of accounts and review prior transaction records to address errors and properly classify transaction sources  In response to our third recommendation to prioritize the development and implementation of monitoring procedures for resident safeguards, HUD agreed that it is important to better document and expedite development and implementation of monitoring procedures  HUD also agreed that additional monitoring was needed to ensure the right of residents to request and move with a tenant-based voucher after a period of residency (choice-mobility)  HUD noted that its Office of Policy Development and Research is seeking funding for additional research on RAD with a focus on the use and effect of choice-mobility options, which would inform HUD's monitoring efforts  Finally, while HUD said that we did not find the safeguards to be weak or inadequate, we did not perform an audit designed to assess the safeguards and therefore cannot opine on their adequacy  On the basis of our findings, we found that HUD’s implementation of these safeguards could be strengthened  Regarding our fourth recommendation that HUD determine how it can use available program-wide data and resident logs for analysis of RAD resident protections, HUD agreed to examine how it could use its existing data systems to further enhance its monitoring efforts  HUD added that the systems have limitations, so that the agency also uses other mechanisms to track and monitor implementation of resident protections  For our fifth recommendation to prioritize the development and implementation of procedures to assess risks to the preservation of unit affordability, HUD agreed that it is important to assess and mitigate risks to unit affordability  HUD stated that it employs robust underwriting standards prior to permitting conversion, and relies on existing procedures to conduct ongoing oversight of Project-Based Rental Assistance (PBRA) properties, which we discussed in the draft  However, as we noted, HUD has not yet developed procedures to more closely monitor RAD properties at risk of foreclosure, though it plans to establish indicators of foreclosure risk and oversight roles and responsibilities within HUD  HUD said that since the summer of 2017, it has been evaluating what additional oversight procedures might be needed for RAD Project-Based Voucher properties  HUD also described plans to augment its existing oversight procedures to preserve affordable units in the event of foreclosure by developing protocols in the following areas: transfer of property ownership to a capable entity, transfer of the rental assistance to another site, and protection of residents in the event a Housing Assistance Payment contract was terminated  As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date  At that time, we will send copies to the Secretary of Housing and Urban Development and other interested parties  In addition, the report will be available at no charge on the GAO website at http://www gao gov  If you or your staff have any questions about this report, please contact me at (202) 512-8678 or garciadiazd@gao gov  Contact points for our Offices of Congressional Relations and Public Affairs are listed on the last page of this report  GAO staff who made major contributions to this report are listed in appendix IV  This report examines aspects of the Department of Housing and Urban Development’s (HUD) Rental Assistance Demonstration (RAD) program  More specifically, this report addresses (1) HUD’s assessment of the physical and financial outcomes of RAD conversion to date; (2) how RAD conversions affected residents and what safeguards were in place to protect them, including while temporarily relocated and during conversion; (3) what challenges, if any, public housing agencies (PHA) faced in implementing RAD; and (4) the extent to which RAD provisions are designed to help preserve the long-term affordability of units  To address all four objectives, we analyzed agency documentation and interviewed officials from HUD  The documentation we reviewed included policies and procedures for RAD; manuals describing HUD data systems; draft policies and procedures for implementing postclosing oversight; and reports on RAD performance  We interviewed HUD headquarters officials from the Office of Recapitalization within the Office of Housing, which oversees the administration of RAD, and the Office of Policy Development and Research (PD&R)  We also interviewed PHA officials and developers involved in RAD transactions, as well as selected experts and other stakeholders to obtain their perspectives on RAD  Additionally, we conducted a literature search to identify publications related to RAD  We visited a nonprobability sample of eight PHAs in Maricopa County, Arizona; Alameda County, California; Montgomery County, Maryland; and in the cities of San Francisco, California; Baltimore, Maryland; New Bern, North Carolina; El Paso, Texas; and Tacoma, Washington, to observe housing units before, during, or after renovation when possible as well as common areas that had planned or undergone renovation  We selected sites to include varying PHA sizes, RAD subsidy types, planned rehabilitation and resident relocation, numbers and sizes of RAD transactions, closing dates, constructions costs, and geographic locations across the United States  At each site, we conducted semistructured interviews with PHA officials and, when available, developers (5 sites)  We also conducted one or two focus-group interviews with groups of 6– 15 residents who lived at the converted properties to obtain their perspectives and experiences  In each location, we asked the PHAs to invite residents to participate in the focus groups based on their availability  We also met with the Resident Advisory Board in each location that had one  For 7 of 8 site visits, we selected two RAD properties to conduct resident focus groups (in Alameda County, California we held one focus group)  We conducted a content analysis based on resident focus group interviews to describe resident experiences and the RAD program’s effects on residents  Utilizing the selection criteria noted above, we conducted semistructured telephonic interviews with an additional nonprobability sample of 10 PHAs in Fresno, California; Fort Collins, Colorado; Dekalb County, Georgia; Chicago, Illinois; Ypsilanti, Michigan; Cuyahoga County, Ohio; Philadelphia; Pennsylvania; Spartanburg, South Carolina; McKinney, Texas; and Yakima, Washington  Because we selected a non-probability sample of PHAs to visit and interview, the information we obtained cannot be generalized more broadly to all PHAs  However, it provides context on RAD particularly on implementation challenges and perspectives on physical and financial impacts, long-term affordability, and resident protections  We also selected the following 11 individuals and organizations as experts and stakeholders: 1  Council of Large Public Housing Authorities 2  National Association of Housing and Redevelopment Officials 3  Center on Budget and Policy Priorities 4  Public Housing Authorities Directors Association 5  National Housing Law Project 6  Community Legal Services of Philadelphia 7  Maryland Legal Aid 8  Disability Rights Maryland 9  Jaime Alison Lee, Associate Professor of Law and Director, Community Development Clinic, University of Baltimore School of Law 10  Yumiko Aratani, Assistant Professor, Columbia University Mailman 11  University of California, Berkeley, Terner Center for Housing We interviewed experts and stakeholders on resident impacts and implementation challenges associated with RAD  The entities may not represent all views on these topics, but their views provide insights on RAD  To select these individuals and groups, we met with three major PHA associations and two resident advocacy groups, and asked for referrals for organizations or individuals with expertise in RAD  We also selected a nonprobability, random sample of 31 RAD conversion files to review  Utilizing HUD RAD Resource Desk data, we randomly selected 31 RAD files for properties that had closed conversion as of June 30, 2017 and that planned to incur construction costs  We used the files to help us determine physical changes to RAD conversions and the impacts of RAD on residents through, for example, relocation  We excluded RAD conversions with no construction costs from the random sample because they would not have physical changes and no resident relocation would occur before or during our review  To address our first objective on the physical and financial outcomes of RAD conversion to date and how HUD measured these outcomes, we first obtained and analyzed HUD data on RAD conversions since RAD’s authorization (from fiscal years 2013 through 2017)  We assessed the reliability of these data by reviewing system documentation, interviewing knowledgeable officials about system controls, and conducting electronic testing  We determined that the data were sufficiently reliable for the purposes of describing rehabilitation and new construction in RAD projects and evaluating RAD leveraging metrics  We included in our analysis all RAD conversions that were active or closed  We used these data to determine the number of closed RAD conversions, associated financial sources and uses, subsidy types, and type of construction (rehabilitation, new construction, and no rehabilitation or new construction)  In addition, during our interviews with PHAs and developers, we obtained their perspectives on potential contributing factors to financial decisions and type of construction pursued through RAD conversion  As noted earlier, we also reviewed 31 randomly selected files of converted properties with construction costs to describe property physical changes in RAD conversions  Furthermore, we reviewed HUD documents, such as HUD and PD&R evaluations, publications, and policies and procedures to gain additional context for how HUD measures RAD outcomes  We also interviewed HUD officials, including PD&R and Office of Recapitalization officials, on RAD data and metrics, as well as other performance monitoring activities  We further analyzed data from the HUD RAD Resource Desk to determine how these data support HUD’s metrics and performance monitoring activities  As previously mentioned, we determined that these HUD data were sufficiently reliable for the purposes of this report  Specifically, we assessed and calculated RAD leverage ratio and construction activity  We assessed HUD’s performance monitoring activities and reporting against the RAD authorizing statute, Standards for Internal Control in the Federal Government  To recalculate estimates of the RAD leverage metric, we obtained documentation from the Office of Recapitalization to review the methodology used to calculate their most recent leverage ratio  We aligned the methodology they provided with RAD Resource Desk Transaction Log data that was downloaded on August 7, 2017  We replicated HUD’s methodology and matched the data utilized with the descriptors from the Transaction Log  To isolate financial sources and manually adjust the “other source” data, we compiled matched descriptors and funding amounts and categorized each observation based on the funding source description, as a federal source, state/county/city source, or PHA source, among others  For additional information and results, see appendix II  To determine how RAD affected residents in converted units, we analyzed HUD public housing and Section 8 household data before and after conversion (demographic characteristics of residents and changes in rent, income, and location)  Specifically, we examined data from 2013— when the first transactions closed—through June 30, 2017  HUD compiled and provided custom extracts of data on households in RAD- converted properties from the Inventory Management System/Public and Indian Housing Information Center (IMS/PIC) (public housing and Section 8 PBV) and Tenant Rental Assistance Certification System (Section 8 PBRA)  We assessed the reliability of the data extracts provided by HUD by (1) performing electronic testing of required data elements, (2) reviewing existing information about the data and the system that produced them, and (3) interviewing agency officials knowledgeable about the data  We determined the data on PBV households were sufficiently reliable for the purposes of our reporting objectives, but that the data on PBRA households was not sufficiently reliable for purposes of describing some characteristics of RAD households  For example, in trying to determine participation in the RAD program by year, we received several thousand PBRA entries that preceded the establishment of the RAD program  Moreover, as we previously mentioned, the postconversion household data for PBRA conversions is in a separate data system, so some variables, such as those related to race, ethnicity, rent, and income, differ from the other household data for that program  Because of these limitations, the data for PBRA households were not reliable for purposes of comparing RAD household characteristics before and after conversion as we had intended  To describe safeguards for residents and help ascertain how HUD implemented protections, we reviewed legal protections and requirements in HUD notices, reviewed selected conversion files, and interviewed HUD officials about monitoring and compliance processes  Finally, as previously described, we held focus groups with residents to better understand any effects on their living conditions and quality of life  To determine challenges PHAs faced in implementing RAD, we reviewed HUD guidance and related documents for PHAs in the program  We also interviewed eight PHAs during our site visits and spoke with another 10 PHAs by telephone about the benefits and challenges of participating in the RAD program  To examine provisions designed to help preserve long-term affordability of units, we reviewed the RAD authorizing statute and amendments and HUD notices and interviewed HUD staff to verify our understanding of agency affordability protections  For a sample of 31 randomly selected properties, we examined templates for contractual agreements for RAD closings and analyzed closing documents and contracts to determine if agreements matched program requirements  We interviewed HUD staff and staff of 18 PHAs to obtain viewpoints on the potential strengths or weaknesses of preservation in the case of default or foreclosure  We conducted this performance audit from February 2016 to February 2018 in accordance with generally accepted government auditing standards  Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives  We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives  The Department of Housing and Urban Development’s (HUD) Office of Recapitalization collects financial sources and use data from Rental Assistance Demonstration (RAD) participants  Table 2 lists the financial source fields collected by HUD  Table 3 lists the financial cost fields collected by HUD  Table 4 provides additional financial source detail pertaining to HUD’s leverage ratio calculation  Table 5 and Table 6 show the total financial source amounts collected by HUD  Specifically, Table 5 shows total financial source amounts prior to recategorization, while Table 6 shows total financial source amounts after manual adjustments  Manual adjustments included isolating funding source observations in “other funding” fields 1-6 and incorporating them into existing fields, as appropriate  Table 7 replicates HUD’s methodology for calculating the RAD leverage metrics after manual adjustments in HUD data  See Table 4, above, to compare changes in each category  Table 8 recalculates the leverage ratio by deducting federal sources as leveraged sources  Table 9 recalculates the leverage ratio by deducting public sources as leveraged sources (compare to Table 8 above)  The Rental Assistance Demonstration (RAD) program has numerous requirements intended to ensure residents whose units are converted through RAD receive certain protections  The following is a description of these safeguards and their reporting and monitoring requirements  The Government Accountability Office, the audit, evaluation, and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people  GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions  GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability  The fastest and easiest way to obtain copies of GAO documents at no cost is through GAO’s website (https://www gao gov)  Each weekday afternoon, GAO posts on its website newly released reports, testimony, and correspondence  To have GAO e-mail you a list of newly posted products, go to https://www gao gov and select “E-mail Updates ” The price of each GAO publication reflects GAO’s actual cost of production and distribution and depends on the number of pages in the publication and whether the publication is printed in color or black and white  Pricing and ordering information is posted on GAO’s website, https://www gao gov/ordering htm  Place orders by calling (202) 512-6000, toll free (866) 801-7077, or TDD (202) 512-2537  Orders may be paid for using American Express, Discover Card, MasterCard, Visa, check, or money order  Call for additional information  Connect with GAO on Facebook, Flickr, Twitter, and YouTube  Subscribe to our RSS Feeds or E-mail Updates  Listen to our Podcasts  Visit GAO on the web at https://www gao gov  James-Christian Blockwood, Managing Director, spel@gao gov, (202) 512-4707 U S  Government Accountability Office, 441 G Street NW, Room 7814, Washington, DC 20548 Please Print on Recycled Paper  In addition to the individual named above, Paul Schmidt (Assistant Director), Julie Trinder-Clements (Analyst in Charge), Meghana Acharya, Enyinnaya David Aja, Alyssia Borsella, Juan J  Garcia, Ron La Due Lake, Amanda Miller, Marc Molino, Barbara Roesmann, Jessica Sandler, MaryLynn Sergent, Rachel Stoiko, and William Woods made major contributions to this report 

ground_truth: HUD administers the Public Housing program, which provides federally assisted rental units to low-income households through PHAs. In 2010, HUD estimated its aging public housing stock had $25.6 billion in unmet capital needs. To help address these needs, the RAD program was authorized in fiscal year 2012. RAD allows PHAs to move (convert) properties in the public housing program to Section 8 rental assistance programs, and retain property ownership or transfer it to other entities. The conversion enables PHAs to access additional funding, including investor equity, generally not available for public housing properties. GAO was asked to review public housing conversions under RAD and any impact on residents. This report addresses, among other objectives, HUD's (1) assessment of conversion outcomes; (2) oversight of resident safeguards; and (3) provisions to help preserve the long-term affordability of units. GAO analyzed data on RAD conversions through fiscal year 2017; reviewed a sample of randomly selected, nongeneralizable RAD property files; and interviewed HUD officials, PHAs, developers, academics, and affected residents. The Department of Housing and Urban Development (HUD) put procedures in place to evaluate and monitor the impact of conversion of public housing properties under the Rental Assistance Demonstration (RAD) program. RAD's authorizing legislation requires HUD to assess and publish findings about the amount of private-sector leveraging. HUD uses a variety of metrics to measure conversion outcomes. But, the metric HUD uses to measure private-sector leveraging—the share of private versus public funding for construction or rehabilitation of assisted housing—has limitations. For example, HUD's leveraging ratio counts some public resources as leveraged private-sector investment and does not use final (post-completion) data. As a result, HUD's ability to accurately assess private-sector leveraging is limited. HUD does not systematically use its data systems to track effects of RAD conversions on resident households (such as changes in rent and income, or relocation) or monitor use of all resident safeguards. Rather, since 2016, HUD has required public housing agencies (PHA) or other post-conversion owners to maintain resident logs and collect such information. But the resident logs do not contain historical program information. HUD has not developed a process for systematically reviewing information from its data systems and resident logs on an ongoing basis. HUD has been developing procedures to monitor compliance with some resident safeguards—such as the right to return to a converted property—and begun a limited review of compliance with these safeguards. However, HUD has not yet developed a process for monitoring other safeguards—such as access to other housing voucher options. Federal internal control standards require agencies to use quality information to achieve objectives, and obtain and evaluate relevant and reliable data in a timely manner for use in effective monitoring. Without a comprehensive review of household information and procedures for fully monitoring all resident safeguards, HUD cannot fully assess the effects of RAD on residents. RAD authorizing legislation and the program's use agreements (contracts with property owners) contain provisions intended to help ensure the long-term availability of affordable units, but the provisions have not been tested in situations such as foreclosure. For example, use agreements between HUD and property owners specify affordability and use restrictions that according to the contract would survive a default or foreclosure. HUD officials stated that HUD intends to develop procedures to identify and respond to risks to long-term affordability, including default or foreclosure in RAD properties. However, HUD has not yet done so. According to federal internal control standards, agencies should identify, analyze, and respond to risks related to achieving goals and objectives. Procedures that address oversight of affordability requirements would better position HUD to help ensure RAD conversions comply with program requirements, detect potential foreclosure and other risks, and take corrective actions. GAO makes five recommendations to HUD intended to improve leveraging metrics, monitoring of the use and enforcement of resident safeguards, and compliance with RAD requirements. HUD agreed with our recommendations to improve metrics and build on existing oversight.