Military Housing: DOD Needs to Strengthen Oversight and Clarify Its Role in the Management of Privatized Housing
Released: 2020-03-26T00:00:00

Letter

Background Military Housing Privatization Authorities and Project Structures

DOD’s policy is to ensure that eligible personnel and their families have access to affordable, quality housing facilities and services consistent with grade and dependent status, and that the housing generally reflects contemporary community living standards. From the inception of MHPI, the military departments were provided with various authorities to obtain private-sector financing and management to repair, renovate, construct, and operate military housing in the United States and its territories. These authorities included the ability to make direct loans to and invest limited amounts of funds in projects for the construction and renovation of housing units for servicemembers and their families. The projects were generally financed through both private-sector financing, such as bank loans and bonds, and funds provided by the military departments. The Army and the Navy generally structured their privatized housing projects as limited liability companies in which the military departments formed partnerships with the developers and invested funds in the partnership.

The Air Force generally provided direct loans to the developers. Because privatized housing projects involve budgetary commitments of the federal government, each project was scored at inception by the Office of Management and Budget to determine the amount of funds that needed to be budgeted for that particular project.

The military departments have flexibility in how they structure their privatized housing projects, but typically the military departments lease land to developers for a 50-year term and convey existing housing located on the leased land to the developer for the duration of the lease. The developer then becomes responsible for renovating and constructing new housing and for the daily management of the housing units. At the end of fiscal year 2017, 14 private partners were responsible for 79 privatized military family housing projects—34 for the Army, 32 for the Air Force, and 13 for the Navy and the Marine Corps. See appendix II for a list of all of these housing projects.

Each privatized housing project is a separate and distinct entity governed by a series of legal agreements that are specific to that project, hereafter referred to as business agreements. These agreements include, among other things, an operating agreement, a property management agreement, and an agreement that describes the management of funds in the projects, including the order in which funds are allocated within the project. However, while each project is distinct, there are some common elements in how projects invest and use funds. Every project takes in revenue, which consists mostly of rent payments. Projects then pay for operating expenses, including administrative costs, day-to-day maintenance, and utilities, among other things. After that, projects generally allocate funds for taxes and insurance, followed by debt payments. Figure 1 shows a typical funding structure for a privatized housing project.

In the typical privatized housing project depicted in figure 1, once debt payments are made, funds are allocated to accounts that fund scheduled maintenance, such as repair and replacement of items like roofs, heating and cooling systems, and infrastructure. After that, funds are allocated to a series of management incentive fees, such as the property management fee. Finally, the project divides these remaining funds according to a fixed percentage between accounts that (1) fund major renovations and rebuilds and (2) are provided to the developer. The percentages may vary across agreements, but according to military department documentation, typically, the majority of funds go toward the accounts funding major renovations and rebuilds.

Most of the projects’ business agreements also include the option for the private partners to receive performance incentive fees based on achieving the performance metrics established in each individual project’s business agreement. These fees are intended to incentivize private partner performance. The incentive fees can be paid to private partners on an annual or quarterly basis and can be withheld in part or in total if the private partner fails to meet the established metrics. The weight each performance metric and underlying indicator carries toward the incentive fee varies by project, so incentive fees for some projects may be heavily dependent on financial performance, while others may be more heavily weighted toward resident satisfaction.

DOD Goals, Roles, and Responsibilities in the Privatized Housing Program

The Deputy Assistant Secretary of Defense for Facilities Management, under the authority, direction, and control of the Assistant Secretary of Defense for Sustainment, is responsible for all matters related to MHPI and is the program manager for all DOD housing, whether DOD-owned, DOD-leased, or privatized. In this capacity, the Deputy Assistant Secretary is to provide both guidance and general procedures related to military housing privatization, as well as required annual reports to Congress on privatized military housing projects. However, it is the responsibility of the military departments to execute and manage the privatized housing projects, including conducting financial management and monitoring their portfolio of projects. Each military department has issued guidance that outlines its responsibilities for privatized housing, such as which offices are responsible for overseeing privatized housing projects. See figure 2 for details on each military department’s roles and responsibilities in the MHPI program.

Prior GAO Work

We have previously reported on DOD’s privatized housing program. In 2002, we reported that although military installation officials were participating with developers in making improvement decisions for selected projects, DOD and military department headquarters oversight of those decisions appeared limited. We recommended, among other things, that DOD implement several changes to enhance government protections in the privatization program, such as requiring service headquarters and the OSD to review and approve privatization project reinvestment account expenditures over an established threshold. DOD generally agreed with our recommendations and took steps to implement them. Specifically, DOD revised guidance to establish new rules and thresholds for review and approval of project reinvestment expenditures, among other things.

In addition, in 2006, we reported that although DOD and the individual military departments implemented program oversight policies and procedures to monitor the execution and performance of privatized housing projects, opportunities existed for improvement. Specifically, we reported that the value of DOD’s semiannual report to Congress was limited because it lacked a focus on key project performance metrics to help highlight any operational concerns. We also reported that data collected on servicemember satisfaction with housing, important for tracking satisfaction over time, were inconsistent and incomplete because DOD had not issued guidance for the standardized collection and reporting of such information. We recommended, among other things, that DOD streamline its report to Congress to focus on key project performance metrics and to provide guidance to the military departments to ensure the consistent collection and reporting of housing satisfaction information from all servicemembers. DOD generally agreed with our recommendations and took steps to implement them. For example, DOD took steps to streamline its report to Congress and update its guidance directing the services to ensure consistent reporting using a numerical rating system to rank housing satisfaction information.

DOD Conducts Some Oversight of the Condition of Privatized Housing, but Efforts Are Limited in Key Areas

OSD and each of the military departments conduct a range of activities to oversee both the condition of privatized housing and performance of the private partners and have recently implemented initiatives to improve this oversight—such as increasing the frequency of the physical inspection of homes and issuing guidance to ensure consistency in the framework used to measure project performance. However, we found that these oversight efforts remain limited. Specifically, our review showed (1) the scope of oversight of the physical condition of privatized housing has been limited; (2) performance metrics focused on quality of maintenance and resident satisfaction may not accurately reflect private partner performance related to the condition of privatized housing; (3) there is a lack of reliable or consistent data on the condition of privatized housing; and (4) past DOD reports to Congress on resident satisfaction are unreliable due to the inconsistent handling and calculation of the data and therefore may be misleading.

Military Departments Conduct Some Oversight of the Physical Condition of Privatized Housing, but the Scope of These Efforts Is Limited

The military departments have taken steps to oversee the condition of their privatized military housing inventory and each has issued guidance that outlines their respective oversight roles and responsibilities, but the scope of these oversight efforts has been limited. Military department oversight activities generally fall into two categories—(1) daily oversight of management and operations and (2) periodic reviews of compliance with each project’s business agreements.

Daily oversight of management and operations. Each installation has a military housing office that is responsible for conducting daily oversight of a project’s management and operations. Military housing officials told us that activities to monitor the physical condition of housing units generally include reviewing sample work order requests, following up with a sample of residents to check on their experience with recently completed work, and inspecting homes during the change-of-occupancy process. However, the implementation and scope of these activities varies and can be limited. For example, during our site visits conducted from June through August 2019, we identified the following installation- specific practices:

The rate of inspections of homes following change-of-occupancy maintenance at the installations we visited varied. For example, at the time of our site visits, military housing office officials at Tinker Air Force Base, Oklahoma, told us that they inspect 100 percent of homes that have completed change-of-occupancy maintenance, while officials from Langley Air Force Base, Virginia, stated that they inspect 10 to 20 percent of these homes. In November 2019, Air Force officials told us that they are moving to a 100-percent inspection policy. Similarly, the Army issued an order in March 2019 directing military housing office officials to inspect 100 percent of homes where change-of-occupancy maintenance has been completed. Officials from Army installations we visited noted that this was an increase from previous practices, and for one installation was a change in practice from conducting inspections only during the move-out process, which occurs prior to change-of-occupancy maintenance.

According to Department of Navy officials, the Navy’s business agreements stipulate that Navy and Marine Corps installations have access to all work order information. However, practices for following up on work order records varied among some of the Navy and Marine Corps installations we visited. For example, military housing office officials at Camp Pendleton, California, told us that for one of the two partners that own housing on the base, they had access to only 3 percent of completed work orders from the previous month. For the other partner that owns housing on the base, military housing office officials noted that the partner provided them with nine work orders of varying priority each month to review. One military housing office official added that these were the minimum requirements needed for monthly reporting and that they were working with the private partner to increase their access to work order records. Following a different practice, military housing office officials at Naval Station Norfolk, Virginia, told us that they had access to the private partner’s maintenance record system and would pull reports on homes that had made six or more maintenance calls in a 30-day period.

Periodic reviews of compliance with each project’s business agreements. Periodic reviews of compliance with a project’s business agreements are a joint effort between the local military housing office, the private partners, military department installation commands, and other echelons of command. These reviews can include neighborhood tours to view project amenities such as community centers, playgrounds, and pools, all of which are owned, maintained, and operated by the private partner companies, as well as exteriors of housing units. However, similar to the daily oversight activities, these annual reviews have been narrow in the scope of their assessment of the physical condition of the housing units, as interior walk-throughs were, at times, focused on just a few homes at each installation. For example:

The Air Force Civil Engineer Center is the primary oversight and governance body for the Air Force’s privatized housing projects. The Air Force oversight process includes periodic compliance reviews of all privatized housing projects. To accomplish this task, the Air Force is to use a compliance checklist to review the private partner’s compliance with a project’s business agreements. In addition to the compliance reviews, guidance states that Air Force Civil Engineer Center officials visit projects annually, and officials told us that they tour a sample of homes and interview private partner representatives, military housing office staff, and residents during these visits. However, according to selected annual site visit reports we reviewed and a discussion with an Air Force official, annual site visit reports typically include only an evaluation of three to four housing units on an installation and can be restricted to empty units or units that have completed change-of-occupancy maintenance, limiting the robustness of the assessment of the installation’s housing units’ physical condition.

According to Department of the Navy officials, the Navy and the Marine Corps provide oversight of privatized housing projects through a tool called the monitoring matrix. Officials from the various organizational entities involved with privatized housing—to include the Commander, Naval Installation Command; the Naval Facilities and Engineering Command; and the military housing office—are to use this monitoring matrix to periodically review private partner compliance with a project’s business agreements. The matrix contains a condition assessment component, which includes a tour of privatized housing neighborhoods and a visual inspection of individual privatized housing units. However, similar to the Air Force, according to select assessments we reviewed and a discussion with a military housing office official, the visual inspections are typically focused on two to three homes in each neighborhood on an installation and to homes that have recently undergone change-of-occupancy maintenance.

Army guidance calls for the U.S. Army Corps of Engineers to conduct an annual ground lease inspection to review private partner compliance with a project’s business agreements. The guidance also calls for the Army’s program manager to conduct an annual installation visit to each project to evaluate performance and ensure a project’s compliance with the business agreements. The visit is to include a recommended site tour, described in guidance as a brief visual inspection tour of community elements, and a walk-through visual inspection of at least four housing units—two renovated and two recently built—including one unit designated as an accessible home under federal guidelines. However, according to a May 2019 report by the Army Inspector General, these requirements were inconsistently met, and the results did not include a follow-up process and were not communicated to senior commanders.

Through the recent housing reviews that they have conducted, each military department’s internal oversight body has recognized that the departments’ oversight guidance has been limited in addressing the condition of privatized homes and provides little clarity to housing officials about their roles and responsibilities in assessing the physical condition of homes. For example, in May 2019, the Department of the Army Inspector General reported that senior commanders and garrison staffs expressed confusion concerning the roles, responsibilities, and authorities regarding privatized housing and that oversight, governance, and synchronization were insufficient to identify current housing challenges. Similarly, an April 2019 report from the Air Force Inspector General noted that ambiguous guidance had resulted in inconsistent action and uneven performance across Air Force housing projects. In addition, a November 2019 report by the Naval Audit Service identified nine separate guidance documents for the oversight of privatized housing and found that personnel at installation and regional levels were unclear on the guidance and requirements for performing oversight of privatized housing.

According to military department officials, each department has completed initiatives and is undertaking initiatives to revise guidance and standardize daily oversight activities in an effort to provide consistent oversight across projects and installations and to increase the focus on the physical condition of housing. In addition, the military departments have initiatives to increase staffing levels, improve training for military housing office officials, and ensure that military department housing officials have independent access to work order data to strengthen their oversight activities. Figure 3 outlines examples of completed and ongoing initiatives by military department to improve the oversight of privatized housing.

However, each military department is working to implement service- specific initiatives with minimal guidance from OSD on the level of oversight expected as it relates to the condition of privatized housing. OSD guidance as it pertains to the condition of privatized housing is limited compared with the guidance OSD provides for monitoring the condition of military-owned housing.

Specifically, OSD guidance is focused on the oversight of the implementation of projects, the construction of new housing units, and project financial monitoring. The guidance stipulates that after privatized housing projects are awarded, monitoring should include descriptions of deal structure and strategies for project monitoring. In contrast, OSD guidance for military-owned housing provides clearly defined objectives to the military departments for oversight, including the physical condition of the homes. For example, the DOD manual for housing management directs the military departments to provide managerial oversight of DOD’s government-owned family housing to ensure that (1) the required inventory is being provided and maintained in good condition, (2) the program is being operated in an effective and cost-efficient manner, and (3) servicemembers and their families have adequate housing choices. Further, the manual provides specific objectives for the condition of DOD’s government-owned family housing, stating that for DOD family housing to be considered adequate overall, it must meet minimum standards for configuration, privacy, condition, health, and safety. It also states that military service condition assessments shall use private-sector housing industry and DOD standards or codes as a basis for assessing inventory adequacy. The manual adds that for DOD government-owned family housing to be considered in adequate condition, the construction cost for all needed repairs and improvements cannot exceed 20 percent of the replacement cost.

According to DOD’s housing manual, program assumptions for privatized housing are that privatization allows the military departments to work with the private sector to generate housing built to market standards. While the military departments’ policies provide for some measureable oversight activities, such as requiring a certain number or type of home to be inspected, OSD has not provided guidance to the military departments clearly defining oversight objectives for monitoring the physical condition of privatized housing units.

DOD’s housing manual further states that because privatization creates a long-term governmental interest in privatized housing, it is essential that projects be attentively monitored. The 50-year term for the ground leases creates a long-term interest in monitoring the privatized housing assets, to include the physical condition of the housing units. However, unless DOD updates its guidance on the oversight of privatized housing with objectives for overseeing the physical condition of housing units, it cannot be assured that the military departments’ oversight activities will be sustained over time or be sufficiently consistent across projects, raising the risk that private partners may not provide adequate quality housing. Notably, the military departments have entered into privatized housing agreements with some of the same companies, and members of different military services may live at installations managed by military services different than their own. As such, it is important that oversight expectations generally be consistent across the military departments and the projects they manage.

Moreover, all military departments have an interest in ensuring that residents feel confident that the private partners will be held to a consistent standard for maintaining the condition of their homes. Participants in 8 of our 15 focus groups stated that they will no longer live in privatized housing following their current experience, and participants in 6 of our 15 focus groups stated that their current experience with privatized housing will affect the future career decisions for their family. One participant stated that he plans to exit the service after 8 years, noting that his decision is largely based on his experience with privatized housing. In addition, in our online tool we asked residents if their experience with privatized housing would impact their future career and housing decisions. For those residents that responded to these questions, the majority said their experience will make them less likely to continue to live in privatized housing in the future. For example, one respondent stated that while living in privatized housing is a benefit to being in the military, living in housing that is subpar and where nothing seems to be getting fixed or at least acknowledged makes the family hesitant to live in privatized housing again. Some residents also indicated that their experience would impact their future career decisions.

DOD Uses Several Metrics to Monitor Private Partner Performance, but the Indicators Underlying Those Metrics May Not Provide Meaningful Information on the Condition of Privatized Housing

The military departments each use a range of project-specific performance metrics to monitor private partner performance. However, the indicators underlying the metrics designed to focus on resident satisfaction and on the quality of the maintenance conducted on housing units may not provide meaningful information or reflect the actual condition of the housing units. For example, in April 2019 the Air Force Inspector General reported that the current incentive structure measures many things with precision, but does not measure the right things. Private partner performance is commonly measured through four key metrics—resident satisfaction, maintenance management, project safety, and financial management. To determine how well the private partners are performing under these metrics, military housing office officials told us that they rely on a range of indicators established in the project business agreements. Table 1 provides examples of various indicators that the performance metrics comprise.

According to officials from each military department, the performance metrics and their underlying indicators are a key tool that each military department uses to hold private partners accountable for providing quality management of the privatized housing projects. However, we found that the indicators themselves may not reflect how the private partner is performing in terms of providing servicemembers and their families with quality services and housing. For example:

Maintenance management: One commonly used indicator of performance in maintenance management measures how often the property manager’s response time to work orders meets required time frames established in the project’s business agreements. While this indicator measures the timeliness of the private partner’s response, it does not measure or take into account the quality of the work that was conducted or whether the resident’s issue was fully addressed. As such, a property manager may fully meet the metric for maintenance management, even if a given repair has not been adequately completed. Residents in 13 of our 15 focus groups noted that they typically have had to submit multiple work order requests before an individual maintenance issue has been fully addressed. For example, a resident who participated in one of our focus groups provided us with a copy of work orders she had submitted related to a single maintenance issue in her home. The first work order was marked completed on time, yet the resident had to submit a work order for the same issue a week later. Further, an official at one Army installation told us that since the incentive fee for the project is awarded on a quarterly basis, judging property managers only on the basis of work orders completed on time for that quarter could mask persistent ongoing housing problems. This is because many smaller work orders get closed out each quarter, while work orders for more complicated issues might stay open over multiple quarters. Some projects include indicators that aim to more directly measure quality, such as the number of work orders placed during the first 5 business days of residency. This type of indicator may more clearly indicate the extent to which change-of-occupancy maintenance was complete on a given home.

Resident satisfaction: One example of an indicator of resident satisfaction is whether a project has met target occupancy rates established in the project’s business agreements. An OSD official and private partner representatives told us they use occupancy as an indicator of satisfaction, based on the assumption that residents would move if they are dissatisfied with their home’s condition. However, according to the Army’s Portfolio and Asset Management Handbook, occupancy rates are not a recommended metric to monitor private partner performance because occupancy rates already impact project finances. Our focus groups and the responses we received to our online tool also indicate that this may not be a reliable assumption. Although most residents are not required to live in military housing, residents in each of our 15 focus groups and responses to our online tool indicated a variety of reasons for choosing to live in privatized housing, many of which do not have to do with their satisfaction with the quality or condition of their homes. For example, residents in our focus groups cited other factors influencing their decision to live in privatized housing, such as living in close proximity to military medical or educational services for children or other family members that are part of the military’s Exceptional Family Member Program, a lack of safe and affordable housing in the surrounding community, and access to quality schools. Volunteers that responded to our online tool also cited accessibility to base services, commute time, and safety as reasons for choosing to live in privatized housing.

Another commonly used indicator of resident satisfaction is the results of various resident satisfaction surveys, such as maintenance surveys and leasing surveys, as well as the annual satisfaction survey. The military departments and the private partners use these survey tools to gauge resident satisfaction with the maintenance conducted on their homes, service provided by property managers, and amenities provided in their community, among other things. However, residents in 4 out of our 15 focus groups indicated that the surveys they receive related to maintenance performed on their homes do not ask questions about the quality of maintenance work. For example, residents told us that maintenance surveys, which they generally receive after maintenance work is completed on their homes, ask if the maintenance worker was courteous, but not about the quality of the work performed on the home.

We reviewed maintenance surveys from 3 of the 10 installations we visited and found that the surveys asked residents to provide feedback on the quality of the work, with questions asking them to rate their satisfaction with the quality of the maintenance work completed. In addition, we reviewed a quarterly Army survey from one of the installations we visited and found that this survey asked residents about their satisfaction with the courteousness and professionalism of the maintenance team and the responsiveness and timeliness of maintenance work, but did not specifically ask about their satisfaction with the quality of the maintenance work completed.

We also found that the information used to support the indicators can vary. For example, officials at one Army installation—Fort Huachuca, Arizona—use quarterly resident surveys, the Army’s annual survey, and action plans on Army annual survey results as indicators of resident satisfaction. However, officials at another Army installation—Fort Knox, Kentucky—use residential community office relationship management and point of service surveys. Similarly, we found differences in the information used as indicators of the maintenance management metric. For example, officials at both Hickam Air Force Base, Hawaii, and Davis- Monthan Air Force Base, Arizona, rely on the timeliness and quality of change-of-occupancy maintenance as an indicator of maintenance management. However, officials at Hickam Air Force Base also use work order response and completion times as indicators of the maintenance management metric, whereas officials at Davis-Monthan Air Force Base, Arizona, only use work order response times.

Standards for Internal Control in the Federal Government state that management should evaluate performance and hold individuals accountable for their internal control responsibilities. If management establishes incentives, management should recognize that such actions can yield unintended consequences and evaluate incentives so that they align with the entity’s standards of conduct. The standards further state that management should use quality information to achieve the entity’s objectives, including relevant data from reliable sources.

In October 2019, OSD, in collaboration with the military departments and private partners, issued new guidance standardizing the performance incentive fee framework across the military departments. The new guidance provides a framework for standardizing the minimum and maximum percentages of the fee that each metric can account for, allowing for some flexibility in the weight each metric will carry for an individual project. Specifically, maintenance management and resident satisfaction can account for between 60 and 90 percent of the fee, project safety can account for between 5 and 15 percent of the fee, and financial performance can account for between 5 and 15 percent of the fee.

However, despite DOD’s efforts to ensure more focus on the condition and quality of, and resident satisfaction with, privatized housing through the standardization of metrics across the military departments, the metrics may be misleading if the specific underlying indicators used to determine whether a metric has been reached are not reevaluated on an ongoing basis to ensure they are accurate measures of the private partners’ performance and an accurate reflection of the condition and quality of privatized homes. OSD and military department officials have recognized that the current indicators for measuring performance do not consistently focus on or prioritize the private partners’ performance with maintaining housing units and ensuring resident satisfaction. For example, Army officials told us they are no longer using occupancy rate as an indicator of resident satisfaction and have taken steps to standardize performance indicators across all Army projects, while still allowing for flexibility at the installation level to modify the weight of indicators to provide incentives reflective of the specific needs of the installation. Limitations to the current indicators may hinder the military departments’ ability to accurately determine private partner performance. OSD and the military department officials told us they have not yet reevaluated the specific indicators used to determine whether a private partner has met a specific metric because doing so will require negotiation with each of the private partners for each project. However, without reviewing the specific indicators used to award performance incentives, OSD and the military departments do not have assurance that the information the military departments are using to award these incentives reflects the actual condition of the housing.

DOD and Private Partners Collect Maintenance Data on Privatized Housing, but These Data Are Not Captured Reliably or Consistently for Use in Ongoing Monitoring of Housing Units

Maintenance data collected by the private partners are not captured consistently or reliably across projects for use in ongoing monitoring of the condition of privatized housing units over time. The privatized housing projects’ business agreements typically include a requirement for the private partner to maintain a records management system to record, among other things, maintenance work requested and conducted on each housing unit. According to private partner representatives from all 14 companies, each company uses commercial property management software platforms for activities such as initiating maintenance work orders and dispatching maintenance technicians. Some private partner representatives stated that while data from the work order tracking systems are primarily used to prioritize and triage maintenance work, the data were never intended to monitor the overall condition of privatized housing units.

Military department officials told us that efforts are underway to monitor work order data from the private partners’ work order tracking systems in an effort to increase the military departments’ oversight and accountability of the private partners for providing quality housing to servicemembers. For example, the Army and the Navy are taking steps to create data dashboards to track installations’ work orders by priority, status, and category. However, while data from these work order tracking systems may be useful for point-in-time assessments of work order volume at a given installation, we found that these data are not captured reliably or consistently for use in ongoing monitoring of the condition of privatized housing units across projects and over time.

We received and reviewed data from each of the 14 private partners’ work order tracking systems covering each of the 79 privatized housing projects. Based on our review of these data and discussions with private partner representatives, we found two primary factors that would limit the reliability or consistency of using these data for ongoing monitoring of the condition of privatized housing units over time—(1) inconsistent use of terminology in work order records and (2) differing practices for opening and closing work orders.

Inconsistent Use of Terminology in Work Order Records

Data in these work order tracking systems include information such as records of resident requests for service, history of work conducted on specific housing units, change-of-occupancy maintenance performed, and work completed on common areas. Residents may request service for a broad range of issues, such as lost keys, broken appliances, ceiling or wall damage, lack of hot water, or water leaks or floods. According to private partner representatives, work orders can be entered into the system by property management office staff, maintenance technicians, or call center representatives for those companies that use offsite call centers to process resident service request calls. At some installations, residents can also enter work orders into the work order tracking system through online portals or mobile applications.

However, we noted cases where work orders were inconsistently entered into the work order tracking systems with respect to two primary factors— (1) how the request is described by the resident or interpreted by the individual entering the data, which can differ for each work order; and (2) the existing range of pre-established service category options in the private partner’s work order tracking system, which differ among the partners. According to private partner representatives, the individual responsible for entering the work order into the system—property management office staff, maintenance technicians, call center representatives, or residents—makes a judgment on how to categorize the work order.

These factors create challenges for looking at the data across projects. Private partner representatives from one installation we met with stated that the quality of the work order data is dependent on the data input into the system. In some cases, the data input can be inaccurate or imprecise, depending on the specificity with which a resident describes his or her maintenance issue or how a staff person enters the data into the system. A private partner representative from another installation we visited stated that reporting on data from the work order tracking system can be challenging because individuals across installations inputting data may have a different interpretation of a resident’s reported issue. Private partner representatives from another installation noted that the work order tracking system they used could not be easily updated with a new category if needed, making it more difficult to identify systemic issues. For example, there is one category for all exterior repairs, but no way to break that category down into what the specific repairs are, such as roofs. In the event that there is an issue with several roofs in the same area, the private partner representative said it would be hard to identify the issue because the only option available is to look through the notes section. According to this individual, the regional maintenance technicians, not the work order tracking system, are the best resource for identifying trends or recurring issues.

This inconsistent entering of information into the work order tracking systems, which occurs both within and across installations, means that the military departments cannot readily use the data to capture the prevalence of a particular issue, such as mold, among the homes in a reliable manner. For example, if someone wanted to use work order data to track instances of mold, he or she would find that these may be represented in the work order systems under a variety of service categories, such as mold or mildew, plumbing and bath, heating and cooling, or general. To isolate service requests related to mold, one may have to rely on using the service comments for each request, which can vary in their level of detail. In addition, service requests for mold issues may be entered into the work order systems under different priority levels, such as routine, urgent, or emergency. As a result of the variation in the type and amount of information collected in the work order tracking systems, work order data alone cannot be used to determine the validity of a service request, the severity of the problem, or whether the work was completed to a quality standard. Figure 4 shows examples of differences in how a perceived mold issue can be captured in these systems based on our review of the data provided by the private partners.

Military department officials found similar limitations when analyzing the work order data. According to some officials, one challenge in using the work order data for oversight is that, while there are good data in the individual records, people report and record things differently. Specifically, a Navy official working with these data told us they have to consider these differences and create unique algorithms to query data for each partner.

Differing Practices for Opening and Closing Work Orders

At some installations we visited, private partners noted changes in practices for opening or closing work orders, limiting the usefulness of data in monitoring the status of work orders over time and thus the condition of privatized housing. For example, according to private partner representatives at one installation we visited, a practice for tracking emergency work orders in the work order tracking system had changed in 2013. Work that comes in under an emergency priority may take several steps to complete: A maintenance technician may first have to stop the emergency, then clean up any resulting damage, before repairing the root cause and completing any finishing work. Prior to 2013, maintenance technicians would open and close new work orders for each step in the process. Under the new practice, the original work order is kept open until completion. Representatives from a different private partner described a similar change in practices, noting that if a work order was closed or recategorized before the work was finished there could be issues for how it is tracked, such as getting dropped out of the system and the work not getting done. A third partner noted the same practice, but added that an emergency work order can be downgraded to urgent or routine status during the time that the work is taking place. As a result, work order data alone may not accurately identify the number of open work orders at any given time, the time it took to address a maintenance issue, or if a maintenance request has been fully completed.

Additionally, we identified anomalies in the work order data provided to us from each of the 14 partners. For example, we identified instances of, among other things, duplicate work orders, work orders with completion dates prior to the dates that the resident had submitted the work order, and work orders still listed as in-progress for more than 18 months. According to military department officials, they have increased their efforts to review data from the private partners’ work order tracking systems and have found similar anomalies. For example, a Navy official working with work order data found that a couple of homes had six or seven unique work order records in the system, but each contained identical information in the various data fields. Officials from both the Navy and Air Force have come across work order records that were marked as complete within minutes of being entered into the system or marked as complete with a date prior to the work order being open, which signaled the need for further scrutiny.

Each military department has efforts underway to monitor private partner work order data in an effort to increase oversight of the quality of privatized housing. However, because neither OSD nor the military departments have identified minimum data requirements, established consistent terminology or practices for data collection, or developed processes for the military departments to validate the work order data collected by the private partners, data from these work order tracking systems are not reliable for use in the ongoing monitoring of the condition of privatized homes. Further, military department data monitoring efforts are department-specific, even though the departments have entered into privatized housing agreements with some of the same companies.

Standards for Internal Control in the Federal Government state that management should use quality information to achieve the entity’s objectives and design information systems and related control activities to achieve objectives and respond to risks. Information, among other things, should be complete and accurate. The standards also state that management should define the identified information requirements at the relevant level and requisite level of specificity for appropriate personnel. Without direction from OSD to establish minimum data requirements and consistent terminology or practices for data collection, as well as a requirement for the military departments to validate the data, the military departments’ ability to use data from the private partners’ work order tracking systems to monitor the condition of privatized homes over time is limited and may vary across projects.

DOD Provides Reports to Congress on Resident Satisfaction with Privatized Housing, but Data in These Reports Are Unreliable, Leading to Misleading Results

DOD has provided periodic reports to Congress on the privatized housing program; however, reported results on resident satisfaction have been unreliable and are misleading due to (1) variances in the data the military departments collect and provide to OSD and (2) OSD’s calculation and presentation of the data. DOD is statutorily required to provide reports to Congress that include, among other things, information about military housing privatization projects’ financial health and performance and the backlog, if any, of maintenance and repairs. These reports have also included information on resident satisfaction based on the results of the annual military department satisfaction surveys. In May 2019, DOD issued its report for fiscal year 2017, which stated that overall resident satisfaction for calendar year 2017 was 87 percent. However, this number is misleading due to issues associated with the collection and calculation of the data DOD used.

The military departments provide data on resident satisfaction to OSD for inclusion in DOD’s submission to Congress based on information from the annual resident satisfaction surveys. Specifically, OSD’s instructions to the military departments for the fiscal year 2017 report required the military departments to report the following:

The month and year of the most recently completed tenant satisfaction survey.

The number of residents surveyed and the total number of tenants who completed the survey during the reporting period.

Resident responses to the question that asks: “Would you recommend privatized housing?” Results should indicate how many tenants responded “Yes,” “No,” or “Don’t Know.”

However, instead of asking whether residents would recommend privatized housing, the military departments’ annual resident satisfaction survey asked residents the following: “How much do you agree or disagree with the following statement, ‘I would recommend this community to others.’”

The difference in the wording between the question asked of residents and the question reported to Congress is notable, as a resident’s satisfaction with his or her community and inclination to recommend it to others may not be reflective of satisfaction with either the privatized housing unit or privatized housing in general.

We also found differences in how the military departments interpreted responses to the question they asked. When asked whether they would recommend their community to others, residents were provided the following response categories on a scale of five to zero: (5) strongly agree, (4) agree, (3) neither agree nor disagree, (2) disagree, (1) strongly disagree, and (0) not applicable, no opinion, don’t know, or no answer. However, we found that the ways in which the military departments translated these responses into the “yes,” “no,” or “do not know” categories differed across the military departments, and in the case of the Army differed from year to year.

For the fiscal years 2015 through 2017 reports, Navy officials told us they counted responses reported in categories 5 (strongly agree) and 4 (agree) as “yes,” responses in categories 2 (disagree) and 1 (strongly disagree) as “no,” and responses in categories 0 (not applicable, no opinion, don’t know, or no answer) and 3 (neither agree nor disagree) as “don’t know.”

For the same time period, Air Force officials told us they counted responses in categories 5 (strongly agree), 4 (agree), and 3—neither agree nor disagree—as “yes,” responses in categories 2 (disagree) and 1 (strongly disagree) as “no,” and responses in category 0 (not applicable, no opinion, don’t know, or no answer) as “don’t know.” If 3 had not been counted as “yes,” the reported resident satisfaction rate would have been lower. For example, for Lackland Air Force Base, Texas, if officials had not counted responses in category 3 as “yes,” the resident satisfaction rate for newly constructed units would have been more than 20 percent lower than what was reported.

The Army calculated responses differently for fiscal years 2015, 2016, and 2017. Specifically:

For the fiscal year 2017 report, the Army counted responses in categories 5 (strongly agree) and 4 (agree) as “yes,” responses in categories 2 (disagree) and 1 (strongly disagree) as “no,” and responses in categories 0 (not applicable, no opinion, don’t know, or no answer) and 3 (neither agree nor disagree) as “don’t know.”

For the fiscal year 2016 report, the Army counted responses in categories 5 (strongly agree) and 4 (agree) as “yes,” responses in categories 2 (disagree), 1 (strongly disagree), and 0 (not applicable, no opinion, don’t know, or no answer) as “no,” and responses in category 3 (neither agree nor disagree) as “don’t know.”

For the fiscal year 2015 report, the Army counted responses in categories 5 (strongly agree), 4 (agree), and 3 (neither agree nor disagree) as “yes,” responses in categories 2 (disagree) and 1 (strongly disagree) as “no,” and responses in category 0 (not applicable, no opinion, don’t know, or no answer) as “don’t know.”

In addition, we identified errors and inaccuracies in how OSD calculates these data and reports them to Congress. Specifically, we found missing data points and incorrect formulas, among other errors, in OSD’s calculation of the data submitted by the military departments. For example:

The formula used by OSD to calculate overall resident satisfaction for fiscal year 2017 did not include data for several projects, including four Army projects—Fort Bragg, North Carolina; Fort Knox, Kentucky; Joint Base Lewis-McChord, Washington; and Presidio of Monterey/Naval Postgraduate School, California. As of September 30, 2017, these four projects accounted for over 18 percent of the Army’s total housing inventory.

The formula used by OSD to calculate resident satisfaction by project double counted resident satisfaction data for new and unrenovated homes for Vandenberg Air Force Base, California, by incorrectly using the Vandenberg Air Force Base data for both the Vandenberg and for the Fort Huachuca/Yuma Proving Ground project. As a result, incorrect data were reported for the Fort Huachuca/Yuma Proving Ground project for some categories of homes.

OSD did not include resident satisfaction data for New Orleans Naval Complex, Louisiana, in its fiscal year 2017 report to Congress, even though the Navy had included data for this project when submitting its data to OSD. OSD also reported identical resident satisfaction data for Wright-Patterson Air Force Base, Ohio, in fiscal years 2015, 2016, and 2017, despite the fact that Air Force officials noted in their submissions to OSD that the annual resident satisfaction data was from the annual resident satisfaction survey for Wright-Patterson Air Force Base conducted December 2013.

Further, Army data provided to OSD had calculation errors that OSD did not reconcile. Specifically, the Army provided OSD the total number of surveys received for a project, as well as the number of surveys broken out by different housing categories. However, we found instances where the sum of the data broken out by different housing categories was not equal to the reported total number of surveys received. For example, when we reviewed data for Fort Rucker, Alabama, the calculated sum of surveys broken out by different housing categories was 1,372, but the Army reported a total of 530 surveys received, a difference of 842 surveys.

Further, the presentation of data in OSD’s report to Congress is misleading because OSD did not explain the methodology it used to calculate the overall resident satisfaction percentage or include caveats to explain limitations to the data. Specifically, OSD did not include information on overall response rates to the annual satisfaction survey for each military department, nor did it include response rates by project. Low response rates can create the potential for bias in survey results. For example, in its fiscal year 2017 report, OSD reported that 25 percent of residents living in renovated homes at the MHPI project including Fort Detrick, Maryland/Walter Reed Army Medical Center, Washington, D.C., were satisfied with their housing. However, only four residents provided responses to this question, meaning that just one resident reported being satisfied. In addition, we found that OSD did not include an explanation in the report for why five projects were listed as not applicable. According to OSD officials, this error was a quality control issue that they plan to address, but the officials told us there are no plans for quality control in development at this time. The Fiscal Year 2020 NDAA includes a provision requiring each military installation to use the same satisfaction survey for tenants of military housing—including privatized military housing—the results of which are not to be shared with private partners until reviewed by DOD. The statute also states that DOD’s reports to Congress shall include additional information, such as the results of residence surveys, as well as assessments of maintenance response times, completion of maintenance requests, the dispute resolution process, overall customer service for tenants, and other factors related to the condition of privatized housing.

OSD’s report to Congress states that, given DOD’s objective of improving the quality of life for its servicemembers, the degree of satisfaction military families experience in privatized housing is a critical indicator of overall program success and the military departments and private partners use tenant surveys to help assess the quality of privatized housing. Additionally, Standards for Internal Control in the Federal Government state that management should obtain relevant data from reliable internal and external sources in a timely manner based on identified information requirements. Relevant data have a logical connection with, or bearing upon, the identified information requirements. Reliable internal and external sources provide data that are reasonably free from error and bias and faithfully represent what they purport to represent. Management should evaluate both internal and external sources of data for reliability, and obtain data on a timely basis so they can be used for effective monitoring.

However, the errors we identified in OSD’s data calculations, as well as the differences in how the military departments translate data provided to OSD, indicate the need for better internal controls, including a process for collecting and calculating resident satisfaction data from the military departments, and explanation of the data collected and reported on resident satisfaction to ensure they are reasonably free from error and bias and represent what they purport to represent. According to an OSD official responsible for preparing the reports to Congress, her office inherited the MHPI report process from its predecessors and had to quickly catch up on reports because DOD was behind on its reporting requirement. However, she noted her office is working with the military departments to review the resident satisfaction survey questions and will be identifying and implementing measures to ensure an accurate and reliable process to compile, calculate, report, and compare MHPI residents’ satisfaction by military department and across DOD. Additionally, for future survey data reporting, OSD officials told us they plan to research the possibility of directly collecting resident survey data from the survey administrator to minimize data transcription errors. Until OSD makes these changes to the data collection and calculation efforts that make up the department’s report to Congress and provides explanations of the data in the reports, OSD will not be able to provide Congress with an accurate picture of resident satisfaction with privatized housing.

Military Housing Offices Have Not Effectively Communicated Their Role as a Resource for Servicemembers Experiencing Challenges with Privatized Housing

Military housing offices located at each installation are available to provide resources to servicemembers experiencing challenges with their privatized housing, among other services, but these offices have not always clearly and systematically communicated this role to residents of privatized housing. Military housing office officials noted that servicemembers living in privatized military housing primarily interact with their installation’s military housing office when they first receive orders to move to an installation. The military housing office provides new residents with information on their local housing options, to include referral services for housing options. However, military department guidance calls for the military housing office to provide continued assistance to servicemembers and their families living in privatized housing. For example, each military department has guidance that establishes the role of its housing offices in the resident dispute resolution process—specifically, if servicemembers are experiencing a dispute with the private partner:

Army policy states that each installation should have an official tasked with providing support to servicemembers regarding resident issues that cannot be resolved by the private property manager. This individual is also in charge of resolving every resident complaint, and the military housing office, if required, can request mediation by the garrison commander.

Air Force policy directs installation commanders to establish regular meetings with the private partners to discuss resident disputes and develop resolutions for residents’ issues. Also, the Air Force business agreements for each project are to establish Management Review Committees, in which the private project owner, Air Force housing office officials, and the Air Force Civil Engineer Center meet quarterly to review and facilitate the resolution of prevalent issues.

The Navy announced a standardized two-step resolution process in May 2019 for housing residents who have issues or concerns with their current homes. The first step is to report any issue to the local property manager. If the issue is not resolved in either a timely manner or to quality standards, residents are asked to contact their local Navy housing service center, which directly reports to the installation commanding officer, or the servicemember’s chain of command. Prior to the standardization of this process, Navy guidance established a general responsibility to assist residents in the dispute resolution process and each project’s tenant lease includes specific dispute resolution processes.

The Marine Corps has established a three-step dispute resolution process for residents to follow when they are experiencing a dispute with the private partner. Further, Marine Corps policy calls for each of the private partners to establish standard operating procedures that should include complaint resolution procedures.

Despite established military department guidance, we found that residents were sometimes confused and lacked awareness of the availability of the military housing office to assist them with issues they were experiencing with privatized housing. For example, residents who participated in our focus groups and responded to our online tool expressed the following concerns:

At least one resident in each of our focus groups noted being sometimes confused about the military housing office’s roles and responsibilities with regard to the maintenance of their home. These residents indicated they did not know the military housing office existed or could serve as a resource. Further, some individuals that responded to our online tool indicated that they did not know they could reach out to military housing office officials or their chain of command with issues related to the condition of their home.

Residents in at least three of our focus groups indicated they perceived that the military housing office was not working independently of the partner or in the residents’ best interest. For example, residents in at least three focus groups noted that they viewed the military housing office as an extension of the private partner. Other residents noted that they did not know what the military housing office was or what role the office plays in managing privatized housing. In addition, residents we solicited information from through our online tool indicated that they felt they have not had any recourse in resolving issues and disagreements with private partners. For example, one individual who responded to our online tool stated that she was glad she moved off post because she now has legal recourse if the landlord does not meet maintenance requirements.

The military department oversight agencies have found that the military departments have not clearly and systematically communicated their roles to residents, and resident confusion and a lack of awareness regarding the role of the military housing offices is an issue. In April 2019, the Air Force Inspector General reported that less than half of the residents interviewed used their military housing office to resolve complaints, and at some installations officials visited many residents did not know the military housing office had an oversight role. Similarly, in May 2019, the Army Inspector General reported to the Secretary of the Army that at 82 percent of Army installations with privatized housing, residents did not know how to escalate issues with either the private partner or the Army housing office. Additionally, the Army Inspector General reported that installation command teams and staff cited multiple circumstances where military housing offices and tenant advocacy roles and responsibilities were unclear.

Further, military housing office officials with whom we spoke during our site visits acknowledged the gap in resident awareness regarding the existence and purpose of the military housing office. Officials also noted that at times residents were unaware of the difference between the military housing office and the private partner office due, in part, to their physical co-location and unclear building signage. For example, a military housing office official at Fort Bragg, North Carolina, told us the military housing office was the best kept secret on the installation. Moreover, residents that participated in our four focus groups at Fort Bragg expressed confusion in differentiating the Army military housing office officials from private partner representatives. Similarly, officials at the military housing office at Tinker Air Force Base, Oklahoma, told us that many residents were confused by their office’s role because the private partner office goes by the name “Tinker Housing Office.” Further, we observed that both private partner representatives and some military housing office officials are located in the same building, and signage does not distinctly indicate that the office houses both military officials and private partner representatives. In contrast, the military housing office at Camp Pendleton, California, is intentionally branded as the “Camp Pendleton Joint Housing Office” and signage indicates the office houses officials from both the Marine Corps and the installation’s private partners. See figure 5 for examples of the varying level of detail in military housing office signage.

Some military housing office officials told us they have taken steps to improve resident awareness, such as increasing advertising of the military housing office’s role and contact information, using town hall meetings to inform residents of their roles and responsibilities, and rebranding their military housing offices to differentiate them from the private partners. For example, the Army housing office at Fort Sill, Oklahoma, changed its name from the “Residential Communities Initiative Housing Office” to the “Garrison Housing Office” to more clearly denote that the military housing office is not associated with the private partner. In addition, a Marine Corps housing office official provided us with a flyer, which is distributed to residents by the private partner, informing residents of housing office contact information and the service’s three-step dispute resolution process. See figure 6 for a copy of the flyer.

According to DOD officials, the military departments generally decreased their staffing and oversight of daily privatized housing operations after the MHPI was enacted, which led to less ongoing resident interaction. For example, Army officials we spoke with in January 2019 told us they typically filled 80 percent of available military housing office positions across their installations. Additionally, officials stated that housing offices were generally staffed with two or three officials responsible for assisting servicemembers with housing needs both on the installation as well as in the local community. Further, the officials told us that the team at Fort Bragg, North Carolina, was decreased from about 15 to 3 positions. According to OSD officials, while housing offices should generally not require the number of personnel that were necessary prior to privatization, reductions following sequestration reduced housing staff below the level necessary to fully perform required privatized housing oversight as it was originally envisioned at the outset of the program.

OSD has also recognized that the military departments’ communication with residents about their role as a resource for residents has not been clear or systematic. In February 2019, the Assistant Secretary of Defense for Sustainment testified before Congress that a way forward in addressing resident concerns would require focus in three key areas: communication, engagement, and responsiveness. In support of this, OSD and the military departments are collaborating with each of the private partners on several initiatives aimed at improving the residents’ experience with privatized housing and ensuring a consistent resident experience across installations. These initiatives include: establishing a tenant bill of rights that will clearly define tenants’ rights establishing a resident advocate position that is planned to provide advice, education, and support to the resident and advocate on the resident’s behalf in disputes with private partners; developing a common lease that provides a common framework and language in residential leases across all privatization projects; and developing a standardized formal dispute resolution process to ensure the prompt and fair resolution of disputes that arise between privatized housing landlords and residents.

Despite the development of initiatives aimed at improving the resident’s experience with privatized housing and various ad hoc efforts to better brand and advertise the roles and responsibilities of some military housing offices, the military departments have not systematically or clearly communicated these efforts to residents, and military officials we met with acknowledged that there still appears to be a gap in residents’ awareness of the military housing office and its role in the dispute resolution process.

Standards for Internal Control in the Federal Government state that management should externally communicate the necessary quality information to achieve the entity’s objectives. Management communicates this externally through reporting lines so that external parties can help the entity achieve its objectives and address related risks. Moving forward, having plans in place to clearly and systematically communicate the difference between the military housing office and the private partners— including the military departments’ roles, responsibilities, and military housing office locations and contact information—will better position the military departments to achieve the intended objectives of the initiatives they are currently developing with OSD.

DOD and Private Partners Are Implementing Initiatives to Improve Privatized Housing, but May Face Challenges DOD and Private Partners Are Implementing Initiatives to Improve MHPI

OSD, the military departments, and the private partners have identified and begun collaborating on a series of initiatives aimed at improving residents’ experience with privatized housing. According to an OSD official, a series of initiatives have been identified and are currently in various phases of development and implementation. Tri-service working groups, each chaired by a designated military department and including officials and legal counsel from each military department as well as private partner representatives, are leading efforts to develop and implement the initiatives. In addition, in the Fiscal Year 2020 NDAA, Congress established several requirements aimed at addressing military privatization housing reform. Several of the statutory requirements provide specific provisions that DOD will need to incorporate into its development and implementation of existing MHPI initiatives, as well as additional requirements aimed at improving the oversight of privatized housing. Table 2 outlines key initiatives aimed at improving privatized housing, as well as additional selected requirements mandated by the Fiscal Year 2020 NDAA.

In addition to the provisions noted in table 2, the Fiscal Year 2020 NDAA included requirements for increased oversight of the physical condition of privatized housing. Specifically, the legislation required the following:

The Secretary of Defense is to designate a Chief Housing Officer to oversee housing units, including the creation and standardization of policies and processes regarding housing units.

The Secretary of Defense is required to establish a uniform code of basic standards for privatized military housing and plans to conduct inspections and assessment of the condition of privatized homes.

The military departments are required to create a council on privatized military housing for the purposes of maintaining adequate oversight of the military housing program and serving as a mechanism to identify and resolve problems regarding privatized military housing.

The head of the installation military housing office is responsible for conducting a physical inspection and approving the habitability of a vacant housing unit for the installation before the landlord managing the housing unit is authorized to offer the housing unit available for occupancy; conducting a physical inspection of the housing unit upon tenant move-out; and initiating contact with a tenant regarding the tenant’s satisfaction with his or her housing unit not later than 15 days after move-in, and again 60 days after move-in.

Each installation is required to use the same satisfaction survey for tenants of military housing, including privatized military housing, and results are not to be shared with partners until reviewed by DOD.

Initiatives to Improve MHPI May Face Implementation Challenges

DOD and private partner representatives have cited several challenges that could affect their ability to implement initiatives aimed at improving MHPI. Specifically, key challenges include the timeliness with which they are able to implement initiatives, a lack of resources needed for implementation, and concerns that implementation could have unintended negative impacts on the financial viability of the privatized housing projects.

Timeliness of implementation due to the need to collaborate with and obtain input and agreement from the large number of stakeholders involved in privatized housing. According to DOD officials and private partner representatives, many of the initiatives designed to improve privatized housing require not only agreement between DOD and the private housing partners, but also discussion with and, in some cases, approval by the project bond holders. Because DOD does not have the ability to unilaterally make changes to existing business agreements, this need for stakeholder agreement limits DOD’s control over the implementation timeline of any initiative that requires changes to a project’s business agreement—such as the implementation of a standardized dispute resolution process. Additionally, the private partners noted that the bond holders may be reluctant to agree to changes to the business agreements that could result in higher project costs.

The need for more military department staff with targeted expertise. As noted earlier, the military departments had reduced their involvement in daily privatized military housing operations as part of the overall privatization effort. This included reducing staffing levels at the installations, and military housing office officials at over half of the installations we visited stated that reduced staffing levels had impacted their ability to carry out oversight duties, such as work order data analysis and housing inspections. Further, until recent issues surfaced over concerns of the quality of privatized housing, the military departments had distanced themselves from involvement in daily military housing operations. For example, the Army issued a memorandum in 2013, which has since been rescinded, stating that garrison commanders were not to authorize, direct, or permit Army representatives to initiate health and welfare inspections of privatized housing. Each of the military departments has plans to increase the military housing office staffing at each installation to allow for enhanced oversight. In particular, according to military department officials, these positions will focus on quality control and quality assurance of the maintenance of privatized homes. However, improvements to the condition of privatized housing may not be fully realized until DOD establishes a uniform code of basic standards for privatized military housing, as required by the Fiscal Year 2020 NDAA, and these new personnel are trained in these standards.

The potential for unintended negative financial impacts on the projects that could outweigh the intended benefits of the initiatives. OSD officials and private partner representatives have expressed concern that some proposed initiatives could result in unintended financial consequences for the housing projects. In particular, private partner representatives noted costs from things such as legal fees associated with the development of a common lease and the various addendums that would be required and the unanticipated costs of hiring outside third party inspections. In particular, some of the private partners noted that the financial impact of unfunded requirements to projects that are already experiencing financial distress could result in even fewer funds available to reinvest in improvements to the current and future physical condition of the homes.

Moreover, OSD officials told us they have concerns that some initiatives—such as increased frequency of change-of-occupancy inspections that may result in homes remaining vacant longer than planned and therefore not collecting rent—may unintentionally impact a project’s cash flow. Officials noted that some installations have large-scale housing turn over at the same time and inspections may not be able to be completed in the required time frames. For example, OSD officials said that at Fort Leavenworth, Kansas, the vast majority of homes generally turn over during a 2-week time period. Officials said that in a location like this, new oversight requirements may have a negative impact on residents’ move-in timelines, which could subsequently impact occupancy rates and project cash flow as a result of delays in rent payments. OSD officials also stated that residents’ ability to have their basic allowance housing payments segregated and held in escrow may present financial challenges to both the resident and the project. These officials noted that they did not yet know how the withholding of these payments would be implemented. According to OSD officials, as of January 2020, there are many questions surrounding the implementation of the Fiscal Year 2020 NDAA provisions. Officials told us that they have not yet assessed the impact of increased oversight on the financial viability of the MHPI projects, but stated that as they develop processes to implement each new statutory provision, the financial impact is something that needs to be considered.

DOD’s Military Housing Privatization Initiative Performance Evaluation Report for fiscal year 2017 stated that the government’s interests are not always aligned with those of the private sector, and that oversight and engagement are required and expected in a public-private partnership over the long term to ensure success. We have previously reported that the military departments have not defined their risk tolerance levels for privatized housing relative to the program’s objective of providing quality housing that reflects community living. Specifically, we recognized that the Office of Management and Budget guidance on the preparation, submission, and execution of the federal budget suggests that public- private partnerships, such as privatized military housing projects, contain some elements of risk to the government. Standards for Internal Control in the Federal Government state that management should identify, analyze, and respond to risks related to achieving defined program objectives. While DOD is in the process of developing and implementing initiatives to improve privatized military housing, OSD and the military departments have not assessed the risk of the proposed initiatives on the financial viability of the privatized housing projects.

According to an OSD official, the intention of privatization was to reduce the government’s role in the management of military housing and put more responsibility on the private partners. As described earlier in this report, the military departments have ramped up their oversight efforts in response to recent concerns about the condition of privatized housing by, for example, revising guidance and hiring additional staff. However, OSD has not assessed the impact of these activities on the financial viability of the MHPI projects. For example, OSD has not determined how increasing the frequency of housing office inspections and residents’ withholding of rent could impact the bottom line of some of its privatized projects. Without assessing risks to the financial viability of the MHPI projects associated with the implementation of these initiatives aimed at improving privatized housing, DOD’s efforts to improve the privatized housing program could be compromised. Further, DOD has a long-term interest in ensuring the financial health of the properties privatized under MHPI. As we have reported, typically the titles to the homes that were conveyed to the private partners and any improvements made to these homes during the duration of the ground leases will automatically revert to the military departments upon expiration or termination of the leases.

Conclusions

DOD’s oversight of privatized housing is critical to ensure that residents are being provided with affordable, quality housing that generally reflects contemporary community living standards. In light of recent concerns about the effect of inadequate and poor quality housing on servicemembers and their families, the military departments have recently implemented steps to increase the oversight of the condition of privatized housing. However, OSD has not provided the military departments with specific objectives for this monitoring. The newly established Chief Housing Officer position, intended to standardize guidance and processes for the oversight of privatized housing, provides DOD with an opportunity to ensure that revised guidance provided to the military departments includes objectives for increased oversight. In addition to oversight of the condition of homes, DOD has taken initial steps to standardize performance incentive metrics across the military departments. However, unless efforts are made to ensure that the indicators driving these metrics accurately reflect private partners’ performance in maintaining the condition and quality of privatized homes, DOD’s ability to hold private partners accountable will remain limited. Further, while the military departments continue to increase the access to and use of work order data to monitor and track the condition of privatized housing, without consistent terminology and practices for work order data collection and processes for validating data collected from the private housing partners, the use of these data may not result in reliable findings. Finally, DOD has frequently reported high customer resident satisfaction rates as a key indicator of the success of the privatization initiative. However, the process used to collect and calculate the data used for determining these rates and limitations in its presentation to Congress raise questions about the reliability of DOD’s reports and their usefulness as an indicator of program success. By improving oversight guidance, mechanisms for measuring private partner performance, the reliability of housing data, and reporting on resident satisfaction, DOD can better ensure that MHPI is providing servicemembers with quality housing.

Despite a decreased role in the daily management of privatized housing, the military departments still maintain responsibility for providing servicemembers with resources for seeking resolution to any issues identified in their privatized homes. However, without plans in place to communicate military housing office roles, responsibilities, and locations to residents of privatized housing, these individuals may not receive the full benefits of the assistance that the military housing offices provide. In light of the increased focus by DOD and Congress in ensuring that residents are aware of their rights and responsibilities, improved communication with residents about the military housing offices’ roles and responsibilities can help ensure that residents are utilizing the full range of resources available to them if they have issues with privatized housing.

As OSD, the military departments, and the private partners take steps to improve the resident experience with privatized military housing and increase the department’s focus on the condition of privatized homes, ensuring that their efforts do not inadvertently harm the financial viability of these projects is key. Without assessing and mitigating the potential risk program improvements may have on the financial viability of the MHPI projects, DOD cannot ensure that these initiatives and the implementation of new statutory requirements will ultimately result in improvements to the condition of privatized housing.

Recommendations for Executive Action

We are making a total of 12 recommendations—six to the Office of the Secretary of Defense, two to the Secretary of the Army, two to the Secretary of the Air Force, and two to the Secretary of the Navy: The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment, in collaboration with the military departments, provide updated guidance for the oversight of privatized military housing, to include oversight objectives for each service to monitor the physical condition of privatized homes over the remaining duration of the ground leases. (Recommendation 1)

The Secretary of the Army should take steps, in collaboration with the Army’s private housing partners, to review the indicators underlying the privatized housing project performance metrics to ensure they provide an accurate reflection of the condition and quality of the homes. (Recommendation 2)

The Secretary of the Air Force should take steps, in collaboration with the Air Force’s private housing partners, to review the indicators underlying the privatized housing project performance metrics to ensure they provide an accurate reflection of the condition and quality of the homes. (Recommendation 3)

The Secretary of the Navy should take steps, in collaboration with the Navy and Marine Corps’ private housing partners, to review the indicators underlying the privatized housing project performance metrics to ensure they provide an accurate reflection of the condition and quality of the homes. (Recommendation 4)

The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment, in collaboration with the military departments and private housing partners, establish minimum data requirements and consistent terminology and practices for work order data collection for comparability across installations and projects and to track trends over time. (Recommendation 5)

The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment require the military departments to establish a process to validate data collected by the private housing partners to better ensure the reliability and validity of work order data and to allow for more effective use of these data for monitoring and tracking purposes. (Recommendation 6)

The Secretary of Defense should ensure the Assistant Secretary of Defense for Sustainment, in collaboration with the military departments, develop a process for collecting and calculating resident satisfaction data from the military departments to ensure that the data are compiled and calculated in a standardized and accurate way. (Recommendation 7)

The Secretary of Defense should ensure the Assistant Secretary of Defense for Sustainment provides additional explanation of the data collected and reported in future reports to Congress, such as explaining the limitations of available survey data, how resident satisfaction was calculated, and reasons for any missing data, among other things. (Recommendation 8)

The Secretary of the Army should develop and implement a plan to clearly and systematically communicate to residents the difference between the military housing office and the private partner. At a minimum, these plans should include the Army housing office’s roles, responsibilities, locations, and contact information and should ensure that all residents are aware that they can directly contact Army housing office officials. (Recommendation 9)

The Secretary of the Air Force should develop and implement a plan to clearly and systematically communicate to residents the difference between the military housing office and the private partner. At a minimum, these plans should include the Air Force housing office’s roles, responsibilities, locations, and contact information and should ensure that all residents are aware that they can directly contact Air Force housing office officials. (Recommendation 10)

The Secretary of the Navy should develop and implement a plan to clearly and systematically communicate to residents the difference between the military housing office and the private partner. At a minimum, these plans should include the Navy housing office’s roles, responsibilities, locations, and contact information and should ensure that all residents are aware that they can directly contact Navy housing office officials. (Recommendation 11)

The Secretary of Defense should ensure that the Assistant Secretary of Defense for Sustainment, in collaboration with the military departments, assess the risks of proposed initiatives aimed at improving the privatized military housing program on the financial viability of the projects. (Recommendation 12)

Agency Comments

We provided a draft of this report to DOD for review and comment. In written comments, reprinted in their entirety in appendix III, DOD concurred with 10 of our recommendations and partially concurred with 2, identifying actions it plans to take to address each of them. DOD also provided technical comments, which we incorporated as appropriate.

DOD partially concurred with our recommendation that the Assistant Secretary of Defense for Sustainment, in collaboration with the military departments and private housing partners, establish minimum data requirements and consistent terminology and practices for work order collection. The department noted that neither the Assistant Secretary of Defense for Sustainment nor the military departments could mandate changes to existing privatized housing project ground leases or legal agreements. DOD further noted that it cannot unilaterally make changes to the project ground leases and associated legal documents without concurrence from the private partners. However, the department noted that to the maximum extent practical, it would work to establish minimum data requirements and consistent terminology and practices for work order collection.

DOD also partially concurred with our recommendation that the Under Secretary of Defense for Sustainment, in collaboration with the military departments, develops a process for collecting and calculating resident satisfaction data because there is no Under Secretary of Defense for Sustainment. Based on the department’s comments, we revised the addressee of this recommendation, directing action to the Assistant Secretary of Defense for Sustainment. However, the department noted that effective with the survey collection effort for Fiscal Year 2021, it would refine the process for collecting and calculating resident satisfaction data from the military departments to ensure that DOD compiles and calculates data in a standardized and accurate way.

We are sending copies of this report to the appropriate congressional committees; Senator Catherine Cortez Masto; Senator Mark Warner; Representative Gus Bilirakis; the Secretary of Defense; and the Secretaries of the Departments of the Army, the Navy, and the Air Force. In addition, the report is available at no charge on our website at https://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2775 or FieldE1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV.

Appendix I: Objectives, Scope, and Methodology

The Conference Report accompanying a bill for the Fiscal Year 2019 Department of Defense Appropriations Act included a provision for us to review ongoing issues within privatized military housing. This report examines the extent to which the Office of the Secretary of Defense (OSD) and the military departments (1) conduct oversight of privatized military housing for servicemembers and their families, (2) have communicated their roles and responsibilities to servicemembers and their families, and (3) have developed and implemented initiatives to improve privatized housing.

We included all privatized housing projects in each military department. For each of our objectives, we reviewed OSD and military department policies and guidance for the implementation of the Military Housing Privatization Initiative (MHPI) program, including guidance on the authority, roles, and responsibilities for oversight and management of privatized housing. We evaluated the extent to which the evidence we collected aligned with OSD policy and stated goals for oversight and management of privatized housing, and whether the evidence adhered to the principles in Standards for Internal Control in the Federal Government. We conducted interviews with officials from the Office of the Assistant Secretary of Defense for Sustainment, Office of Facilities Management; the Office of the Deputy Assistant Secretary of the Army (Installations, Housing and Partnerships); the Army Installation Management Command; the Army Assistant Chief of Staff for Installation Management; the Assistant Secretary of the Air Force for Installations, Environment, and Energy; the Air Force Civil Engineer Center; the Commander, Navy Installations Command; the Commander, Naval Facilities Engineering Command; the Marine Corps Installation Command; and representatives from each of the 14 private partners that are currently responsible for privatized housing projects.

We visited a non-generalizable sample of 10 installations selected to represent each of the military departments, six private partners—including the five largest who own the majority of privatized military housing—and geographic and climate diversity. The selected sites in our non- generalizable sample were three Army installations—Fort Bragg, North Carolina; Fort Huachuca, Arizona; and Fort Sill, Oklahoma; two Navy installations—Naval Station Norfolk, Virginia, and Naval Base San Diego, California; two Marine Corps installations—Marine Corps Base Camp Lejeune, North Carolina, and Marine Corps Base Camp Pendleton, California; and three Air Force installations—Davis-Monthan Air Force Base, Arizona; Langley Air Force Base, Virginia; and Tinker Air Force Base, Oklahoma. We reviewed the ground leases and other MHPI project documents for housing projects at each of these locations, and at each installation we met with officials from the installation commander’s office and conducted interviews with officials from both the installation military housing office and representatives from the private partners.

To collect input from residents of privatized housing, we facilitated 15 focus groups with a self-selected group of current residents of privatized military housing. During the focus groups, a methodologist led participants through a structured questionnaire, which we pretested with 11 residents of privatized housing prior to the first focus group. To solicit participants for our focus groups, we requested that local military housing office officials email all current residents of privatized housing prior to our visit to inform them of our focus groups. Individuals interested in participating in our focus group sessions were instructed to contact us directly for further information. We had over 70 residents participate in our focus groups. In addition to the 15 focus groups, we conducted an additional five sessions in which fewer than three residents attended. We collected information from these residents, but we did not include their input in our focus group analysis. Comments from focus group participants are not generalizable to all residents of privatized military housing.

We also developed and administered a publically available online tool that provided an opportunity for any resident of privatized military housing to voluntarily submit information on their experiences. Participants had the option to remain anonymous and make multiple submissions in order to provide us information on their experience at more than one installation. We developed our tool in conjunction with a survey methodologist to ensure it met our requirements for publically available anonymous data collection instruments, and conducted five pretests of the questions with residents of privatized housing. Our online tool was made available to the public from June 17, 2019, through August 31, 2019. We received a total of 658 responses. In analyzing information provided through the online tool, we took steps to identify responses that did not meet our criteria, including removing 13 responses for reasons such as responses with duplicative usernames or Internet Protocol (IP) addresses that described the same experience or had been started but not fully completed, responses from DOD officials that informed us they had provided responses to test our tool, and responses from residents living on installations outside of the United States. In reporting results from our online tool, we used the following qualifiers in presenting our results— most (to indicate 80 percent or higher); majority (to indicate 51-79 percent); and some (to indicate less than 50 percent). Findings from our focus groups and online tool are not generalizable to all privatized military housing residents.

To determine the extent to which DOD conducts oversight of privatized military housing for servicemembers and their families, we conducted the following additional data analysis. Through the steps described in the following bullets, we determined these data to be reliable for the purposes of our findings:

To determine the extent to which performance incentive fee metrics assessed the condition of privatized housing, we collected information on the structure of the incentive fees from private partners for 74 privatized housing projects and received confirmation that there are 5 projects that do not have incentive fee plans as part of their business agreements. We reviewed all of the incentive fee plans and identified commonly used metrics and indicators. We met with officials from the military housing offices, the military departments, and private partner representatives to discuss the administration and measurement of the incentive fee structures. To gain an understanding of how performance incentive fees are used, we reviewed documents and guidance from OSD and the military departments that explains the processes for developing and awarding performance incentive metrics and fees. In addition, we obtained information from residents through focus groups and our online tool and spoke with military housing office officials to obtain anecdotal information regarding the extent to which the metrics are adequately measuring the condition of the housing.

To assess the extent to which private partner work order data could be used to monitor and track the condition of privatized homes, we collected and reviewed private partner work order data from October 2016 through April 2019 from each of the 79 MHPI projects and discussed these data with the private partners and military department officials. Given that we requested the work order data from the private partners in April and May 2019, we selected the October 2016 through April 2019 time frame to include complete data for fiscal years 2017 and 2018 and the most comprehensive data available at the time for fiscal year 2019. Prior to requesting these data, we contacted representatives from each of the 14 private partner companies to discuss our forthcoming data request and to better understand each company’s data system and potential limitations for using the data. Subsequently, we requested that each partner provide us with data for all work orders across all data elements for each installation under their management. We received data on over 8 million work orders among the 14 private partners. We performed manual testing on initial data files received by each partner to identify issues that would impact the validity and reliability of using these data for ongoing monitoring and tracking of the condition of privatized housing units. In doing so, we identified instances of anomalies in work order data from each of the 14 partners. For 12 of the 14 partners, we found at least one of the following anomalies in the initial work order data files received for the time period requested: (1) duplicate work orders; (2) work orders with completion dates prior to the dates that a resident had submitted the work order; and (3) work orders still listed as in-progress for more than 18 months. We reviewed work order data from at least one installation for each private partner to check for instances of these anomalies. We also held follow-up discussions with 10 of the 14 private partners to discuss anomalies found in the data and potential factors contributing to the presence of these anomalies. In addition to the initial data collected on all of the work orders, we requested a second data run of work orders over the same time period—October 1, 2016, through April 30, 2019—for service requests related to lead- based paint, mold, and pest/rodent/vermin infestation. As part of this request, we asked that partners provide the criteria used for querying the data they provided us. We reviewed these data to determine how requests for data on specific hazards were getting sorted into the work order tracking systems by category and priority among the various partners.

To identify differences in terminology and business practices used by the private partners in their work order tracking systems, we requested and reviewed private partner documentation of data definitions and protocols for managing work order data. In addition, we conducted interviews with military department officials to discuss ongoing efforts by the military departments to collect and analyze work order data.

To evaluate resident satisfaction data reported in OSD’s reports to Congress on privatized housing, we reviewed the processes for collecting, calculating, and reporting these data for the three most recently issued reports for fiscal years 2015, 2016, and 2017. We reviewed the instructions OSD provided to the military departments outlining how the military departments are to submit resident satisfaction data to OSD. We also reviewed the question the military departments asked on their annual surveys to gauge resident satisfaction. We then requested the survey data the military departments submitted to OSD to be included in the annual report to Congress for fiscal years 2015, 2016, and 2017. We performed data quality checks and identified inaccuracies on a random sample of data reported by OSD. We reviewed how military departments calculated overall resident satisfaction for each privatized housing project. Further, we discussed these data with OSD and the military departments to assess the validity and reliability of using these data for identifying overall tenant satisfaction with the condition of privatized housing.

To determine the extent to which the military departments have communicated their respective military housing office roles and responsibilities to residents, we reviewed military department policies and guidance related to their roles and responsibilities for working with residents of privatized housing. During our site visits to 10 installations, we interviewed military department housing office officials and private partner representatives to discuss their specific roles and responsibilities. We asked questions soliciting information about residents understanding of the roles and responsibilities of the military housing office and the dispute resolution process during our 15 focus groups. We also solicited resident feedback in our online tools regarding residents’ experience reporting maintenance issues and working with military housing offices and private partners to get maintenance issues resolved.

To determine the extent to which DOD and private partners have developed and implemented initiatives to improve privatized housing, we interviewed OSD and military department officials to discuss ongoing initiatives developed over the course of our audit work aimed at improving MHPI and reviewed relevant guidance. We met with private partner representatives to discuss their involvement in developing these initiatives, as well as to gain an understanding of any challenges or concerns that may impact the implementation of these initiatives. Following the passage of the National Defense Authorization Act for Fiscal Year 2020, we reviewed provisions of the statute designed to improve the condition of privatized housing and evaluated the extent to which these provisions would impact ongoing or planned DOD initiatives or provide new oversight roles and responsibilities for OSD and the military departments. We discussed these provisions with OSD officials and private partner representatives to understand how, if at all, their implementation may impact the privatized housing projects, as well as any potential barriers to implementation in the current legal construct of the program.

We conducted this performance audit from November 2018 to March 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: List of Privatized Military Housing Projects as of September 30, 2017

In table 3, we provide the complete listing of the Department of Defense’s 79 privatized military housing projects, as of September 30, 2017. This list reflects information that the Office of the Assistant Secretary of Defense for Sustainment provided in its annual report to Congress for the time period of October 1, 2016, through September 30, 2017. The report was provided to Congress in May 2019. The projects can consist of one or multiple installations.

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact

Elizabeth A. Field, Director, (202) 512-2775 or FieldE1@gao.gov.

Staff Acknowledgments

In addition to the contact above, the following are key contributors to this report: Kristy Williams (Assistant Director), Tida Barakat Reveley (Analyst in Charge), Austin Barvin, Ronnie Bergman, Vincent Buquicchio, William Carpluk, Juliee Conde-Medina, Mae Jones, Jordan Tibbetts, Kelly Rubin, Monica Savoy, and John Van Schaik.

Related GAO Products

Military Housing: Preliminary Recommendations to Strengthen DOD's Oversight and Monitoring of Privatized Housing. GAO-20-471T. Washington, D.C.: March 3, 2020.

Military Housing Privatization: Preliminary Observations on DOD’s Oversight of the Condition of Privatized Military Housing. GAO-20-280T. Washington, D.C: December 3, 2019.

Defense Infrastructure: Additional Actions Could Enhance DOD’s Efforts to Identify, Evaluate, and Preserve Historic Properties. GAO-19-335. Washington, D.C.: June 19, 2019.

Military Housing Privatization: DOD Should Take Steps to Improve Monitoring, Reporting, and Risk Assessment. GAO-18-218. Washington, D.C.: March 13, 2018.

Defense Infrastructure: Army Has a Process to Manage Litigation Costs for the Military Housing Privatization Initiative. GAO-14-327. Washington, D.C.: April 3, 2014.

Military Housing: Information on the Privatization of Unaccompanied Personnel Housing. GAO-14-313. Washington, D.C.: March 18, 2014.

Military Housing: Enhancements Needed to Housing Allowance Process and Information Sharing among Services. GAO-11-462. Washington, D.C.: May 16, 2011.

Military Housing Privatization: DOD Faces New Challenges Due to Significant Growth at Some Installations and Recent Turmoil in the Financial Markets. GAO-09-352. Washington, D.C.: May 15, 2009.

Military Housing: Management Issues Require Attention as the Privatization Program Matures. GAO-06-438. Washington, D.C.: April 28, 2006.

Military Housing: Further Improvement Needed in Requirements Determination and Program Review. GAO-04-556. Washington, D.C.: May 19, 2004.

Military Housing: Better Reporting Needed on the Status of the Privatization Program and the Costs of Its Consultants. GAO-04-111. Washington, D.C.: October 9, 2003.

Military Housing: Opportunities That Should Be Explored to Improve Housing and Reduce Costs for Unmarried Junior Servicemembers. GAO-03-602. Washington, D.C.: June 10, 2003.

Military Housing: Management Improvements Needed as the Pace of Privatization Quickens. GAO-02-624. Washington, D.C.: June 21, 2002.

Military Housing: DOD Needs to Address Long-Standing Requirements Determination Problems. GAO-01-889. Washington, D.C.: August 3, 2001.

Military Housing: Continued Concerns in Implementing the Privatization Initiative. GAO/NSIAD-00-71. Washington, D.C.: March 30, 2000.

Military Housing: Privatization Off to a Slow Start and Continued Management Attention Needed. GAO/NSIAD-98-178. Washington, D.C.: July 17, 1998.