You are given two reports by the government agency.  

Report 1:
The EFMP provides support to families with special needs at their current and proposed locations. Servicemembers relocate frequently, generally moving every 3 years if in the Army, Marine Corps, and Navy, and every 4 years if in the Air Force. In fiscal year 2016, the Military Services relocated approximately 39,000 servicemembers enrolled in the EFMP to CONUS installations. To implement DOD’s policy on support for families with special needs, DOD requires each Service to establish its own EFMP for active duty servicemembers. EFMPs are to have three components—identification and enrollment, assignment coordination, and family support. Identification and enrollment: Medical and educational personnel at each installation are responsible for identifying eligible family members with special medical or educational needs to enroll in the EFMP. Once identified by a qualified medical provider, active duty servicemembers are required to enroll in their service’s EFMP. Servicemembers are also required to self-identify when they learn a family member has a qualifying condition. Assignment coordination: Before finalizing a servicemember’s assignment to a new location, DOD requires each Military Service to consider any family member’s special needs during this process, including the availability of required medical and special educational services at a new location. Family support: DOD requires each Military Service’s EFMP to include a family support component through which it helps families with special needs identify and gain access to programs and services at their current, as well as proposed, locations. Servicemembers assigned to a joint base would receive family support from the Service that is responsible for leading that installation. For example, an Airman assigned to a joint base where the Army is the lead would receive family support from the Army installation’s EFMP. As required by the NDAA for Fiscal Year 2010, DOD established the Office of Community Support for Military Families with Special Needs (Office of Special Needs or OSN) to develop, implement, and oversee a policy to support these families. Among other things, this policy must (1) address assignment coordination and family support services for families with special needs; (2) incorporate requirements for resources and staffing to ensure appropriate numbers of case managers are available to develop and maintain services plans that support these families; and (3) include requirements regarding the development and continuous updating of a services plan for each military family with special needs. OSN is also responsible for collaborating with the Services to standardize EFMP components as appropriate and for monitoring the Services’ EFMPs. OSN has been delegated the responsibility of implementing DOD’s policy for families with special needs by the Undersecretary of Defense for Personnel and Readiness through the Assistant Secretary for Manpower and Reserve Affairs according to DOD officials. Currently, OSN is administered under the direction of the Deputy Assistant Secretary of Defense for Military Community and Family Policy through the Office of Military Family Readiness Policy. In addition, each Military Service has designated a program manager for its EFMP who is also responsible for working with OSN to implement its EFMP (see fig. 1). DOD’s guidance for the EFMP (1) identifies procedures for assignment coordination and family support services; (2) designates the Assistant Secretary of Defense for Manpower and Reserve Affairs as being responsible for monitoring overall EFMP effectiveness; (3) assigns the OSN oversight responsibility for the EFMP, including data review and monitoring; and (4) directs each Service to develop guidance for overseeing compliance with DOD requirements for their EFMP. Table 1 provides an overview of the procedures each Service must establish for the assignment coordination and family support components of the EFMP. As a part of its guidance for monitoring military family readiness programs, DOD also requires each Military Service to certify or accredit its family readiness services, including family support services provided through the EFMP. In addition, DOD states that each Service must balance the need for overarching consistency across EFMPs with the need for each Service to provide family support that is consistent with their specific mission. To accomplish this, each Service is required to jointly work with DOD to develop a performance strategy, which is a plan that assesses the elements of cost, quality, effectiveness, utilization, accessibility, and customer satisfaction for family readiness services. In addition, each Military Service is required to evaluate their family readiness services using performance goals that are linked to valid and reliable measures such as customer satisfaction and cost. DOD also requires each Service to use the results of these evaluations to inform their assessments of the effectiveness of their family readiness services for families with special needs. According to DOD officials, each Military Service provides family support services in accordance with DOD guidance as well as Service-specific guidance. However, we found wide variation in each Service’s requirements for family support personnel as well as the practices and expectations of EFMP staff. As a result the type, amount, and frequency of assistance enrolled families receive varies from Service to Service and when a servicemember from one Service is assigned to a joint base led by another Service (see table 2). For example, in terms of a minimum level of contact for families with special needs enrolled in the EFMP, the Services vary in the frequency with which they require family support providers to contact families with special needs: The Marine Corps specifies a frequency (quarterly) with which families with special needs should be contacted by their family support providers. The Air Force has each installation obtain a roster of families with special needs enrolled in the EFMP on a monthly basis, but it does not require family support providers to, for example, use this information to regularly contact these families. The Navy assigns one of three service levels to each family member enrolled in the EFMP. These service levels are based on the needs of each family with special needs; family support providers are responsible for assigning a “service level” that directs the frequency with which the family must be contacted. The Army has no requirements for how often families with special needs should be contacted. The Services also vary as to whether they offer legal assistance to families with special needs as follows: The Marine Corps employs two attorneys who can represent families with special needs who fail to receive special education services from local school districts, as specified in their children’s individualized education programs (IEP). They can also advise EFMP-enrolled families on their rights and options if a family believes their child needs special education services from a local school district (e.g., an IEP). The Air Force, Army, and Navy choose not to employ special education attorneys. Officials with whom we spoke said families with special needs in these Services can receive other types of assistance that may help them resolve special education legal issues. For example, Air Force officials said servicemembers and their families can receive support from attorneys that provide general legal assistance on an installation, Army officials said installation EFMP managers can refer families with special needs to other organizations that provide legal support, and Navy officials said families can find support through working with their installation’s School Liaison Officers. The NDAA for Fiscal Year 2010 requires DOD’s policy to include requirements regarding the development and continuous updating of a services plan (SP) for each family with special needs, and DOD has specifically required these plans as part of the provision of family support services. These plans describe the necessary services and support for a family with special needs and document and track progress toward meeting related goals. According to DOD guidance, these plans should also document the support provided to the family, including case notes. In addition, the DOD reference guide for family support providers emphasizes that timely, up-to-date documentation is especially important each time a family relocates, as military families regularly do. Therefore, SPs are an important part of providing family support during the relocation process, and provide a record for the gaining installation. Requiring timely and up-to-date documentation is consistent with federal internal control standards, which state that agencies should periodically review policies, procedures, and related control activities for continued relevance and effectiveness in achieving their objectives. SPs follow families with special needs each time they relocate and without timely and up-to-date documentation, DOD cannot ensure that all families continue to receive required medical and/or special educational services once they relocate to another installation. For every Service the number of SPs was relatively few when compared to the number of servicemembers (known as sponsors) or the number of family members enrolled in the EFMP (see table 3). The Services and OSN provided a range of reasons as to why the Services do not develop and maintain a SP for each family with special needs. For example, Air Force officials said their family support providers consider the needs of each family with special needs before determining whether a SP will help them receive the required services. In addition, Army and Marine Corps officials said they may not develop these plans if families do not request them. Further, according to a Navy official, some families lack the required SPs because installations may not have the staff needed to develop them—even though DOD requires the Services to maintain sufficient staff and certify their EFMPs. OSN officials with whom we spoke also said that the Services may not have developed many SPs during fiscal year 2016 because DOD had not yet approved a standardized form that could be used to meet this requirement. Finally, OSN officials also said that each family with special needs enrolled in the EFMP may not need a SP because their condition does not require this type of family support. To meet requirements of the NDAA for Fiscal Year 2010, in April 2017, DOD issued to the Services guidance that directed them to “rogram, budget, and allocate sufficient funds and other resources, including staffing,” to meet DOD’s policy objectives for the EFMP. According to OSN officials, DOD relies on each Service to determine what level of funds and resources is sufficient and what constitutes an appropriate number of family support personnel. To determine family support providers and related personnel staffing levels, the Service officials with whom we spoke said they consider a number of factors, including the number of families with special needs enrolled in the EFMP at any given installation (see app. II for more information about the EFMP data by installation). See Table 4 for a summary of EFMP family support providers and other key personnel at CONUS installations. As required by DOD, all of the Services employ family support providers to assist families with special needs. In addition, some Services employ additional personnel to support implementation of the EFMP (see sidebar). For example, the Air Force employs family support coordinators to administer its EFMP and no other personnel are dedicated to assisting these coordinators or enrolled families. The Army employs “system navigators” who provide individualized support to families with special needs at selected installations through its EFMP, as well as other personnel to administer the EFMP. workers at most of its CONUS installations to administer individualized support to families with special needs. In addition, the Marine Corps employs program managers, administrative assistants, as well as training and education outreach specialists. The Navy contracts regional case liaisons and case liaisons at selected CONUS installations to administer individualized support to families with special needs. In addition, the Navy employs collateral duty case liaisons who assist with the delivery of family support services at all other CONUS installations. Senior OSN officials said they rely on each Service to determine the extent to which its EFMP complies with DOD’s policy for families with special needs because they consider OSN to be a policy-making organization that is not primarily responsible for assessing compliance. In addition, these officials said the Services need flexibility to implement DOD’s policy for families with special needs because they each have unique needs and the number of enrolled families in the EFMP is constantly changing. However, DOD has not developed a standard for determining the sufficiency of funding and resources each Service allocates for family support. Air Force officials at one of the installations we visited said the Air Force identified the lack of staff and funding to provide individualized support to most families with special needs as an issue. In addition, officials from the Army and Navy said they have not received any guidance from OSN officials about their Service-specific guidance, including requirements for resources and services plans. Further, the Services may not know the extent to which their Service- specific guidance complies with DOD’s policy for families with special needs. The NDAA for Fiscal Year 2010 requires DOD to identify and report annually to the congressional defense committees on gaps in services for military families with special needs and to develop plans to address these gaps. However, DOD’s most recent reports to the congressional defense committees did not address the relatively few SPs being created for families with special needs, or whether the Services are providing sufficient resources to ensure an appropriate number of family support providers. Federal internal control standards require that agencies establish control activities, such as developing clear policies, in order to accomplish agency objectives such as those of the Services’ EFMPs. Without fully identifying and addressing potential gaps in family support across these programs, some families with special needs may not get the assistance they require, particularly when they relocate. Each Service monitors EFMP assignment coordination and family support using a variety of mechanisms, such as regularly produced internal data reports. However, DOD has not yet established common performance measures to track the Services’ progress in implementing its standard procedures over time or developed a process to evaluate the overall effectiveness of each Service’s assignment coordination and family support procedures. DOD requires each Service to monitor implementation of their EFMP, including their procedures for assignment coordination and family support. To comply with this requirement, each Service has developed guidance that establishes monitoring protocols and assigns oversight responsibilities. Officials from each Service told us they use internal data reports from each installation to monitor assignment coordination and family support. To monitor assignment coordination, officials from each Service told us their headquarters reviews proposed assignment locations for families with special needs enrolled in the EFMP. These officials said monitoring proposed assignment locations helps ensure that enrolled families will be able to access required services at their new installations. In addition, Army officials said each Army unit commander is responsible for tracking the number of families with special needs that have expired enrollment paperwork because it affects assignment coordination worldwide. Several years ago, the Army determined that 25 percent of soldiers (over 13,000) enrolled in the EFMP had expired enrollment paperwork, complicating the task of considering each enrolled family’s special medical or educational needs as part of proposed relocations. In response, in August 2011, the Army revised its policies and procedures for updating enrollment paperwork which would help ensure a family member’s special needs are considered during the assignment coordination process. To monitor family support provided by installations worldwide, each Military Service told us they use a variety of mechanisms (see table 5). The Marine Corps pays particular attention to customer satisfaction. Marine Corps officials told us that every three years Marine Corps headquarters administers a survey of family members enrolled in the EFMP. We previously reported that organizations may be able to increase customer satisfaction by better understanding customer needs and organizing services around those needs. This survey is one of the primary ways Marine Corps headquarters measures customer satisfaction with family support services at installations worldwide. Marine Corps officials also said this survey helps ensure its EFMP is based on the current needs of families with special needs. To improve its oversight of the EFMP and implement its policy for families with special needs, DOD, through OSN, has several efforts under way to standardize the Services’ procedures for assignment coordination and family support. However, DOD has not developed common performance measures to monitor its progress toward these efforts and has not developed a process for assessing the Services’ related monitoring activities. Federal internal control standards emphasize the importance of assessing performance over time and evaluating the results of monitoring activities. To help improve family member satisfaction by addressing gaps in support that may exist between Services, OSN has begun to standardize procedures for assignment coordination and family support. To date, OSN’s efforts have focused on ensuring each Service’s EFMP considers the needs of family members during the assignment process and helps family members identify and gain access to community resources. According to OSN’s April 2017 Report to Congress, the long-term goal of these efforts is to help ensure that all families with special needs enrolled in the EFMP receive the same level of service regardless of their Military Service affiliation or geographic location. In addition, OSN officials told us its standardized procedures will also help DOD perform required oversight by improving its access to Service-level data and its ability to validate each Service’s monitoring activities. To date, efforts to standardize assignment coordination and family support have included efforts such as developing new family member travel screening forms which will be the official documents used during the assignment coordination process and completing a DOD-wide customer service satisfaction survey on EFMP family support (see table 6). Despite its efforts to begin standardizing assignment coordination and family support services, DOD is unable to measure its progress in standardizing assignment coordination and family support procedures for families with special needs and assessing the Services’ performance of these processes because it has not yet developed common metrics for doing so. Federal internal control standards emphasize the importance of agencies assessing performance over time. We have also reported on the importance of federal agencies engaging in large projects using performance metrics to determine how well they are achieving their goals and to identify any areas for improvement. By using performance metrics, decision makers can obtain feedback for improving both policy and operational effectiveness. Additionally, by tracking and developing a baseline for all measures, agencies can better evaluate progress made and whether or not goals are being achieved—thus providing valuable information for oversight by identifying areas of program risk and causes of risks or deficiencies to decision makers. Through our body of work on leading performance management practices, we have identified several attributes of effective performance metrics relevant to OSN’s work (see table 7). OSN officials said each Service is currently responsible for assessing the performance of its own EFMP, including the development of Service- specific goals and performance measures. OSN officials said that they recognize the need to continually measure the department’s progress overall in implementing its policy for families with special needs, and are considering ways to do so. They also said they have encountered challenges to developing common performance measures. In addition, OSN officials said its efforts to reach consensus among the Services about performance measures for the overall EFMP are still ongoing because each Service wants to maintain its own measures, and DOD has not required them to reach a consensus. Absent common performance measures, DOD is unlikely to fully determine whether its long-standing efforts to improve support for families with special needs are being implemented as intended. DOD requires each Service to monitor its own family readiness programs, including procedures for assignment coordination and family support through the EFMP, but lacks a systematic process to evaluate the results of these monitoring activities. To monitor family readiness services, as required by DOD, each Service must accredit or certify its family support services, including the EFMP, using standards developed by a national accrediting body not less than once every 4 years. In addition, personnel from each Service’s headquarters are required to periodically visit installations as a part of their monitoring activities for assignment coordination, among other things. The Services initially had the Council on Accreditation accredit family support services provided through their installations’ EFMPs using national standards developed for military and family readiness programs, according to the officials with whom we spoke. However, by 2016, each Service was certifying installations’ family support services using standards that meet those of a national accrediting body, Service-specific standards, and best practices. According to officials from each Service with whom we spoke, this occurred due to changes in the funding levels allocated to this activity. Table 8 provides an overview of the certification process currently being used by each Service. OSN officials said they do not have an ongoing process to systematically review the results of the Services’ activities, including the certification of EFMPs because they choose to rely on the Services to develop their own monitoring activities and ensure they provide the desired outcomes. In doing so, DOD allows each Service to develop its own processes for certifying installations’ family support services, including the selection of standards. In addition, OSN officials told us that efforts to standardize certification of EFMPs are ongoing because the Military Services have not been able to reach consensus on a set of standards that can be used across DOD for installations’ family support services. Further, OSN has not established a process to assess the results of the Services’ processes for certifying installations’ family support services. Federal standards for internal control state that management should evaluate the results of monitoring efforts—such as those the Services are conducting on their own—to help ensure they meet their strategic goals. The lack of such a process hampers OSN’s ability to monitor the Services’ EFMPs and determine the adequacy of such programs as required by the NDAA for Fiscal Year 2010. OSN’s job of developing a policy for families with special needs that will work across DOD’s four Services is challenging given the size, complexity, and mission of the U.S. military. It has had to consider, among other things, the Services’ mission requirements, resource constraints, and the myriad demands on servicemembers and their families during their frequent relocations. Anything that further complicates a relocation—such as not receiving the required family support services for family members with special needs—potentially affects readiness or, at a minimum, makes an already stressful situation worse. By providing little direction on how the Services should provide family support or what the scope of family support services should be, some servicemembers get more—or less—from the EFMP each time they relocate, including when a servicemember from one Service is assigned to a joint base led by another Service. By largely deferring to the Services to design, implement, and monitor their EFMPs’ performance, DOD cannot, as required by the NDAA for Fiscal Year 2010, fully determine the adequacy of the Services’ EFMPs in serving families with special needs, including any gaps in services these families receive, because it has not built a systematic process to do so. Instead, it relies on the Services to self-monitor and address, within each Service, the results of monitoring activities. However, because servicemembers relocate frequently and often depend on the EFMP of a Service other than their own, a view of EFMP performance across all of the Services is essential to ensuring, for example, that relocating servicemembers get consistent EFMP service delivery no matter where they are stationed. Evaluating and developing program improvements based on the results of the Services’ monitoring would help DOD ensure the Services’ EFMPs achieve the desired outcomes and improve its ability to assess the overall effectiveness of the program. We are making the following three recommendations to DOD: We recommend the Secretary of Defense direct the Office of Special Needs (OSN) to assess the extent to which each Service is (1) providing sufficient resources for an appropriate number of family support providers, and (2) developing services plans for each family with special needs, and to include these results as part of OSN’s analysis of any gaps in services for military families with special needs in each annual report issued by the Department to the congressional defense committees. (Recommendation 1) We recommend that the Secretary of Defense direct the Office of Special Needs (OSN) to develop common performance metrics for assignment coordination and family support, in accordance with leading practices for performance measurement. (Recommendation 2) We recommend that the Secretary of Defense implement a systematic process for evaluating the results of monitoring activities conducted by each Service’s EFMP. (Recommendation 3) We provided a draft of this report to the Department of Defense (DOD) for comment. DOD provided written comments, which are reproduced in appendix IV. DOD also provided technical comments, which we incorporated as appropriate. DOD agreed with all three of our recommendations. In its written comments, DOD stated that additional performance metrics need to be developed for assignment coordination and that it is in the process of measuring families’ satisfaction with family support provided through the EFMP. DOD also stated that it is developing plans for evaluating the results of each Service’s monitoring activities for the EFMP. We are sending copies of this report to the appropriate congressional committees, the Secretaries of Defense and Education, and other interested parties. The report also is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions about this report, please contact me at (617) 788-0580 or nowickij@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V. The National Defense Authorization Act (NDAA) for Fiscal Year 2017 includes a provision for GAO to assess the effectiveness of the Department of Defense’s (DOD) Exceptional Family Member Programs (EFMP). This report focuses on the assignment coordination and family support components of the EFMP for dependents with special needs and examines: (1) the extent to which each Service has provided family support as required by DOD, and (2) the extent to which the Services monitor and DOD evaluates assignment coordination and family support. To address these objectives, we used a variety of data collection methods. Key methods are described in greater detail below. For both objectives, we reviewed relevant federal laws, regulations, and DOD guidance and documentation that pertain to the EFMP, including the following: The NDAA for Fiscal Year 2010, which established the Office of Special Needs and defined program requirements for assisting families with special needs, including assignment coordination and family support. DOD’s guidance for administering the EFMP. We assessed how DOD implements the requirements in the NDAA for Fiscal Year 2010; how each Service implements assignment coordination and family support; and how the Services and DOD monitor assignment coordination and family support using performance measures. Specially, we reviewed DOD Instruction 1315.19 - Exceptional Family Member Program; Service-specific guidance and related documents from the Air Force, Army, Marine Corps, and Navy; and DOD Instruction 1342.22 - Military Family Readiness. Standards for internal control in the federal government related to the documentation of responsibilities through policies, performance measures, and evaluating the results of monitoring activities. We compared each Service’s procedures for monitoring assignment coordination and family support to these standards. To determine the extent of the Services’ EFMP family support, we obtained and analyzed fiscal year 2016 EFMP data (the most recent available) for each Service. We reviewed DOD policy to identify data variables that each Service maintains related to its EFMP. We used these data to summarize key characteristics of each Service’s EFMP. The selected variables provided Service-wide and installation-specific EFMP information on, the number of continental United States (CONUS) and outside the continental United States (OCONUS) installations; the number of servicemembers (sponsors) enrolled in the EFMP; the number of family members with special needs enrolled in the EFMP; the number of EFMP family support personnel; and the number of services plans created for families with special needs enrolled in the EFMP. We determined that the selected data variables from each Service are sufficiently reliable for the purposes of providing summary results about family support for fiscal year 2016. To learn more about how the Services implement their EFMPs, we visited seven installations in five states. We selected the seven installations based on their location in states with the largest number of military- connected students in school year 2012-2013 (the most recent available and reliable data) or in states with the largest percentage of students enrolled in U.S. DOD schools as of May 2017, as well as their status as a joint base. At each installation, we interviewed installation officials, EFMP managers, selected family support personnel, and family members and caregivers enrolled in the program. In states we visited that had the largest number of military-connected students, the EFMP personnel we interviewed collectively served 66 percent of students who attend local public schools and 42 percent of the students attending U.S. DOD schools. To obtain illustrative examples about how the EFMP serves families with special needs, we conducted seven group interviews with EFMP-enrolled family members and caregivers (one at each of the seven installations we visited). Using a prepared script, we asked participants to describe how they were identified and enrolled in the EFMP, how they were assigned to new installations, and the types of family support services they received. We also asked about how these services aligned with their family member’s EFMP-eligible condition, the benefits and challenges they experienced, as well as their overall satisfaction. A total of 38 self- selected volunteers participated in the seven group discussions. While the participants in these groups included a variety of family members and caregivers, the number of participants and groups were very small relative to the total number of family members enrolled in the EFMP. Their comments are not intended to represent all EFMP-enrolled family members or caregivers. Other EFMP-enrolled family members and caregivers may have had other experiences with the program during the same period. Finally, for both objectives, we conducted interviews with a variety of DOD, Service-level, and nonfederal officials. We spoke with DOD officials from the Office of the Assistant Secretary of Defense–Offices of Manpower and Reserve Affairs, Military Community and Family Policy, Military Family Readiness Policy, and Special Needs. We also spoke with EFMP Managers from Air Force, Army, Marine Corps, and Navy headquarters. We also met with officials from selected national military family advocacy organizations including the National Military Family Association; the Military Family Advisory Network; and the Military Officers Association of America to discuss the EFMP. We conducted this performance audit from February 2017 to May 2018 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives. Each Service has an Exceptional Family Member Program (EFMP) that provides support to military families with special needs. The tables below present the following information on selected EFMP and family support categories for each Service’s program at continental United States (CONUS) and outside the continental United States (OCONUS) installations in fiscal year 2016: City, state or country; Number of exceptional family members; Number of family support providers (by Full-Time Equivalent); Number of family support provider vacancies; Number of services plans; Number of indirect contacts; and Number of direct contacts. The information below is listed sequentially in alphabetical order by Service. We held small group discussions with Exceptional Family Member Program (EFMP) participants at the seven military installations we visited. Family members and caregivers who attended each session reported they had children or spouses with EFMP-eligible conditions. The discussion group participants were self-selected; and their comments are not intended to represent all EFMP -enrolled family members or caregivers in fiscal year 2016. In addition, other EFMP -enrolled family members and caregivers may have had different experiences with the program during the same period. There were a total of 38 participants representing all the Services. The following issues were discussed by one or more participants during the small group discussions at the installations we visited. The issues that emerged relate to the current and future overall effectiveness of the EFMP. Overall Satisfaction with EFMP (Discussed by 30 of 38 participants): Measure of participants’ approval of the family support services offered and experience with the EFMP. Many participants expressed overall satisfaction with the EFMP. Several participants expressed dissatisfaction with the EFMP. A participant expressed dissatisfaction with the lack of consistency in the provision of family support services (i.e., special education advocacy) across installations. School Liaison Officers (Discussed by 20 of 38 participants): Serve as the primary point of contact for school-related matters as well as assist military families with school issues. Several participants noted that they received no response to their request for assistance from their School Liaison Officer or they only received general information. Several participants said School Liaison Officers were not helpful. Some participants found School Liaison Officers were helpful. Some participants were unaware of School Liaison Officers being available at their installation and the service(s) they provide. A few participants said School Liaison Officers did not follow up on requests for information. A participant noted there seems to be a disconnect between family support services provided through the EFMP and services provided by School Liaison Officers. Family Support Personnel (Discussed by 12 of 38 participants): Provide information and referral to military families with special needs. Some participants at one installation noted that the EFMP was understaffed. Some participants at one installation noted high turnover of family support personnel. Some participants noted family support personnel did not provide support for their family with special needs. Stigma (Discussed by 12 of 38 participants): A perception that participating in the EFMP may limit a soldier’s assignment opportunities and/or compromise career advancement. Several participants believe there is still stigma associated with participating in the EFMP. Some participants said participating in the EFMP has not affected career advancement. Assignment Coordination (Discussed by 10 of 38 participants): The assignment of military personnel in a manner consistent with the needs of armed forces that considers locations where care and support for family members with special needs are available. Some participants found the assignment coordination process challenging. Some participants described limitations with the assignment coordination process. A few participants noted there is a lack of information among families with special needs regarding how to express the need for stabilization and /or continuity of care. A few participants cited the challenges of assignment coordination as contributing to their decision to retire. One participant commented that the opinion of a medical professional was not reflected in the assignment coordination process. Special Education Services (Discussed by 10 of 38 participants): The provision of staff capable of assisting families with special needs with special education and disability law advice and/or assistance and attendance at individualized education program (IEP) meetings where appropriate. Several participants who had a family support provider assist them with preparing for or attending a school-based meeting, including IEP meetings, spoke positively of their experience(s). Some participants at one installation agreed that assistance from family support providers during meetings with school officials regarding special education services is helpful. A few participants who were unable to get assistance with special education services from the EFMP sought the services of private attorneys at their own expense. Family Support Services (Discussed by 9 of 38 participants): The non-clinical case management delivery of information and referral for families with special needs, including the development and maintenance of a services plan. Some participants found that family support providers were helpful. Some participants could not identify needed resources or were unaware of the resources or services available to them. One participant noted that the family support provider had minimal contact. One participant said navigating the system can be challenging. Surveys (Discussed by 8 of 38 participants): The process of collecting data from a respondent using a structured instrument and survey method to ensure the accurate collection of data. Several participants noted that they had not or rarely had the opportunity to evaluate the family support services provided through the EFMP. One participant noted that comment cards used by each service are not effective for evaluating the EFMP. Warm hand-off (Discussed by 6 of 38 participants): Assistance to identify needed supports or services and facilitating the initial contact or meeting with the next program. Many participants at one installation agreed that the warm hand-off process worked well for them. Several participants said they found the warm hand-off process helpful when moving from one installation to the next. Outreach (Discussed by 5 of 38 participants): Developing partnerships with military and civilian agencies and offices (local, state, and national), improving program awareness, providing information updates to families, and hosting and participating in EFMP family events. Some participants found it difficult to obtain information regarding the types of family support services that are available. A few participants noted that communications regarding the EFMP were not targeted to address their needs. A few participants noted communications regarding the EFMP are untimely, (e.g., newsletters not issued periodically). Joint Base Family Support Services (Discussed by 1 of 38 participants): Family support services provided by the lead Service of the Joint Base that is different from that of the servicemember enrolled in the EFMP. One participant said that using family support services on joint bases may pose a challenge as each Service has different rules and procedures and as a result provides different types of family support services. In addition to the contact named above, Bill MacBlane (Assistant Director), Brian Egger (Analyst-in-Charge), Patricia Donahue, Holly Dye, Robin Marion, James Rebbe, Shelia Thorpe, and Walter Vance made significant contributions to this report. Also contributing to this report were Lucas Alvarez, Bonnie Anderson, Connor Kincaid, Brian Lepore, Daniel Meyer, and Mimi Nguyen.

 

 Report 2:
JWST is envisioned to be a large deployable space telescope, optimized for infrared observations, and the scientific successor to the aging Hubble Space Telescope. JWST is being designed for a 5-year mission to find the first stars, study planets in other solar systems to search for the building blocks of life elsewhere in the universe, and trace the evolution of galaxies from their beginning to their current formation. JWST is intended to operate in an orbit approximately 1.5 million kilometers—or 1 million miles—from the Earth. With a 6.5-meter primary mirror, JWST is expected to operate at about 100 times the sensitivity of the Hubble Space Telescope. JWST’s science instruments are designed to observe very faint infrared sources and therefore are required to operate at extremely cold temperatures. To help keep these instruments cold, a multi-layered tennis court-sized sunshield is being developed to protect the mirrors and instruments from the sun’s heat. The JWST project is divided into three major segments: the observatory segment, the ground segment, and the launch segment. When complete, the observatory segment of JWST is to include several elements (Optical Telescope Element (OTE), Integrated Science Instrument Module (ISIM), and spacecraft) and major subsystems (sunshield and cryocooler). The hardware configuration referred to as OTIS was created when the Optical Telescope Element and the Integrated Science Instrument Module were integrated. Additionally, JWST is dependent on software to deploy and control various components of the telescope, and to collect and transmit data back to Earth. The elements, major subsystems, and software are being developed through a mixture of NASA, contractor, and international partner efforts. See figure 1 for the elements and major subsystems of JWST and appendix 1 for more details, including a description of the elements, major subsystems, and JWST’s instruments. For the majority of work remaining, the JWST project is relying on two contractors: Northrop Grumman and the Association of Universities for Research in Astronomy’s Space Telescope Science Institute. Northrop Grumman plays the largest role, developing the sunshield, the Optical Telescope Element, the spacecraft, and the Mid-Infrared Instrument’s cryocooler, in addition to integrating and testing the observatory. Space Telescope Science Institute’s role includes soliciting and evaluating research proposals from the scientific community, and receiving and storing the scientific data collected, both of which are services that it currently provides for the Hubble Space Telescope. Additionally, the Institute is developing the ground system that manages and controls the telescope’s observations and will operate the observatory on behalf of NASA. JWST will be launched on an Ariane 5 rocket, provided by the European Space Agency. JWST depends on 22 deployment events—more than a typical science mission—to prepare the observatory for normal operations on orbit. For example, the sunshield and primary mirror are designed to fold and stow for launch and deploy once in space. Due to its large size, it is nearly impossible to perform deployment tests of the fully assembled observatory, so the verification of deployment elements is accomplished by a combination of lower level component tests in flight-simulated environments; ambient deployment tests for assembly, element, and observatory levels; and detailed analysis and simulations at various levels of assembly. We have previously found that complex development efforts like JWST face numerous risks and unforeseen technical challenges, which can often become apparent during integration and testing. To accommodate unanticipated challenges and manage risk, projects reserve extra time in their schedules, which is referred to as schedule reserve, and extra funds in their budgets, which is referred to as cost reserve. Schedule reserve is allocated to specific activities, elements, and major subsystems in the event of delays or to address unforeseen risks. Each JWST element and major subsystem has been allocated schedule reserve. When an element or major subsystem exhausts schedule reserve, it may begin to affect schedule reserve on other elements or major subsystems whose progress is dependent on prior work being finished for its activities to proceed. Cost reserves are additional funds within the project manager’s budget that can be used to address unanticipated issues for any element or major subsystem, and are used to mitigate issues during the development of a project. For example, cost reserves can be used to buy additional materials to replace a component or, if a project needs to preserve schedule reserve, reserves can be used to accelerate work by adding shifts to expedite manufacturing. NASA’s Goddard Space Flight Center— the NASA center with responsibility for managing JWST—has issued procedures that establish the requirements for cost and schedule reserves. In addition to cost reserves held by the project manager, management reserves are funds held by the contractors that allow them to manage program risks and to address unanticipated cost increases throughout development. We have previously found that management reserves should contain 10 percent or more of the cost to complete a project and are generally used to address various issues tied to the contract’s scope. NASA’s cost-plus-award-fee contract with Northrop Grumman has spanned almost two decades, during which there have been significant variances in contractor performance. Cost-reimbursement contracts are suitable when uncertainties in the scope of work or cost of services prevent the use of contract types in which prices are fixed, known as fixed-price contracts. Award fee contracts provide contractors the opportunity to obtain monetary incentives for performance in designated areas identified in the award fee plan. Award fees may be used when key elements of performance cannot be defined objectively, and, as such, require the project officials’ judgment to assess contractor performance. For JWST’s contract with Northrop Grumman, these areas include cost, schedule, technical, and business management and are established in the contracts’ performance evaluation plans. In December 2013, the JWST program and the contractor agreed to replace a $56 million on-orbit incentive—incentives based on successful performance in space—with award fees. The award fees are to incentivize cost and schedule performance during development. This shift increased the available award fee for the entire contract to almost a quarter of a billion dollars. According to officials, restructuring the incentives gave NASA more flexibility to incentivize the contractor to prioritize the cost and schedule performance over exceeding technical requirements. In December 2014, we found that NASA award fee letters of award fee periods from February 2013 to March 2014 indicated that the contractor had been responsive to interim award fee period criteria provided by NASA and that contractor officials confirmed that they pay close attention to this guidance in prioritizing their work. For example, Northrop Grumman officials reported that they had made specific changes to improve communications in direct response to this guidance, which was validated by award fee letters from NASA. The JWST program has a history of significant schedule delays and increases to project costs, which resulted in replans in 2011 and 2018. Before 2011, early technical and management challenges, contractor performance issues, low levels of cost reserves, and poorly phased funding caused the JWST program to delay work. As a result, the program experienced schedule overruns, including launch delays, and cost growth. The JWST program underwent a replan in September 2011, and a rebaseline in November of that same year, and Congress placed an $8 billion cap on the formulation and development costs for the project. On the basis of the replan, NASA rebaselined JWST with a life- cycle cost estimate of $8.835 billion, which included additional money for operations and a planned launch in October 2018. Congress also required that NASA treat any cost increase above the cap according to procedures established for projects that exceed their development cost estimates by at least 30 percent. This process is known as a rebaseline. Congress must authorize continuation of the JWST program if formulation and development costs increase over the $8 billion cost cap. In June 2018, after a series of launch delay announcements due to technical and workmanship issues identified during spacecraft element integration, NASA notified Congress that it had again revised the JWST program’s cost and schedule estimates. NASA estimated that it now required $828 million in additional resources and 29 more months to complete beyond those estimates agreed to in the 2011 rebaseline. As of November 2018, NASA had funding to continue to execute the program and was waiting to see if Congress would authorize the program’s continuation and appropriate funds for the program in fiscal year 2019. Figure 2 shows the project’s history of changes to its cost or schedule and key findings from two external independent review teams and our prior work. As discussed above, various technical and workmanship errors drove some of the more recent delays. Examples of some of the workmanship issues we found in the past include: In October 2015, the project reported that a piece of flight hardware for the sunshield’s mid-boom assembly was irreparably damaged during vacuum sealing in preparation for shipping. The damaged piece had to be remanufactured, which consumed 3 weeks of schedule reserve. In April 2017, a contractor technician applied too much voltage and irreparably damaged the spacecraft’s pressure transducers, components of the propulsion system that help monitor spacecraft fuel levels. The transducers had to be replaced and reattached in a complicated welding process. At the same time, Northrop Grumman also addressed several challenges with integrating sunshield hardware. These issues combined took up another 1.25 months of schedule reserve. In May 2017, some of the valves in the spacecraft propulsion system’s thruster modules were leaking beyond permissible levels. Northrop Grumman determined that the most likely cause was the use of an improper cleaning solution, and the thruster modules were returned to the vendor for investigation and refurbishment. Reattaching the refurbished modules was expected to be complete by February 2018, but was delayed by one month when a technician applied too much voltage to one of the components in a recently refurbished thruster module. NASA and Northrop Grumman reported that resolving the thruster module issue resulted in a 2-month delay to the project’s overall schedule. In October 2017, when conducting folding and deployment exercises on the sunshield, Northrop Grumman discovered several tears in the sunshield membrane layers. According to program officials, a workmanship error contributed to the tears. The tears resulted in another 2-month delay to the project’s overall schedule. In addition, some first-time efforts took longer than planned. For example, in fall 2017, the project determined that it would need to use up to 3 months of schedule reserve based upon lessons learned from the contractor’s initial sunshield folding operation. This first deployment, or unfolding, took 30 days longer than planned. The sunshield has since undergone another deployment, and will be deployed twice more before launch. The IRB took into account these technical and workmanship errors, as well as other considerations, when it analyzed the project’s organizational and technical issues. The board’s final report, issued in May 2018, included 31 recommendations that addressed a range of factors. For example, the IRB recommended that the project: Conduct an audit to identify potential embedded design flaws— problems that have not been detected through analysis, inspection, or test activities and pose a significant risk to JWST schedule, cost, and mission success; Establish corrective actions to detect and correct human mistakes during integration and test; Establish a coherent, agreed-upon, and factual narrative on project status and communicate that status regularly across to all relevant stakeholders; and Augment integration and test staff to ensure adequate long-term staffing and improve employee morale. In its response to the IRB’s report, NASA stated that it accepted the report’s recommendations and had already begun implementing action in response to many of them. Further, project officials told us that some of the actions were underway before the IRB completed its review. To develop a new schedule for JWST’s 2018 replan, NASA took into account the remaining integration and test work and added time to the schedule to address threats that were not yet mitigated. This includes 5.5 months to address an anomaly that occurred on the sunshield’s cover in 2018. The project also replenished its schedule reserves—which we found in February 2018 had been consumed—so that they now exceed the recommended levels. Both the project and IRB conducted schedule risk assessments that produced similar launch dates. The project relied on the replan schedule to determine its remaining costs because the workforce necessary to complete the observatory represents most of the remaining cost. Following is additional information on the schedule and cost considerations. Schedule: JWST’s revised launch readiness date of March 2021 reflects a consideration of the hardware integration and test challenges the project has experienced, including adding time to: Add snag guards for the membrane tensioning system—which helps deploy the sunshield and maintain its correct shape—to prevent excess cable from snagging, Repair tears of the sunshield membrane, Deploy, fold, and stow the sunshield, and Mitigate contractor schedule threats. In addition, the project added extra time to the schedule to complete repairs to the membrane cover assembly, which did not perform as expected during acoustics testing in April 2018. The membrane cover assembly shown in figure 3 is used to cover the sunshield membrane when in the stowed position to provide thermal protection during launch. After the anomaly occurred, the project halted spacecraft element testing, investigated the anomaly, and found that the fasteners had come loose due to a design change made to prevent the fasteners from damaging the sunshield membrane. The design change caused the nuts to not lock properly. According to project officials, due to the design of the membrane cover assembly, the project was not able to conduct flight-like, stand- alone testing on the cover prior to spacecraft element testing. As a result, the project did not discover the design issue until the hardware came loose while installed on the spacecraft element. The project determined that the repairs would take approximately 5.5 months. The project’s replan also reflected schedule reserves above the level required by Goddard Space Flight Center policy, which would have been approximately 5 months at that time. The new schedule includes a total of 293 days or 9.6 months of schedule reserves leading up to its committed launch readiness date of March 2021. NASA approved a JWST launch date of March 2021, but the project and the contractor are working toward a launch date in November 2020. Figure 4 shows the project’s new schedule following the 2018 replan, including how the project distributed its schedule reserves through different integration and test activities. As part of its May 2018 study, the IRB reviewed the project’s schedule and recommended a launch date of March 2021, which was subsequently reflected in NASA’s new schedule for the program. In reviewing the project’s schedule, the IRB found that the project had robust scheduling practices for ensuring that the schedule represented a complete and dynamic network of tasks that could respond automatically to changes. This schedule also passed a standard health check with minimal errors indicating that it was well constructed. However, the IRB noted that this schedule does not account for certain types of unknown risks to the program such as integration and test errors which can take many months to resolve, or the potential need to remove a science instrument from the observatory, which can have about a 1 year impact. As a result, the program could experience additional delays if a risk of this magnitude is realized. Cost: The project’s new $9.7 billion life-cycle cost estimate is principally driven by the schedule extension, which requires keeping the contractor’s workforce to complete integration and test longer than expected. Specifically, the project determined that almost all of the hardware had been delivered and the remaining cost was predominantly the cost for the workforce necessary to complete and test the observatory. For the past 3 years, we have reported that Northrop Grumman’s ability to decrease its workforce was central to JWST’s capacity to meet its long- term cost commitments. However, Northrop Grumman’s actual workforce continued to exceed its projections. This was because it needed to maintain higher workforce levels due to technical challenges, including problems with spacecraft and sunshield integration and test. It also needed to keep specialized engineers available when needed during final assembly and test activities. In developing the cost estimate supporting the 2018 replan, the project used a Northrop Grumman workforce profile that is higher than previous projections because Northrop Grumman now plans to maintain personnel longer during integration and test. According to project officials, the planned reduction of Northrop Grumman’s workforce is now more gradual and conservative than the prior plan. For example, the Northrop Grumman workforce will not start to significantly decline until the observatory ships to the launch site, which is expected to occur in August 2020. As shown in Figure 5, the JWST workforce assembling the observatory declines and the government and contractor workforce necessary to manage and operate the observatory remains after the internal launch readiness date of November 2020. As seen in the above figure, the Space Telescope Science Institute workforce, the contractor responsible for operating JWST, will remain generally flat between fiscal years 2021 to 2026 when it operates the observatory. The NASA civil service and support contractor will remain relatively flat through November 2020 launch date and then decline. In addition, the new cost estimate also took into account $61 million for implementing the IRB recommendations and mission success enhancements, funding for project cost reserves, and operations costs. In June 2018, the NASA associate administrator—who is the project’s decision authority—approved the project to proceed with its replan with a March 2021 launch date and $9.7 billion in life-cycle costs based on the Agency Program Management Council review and replan documents. The associate administrator did not require the project to conduct an updated Joint Cost and Schedule Confidence Level (JCL) analysis for this replan. A JCL is an integrated analysis of a project’s cost, schedule, risk, and uncertainty whose result indicates the probability of a project’s success of meeting cost and schedule targets. NASA policy states that a JCL should be recalculated and approved as a part of the rebaselining approval process, but it is not required. In its replan decision memo, NASA’s associate administrator explained that he did not require the project to update the JCL because project costs are almost entirely related to the workforce and most of the remaining planned activities will be performed generally in sequence. Therefore, according to NASA’s associate administrator, the total cost would be driven almost entirely by the schedule because the workforce levels will remain the same through delivery of the observatory. Both the project and independent estimators used multiple schedule estimating methods to analyze the schedule for the remaining work, and NASA’s associate administrator said these analyses returned consistent, high confidence launch dates. The project’s ability to execute to its new schedule will be tested as it progresses through the remainder of challenging integration and test work. The project has yet to complete three of five integration and test phases. The remaining phases include integration and test of OTIS, the spacecraft element, and the observatory. Our prior work has shown that integration and testing is the phase in which problems are most likely to be found and schedules tend to slip. For a uniquely complex project such as JWST, this risk is magnified as events start to become more sequential in nature. As a result, it will continue to become more difficult for the project to avoid schedule delays by mitigating issues in parallel. As of November 2018, the project is about a week behind its replanned schedule because repairs on the membrane cover assembly took longer than planned. Completing the membrane cover assembly repairs and returning the spacecraft to vibration testing was a key event for the project to demonstrate that it could execute to its new schedule. When the project developed its 2018 replanned schedule, it had planned to complete the membrane cover assembly repairs and reinstall the assembly onto the sunshield and restart spacecraft element integration and test activities by November 6, 2018. The project allocated 4 weeks of schedule reserves specifically for these repairs. However, the membrane cover repairs proved more difficult than anticipated. For example, the program had to address unanticipated technical challenges on the membrane cover assemblies, including repairing tears and pin holes in the covers discovered after the covers were removed. The project also had to allot time to install bumpers, which are kapton tubes, to the assembly to protect the composite material on a sunshield structure during launch. The project identified the need to add the bumpers during subassembly vibration testing. As a result, as of November 2018, the project had used about 4.5 weeks of schedule reserves to cover delays associated with these activities. The use of reserves beyond what the project had planned for the repairs pushed the restart of spacecraft element integration and test activities out about a week to November 14, 2018. Figure 6 compares the project’s initial membrane cover assembly schedule in June 2018 to the actual schedule in November 2018. While the project repaired the membrane cover assembly, it also used this time to conduct risk mitigation activities on OTIS. For example, the project worked to mitigate a design issue on the frill connections. The frill is composed of a single layer of blankets placed around the outside of the primary mirror used to block stray light (see figure 7). A combination of modeling and inspections revealed that most of the frill sections did not have as much slack as expected at the near-absolute zero cryogenic temperatures of space. This caused shrinkage that put stress on the edges of the outer ring of mirrors, which could affect the stability of the optical mirror and image quality. The project loosened these outer connections by adding a ring to the connecting points. As of November 2018, project officials said they were in the process of verifying the fix through inspections. Examples of technical issues and risks that the project continues to face during the remaining phases of integration and test include: The project is working to mitigate a design issue on the sunshield membrane tensioning system—which helps deploy the sunshield and maintain its correct shape. In our February 2018 report, we found that Northrop Grumman was planning to modify the design of the membrane tensioning system after one of the sunshield’s six membrane tensioning systems experienced a snag when conducting folding and deployment exercises on the sunshield in October 2017. The project and Northrop Grumman determined that a design modification was necessary to fully mitigate the issue, which includes modifying clips used to progressively release the cable tension and adding guards to control the excess cable. The project identified a concern that the depressurization of trapped air in the folded sunshield membrane when the fairing separates to release the JWST observatory may overly stress the membrane material. The project is working with Arianespace—the company responsible for operating JWST’s launch vehicle—and experts at the Kennedy Space Center to resolve this concern. Officials estimated that a design solution would be in place in mid-2019. However, if the project determines that it needs to reinforce the membrane covers to survive excessive residual pressure as it works on this design solution, a multi-month schedule delay could occur. As of November 2018, the project has mitigated 21 of its 47 hardware and software risks to acceptable levels, and reviews these risks monthly for any changes that might affect the continued acceptability of the risk. Five of these 21 risks are related to the project’s more than 300 potential single point failures—several of which are related to the deployment of the sunshield. The project is actively working to mitigate the remaining 26 risks to acceptable levels or closure prior to launching. The project also has several first-time and challenging integration and test activities remaining. For example, the project must integrate OTIS and the completed spacecraft element and test the full observatory in the final integration phase, which includes another set of challenging environmental tests. See figure 8 for an image of OTIS and the spacecraft element prior to being integrated. As previously discussed, the project also has two remaining deployments of the sunshield, and prior deployments have taken longer than planned. To help mitigate the risks associated with the deployments, the project added additional time for deployments in the 2018 replanned schedule based on lessons learned from prior deployments. The two remaining deployments are to occur after spacecraft element integration and test and again after observatory integration and test. The JWST project office is required to evaluate whether the project can complete development within its revised cost and schedule commitments at its next major review—the system integration review—planned for August 2019. This review is to occur after the project has completed two major tasks—OTIS and spacecraft element integration and test. The review is to evaluate whether the project (1) is ready to enter observatory integration and test, and (2) can complete remaining project development with acceptable risk and within its cost and schedule constraints. NASA guidance does not require projects to conduct a JCL at this review. However, project officials said that they plan to conduct another schedule risk analysis in the future. They do not intend to complete a new JCL for the same reasons they did not complete one for the 2018 replan— because costs are almost entirely related to the workforce and can be derived from a schedule that takes into account known risk. While not required, conducting a JCL prior to the system integration review would inform NASA about the probability of meeting both its cost and schedule commitments. If the project proceeds with its plan to conduct only a schedule risk analysis, NASA would be provided only with an updated probability of meeting its schedule commitments. Our cost estimating best practices recommend that cost estimates should be updated to reflect changes to a program or kept current as it moves through milestones and as new risks emerge. In addition, government and industry cost and schedule experts we spoke with noted that integration and testing is a critical time for a project when problems can develop. These experts told us that completing a JCL is a best practice for analyzing major risks at the most uncertain part of project execution. Conducting a JCL at system integration review—a review that occurs during the riskiest phase of development, the integration and test phase— would allow the project to update its assumptions of risk and uncertainty based on its experiences in OTIS and spacecraft element integration and test. The project could then determine how those updated assumptions affect overall cost and schedule for the JWST project. As noted above, the project has many risks to mitigate, technical challenges to overcome, and challenging test events to complete, which could affect the project’s schedule and risk posture. Further, the project has an established history of significant cost growth and schedule delays. In its June 2018 letter notifying an appropriate congressional committee of its updated cost and schedule commitments, NASA acknowledged that recent cost growth for the project will likely impact other science missions. Conducting a JCL at system integration review would provide NASA and Congress with critical information for making informed resource decisions on the JWST project and its affordability within NASA’s portfolio of projects more broadly. NASA has taken steps to augment oversight of the contractor and project following the discovery of the embedded design flaws and workmanship errors that contributed to the project’s most recent schedule delays and cost increases. See table 1 for examples of changes NASA has made to contractor and project oversight—some of which NASA self-identified and others that were in response to IRB recommendations. The IRB made 31 recommendations that ranged from improving employee morale to improving security during transporting JWST to its launch site. NASA has also used award fees to try to incentivize Northrop Grumman to improve its performance. In a July 2018 hearing on the JWST program before the House Science, Space, and Technology Committee, Administrator Bridenstine stated that NASA had reduced the available award fee through commissioning by $28 million out of a total of about $60 million. Northrop Grumman also did not earn its full award fee in the two most recent periods of performance that NASA assessed. For the performance period of April 1, 2017 to September 30, 2017, Northrop Grumman earned approximately 56 percent of the available award fee. Reasons that NASA cited for its evaluation of award fees in this period included workmanship errors on the propulsion system, schedule delays, as well as issues with schedule execution, management, and quality control. For the period of October 1, 2017 to March 31, 2018, Northrop Grumman earned none of the available award fee. Northrop Grumman’s overall score was driven by an “unacceptable” rating in schedule and cost due to delays and in anticipation of exceeding the project’s $8 billion cost cap. Northrop Grumman received an “excellent” rating under the technical category, but the evaluation noted ongoing issues with quality controls, which resulted in delays. For example, the process steps for applying voltage to the spacecraft’s pressure transducers were not clear enough, which resulted in technician error and irreparable damage to the hardware. According to Northrop Grumman officials, the contractor has started to take action to try to improve its quality assurance processes. Officials described actions that ranged from rewriting hardware integration and test procedures to starting efforts to change aspects of the company’s culture that contributed to quality control issues. For example, in July 2018, Northrop Grumman initiated a JWST mission assurance culture change campaign to increase focus on product quality and process compliance. This effort includes having inspectors affirm by signature that they have personally inspected, verified, and confirmed that all aspects of an activity meet quality standards. According to the form instructions, if the inspector is uncertain on compliance or if instructions are unclear, workers are to halt work, investigate and assess the situation, and request help to resolve the situation. Project and Northrop Grumman officials provided an example of these changes working. During a manual deployment of a radiator panel, a Northrop Grumman employee discovered that a flap used as thermal protection for a radiator was installed incorrectly and reported the error. Northrop Grumman technicians found that this flap had been swapped with another flap in the process of moving them to be installed and corrected the problem before work proceeded. Further, NASA and Northrop Grumman are conducting audits to try to minimize the risk of failures during the remaining phases of integration and test. These audits are conducted on items that have not been fully tested, are in workmanship-sensitive areas, or have had a late design change. The first phase of the audit was completed in September 2018 and found no major design issues or hardware rework required. The project plans to audit other areas through at least spring 2019, but will add audits if needed. The JWST oversight structure includes a number of positions that could be responsible for ensuring that the recent augmentations to contractor and project oversight are sustained through launch (see table 2). In response to our review, NASA officials clarified that the project manager has sole responsibility for ensuring that these improvements are sustained through launch. Further, these officials stated that the project office is responsible for monitoring these changes at the project level and at Northrop Grumman. The project manager’s continued focus on these efforts will be important because: The project is implementing a wide span of improvement efforts, ranging from more on-site coverage at the contractor facility to cultural improvements, which will now need to be sustained for an additional 29 months. The project has had recurring issues with effective internal and external communication as well as defining key management and oversight responsibilities, both of which are important to sustaining oversight. For example, the Independent Comprehensive Review Panel identified communication problems—between the JWST project and Science Mission Directorate management as well as between NASA and Northrop Grumman—and that the project’s governance structure lacked clear lines of authority and accountability. In December 2012, we found the JWST project had taken several steps to improve communication—such as instituting meetings that include various levels of NASA, contractor, and subcontractor management— but the IRB’s findings in 2018 indicate that communication and governance issues have resurfaced in some areas. For example, the IRB found that communication with key stakeholders including the science community, Congress, and NASA leadership, has been variable and at times inconsistent. The project may encounter new schedule pressures as it proceeds through integration and test. A senior NASA official with expertise in workmanship issues told us that schedule pressure is a key reason for increased quality problems on projects. For example, this official said that companies tend to give experts leniency to operate without the burden of quality assurance paperwork when schedule pressures arise, which can lead to workmanship errors. While JWST project officials told us they do not view this as applicable to their project, the perspective regarding potential schedule pressures and workmanship is important to keep focus on given the magnitude of technical challenges and delays the project has faced. We will continue to monitor the project’s efforts at maintaining these oversight augmentations in future reviews, given that less than a year has passed since the project began implementing many of them. Moreover, the project may find that some actions will be required of officials outside the project, particularly since the communication problems identified by the IRB may well extend to headquarters’ interaction with stakeholders from the science community, industry, and the Congress. JWST is one of NASA’s most expensive and complex science projects, and NASA has invested considerable time and resources on it. The project first established its cost and schedule baseline in 2009. Since then, the project made progress by completing two of five phases of integration and test, but has also experienced significant cost growth and schedule delays. However, the project did not complete a JCL analysis as part of its second replan. Between now and its system integration review planned for August 2019, the JWST program will have to continue to address technical challenges and mitigate risks. Conducting a JCL would better inform decision makers on the status of the project as they determine whether the project can complete remaining project development with acceptable risk and within its cost and schedule constraints. Given the project is now on its third iteration of cost and schedule commitments, conducting a JCL is a small step that NASA can take to demonstrate it is on track to meet these new commitments. We are making the following recommendation to NASA: The NASA Administrator should direct the JWST project office to conduct a JCL prior to its system integration review. (Recommendation 1) We provided a draft of this report to NASA for comment. In written comments, NASA agreed with our recommendation. NASA expects to complete the JCL by September 2019, prior to the system integration review. The comments are reprinted in appendix II. NASA also provided technical comments, which have been addressed in the report, as appropriate. We are sending copies of this report to the appropriate congressional committees, the NASA Administrator, and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov. If you or your staff have any questions on matters discussed in this report, please contact me at (202) 512-4841 or chaplainc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III. In addition to the contact named above, Molly Traci (Assistant Director), Karen Richey (Assistant Director), Jay Tallon (Assistant Director), Brian Bothwell, Daniel Emirkhanian, Laura Greifner, Erin Kennedy, Jose Ramos, Sylvia Schatz, Roxanna Sun, and Alyssa Weir made key contributions to this report.

Question: Write a one-page summary of the report 1.

Answer: Military families with special medical and educational needs face unique challenges because of their frequent moves. To help assist these families, DOD provides services plans, which document the support a family member requires. The National Defense Authorization Act for Fiscal Year 2017 included a provision for GAO to review the Services' EFMPs, including DOD's oversight of these programs. This report examines the extent to which (1) each Service provides family support and (2) the Services monitor and DOD evaluates assignment coordination and family support. GAO analyzed DOD and Service-specific EFMP guidance and documents; analyzed fiscal year 2016 EFMP data (the most recent available); visited seven military installations, selected for their large numbers of military-connected students; and interviewed officials responsible for implementing each Service's EFMP, as well as officials in OSN that administer DOD's EFM policy. The support provided to families with special needs through the Department of Defense's (DOD) Exceptional Family Member Program (EFMP) varies widely for each branch of Military Service. Federal law requires DOD's Office of Special Needs (OSN) to develop a uniform policy that includes requirements for (1) developing and updating a services plan for each family with special needs and (2) resources, such as staffing, to ensure an appropriate number of family support providers. OSN has developed such a policy, but DOD relies on each Service to determine its compliance with the policy. However, Army and Navy officials said they have not received feedback from OSN about the extent to which their Service-specific guidance complies. Federal internal control standards call for developing clear policies to achieve agency goals. In addition, DOD's most recent annual reports to Congress do not indicate the extent to which each Service provides services plans or allocates sufficient resources for family support providers. According to GAO's analysis, the Military Services have developed relatively few services plans, and there is wide variation in the number of family support providers employed, which raises questions about potential gaps in services for families with special needs (see table). Each Service uses various mechanisms to monitor how servicemembers are assigned to installations (assignment coordination) and obtain family support, but DOD has not established common performance measures to assess these activities. DOD has taken steps to better support families with special needs, according to the DOD officials GAO interviewed. For example, DOD established a working group to identify gaps in services. However, OSN officials said that DOD lacks common performance measures for assignment coordination and family support because the Services have not reached consensus on what those measures should be. In addition, OSN does not have a process to systematically evaluate the results of the Services' monitoring activities. Federal internal control standards call for assessing performance over time and evaluating the results of monitoring activities. Without establishing common performance measures and assessing monitoring activities, DOD will be unable to fully determine the effect of its efforts to better support families with special needs and the adequacy of the Services' EFMPs as required by federal law. GAO makes a total of three recommendations to DOD. DOD should assess and report to Congress on the extent to which each Service provides sufficient family support personnel and services plans, develop common performance metrics for assignment coordination and family support, and evaluate the results of the Services' monitoring activities. DOD agreed with these recommendations and plans to develop performance metrics for assignment coordination and develop plans to evaluate the Services' monitoring activities.